Minibatch and Momentum Model-based Methods for Stochastic Weakly Convex Optimization

Abstract: Stochastic model-based methods have received increasing attention lately due to their appealing robustness to the stepsize selection and provable efficiency guarantee. We make two important extensions for improving model-based methods on stochastic weakly convex optimization. First, we propose new minibatch model- based methods by involving a set of samples to approximate the model function in each iteration. For the first time, we show that stochastic algorithms achieve linear speedup over the batch size even for non-smooth and non-convex (particularly, weakly convex) problems. To this end, we develop a novel sensitivity analysis of the proximal mapping involved in each algorithm iteration. Our analysis appears to be of independent interests in more general settings. Second, motivated by the success of momentum stochastic gradient descent, we propose a new stochastic extrapolated model-based method, greatly extending the classic Polyak momentum technique to a wider class of stochastic algorithms for weakly convex optimization. The rate of convergence to some natural stationarity condition is established over a fairly flexible range of extrapolation terms.While mainly focusing on weakly convex optimization, we also extend our work to convex optimization. We apply the minibatch and extrapolated model-based methods to stochastic convex optimization, for which we provide a new complexity bound and promising linear speedup in batch size. Moreover, an accelerated model-based method based on Nesterov’s momentum is presented, for which we establish an optimal complexity bound for reaching optimality.


Reviews and author responses:

 Thank you for the detailed response!

For Question 1, I understand that searching for the best initial stepsize is needed, and I am more clear about it after reading your explanation. I assume that the choice for stepsize should decrease w.r.t. $m$ since $\sqrt{m}$ is in the denominator, but in Figure 2, we can see that the two ends of the preferable stepsize range actually become greater when the batchsize is larger. There could be a gap here between theory and practice, but I am okay with it for now.

For Question 2, if I understand correctly, the potential function for Theorem 3.3 is actually $f_{1/\rho}$, and the new potential function for Lemma 4.1 is the one in your response. 

For Question 3, I did not expect the difficulty in solving SPP subproblem until reading your explanation. For the stepsize range choices in Figure 1, 2, 3, 4, I agree with your answer to my first question, and therefore it is understandable that you choose different x-axis limits in these figures. 

Thanks again for answering my questions!

 We thank the reviewer Cw8j for the support and the additional suggestions. The reviewer shares the concern that the claim is on general non-smooth problems but the actual problem is well-structured. While we already stated on the first page of paper that we are dealing with weakly-convex functions--a special class of non-smooth non-convex functions, we will further clarify our target. In the title we will replace "stochastic non-smooth and non-convex optimization" by "stochastic weakly-convex optimization"; abstract and the other parts of the introduction will be revised accordingly to reflect that we are dealing with weakly-convex functions. Moreover, we agree with the reviewer that the problem is well-structured. However, due to the difficulty in dealing with non-smoothness and non-convexity, we feel that identifying the above structure is actually very important and necessary to improve the performance. We also refer the reviewer to the additional comments for 5Yxy on the difficulty of general non-smooth problems.

We also thank the reviewer for the thought-provoking questions about the difference with work [ACCD21, Asi20] and their results advance our understanding of minibatching in stochastic convex optimization.  We will gladly expand the related work part, make a more detailed comparison with [ACCD21, Asi20] and clarify the differences in assumptions as well as the connections. We also promise to the reviewer we will simplify our results by removing A6 and A7.

 We thank the reviewer 5Yxy for the comment and update of score. The earlier comments indeed helped us improve the applicability of SPP.

The reviewer is concerned that the studied problem is not so general as the title seems to imply. We will gladly change the paper title to "Minibatch and Momentum Model-based Methods for Stochastic Weakly-convex Optimization", hopefully this can address the issue.

We thank the reviewer again for pointing out the work [ZX20]. Therein the problem indeed has many interesting applications. [ZX20] motivates us to consider composition problem where both inner part and outer part are in the expectation form. We will leave this interesting direction to the future work.


 



**Additional comments on general SGD and difficulty** 

The reviewer commented that for the general case of SGD, the rate is not better than the existing works. While we agree with the reviewer upon this comparison, we feel that improving the rate of SGD is unlikely in the general case, and it is not due to the limitation of our technique. Strong evidence includes 

**1)** in the weakly-convex case, both stochastic/deterministic subgradient converge at rate of $O(1/\sqrt{K})$ to stationarity;

**2)** in the convex case both stochastic/deterministic subgradient converge at rate of $O(1/\sqrt{K})$ in function gap;

**3)** for non-convex and non-weakly-convex problems, it is impossible to establish a polynomial iteration complexity ($\mathcal{O}(\text{Poly}({1}/{{\varepsilon}}))$)  to our stationary measure in the worst case [Ohad20]. 

These results imply that a fundamental difficulty in improving SGD probably stems from the non-smooth structure rather than the stochasticity. We anticipate that any technique or assumption that drives faster rate of general SGD should benefit deterministic subgradient method as well. This is also an interesting direction for the future work.

We hope that the general case of SGD will not be a concern to the reviewer. Despite much discussion on SGD, it is indeed not our primary target to lay emphasis on SGD. Recall that a strong motivation to promote model-based optimization is the robustness to the hyper-parameter. In this perspective, SPL and SPP are better alternatives to SGD in the SMOD family. Hence, we think that our contribution in improving SPL and SPP by minibatching should be more impactful to the practitioners.

 Thank you for the very detailed response. I share the concern with Reviewer 5Yxy that while this paper claims to target non-smooth optimization, the actual problems considered are very well-structured and are quite close to those that are smooth. I encourage the authors to rephrase the words and add a proper comparison with the provided extra references. Also, as the authors mentioned, it would be nice to clear the unnecessary confusion caused by Assumptions A6 + A7. I will keep my score unchanged in the current iteration and may re-evaluate after the discussion with other reviewers.   

This paper studies minibatch and momentum algorithm for non-smooth non-convex optimizations. A linear speedup property of minibatch is achieved in non-smooth non-convex optimization.  Most interestingly the usage of minibatch is demonstrated to have a linear speed up property in nonsmooth setting, which is rare in the area. However, there are several limitations about the problem setting being considered. Although this paper claims to target on non-smooth optimization, the problems considered are several special classes of problems. 

1. The SSP assumes exact solution of sub-problem at each iteration, which is a very strong assumption of power of the subroutine. Taking the complexity of subroutines into consideration, it is very likely that the complexity will be much higher. 

2. The Prox. SGD requires smoothness of \ell(.), which setting has been well studied in existing works including in [8]. 

3. What is more, SPL requires smoothness of inner function C(.) (refer to line 133 of the submission), this should be clearly pointed out in early sections and explicitly in assumptions. Because the fact that the inner function where variance arise is smooth makes the algorithms quite restricted to this special class of problem. 

4. I also noticed a paper that studies a similar setting as the SPL and gets better sample complexity of O(1/\epsilon^3), while this paper gets O(1/\epsilon^4) sample complexity. See [Zhang, Junyu, and Lin Xiao. "Stochastic variance-reduced prox-linear algorithms for nonconvex composite optimization." arXiv preprint arXiv:2004.04357 (2020)]. Since this reference uses variance techniques, it is understandable that they get better rates, but it would be helpful to mention and compare with it. 


 Yes

 Overall, I raise my score to 6. Thank you for the well clarified response. The authors' feedback has addressed some of my concerns. However, I still think it is limited to special classes of problems instead of general problems which the title seems to imply. For the general case of SGD, the rate is not better than exiting works. By the way, for the reference [ZX20], I understand that you are under different setting, but I think your work is related to that one in terms of techniques to address the non-smoothness of outer function for SPL. 

 #### Reviewer 5Yxy

We thank the reviewer for the effort in reviewing our paper. 

**Question 1**

We agree the minibatch proximal point subproblem may cause extra complexity. However, we argue that at least for the composition function $h(c(x))$​​​​​​​​, the SPP subproblems can be efficiently solved by prox-linear methods with rapid convergence.  More specifically, the objective in the SPP subproblem (6) is strongly convex. It is easy to show that the deterministic prox-linear (DPL) method ,when applied to SPP subproblem, exhibits linear convergence to the optimum. Remarkably,  the subproblem is extremely well-conditioned because the strongly convex parameter $\\gamma-\\lambda$​​​​​​​​ is exceedingly large due to the choice of  $\\gamma$​​​​​​​​​​. Therefore, we can show that SPP has nearly the same complexity as SPL with difference up to some logarithmic factor. (In a preliminary experiment, we do observe linear convergence; we find that 4-5 DPL iterations give a solution with accuracy between 1e-05 and 1e-07). Since most of the interesting weakly convex problems in our paper have the composition form, we hope this can resolve the concern about the limitation for practical use.  Furthermore, in a revised version, we  will **1)** add experiments on minibatch SPP and **2)** add a simple proof to show the linear convergence of prox-linear method for SPP subproblem.

**Additional comment on complexity result**

We just argued that prox-linear applied to SPP subproblem has linear rate. The linear convergence of prox-linear method for strongly convex problem should not be a surprise.  As is noted by reviewer Cw8j,  [Dam20] shows that the accelerated prox-linear method achieves $O(1/{K^2})$​​​ for convex optimization. After reading, we find that [Dam20] also shows that standard prox-linear obtain $O(1/K)$​​​​​​​​. Although we did not find the result for strongly convex problems in [Dam20], we are able to show the linear convergence when preparing the review response. We will revise the paper accordingly.

**Question 2**

Prox-SGD is indeed carefully studied by [8]. However, it is not our ultimate goal to simply reproduce their result; we **1)** recover the same convergence rate of Prox-SGD and **2)** improve the rate of many other model-base algorithms in the minibatch setting; and we achieved both goals in a single framework. We feel that this is not easy using the traditional optimization techniques, and at least some more credits should be given to our unified convergence analysis based on statistical stability, which appears to reveal more new insights on minibatching than classic optimization techniques do. 

**Question 3**

As suggested by the reviewer, we will move the smoothness assumption of $C(\\cdot)$​​​ from Line 132 to the previous section where SPL is first introduced.  We agree with the reviewer that $C(\\cdot)$​​​ needs to be Lipschitz smooth. This assumption is actually very important.  Without this assumption, it is possible that the composite function is not weakly convex,  For example,  the composition of identity map $h(x)=x$​​​ and the continuous function $C(x,y)=|x|-|y|$​​​ is $h(C(x,y))=|x|-|y|=C(x,y)$​​​​​, which is not weakly convex (we cannot turn it convex by simply adding a quadratic term).  For the non-convex non-smooth optimization without weakly convexity, designing a *convergent sequential algorithm* is already a substantial challenge, see [Ohad 2020] on the discussion. Therefore, it is far beyond the scope of our paper.  We feel our assumptions are appropriate and beg to disagree with the reviewer on the limitation of the composition function.

While the concept of composition function appears to be in the spotlight recently, it is definitely not new, and it does have rich applications. We argued in the paper that many interesting machine learning problems have such a composition structure with inner smooth map, such as matrix completion, blind deconvolution. Hence they are weakly convex. In fact, convex models such as SVM and logistic regression also have this desired property since the linear map is smooth. We refer to the work [8, 9] for the rich applications in machine learning and signal processing. 



**Question 4**

Thanks for referring to [Zhang and Xiao 2020 (ZX20)]. We would gladly add reference to this paper. However, their problem formulation is quite different from ours, and we feel it is inappropriate to directly compare the complexity results. More detailedly, their setting has the inner part in expectation/finite-sum form while our problem is a special case of the stochastic programming which assumes that the expectation is taken outside of the composition. Notably, the difference in problem formulation to classic stochastic programming has been clarified by [ZX20] in their paper (page 5).

**Additional comments**


We completely understand the reviewer's concern about the limitation of problems and three algorithms and hope our answer resolves this issue. We feel that our work is an important step to make the model-based algorithm from sequential to the more realistic minibatch setting.  In addition, we would like to argue that our *momentum model-based algorithms* are also technically non-trivial and substantially broaden the application of momentum technique. We appreciate the time spent on our paper and sincerely ask the reviewer to re-evaluate the score if he/she finds our response addressing the concerns.

**References**

[Dam20] Damek Davis, Lecture Notes for ORIE 6300: Mathematical Programming I, 2020

[ZX20] Zhang, Junyu, and Lin Xiao. "Stochastic variance-reduced prox-linear algorithms for nonconvex composite optimization." *arXiv preprint arXiv:2004.04357* (2020).

[Ohad20] Shamir, Ohad. “Can We Find Near-Approximately-Stationary Points of Nonsmooth Nonconvex Functions?” ArXiv Preprint ArXiv:2002.11962, 2020.




 #### Reviewer h9LY

We thank the reviewer for recognizing the novelty and contribution of our work. 

**Question 1**

Indeed, parameter $\\gamma$​ can be set in a more intuitive way. Theorem 3.4 currently uses a fixed parameter  $\\gamma$​ because it depends on a theoretically optimized value $\\eta$​.  Indeed, $\\gamma$​ can be set for a range of values if we treat $\\eta$​​ as a free parameter rather than a fixed value.  Let us give some insights. In the proof of Theorem 3.4 Appendix  eq(37),  we showed
$$
\\frac{\\rho-\\lambda-\\tau}{\\rho}\\mathbb{E}[\\|\\nabla f_{1/\\rho}(x^{{k^*}})\\|^{2}]\\le \\frac{(2\\rho-\\lambda)\\Delta}{K}+\\min\\{\\frac{\\eta\\Delta}{K}+\\frac{2\\rho L^{2}}{m\\eta},\\frac{\\Delta\\eta}{K}+\\rho L\\sqrt{\\frac{2MD}{m\\eta}}\\}
$$
For the sake of  simplicity, let us now skip the second part in the $\\min$​ function  and get 

$$
\\frac{\\rho-\\lambda-\\tau}{\\rho}\\mathbb{E}[\\|\\nabla f_{1/\\rho}(x^{{k^*}})\\|^{2}]\\le \\frac{(2\\rho-\\lambda)\\Delta}{K}+\\frac{\\eta\\Delta}{K}+\\frac{2\\rho L^{2}}{m\\eta}
$$
Instead of optimizing $\\eta$​​​​​​​​​​​​, suppose now we take $\\eta=\\sqrt{K}/(\\sqrt{m}\\alpha_0)$​​​​​​​​​​​ where $\\alpha_0$​​​​​​​​​​​ is free, and  we get 
$$
\\frac{\\rho-\\lambda-\\tau}{\\rho}\\mathbb{E}\\|\\nabla f_{1/\\rho}(x^{{k^*}})\\|^{2}\\le \\frac{(2\\rho-\\lambda)\\Delta}{K}+(\\frac{\\Delta}{\\alpha_0}+2\\alpha_0\\rho L^{2})\\frac{1}{\\sqrt{mK}}
$$
Notice that since $\\Delta$​ and $L$​ are usually unknown a-priori, we may need to search for the optimal  $\\alpha_0$​ in practice and this explains why we tune  $\\alpha_0$​​​​​​​​​​​​​​​​​​ in the experiment.

In a later version, we will revise Theorem 3.4 and leave $\\eta$​​ as a free parameter using the above argument. We will also give the theoretically optimized value of $\\alpha_0$​​ and recover (10) as a special case.



**Question 2** 

We first wish to appologize for the abuse of notation and that we use the notion of potential function and Lyapunov function exchangeably. The Lyapunov function was defined in eq(18). Here we also wish to fix a typo in eq(18): our intended Lyapunov function should be
$$
f_{1 / \\rho} (z^k)  +\\frac{\\rho(\\gamma\\beta+\\rho\\beta^{2}\\theta^{-2})}{2(\\gamma\\theta-\\lambda\\theta)}\\|x^{k}-x^{k-1}\\|^{2}, \\quad\\quad (18)
$$
which is also consistent with Lemma 4.1.



**Question 3**

> ... is there any reason that SPP is not shown in this Figure?

We skipped SPP in Figure 1 since the interior point method solver we used for the SPP subproblems encounters numerical difficulty when batchsize $m$​​​​​​​ increases to more than 10. This is largely due to the difficulty of solving the equivalent QCQP by interior point methods. (Please also check Remark 11, Line 689 in the Appendix. D. for a more detailed explanation)   

We remark, while interior point method may be not a great choice for SPP, we have now provided an efficient routine to solve SPP subproblem by prox-linear method. In particular,  the SPP subproblems are well-conditioned and the prox-linear method achieves linear convergence. Therefore, we would expect that SPP has nearly the same efficiency as SPL with difference up to some logarithmic factor. We will revise the paper accordingly and update the experiment results for SPP.  We already have some promising preliminary results. In the table below we present the speedup for SPP with $\\kappa=10, p_\\text{fail}=0.3, m \\in \\{16, 32, 64, 128\\}$​​​​​​​​​ over 5 repetitions. The result is indeed consistent with our observations on other methods.

|    m    |  1   |  16  |  32  |  64  | 128  |
| :-----: | :--: | :--: | :--: | :--: | :--: |
| Speedup |  1   |  18  |  35  |  85  | 200  |

For more details about algorithm complexity please also see our first comment to reviewer 5Yxy. 

>  On different choices of $\\alpha_0$​​​ for Figure 1 and Figure 3 

In our initial experiment design, Figure 1 and 3 are plotted independently to show the effect of minibatch and momentum and we do not use the same stepsize ranges in the setup. More specifically, Figure 1 and 2 are for batchsize $m\\geq1$ without momentum while Figure 3 examines the effect of momentum method with batchsize $m=1$​. 

> On the first sub-figure of Figure 1

As with the first sub-figure of Figure 4, we indeed observe some interesting results from the whole set of experiments. Namely,  momentum empirically accelerates the convergence of the SMOD methods for a mild range of stepsize selections. However, aggressively using momentum with large stepsize may hurt convergence and drive the algorithm unstable. This is indeed observed at the large stepsize $\\alpha_0=10$​​​​​.

We hope that the above can resolve the reviewer's concerns and we thank the reviewer again for the time spent on our paper.



 #### Reviewer Cw8j

We appreciate the effort of reviewer in reviewing our paper.

**Question 1**

We first wish to thank the reviewer for providing the reference to [Dam20]. Indeed, it is possible to accelerate the convergence rate to $O(1/K^2)$​ using Nesterov's acceleration. Interestingly, this result aligns with our additional materials on the stochastic convex optimization in Appendix C. Specifically, we further developed a Nesterov-type accelerated stochastic model-based method in the convex setting, which achieves $O(1/K^2 + 1/\\sqrt{mK})$​. This means if the problem is deterministic, the algorithm has the optimal deterministic rate and if the problem is stochastic, it has the best $1/\\sqrt{mK}$​ rate obtained by minibatch acceleration. We leave it in the appendix and sincerely remind the reviewer of this result in case it is out of the radar.  And we apologize for not mentioning it beforehand.

**Question 2**

Compared with [Asi20] , it is indeed hard to claim which one is more conservative than the other.  Specifically, [Asi20]  requires the objective to have a convex lower quadratic approximation. Although we also have a lower quadratic growth, the quadratic function actually has a negative sign: $f(y,\\xi)\\ge f_x(y,\\xi)-\\frac{\\tau}{2}\\|x-y\\|^2$​​​,  which is less restrictive when $ f_x(y,\\xi)$​​​ is the linear approximation. However, we also require a convex quadratic upper bound, which is not needed in [Asi20]. The more recent work [ACCD21] established accelerated convergence of $O(1/K^2 + 1/\\sqrt{mK})$​​​ under the Lipschitz smoothness condition while we achieve such a complexity only using non-smoothness assumption. Moreover,  [ACCD21]  also considered  interpolation problems which is not discussed in our paper.

**Question 3**

Indeed, we feel that the additional assumptions A6 and A7 are non-standard and  have already caused some confusions to the reviewer. Yes, we can drop A6-A7 very easily. In a later revision, we will remove A6 and A7 and just state the result under A5. Consequently, in Lemma 3.1, the  $\\varepsilon$​ stability will be defined with $\\varepsilon=\\frac{2L^2}{m(\\gamma-\\lambda)}$​. In Theorem 3.4, the bound(10) will have no $C_2$​ term, and parameter $\\eta$​ will be set to $\\eta=L(\\frac{2\\rho K}{m\\Delta})^{1/2}$​. This gives us a rate of the form $O(1/K+1/\\sqrt{mK})$​. Thus our main result still holds.  Note that this $\\eta$​ is a theoretically optimized value (please see our proof in the appendix). As we promise reviewer h9LY,  we will slightly rewrite Theorem 3.4 to set $\\eta$​ as a free parameter, but we will remark that taking $\\eta=L(\\frac{2\\rho K}{m\\Delta})^{1/2}$​​​​ gives an optimized upper bound as desired.

**Additional comments on limitations**

In our analysis and experiments, we focus on the total complexity in terms of the proximal mapping since this is the most time consuming part in the modern large scale setting, unless the the batchsize $m$​​ is overly large. The subproblem can be efficiently solved by parallelizing. Notably, SPL update can be parallelized by solving the dual problem or be efficiently solved by interior point methods. SPP can be solved by a few prox-linear iterations. (please see our first comment to reviewer 5Yxy). We also refer to [Asi20] for the details on parallel implementation of the proximal updates.

We also thank the reviewer for pointing out the typos and wish that the above can resolve the reviewer's concerns.

**References**

[Asi20] Asi, Hilal, et al. "Minibatch Stochastic Approximate Proximal Point Methods." *Advances in Neural Information Processing Systems* 33 (2020).

[ACCD21] Chadha, Karan, Gary Cheng, and John C. Duchi. "Accelerated, Optimal, and Parallel: Some Results on Model-Based Stochastic Optimization." *arXiv preprint arXiv:2101.02696* (2021).

This paper proves the effectiveness of using minibatch and momentum in stochastic model-based methods (SMOD) under the non-smooth weakly-convex setting. The authors utilize the concept in algorithm stability framework to show that minibatch SMOD has linear speedup over the batch size. They also give the first complexity results of momentum stochastic proximal-linear (SPL) and proximal point (SPP) algorithms for non-smooth weakly-convex optimization.  This paper is very well-written and easy to read.
The authors propose to use the concept in algorithm stability for the analysis of convergence, which is novel to me since the algorithm stability is initially proposed for the analysis of generalization. They obtain their main convergence results based on bounding the effect of one-step stochastic proximal mapping, i.e., the stability of each iteration. This aligns with the intuition of using SMOD over SGD, i.e., for better robustness under non-smooth weakly-convex setting. In their analysis, using minibatch in SMOD is able to provide linear speed up since the batch size is shown in the stability (Lemma 3.1) and can help optimize the choice of $\eta$ and the stepsize $\gamma$ in Theorem 3.4. 
They also use the same way to analyze the method of SMOD with heavyball momentum.  Moreover, in the Appendix, they show results on both heavyball and Nesterov momentum under convex setting.
Their experiments are standard among the literatures of SMOD. With the results shown in the experiments, their theoretical results are verified.

My questions:
1. For Theorem 3.4, in the proof, we can see $\gamma$ cannot be too large since we require $\gamma \leq \rho + \tau + \lambda + \eta$. As the main reason to choose SMOD is for the wide range of the choice of step size, to better show the linear speedup of minibatch SMOD, can the author change the condition on $\gamma$ in Theorem 3.4 to a range of $\gamma$ instead of just $\gamma=\max(\rho+\tau, \lambda+\eta)$?  This is also related to the experiments in this paper:  the initial stepsize $\alpha_0$ is in a range (e.g., Figure 2) and it is unknown why this range is useful for verifying the results of Theorem 3.4.

2. In Line 57, for the analysis of SMOD with momentum, the authors mention they develop a unified Lyapunov analysis. Where is the Lyapunov function actually used for analysis? It may be implicit shown in the proof but I did not find it.

3. For the experiments, in Figure 1, the authors only compared SPL and SGD, is there any reason that SPP is not shown in this Figure?  Also in Figure 1, the difference between SPL and SGD is relatively small for $\alpha_0\in[10^{-1}, 10^0]$, so why in Figure 3, the range in x-axis is chosen as $[10^{-2}, 10^0]$? We can see that for the first subfigure in Figure 4, there are some interesting results around $\alpha_0=10^1$. This extreme case ($\alpha_0\geq 10^1$) may help us understand the momentum minibatching SMOD method better (compared with Theorem 4.3).






 No.

The paper studies the convergence of stochastic model-based methods with mini-batching and proves theoretical speedup for quadratically accurate models such as the proximal point and prox-linear ones.
A momentum extension is also studied.   
The keys to their analysis are algorithm stability and the new Assumption A5. Assumption A5 brings us close to a situation as we would have been if Lispchitz continuous gradient was assumed in the analysis of SGD. In particular, we are granted some kind of *decent lemma*. 

Indeed, a similar condition has been used to unify the analysis of several model-based methods [Damek, 2020]. More concretely, the accelerated rate O(1/k^2) can be obtained for the proximal point and prox-linear methods on *non-smooth* deterministic problems. So I would say that under A5, the linear speedup is rather natural; the proof may not be trivial though.

The author(s) mentioned the restricted convexity (RSC) assumption when compared to [Asi et al., 2020], can you comment on the difference with a somewhat more recent version of that paper [ACCD21]? 
It seems that the RSC requires some lower quadratic growth (QG) of the original function f(x, \xi), while A5 requires QG on both sides but with respect to the model f_y(x, xi). Which one is more conservative? Can you give an example? 

In eq. (7), both the samples are \xi'??

As the author(s) mentioned that removing A6+A7 does not affect the main convergence results, can you drop them?


References: 

[Dam20] Damek Davis, Lecture Notes for ORIE 6300: Mathematical Programming I, 2020

[ACCD21] Chadha, Karan, Gary Cheng, and John C. Duchi. "Accelerated, Optimal, and Parallel: Some Results on Model-Based Stochastic Optimization." arXiv preprint arXiv:2101.02696 (2021).  The bound concerns only the number of proximal steps and not the total complexity. The author did not tell us how to carry out the mini-batch update in general? Without an efficient procedure, it would be hard to compete with gradient-based methods. 

Meta-review: This paper analyzes two extensions of the recently studied (stochastic) model based optimization methods: i) minibatch variant, ii) momentum variant. The authors focus on a class of nonsmooth nonconvex functions which become convex after sufficiently strong Tikhonov regularization. The analysis approach is based on leveraging a recent result by Davis and Drusvyatskiy characterizing the degree of stationarity of a point via the gradient of its Moreau envelope, and on utilizing tools from algorithmic stability literature. The authors recover several existing and several new convergence results.

The reviewers assessed the paper favorably. There was a reasonable discussion between the reviewers and the authors. I have read the paper myself and agree that the work is of sufficient quality to be published in NeurIPS. I have noticed a number of grammatical and stylistic issues and would recommend the authors to carefully proofread the paper.

Decision: accept