Title: Closing the Gap: Tighter Analysis of Alternating Stochastic Gradient Methods for Bilevel Problems
Authors: Tianyi Chen
Yuejiao Sun
Wotao Yin
Venue: NeurIPS 2021
Keywords: Stochastic bilevel optimization
min-max optimization
compositional optimization
convergence analysis

Abstract: Stochastic nested optimization, including stochastic compositional, min-max, and bilevel optimization, is gaining popularity in many machine learning applications. 
While the three problems share a nested structure, existing works often treat them separately, thus developing problem-specific algorithms and analyses. 
Among various exciting developments, simple SGD-type updates (potentially on multiple variables) are still prevalent in solving this class of nested problems, but they are believed to have a slower convergence rate than non-nested problems. 
This paper unifies several SGD-type updates for stochastic nested problems into a single SGD approach that we term ALternating Stochastic gradient dEscenT (ALSET) method. By leveraging the hidden smoothness of the problem, this paper presents a tighter analysis of ALSET for stochastic nested problems. 
Under the new analysis, to achieve an $\epsilon$-stationary point of the nested problem, it requires ${\cal O}(\epsilon^{-2})$ samples in total. 
Under certain regularity conditions, applying our results to stochastic compositional, min-max, and reinforcement learning problems either improves or matches the best-known sample complexity in the respective cases. 
Our results explain why simple SGD-type algorithms in stochastic nested problems all work very well in practice without the need for further modifications. 

TL;DR: Our results explain why simple SGD-type algorithms all work very well in practical bilevel problems without the need for further modifications. 

Review discussion:
Overall I think the paper is well written and the idea of proposing a unified analysis for all the optimization problems discussed in this work can have significant impact.
Also closing the gap between the complexity of simple SGD and bilevel optimization is definitely important in the process of delineating which aspects of bilevel optimization are actually hard.
While the smoothness of the lower level problem has been used in prior work (e.g. [27]), it is the smoothness of the gradients proved in Lemma 2 that opens the way to the improved rate.
While Lemma 2 itself is rather straightforward to prove,  how and why it can be used in the proof is definitely not. Small tweaks to existing approaches are not sufficient for the reasons summarized under Equation 20. I suggest to the authors to expand more on this reasons in the Appendix to make the paper more self contained.
Overall a very strong and well written submission.
No
We thank the reviewer for acknowledging that our idea of proposing a unified analysis for all the optimization problems discussed in this work can have significant impact. Below is our reponse to your minor comment.
**Q1.
While Lemma 2 itself is rather straightforward to prove, how and why it can be used in the proof is definitely not. I suggest to the authors to expand more on this reasons in the Appendix to make the paper more self contained.**
As the reviewer correctly pointed out, the smoothness of the gradients proved in Lemma 2 opens the way to the improved rate. Following your suggestion, we will further discuss how to use this smoothness in the proof and why it can lead to the improved rates. We believe this technique can be used in the future analysis of stochastic nested problems.
I found the manuscript to be clearly written and technically sound. The proposed algorithms are also easy to understand. The authors give a detailed comparison between their result and previous results, where the improvement in the sample complexity is clearly stated. The only suggestion I have is can you also provide some numerical experiments to support your theoretical results?
No
Thank you very much for acknowledging that our paper is clearly written and technically sound. We would like to address your only comment below.
**Q1. The only suggestion I have is can you also provide some numerical experiments to support your theoretical results?**
Following your valuable suggestion, during the rebuttal period, we have conducted the experiment using the risk-averse portfolio management task on a benchmark dataset - 100 Book-to-Market. This is a typical application of stochastic nested optimization used in references [39], [40]. We compared the two-timescale nested SGD approach (SCGD) with our single-timescale nested SGD approach ALSET.
We tune the stepsizes $\alpha, \beta$ by following the suggested order in the original SCGD paper and then using a grid search for the constant $c$, that is
SCGD: $\alpha = c\cdot k^{-3/4}$, $\beta = c\cdot k^{-1/2}$
ALSET: $\alpha = c\cdot  k^{-1/2}$, $\beta = c\cdot k^{-1/2}$
The constant $c$ is chosen from the searching grid  \{$10^{-3}, 5\times 10^{-4}, 10^{-4}$\} and optimized for each algorithm in terms of ergodic average gradient norm versus the number of iterations. In the table below, we report the logarithmic value of the average gradient norm performance of SCGD and ALSET with both the above decreasing stepsizes and the constant stepsizes (replacing k with total number of iterations K=1000). We can observe that the empirical gradient-norm convergence rate of ALSET is no worse than the worst-case theoretical rate, and ALSET outperforms SCGD thanks to its single-timescale stepsizes.
| \# of iter | log(iter) |SCGD| ALSET | ALSET (constant)|
| :------------- | :------------- | :----------: | :-----------: | :-----------: |
|  10  | 2.30|  5.32   |  5.31    |  5.63 |
| 100 | 4.61 |  3.78| 3.49   | 3.63 |
| 200| 5.30   |  3.40   | 2.94  |  3.06 |
| 1000| 6.91   | 2.57  |  1.65 |  2.06|
Thanks again for summarizing our paper and recognizing the merits of our paper!
Thank you for providing the numerical experiments.
The contributions of the work is mainly theoretical.
Although the proof technique is fairly standard, it achieves better rates with a simple single-simescale algorithm. This work also points out the importance of the hidden smoothness of the optimizer of the lower level. It can be very important for further works in the field. The general framework allows us to easily generalize different variants of gradient descent methods to different settings.
As a result, there are many possible lines of research following this work.
The paper is clearly written and the results are well-organized. Previous works are cited properly.
No
Thanks a lot for summarizing the contributions of our paper. They are very accurate.
Also thanks for recognizing the merits and the potential impact of our theoretical results!
We very much appreciate your time and efforts on reviewing our paper.
This paper is well-written. I have some comments below.
1. This paper improves the sample complexity for SGD-type algorithms to solve stochastic nested optimization. The algorithm and some results (like observation (6) and estimation (8)) are already presented in previous literature (e.g., Ghadimi and Wang (2018)). Theorem 1 gives a tighter bound of such algorithm where they utilize the observations in Lemma 2. The authors give a clear presentation of their technique and highlight the key steps of the proof towards Theorem 1.
However, it is my understanding that the novelty of this paper is merely the observation of lemma 2 and the tighter analysis following it.
2. Under the unified structure, they extend their theoretical results to stochastic compositional optimization as well as stochastic min-max optimization. They also give an application example of the actor-critic method with linear value approximation.
3.
However, this paper is not complete. A conclusion part should be presented.
References mentioned above:
1. Ghadimi, Saeed, and Mengdi Wang. "Approximation methods for bilevel programming." *arXiv preprint arXiv:1802.02246* (2018).
No
Thanks a lot for the careful review and favorable recommendation.
**Q1. The novelty of this paper is merely the observation of lemma 2 and the tighter analysis following it.**
Indeed, it is the smoothness in Lemma 2 that opens the way to the improved rate.
However, as one of  reviewers has pointed out, how and why it can be used in the proof is definitely not strightforward. We will expand more on this technique and hope it can be used in future proofs.
**Q2.
However, this paper is not complete. A conclusion part should be presented.**
Due to space limitation, we cannot add the conclusion in the initial submission.
But following your suggestion, we will certainly add a conclusion part in the final paper. We provide a tentative conclusion below.
> ``This paper unifies several SGD-type approaches for stochastic nested problems into a single nested SGD approach that we term ALSET, which runs in a single timescale and uses a fixed batch size. The paper further presents a tighter analysis for using it to solve three different stochastic nested problems - stochastic compositional, min-max and bilevel problems.
Under the new analysis, to achieve an $\epsilon$-stationary point of the nested problem, ALSET requires ${\cal O}(\epsilon^{-2})$ samples in total, which is the best sample complexity of the vanilla nested SGD. Under certain regularity conditions, applying the new analysis to an alternating version of the actor-critic algorithm also yields the state-of-the-art sample complexity.
> To overcome the limitations of ALSET, our future work consists of performing comprehensive simulations to validate this sample complexity in practice and relaxing the regularity conditions needed to achieve this theoretical result.’’
Any comments and suggestions on the conclusion are welcome.

Meta-review:
role: PC
data: comment: It is the consensus of the reviewers that this paper makes a worthwhile contribution to stochastic optimization. The analysis of alternating gradient descent is tight. The implications of the analysis on stochastic compositional, min-max, and bilevel optimizations are interesting and useful.

Decision: Accept (Spotlight)