Title: Fast Extra Gradient Methods for Smooth Structured Nonconvex-Nonconcave Minimax Problems
Authors: Sucheol Lee
Donghwan Kim
Venue: NeurIPS 2021
Keywords: Minimax problems
Nonconvex-nonconcave Problems
Extragradient Method
Acceleration

Abstract: Modern minimax problems, such as generative adversarial network and adversarial training, are often under a nonconvex-nonconcave setting, and developing an efficient method for such setting is of interest. Recently, two variants of the extragradient (EG) method are studied in that direction. First, a two-time-scale variant of the EG, named EG+, was proposed under a smooth structured nonconvex-nonconcave setting, with a slow $\mathcal{O}(1/k)$ rate on the squared gradient norm, where $k$ denotes the number of iterations. Second, another variant of EG with an anchoring technique, named extra anchored gradient (EAG), was studied under a smooth convex-concave setting, yielding a fast $\mathcal{O}(1/k^2)$ rate on the squared gradient norm. Built upon EG+ and EAG, this paper proposes a two-time-scale EG with anchoring, named fast extragradient (FEG), that has a fast $\mathcal{O}(1/k^2)$ rate on the squared gradient norm for smooth structured nonconvex-nonconcave problems; the corresponding saddle-gradient operator satisfies the negative comonotonicity condition. This paper further develops its backtracking line-search version, named FEG-A, for the case where the problem parameters are not available. The stochastic analysis of FEG is also provided.

TL;DR: We propose a fast extragradient method, with an accelerated O(1/k^2) rate, for smooth structured nonconvex-nonconcave problems.

Review discussion:
The paper is very well written and developed sound results. The FEG method is the first accelerated algorithm for nonconvex-nonconcave problems with negative-monotonicity.
Therefore, this paper makes a solid contribution to the existing literature.
Below are some questions.
1. From the analysis in sec 7.1, it is not clear how the reused $Fz_k$ term in FEG help achieve an accelerated convergence rate? How substantial would the proof be different from that of EAG?
2. I think it is important to numerically compare this algorithm with the competitive algorithms mentioned in the paper. Given that FEG only introduces a reuse of $Fz_k$, it is not clear if this can easily accelerate the convergence in practice.
3. The result of stochastic FEG has a bad dependence on the stochastic variance $\sigma_k^2$. There are existing accelerated SGD analysis under constant noise variance, e.g., Accelerated Gradient Methods for Nonconvex Nonlinear and Stochastic Programming, Saeed Ghadimi · Guanghui Lan. This might be an interesting reference.
No
We would like to thank the reviewer for
the nice summary of our work and constructive feedback.
$\bullet$ **(Reusing $Fz_k$):**
Just to be clear, reusing $Fz_k$ was a key
for preserving an accelerated rate even under a negative comonotonicity.
Note that EAG is studied for the monotone case ($\rho=0$),
and the corresponding FEG (with $\rho = 0$) does not reuse $Fz_k$.
At this point, we do not have a clear interpretation
why reusing $Fz_k$ is especially useful under a negative comonotonicity,
which we leave as future work.
$\bullet$ **(Difference between analyses of FEG and EAG):**
As stated in the paper,
both methods use potential functions in a same form.
However, due to the specific formulation of class FEG,
our analysis is much simpler than that of EAG,
and we believe this helped us
to (relatively easily) study various extensions such as FEG-A and S-FEG.
$\bullet$ **(Numerical experiments):**
We ran a toy experiment with a function
$f(x,y) = \frac{\rho L^2}{2} x^2 + L\sqrt{1-\rho^2L^2}xy - \frac{\rho L^2}{2} y^2$, which has an $L$-Lipschitz $\rho$-comonotone saddle gradient.
For the case $\rho = -1/3L$ and $L=1$,
we observed that FEG converges with an accelerated rate,
whereas EG+, EAG-C, and EAG-V diverge.
We will add this in the revision.
$\bullet$ **(Acceleration under the stochastic setting):**
Thanks for introducing the nice reference.
We will add it in the revision.
Just to clarify, in line 217-219,
we mentioned that we would like to adjust the step coefficients
of the S-FEG to have a better rate for a constant noise case in near future,
similar to [5] studying a stochastic accelerated gradient method for minimization.
We want to point out here that the suggested reference
does analysis somewhat similar to [5].
This paper studies nonconvex-nonconcave minimax optimization with smooth and comonotonicity assumptions. The authors combine the idea of EG+ and EAG which leads to a two-time-scale EG with anchoring, named fast extragradient (FEG), that has a fast $O(1/k^2)$ convergence rate on the norm of the gradient. The paper also propose a backtracking line-search variant the problem parameters are not available and extended the algorithm to stochastic case.
This paper improves the convergence rate of nonconvex-nonconcave minimax optimization under comonotonicity assumptions. My main consideration is why such setting is useful. Example 1 connects the assumption to $\alpha$-interaction dominate condition, however, it is no clear which types of problem satisfies $\alpha$-interaction.
It is also desired to see some numerical experiment to validate the effectiveness of the theory, since the extension of FEG are proposed for implementation in practice. The audience of NeurIPS are more interested in the optimization algorithm related to machine learning models.
Hence, I hope the author need to provide more discussion for the applications of the proposed algorithms.
No
We would like to thank the reviewer for
the nice summary of our work and constructive feedback.
$\bullet$ **(Importance of comonotonicity and $\alpha\ge0$-interaction condition):**
We believe that studying
the comonotonicity setting is
one of the important steps
for understanding the training dynamics of
the modern nonconvex-nonconcave minimax problems
such as GAN and adversarial training.
However, we agree that this was not well explained in the paper,
and we would like to further explain
the $\alpha\ge0$-interaction dominant condition in [10]
(yielding the comonotonicity).
A convex-concave function is $\alpha\ge0$-interaction dominant.
On the other hand, for any $\gamma$-weakly-convex-weakly-concave function,
the condition holds with $\alpha = -\gamma < 0$.
Its extreme case is $\phi(x,y) = -\frac{\gamma}{2}x^2 + \frac{\gamma}{2}y^2$,
where there is no interaction between $x$ and $y$.
Interestingly, a nonconvex-nonconcave function
becomes $\alpha\ge0$-interaction dominant,
when the second term in the condition is sufficiently positive definite.
In other words, the condition is satisfied
when the interaction term of the Hessian
$\nabla^2_{xy}\phi$ is dominating any negative curvature
in Hessians $\nabla_{xx}^2\phi$ and $-\nabla_{yy}^2\phi$.
In short, the $\alpha\ge0$-interaction dominant condition captures whether
there is a sufficient coupling between $x$ and $y$,
and we believe many practical (adversarial) applications
(at least locally or asymptotically) satisfy such general interaction condition.
$\bullet$ **(Numerical experiments):**
As a starter,
we ran a toy experiment with a function
$f(x,y) = \frac{\rho L^2}{2} x^2 + L\sqrt{1-\rho^2L^2}xy - \frac{\rho L^2}{2} y^2$, which has an $L$-Lipschitz $\rho$-comonotone saddle gradient.
For the case $\rho = -1/3L$ and $L=1$,
we observed that FEG converges with an accelerated rate,
whereas EG+, EAG-C, and EAG-V diverge.
We will add this in the revision.
We agree that many practical audience of NeurIPS
will like seeing more realistic experiments,
but we think that it is beyond the scope of this paper.
We promise to make this work more practical for practitioners
in near future.
In Example 1, the work of Grimmer et al. does not use $\gamma$-smoothness, but rather a weaker condition (called weak convexity-weak concavity therein). The proof in the appendix does not rely on smoothness, so this difference is minor.
After defining comonotonicity, the paper defines three cases (negative, zero, and positive) implying the resulting theory would cover all three, most impressively the negative \rho setting. It was disappointing then in the theory to see the results only cover a very limited range of negative \rho (between -1/2L and 0). The paper should weaken its language when claiming a rate for negative comonotone problems as only a very limited class of these are able to be addressed by the theory presented.
Section 4 presents detailed comparisons between derived guarantees in the three settings (limited negative rho, rho=0, and \rho>0) with existing bounds. Much of the improvements are in terms of small constants: (i) The limited negative \rho range is larger by a factor of four for FEG than EG+, (ii) The FEG constant is smaller than EAG by 4/27. Numerically, are these improvements realized? They are small enough that I suspect these differences may come from proof technique differences rather than actual improvements in optimization power.
No
We would like to thank the reviewer for
the nice summary of our work and constructive feedback.
$\bullet$ **(Work of Grimmer et al.):**
We appreciate the comment on the work of Grimmer et al. [10].
We will revise the paper as suggested.
$\bullet$ **(Lower bound on the comonotonicity):**
We understand that one might view
the lower bound on the comonotonicity as a limitation,
since such condition is not widely studied yet.
First of all, EG+ has a tighter lower bound $\rho>-\frac{1}{8L}$,
unlike our  $\rho>-\frac{1}{2L}$.
In addition,
the case $\rho\le -\frac{1}{L}$ is out of
anyone's interest,
because it contains extreme cases
such as a "concave-convex" function
$f(x,y) = -\frac{L}{2}x^2 + \frac{L}{2}y^2$
without a coupling term.
We hope you now see that the lower bound is necessary
and is not so restrictive.
$\bullet$ **(Practical improvement of FEG):**
Our main contribution is that FEG
works for a class of nonconvex-nonconcave problems
that other methods are not known to converge.
As a practical example, we ran a toy experiment with a function
$f(x,y) = \frac{\rho L^2}{2} x^2 + L\sqrt{1-\rho^2L^2}xy - \frac{\rho L^2}{2} y^2$, which has an $L$-Lipschitz $\rho$-comonotone saddle gradient.
For the case $\rho = -1/3L$ and $L=1$,
we observed that FEG converges with an accelerated rate,
whereas EG+, EAG-C, and EAG-V diverge, as expected by the theory.
For the monotone case with $\rho = 0$ and $L=1$,
we also observed that FEG is faster than EAG-C and EAG-V as expected.
We will add these results in the revision.
$\bullet$ **(Worst-case for AGDA):**
We were not able to come up with a simple way
to further incorporate AGDA in the context of Lemma 8.1.
Instead, we would like to mention
two recent works [Letcher, et. al., 2020] and [Hsieh, et. al, 2021]
that study the fundamental limits of finding a stationary point of the smooth minimax problems. Both
consider some classes of algorithms that includes AGDA,
and construct examples that no method in such class
converge to a stationary point.
We believe that this can be a good starting point
for answering the reviewer's question.
**References**
- Letcher, Alistair. "On the Impossibility of Global Convergence in Multi-Loss Optimization." International Conference on Learning Representations. 2020.
- Hsieh, Ya-Ping, Panayotis Mertikopoulos, and Volkan Cevher. "The limits of min-max optimization algorithms: Convergence to spurious non-critical sets." International Conference on Machine Learning. PMLR, 2021.
Thank you for your careful reply specifying the limitations of the work in types of problems and algorithms that these ideas can apply to. My score (of Good paper, accept) remains unchanged.
However, I do want to pushback on the authors claiming the regime of \rho <= -1/L is out of anyone's interest. I agree there are some degenerate problems (like the mentioned concave-convex one) in this regime, but I suspect most important nonconvex-nonconcave applications actually fall in this difficult range. In my view, the restriction to \rho > -1/L is a particularly heavy limitation that should be acknowledged, hopefully prompting future works to provide algorithms and guarantees for more problem settings beyond this boundary (and consequently apply more realistically to settings like GAN training).
We are happy to hear that you found our response satisfactory. We realized that our statement was a bit strong to make a point, and we agree to your statement that the regime of $\rho\le -1/L$ should be of interest in future. We will make sure to revise the paper so that such limitation is clearly acknowledged in the revision.
This is an interesting result and the iteration complexity of the provided method is optimal. I think it is a solid contribution for a non trivial class of functions that include well studied classes. My main comment is that the authors mention that they tackle structured non convex non concave landscapes (in the abstract and later). They should rephrase it because they focus on a specific class of non convex non concave functions and sounds like overselling. Technically the paper is okay (accelerated extra gradient has appeared in other paper which are cited, nothing of great novelty here) but the authors should put some work on explaining key technical differences between their paper and the literature (on monotone, mvi etc).
Another comment is that the authors mention all their results in terms of the square of the norm of the gradient, i think the norm of the gradient would be more preferable.
No
We are glad to hear that you found our work interesting,
and we appreciate constructive feedback.
$\bullet$ **(Optimality of FEG in convex minimization?):**
We appreciate the reviewer for further studying our result
in convex minimization.
Even though we hope to have optimality even in convex optimization,
as the reviewer claims,
we want to clarify that it is not the case.
The optimal complexity in terms of the gradient norm
is $O(1/\sqrt{\epsilon})$ [Nemirovsky, 1992].
The standard GD has $O(1/\epsilon)$ with respect to the gradient norm,
and Nesterov's method has $O(1/\epsilon^{2/3})$ complexity [Kim and Fessler, 2018].
We are yet unaware whether FEG has a complexity
better than $O(1/\epsilon)$
for convex minimization.
$\bullet$ **(Highlighting nonconvexity-nonconcavity):**
We added the term "structured" in front of nonconvexity-nonconcavity
to concisely illustrate
that we are considering a specific class of nonconvex-nonconcave problems,
as it is done in [7].
However, we understand that one might think this to be overselling.
We will revise the abstract and the beginning of the introduction accordingly,
so that the readers see the exact condition
from the beginning.
$\bullet$ **(Key technical differences):**
We thought that the key technical differences
between our work and other related works
(under the monotonicity or MVI)
are well illustrated in Table 1 and in Section 4.
Could you let us know
which technical aspects you would like to further hear from us?
We will be happy to answer them.
$\bullet$ **(Squared gradient norm):**
We agree that the norm of the gradient
can be preferred.
However, since the main related works [6,7,38]
use the squared gradient norm,
we would like to follow them for consistency and readability.
1. Nemirovsky, "Information-based complexity of linear operator equations", Journal of Complexity, 1992.
2. Kim and Fessler, "Generalizing the optimized gradient method for smooth convex minimization", SIAM J. Optimization, 2018.
Thank you so much! I messed things up because I was thinking about the value of the function squared (instead of the norm of the gradient).
Of course even if the rates are not optimal (as I thought) the result is still good and I keep my score. My other comment was also addressed
(after more careful reading on section 4).
We are glad to hear that you are satisfied with our response.

Meta-review:
role: PC
data: comment: This paper proposes a new algorithm that can get 1/k^2 convergence rate for a class of structured smooth nonconvex-nonconcave minimax optimization problems, in terms of the squared gradient norm, improving over the currently known 1/k rate algorithm for the same problem. While the paper does not provide any practical example that falls into the class being analyzed here, the reviewers felt that the paper makes an interesting theoretical progress on the hard and challenging problem of nonconvex-nonconcave minimax optimization. Furthermore, the techniques developed in this paper can potentially lead to further work in this area. Consequently, I am recommending the paper for acceptance.

Decision: Accept (Poster)