Fast Extra Gradient Methods for Smooth Structured Nonconvex-Nonconcave Minimax Problems

Abstract: Sucheol Lee, Donghwan Kim

Reviews and author responses:

 We are happy to hear that you found our response satisfactory. We realized that our statement was a bit strong to make a point, and we agree to your statement that the regime of $\rho\le -1/L$ should be of interest in future. We will make sure to revise the paper so that such limitation is clearly acknowledged in the revision.

 We are glad to hear that you are satisfied with our response.

 Thank you for your careful reply specifying the limitations of the work in types of problems and algorithms that these ideas can apply to. My score (of Good paper, accept) remains unchanged.

However, I do want to pushback on the authors claiming the regime of \rho <= -1/L is out of anyone's interest. I agree there are some degenerate problems (like the mentioned concave-convex one) in this regime, but I suspect most important nonconvex-nonconcave applications actually fall in this difficult range. In my view, the restriction to \rho > -1/L is a particularly heavy limitation that should be acknowledged, hopefully prompting future works to provide algorithms and guarantees for more problem settings beyond this boundary (and consequently apply more realistically to settings like GAN training).

 Thank you so much! I messed things up because I was thinking about the value of the function squared (instead of the norm of the gradient). Of course even if the rates are not optimal (as I thought) the result is still good and I keep my score. My other comment was also addressed 
(after more careful reading on section 4).

 We would like to thank the reviewer for 
the nice summary of our work and constructive feedback.

$\bullet$ **(Reusing $Fz_k$):**
Just to be clear, reusing $Fz_k$ was a key
for preserving an accelerated rate even under a negative comonotonicity.
Note that EAG is studied for the monotone case ($\rho=0$),
and the corresponding FEG (with $\rho = 0$) does not reuse $Fz_k$.
At this point, we do not have a clear interpretation
why reusing $Fz_k$ is especially useful under a negative comonotonicity,
which we leave as future work.

$\bullet$ **(Difference between analyses of FEG and EAG):**
As stated in the paper,
both methods use potential functions in a same form.
However, due to the specific formulation of class FEG,
our analysis is much simpler than that of EAG,
and we believe this helped us
to (relatively easily) study various extensions such as FEG-A and S-FEG. 

$\bullet$ **(Numerical experiments):**
We ran a toy experiment with a function
$f(x,y) = \frac{\rho L^2}{2} x^2 + L\sqrt{1-\rho^2L^2}xy - \frac{\rho L^2}{2} y^2$, which has an $L$-Lipschitz $\rho$-comonotone saddle gradient.
For the case $\rho = -1/3L$ and $L=1$, 
we observed that FEG converges with an accelerated rate, 
whereas EG+, EAG-C, and EAG-V diverge.
We will add this in the revision.

$\bullet$ **(Acceleration under the stochastic setting):**
Thanks for introducing the nice reference.
We will add it in the revision.
Just to clarify, in line 217-219, 
we mentioned that we would like to adjust the step coefficients
of the S-FEG to have a better rate for a constant noise case in near future,
similar to [5] studying a stochastic accelerated gradient method for minimization.
We want to point out here that the suggested reference 
does analysis somewhat similar to [5].

 We would like to thank the reviewer for 
the nice summary of our work and constructive feedback.

$\bullet$ **(Importance of comonotonicity and $\alpha\ge0$-interaction condition):**
We believe that studying
the comonotonicity setting is
one of the important steps
for understanding the training dynamics of
the modern nonconvex-nonconcave minimax problems 
such as GAN and adversarial training.

However, we agree that this was not well explained in the paper,
    and we would like to further explain 
    the $\alpha\ge0$-interaction dominant condition in [10]
    (yielding the comonotonicity).
    A convex-concave function is $\alpha\ge0$-interaction dominant.
    On the other hand, for any $\gamma$-weakly-convex-weakly-concave function,
    the condition holds with $\alpha = -\gamma < 0$.
    Its extreme case is $\phi(x,y) = -\frac{\gamma}{2}x^2 + \frac{\gamma}{2}y^2$,
    where there is no interaction between $x$ and $y$.
    Interestingly, a nonconvex-nonconcave function 
    becomes $\alpha\ge0$-interaction dominant,
    when the second term in the condition is sufficiently positive definite.
    In other words, the condition is satisfied 
    when the interaction term of the Hessian
    $\nabla^2_{xy}\phi$ is dominating any negative curvature 
    in Hessians $\nabla_{xx}^2\phi$ and $-\nabla_{yy}^2\phi$.
    In short, the $\alpha\ge0$-interaction dominant condition captures whether 
    there is a sufficient coupling between $x$ and $y$,
    and we believe many practical (adversarial) applications
    (at least locally or asymptotically) satisfy such general interaction condition.

$\bullet$ **(Numerical experiments):**
As a starter,
we ran a toy experiment with a function
$f(x,y) = \frac{\rho L^2}{2} x^2 + L\sqrt{1-\rho^2L^2}xy - \frac{\rho L^2}{2} y^2$, which has an $L$-Lipschitz $\rho$-comonotone saddle gradient.
For the case $\rho = -1/3L$ and $L=1$, 
we observed that FEG converges with an accelerated rate, 
whereas EG+, EAG-C, and EAG-V diverge.
We will add this in the revision.
We agree that many practical audience of NeurIPS
will like seeing more realistic experiments,
but we think that it is beyond the scope of this paper.
We promise to make this work more practical for practitioners
in near future.



 We would like to thank the reviewer for 
the nice summary of our work and constructive feedback.

$\bullet$ **(Work of Grimmer et al.):**
We appreciate the comment on the work of Grimmer et al. [10]. 
We will revise the paper as suggested.

$\bullet$ **(Lower bound on the comonotonicity):**
We understand that one might view 
the lower bound on the comonotonicity as a limitation,
since such condition is not widely studied yet.
First of all, EG+ has a tighter lower bound $\rho>-\frac{1}{8L}$,
unlike our  $\rho>-\frac{1}{2L}$.
In addition,
the case $\rho\le -\frac{1}{L}$ is out of 
anyone's interest,
because it contains extreme cases
such as a "concave-convex" function 
$f(x,y) = -\frac{L}{2}x^2 + \frac{L}{2}y^2$
without a coupling term.
We hope you now see that the lower bound is necessary
and is not so restrictive.

$\bullet$ **(Practical improvement of FEG):**
Our main contribution is that FEG
works for a class of nonconvex-nonconcave problems
that other methods are not known to converge.
As a practical example, we ran a toy experiment with a function
$f(x,y) = \frac{\rho L^2}{2} x^2 + L\sqrt{1-\rho^2L^2}xy - \frac{\rho L^2}{2} y^2$, which has an $L$-Lipschitz $\rho$-comonotone saddle gradient.
For the case $\rho = -1/3L$ and $L=1$, 
we observed that FEG converges with an accelerated rate, 
whereas EG+, EAG-C, and EAG-V diverge, as expected by the theory.
For the monotone case with $\rho = 0$ and $L=1$, 
we also observed that FEG is faster than EAG-C and EAG-V as expected.
We will add these results in the revision.

$\bullet$ **(Worst-case for AGDA):**
We were not able to come up with a simple way 
to further incorporate AGDA in the context of Lemma 8.1.
Instead, we would like to mention
two recent works [Letcher, et. al., 2020] and [Hsieh, et. al, 2021] 
that study the fundamental limits of finding a stationary point of the smooth minimax problems. Both 
consider some classes of algorithms that includes AGDA, 
and construct examples that no method in such class 
converge to a stationary point.
We believe that this can be a good starting point
for answering the reviewer's question.

**References**
- Letcher, Alistair. "On the Impossibility of Global Convergence in Multi-Loss Optimization." International Conference on Learning Representations. 2020.
- Hsieh, Ya-Ping, Panayotis Mertikopoulos, and Volkan Cevher. "The limits of min-max optimization algorithms: Convergence to spurious non-critical sets." International Conference on Machine Learning. PMLR, 2021.

 We are glad to hear that you found our work interesting,
and we appreciate constructive feedback.

$\bullet$ **(Optimality of FEG in convex minimization?):**
We appreciate the reviewer for further studying our result
in convex minimization.
Even though we hope to have optimality even in convex optimization, 
as the reviewer claims,
we want to clarify that it is not the case.
The optimal complexity in terms of the gradient norm
is $O(1/\sqrt{\epsilon})$ [Nemirovsky, 1992].
The standard GD has $O(1/\epsilon)$ with respect to the gradient norm,
and Nesterov's method has $O(1/\epsilon^{2/3})$ complexity [Kim and Fessler, 2018].
We are yet unaware whether FEG has a complexity
better than $O(1/\epsilon)$
for convex minimization.

$\bullet$ **(Highlighting nonconvexity-nonconcavity):**
We added the term "structured" in front of nonconvexity-nonconcavity
to concisely illustrate 
that we are considering a specific class of nonconvex-nonconcave problems,
as it is done in [7].
However, we understand that one might think this to be overselling.
We will revise the abstract and the beginning of the introduction accordingly,
so that the readers see the exact condition
from the beginning.

$\bullet$ **(Key technical differences):**
We thought that the key technical differences 
between our work and other related works 
(under the monotonicity or MVI)
are well illustrated in Table 1 and in Section 4.
Could you let us know
which technical aspects you would like to further hear from us? 
We will be happy to answer them.

$\bullet$ **(Squared gradient norm):**
We agree that the norm of the gradient
can be preferred.
However, since the main related works [6,7,38]
use the squared gradient norm, 
we would like to follow them for consistency and readability.

1. Nemirovsky, "Information-based complexity of linear operator equations", Journal of Complexity, 1992.
2. Kim and Fessler, "Generalizing the optimized gradient method for smooth convex minimization", SIAM J. Optimization, 2018.

The authors focus on the problem of unconstrained min max optimization by finding approximate stationary points of non convex non concave landscapes. These landscapes are not arbitrary, they satisfy the \rho- comonotonicity ( a notion that captures  cocoercivity and monotonicity, and many more classes of functions). They design an accelerated extra gradient method that gives an \eps approximate stationary point (norm of gradient is at most \eps) in time 1\eps. Note this is optimal even in convex optimization for smooth functions (standard GD gives 1/eps^2, Nesterov's method gives 1/\eps). Stochastic analysis of the method is also provided.  This is an interesting result and the iteration complexity of the provided method is optimal. I think it is a solid contribution for a non trivial class of functions that include well studied classes. My main comment is that the authors mention that they tackle structured non convex non concave landscapes (in the abstract and later). They should rephrase it because they focus on a specific class of non convex non concave functions and sounds like overselling. Technically the paper is okay (accelerated extra gradient has appeared in other paper which are cited, nothing of great novelty here) but the authors should put some work on explaining key technical differences between their paper and the literature (on monotone, mvi etc). Another comment is that the authors mention all their results in terms of the square of the norm of the gradient, i think the norm of the gradient would be more preferable. The paper is theoretical and the reviewer does not see any societal impact.

This paper studied nonconvex-nonconcave minimax optimization with smooth and comonotonicity assumptions and proposed fast extragradient (FEG) algorithm which has O(1/k^2) convergence rate. The authors also proposed a line-search strategy to improve the algorithm in practice and extended FEG to stochastic case.  This paper studies nonconvex-nonconcave minimax optimization with smooth and comonotonicity assumptions. The authors combine the idea of EG+ and EAG which leads to a two-time-scale EG with anchoring, named fast extragradient (FEG), that has a fast $O(1/k^2)$ convergence rate on the norm of the gradient. The paper also propose a backtracking line-search variant the problem parameters are not available and extended the algorithm to stochastic case.

This paper improves the convergence rate of nonconvex-nonconcave minimax optimization under comonotonicity assumptions. My main consideration is why such setting is useful. Example 1 connects the assumption to $\alpha$-interaction dominate condition, however, it is no clear which types of problem satisfies $\alpha$-interaction. 

It is also desired to see some numerical experiment to validate the effectiveness of the theory, since the extension of FEG are proposed for implementation in practice. The audience of NeurIPS are more interested in the optimization algorithm related to machine learning models. Hence, I hope the author need to provide more discussion for the applications of the proposed algorithms.
 This work is purely theoretical.

This work proposes an accelerated first-order method for nonconvex-nonconcave minimax optimization. Critically the convergence theory relies on the objective being L-smooth and having at most a modest level of nonconvexity/nonconcavity (based on the comonotonicity being at most a 1/2L). For such problems, the method converges in gradient norm squared at a rate of 1/T^2, improving on the typical 1/T rate of similar extragradient-type methods. Extensions of this framework to utilize backtracking and stochastic oracle evaluations are provided. As a tertiary result, a clever example is given showing a gradient oracle alone can be trapped in a subspace without any stationary points.
  In Example 1, the work of Grimmer et al. does not use $\gamma$-smoothness, but rather a weaker condition (called weak convexity-weak concavity therein). The proof in the appendix does not rely on smoothness, so this difference is minor.

After defining comonotonicity, the paper defines three cases (negative, zero, and positive) implying the resulting theory would cover all three, most impressively the negative \rho setting. It was disappointing then in the theory to see the results only cover a very limited range of negative \rho (between -1/2L and 0). The paper should weaken its language when claiming a rate for negative comonotone problems as only a very limited class of these are able to be addressed by the theory presented.

Section 4 presents detailed comparisons between derived guarantees in the three settings (limited negative rho, rho=0, and \rho>0) with existing bounds. Much of the improvements are in terms of small constants: (i) The limited negative \rho range is larger by a factor of four for FEG than EG+, (ii) The FEG constant is smaller than EAG by 4/27. Numerically, are these improvements realized? They are small enough that I suspect these differences may come from proof technique differences rather than actual improvements in optimization power.
 AGDA (Alternating Gradient Descent Ascent) provides a common first-order method that violates the requirements in Lemma 8.1. It would be interesting if a modified example could show limitations for methods in the span of the x and y gradient components separately.

The stochastic version of the algorithm presented is fairly weak. Requiring both convexity/concavity and the level of stochasticity to decrease as the method converges effectively removes much of the interesting stochastic benefits. The paper already acknowledges itself the importance of handling constant noise as a future direction.


This paper develops the FEG method for solving smooth structured nonconvex-nonconcave minimax problems. It is proved that FEG has an accelerated convergence rate $O(1/k^2)$ on the squared gradient norm, which is the first accelerated method under the nonconvex-nonconcave setting. Moreover, it is shown that the FEG method has a rate faster than those of the EAG and the Halpern
iteration for smooth convex-concave problems. The authors further develop and study backtracking line-search and stochastic versions of FEG.  The paper is very well written and developed sound results. The FEG method is the first accelerated algorithm for nonconvex-nonconcave problems with negative-monotonicity. Therefore, this paper makes a solid contribution to the existing literature. 
Below are some questions.

1. From the analysis in sec 7.1, it is not clear how the reused $Fz_k$ term in FEG help achieve an accelerated convergence rate? How substantial would the proof be different from that of EAG?

2. I think it is important to numerically compare this algorithm with the competitive algorithms mentioned in the paper. Given that FEG only introduces a reuse of $Fz_k$, it is not clear if this can easily accelerate the convergence in practice. 

3. The result of stochastic FEG has a bad dependence on the stochastic variance $\sigma_k^2$. There are existing accelerated SGD analysis under constant noise variance, e.g., Accelerated Gradient Methods for Nonconvex Nonlinear and Stochastic Programming, Saeed Ghadimi · Guanghui Lan. This might be an interesting reference.
 Yes

Meta-review: This paper proposes a new algorithm that can get 1/k^2 convergence rate for a class of structured smooth nonconvex-nonconcave minimax optimization problems, in terms of the squared gradient norm, improving over the currently known 1/k rate algorithm for the same problem. While the paper does not provide any practical example that falls into the class being analyzed here, the reviewers felt that the paper makes an interesting theoretical progress on the hard and challenging problem of nonconvex-nonconcave minimax optimization. Furthermore, the techniques developed in this paper can potentially lead to further work in this area. Consequently, I am recommending the paper for acceptance.

Decision: accept