Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper,
the authors first formulate the optimization problem of GANs as an abstract minimax problem (Equation(1)).
As compared to the original optimization objective of Goodfellow's GANs,
there is an additional term $h(y)$.
Why do you introduce $h(y)$ here? Just for facilitating the adoption of the MONOTONE INCLUSIONS on it?
The authors should provide a clear explanation about this point.

Later the author restrict Equation(1) to a deterministic version,
which means that the noise input of GANs will no longer be considered.
The noise input is an important ingredient of GANs.
Despite many new variants of the GANs,
at least, the noise input is import for the Goodfellow's GANs, which is adopted in this submission as a staring point.
It is very hard for me to decide whether this simplification is appropriate.
Except Algorithm 3.2 and Theorem 3.2 which suddenly provide stochastic versions,
all the following results are on this deterministic version.
In my opinion, the authors should provide more explanations on this point.

I cannot understand the first sentence of Section 2.2, after read it many times.
What is the exact necessary and sufficient optimality condition for the coupling function being convex-concave and differentiable?
Before Equation (2), the authors didn't explain what is monotone and what is monotone inclusion.
I don't think these concepts are very famous in machine learning community.

In Section 2.3,
the authors provided the introduction of lots of iterative methods.
It is difficult for me to distinguish which are completely new findings for solving their inclusion problem and which are the existing results.
The authors completed a literature review here.

In Section 2.4,
I don't know why the problem (1) can be written as Equation (8).
Could you provide more explanations about this point?
Limited space of one submission should be on important points.

In Section 3,
could you explain more about why $G_B(w)$ in Equation (10) is defined as this form?
Just for facilitating the proof of Theorem 3.1 and Theorem 3.2?

In Section 3.2,
the authors provided a generalized FBF algorithm.
Isn't this Algorithm 3.1 a combined re-written version of Equation (4) and Equation (5)?

There is a big gap between Equation (9), Theorem 3.1 and Theorem 3.2 and the experimental results shown in Section 4.
Besides, there are no open source codes provided.
It is very hard for me to figure out the details of the experiments and meantime to check the reproducibility of this paper.

Although, the authors stated that "Due to the theoretical nature of this work, the aim of this section is not to achieve new state-of-the-art
results."
I don't think optimization is a theoretical branch of our machine learning community.
If a proposed optimization method cannot be proved to be very useful in certain ares or specific tasks,
it will be very doubting.
If we intend to do theoretical contributions,
we should try to prove the theoretical properties or convergence bounds for the existing useful optimization methods.

Since ICLR is a highly selective conference,
the originality and significance of one submission will always be in the first priority.
I cannot accept this paper in current state.
review_title: Many unclear or doubting points

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: This work studies minimax optimization (a.k.a saddle-point problems) with nonsmooth regularizers. By leveraging the monotone operator theory, the authors propose to use the forward-backward-forward method so as to avoids the notorious limit cycling problem. The classical FBF method requires two gradient evaluations per step, the authors introduce a new algorithm which reuses the past gradient in the same way as OGDA. In the setting of convex-concave minimax optimization, the authors claim to prove novel convergence rates for both methods. 

Review: This paper is well-written, though I think the difference with extra-gradient can be made much clearer. For example, the authors could compare FBF with EG for a toy example like Eqn. (13). As far as I can tell, the main difference with extra-gradient is just the regularization term. From this standpoint, the authors should clarify the motivations and importance of adding regularizers. In the current version of the paper, section 2.4 is vague and doesn't explain the role of regularizers well in GAN training.
Besides, a similar convergence analysis has been done for EG and OGDA (the authors don't even cite properly), so I believe the convergence analysis of this paper is not novel. Given these reasons, I suggest the rejection of this paper and give a score of 4.

Comments:
- It was shown in [1] that the ergodic (averaged) iterates of extra-gradient converge with a rate of O(1/k) which is the same as FBF. Could the authors clarify the novelty/difference of your proof technique for FBF?

- It was shown in [2] that extra-gradient is NOT robust to gradient noise in convex-concave minimax optimization. Could the authors comment on that and explain why FBF could achieve the rate of O(1/sqrt(k)) in the stochastic setting (while EG fails)?

- In GAN training, we typically care more about the last iterate since averaging could actually hurt the performance when the loss surface is highly nonconvex. Is it possible to derive the last iterate convergence rate? 


I'm willing to increase my rating if the authors could resolve some of my concerns, especially my concern on the novelty of the analysis.


-------------
**I've read the authors' response. I'm still concerned with the novelty of the paper given there are similar results for EG/OGDA. Therefore, I stick to my original rating.**

References:

[1] Convergence Rate of O(1/k) for Optimistic Gradient and Extra-gradient Methods in Smooth Convex-Concave Saddle Point Problems, 2019.

[2] Explore Aggressively, Update Conservatively: Stochastic Extragradient Methods with Variable Stepsize Scaling, 2020.

review_title: Contributions seem incremental

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: This paper introduces the forward-backward-forward splitting for variational inequalities. The main results are an asymptotic convergence results and a non-asymptotic convergence results using a restricted merit function. A new method, FBFp, is introduced and studied. Complete proofs are given. Preliminary numerical results obtained by training GANs are reported. 

Pros:
+ complete proofs 
+ A new stochastic operator splitting method based on Tseng's FBF is introduced in which the operator needs to be evaluated once per iteration. This splitting, called FBFp, is indeed new, and has the potentially of being of practical relevance.
+ Preliminary numerical results on standard GAN architectures. 

Cons: 
- The paper takes a long time until it becomes clear what actually the monotone inclusion looks like. It seems that the problem of interest is formulation in eq. (9), preceded by a long and unnecessary discussion about existing solvers. It would have been much more accurate to simply start with the problem formulation, then propose your solution method, followed by a critical explanation of the contribution. 
- p.3 claim that FBF has not been rigorously analyzed for saddle point problems. This is of course not true. Even the original paper by Tseng (A MODIFIED FORWARD-BACKWARD SPLITTING METHOD FOR MAXIMAL MONOTONE MAPPINGS, SICON 2000) discusses the application to saddle point problems. See Example 5 in that paper.  
- The stochastic FBF has been studied in Bot et al. Mini-batch Forward-Backward-Forward Methods for solving Stochastic Variational inequalities, forthcoming in Stochastic Systems. Note that the Arxive version of that paper is available since 2019. Overall, the paper contains only marginal contributions to the state-of-the-art. 
- Only convergence rates for the ergodic average is provided. It is known that the ergodic average might destroy important features of the true solution, such as sparsity. For SFBF we know non-asymptotic convergence rates of the last iterate. This is not mentioned at all. 
- I have some doubts that the restricted merit function is the appropriate one here. Note if the aim is to solve the VI over an unconstrained domain, then FBF coincides with EG, and there is nothing to be analyzed. The interesting case is thus only the constrained case. These constraints are usually encoded in the non-smooth part of of eq. (8), so there is no need to write this explicitly. In my opinion it would therefore be cleaner to assume at the outset that the domain of $F+\partial r$ is bounded. The gap function used can in fact be traced back to Facchinei & Pang (2003) and is most likely even longer in use than that. 

review_title: Report on "Two steps at a time --- taking GAN training in stride with Tseng's method" 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors in this paper, inspired by the applications of min-max optimization in GANs, study the problem of min-max optimization for convex-concave functions. The main contribution of the paper is proving novel convergence results for Forward-Backward-Forward (FBF) algorithms as well as Optimistic Gradient Descent Ascent (OGDA) based on tools from monotone inclusion problems. Their convergence results cover both deterministic and stochastic settings and the rates of convergence for suitably chosen gap function are non-asymptotic. Finally, they apply their algorithms both on toy problems but also on training GANs on CIFAR-10.

Pros:
1) To the best of my knowledge, the connection between OGDA and monotone inclusion problems is new.
2) Convergence results are non asymptotic for the specified gap function.

Cons:
1) The connection of this work with GANs is a bit tenuous because, as the authors also acknowledge, training GANs is a non-convex non-concave min-max problem. The authors should try to express why they believe their  unification result informs practical applications.
2) This lack of connection is reflected in the experimental section as well. Most experiments re-establish that optimism, extragradient updates or regularization are beneficial for min-max optimization, observations that are already widely known. Again, any experimental insight that is particular to this work, would go a long way towards closing this gap.
3) It is not clear that the connection of OGDA to FBF and monotone inclusion provides any new insights about the convergence properties of either method.  It would be very helpful, if the authors provided any additional intuition why their result could be used to answer open questions related to OGDA or FBF.  For example, last iterate convergence of OGDA is still an open problem even in convex-concave problems.
4) While the gap function used allows the authors to provide non asymptotic guarantees, the intuition behind this gap function when its value is non-zero is unclear. Does this gap function have any game theoretic interpretation?

For now, I am assigning a weak reject score mainly because it is unclear to me if there are significant implications of this unification result either in theory or practice. I am willing to increase my score substantially if the authors provide additional details that address my concerns outlined above.    

---------------------------------
Post-Rebuttal evaluation.

I would like to thank the authors for their detailed answers, especially regarding the interpretation of the gap function.
Based on their answers, I decided to increase my score to a 6.
review_title: Unifying Optimistic Gradient Descent Ascent and Forward Backward Forward For Convex-Concave Optimization

Meta-review: This paper provides a unified view of some known methods for monotone operator inclusion problems like Forward-Backward-Forward (FBF) and OGDA, and provides new convergence results for the stochastic version of a variant of FBF called FBFp. All reviewers initially recommended rejection. The rebuttal and the manuscript update addressed several concerns from the reviewers, though the general consensus after rebuttal was still that the paper lacked in significance for the ICLR community. The AC thinks that the paper could make an interesting overview paper in a more optimization / theoretically minded venue.
Decision: Reject