SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We empirically demonstrate that this scheme can converge for nonmonotone problems for certain choices of parameters (see Figure 2). However, what global guarantees can we provide theoretically?
It turns out that for LA-GDA with two inner steps  $(\tau = 2)$  we have an affirmative answer. After some algebraic manipulation it is not difficult to see that the update can be simplified as follows
$$
z ^ {k + 1} = \frac {1}{2} \left(z ^ {k} - 2 \lambda \gamma F z ^ {k}\right) + \frac {1}{2} \left(z ^ {k} - 2 \lambda \gamma F \left(z ^ {k} - \gamma F z ^ {k}\right)\right). \tag {9}
$$
Figure 2: LA-GDA and RAPP can converge for Hsieh et al. (2021, Ex. 5.2). Interestingly, we can set the stepsize  $\gamma$  larger than  $1 / L$  while RAPP remains stable. Approximate proximal point (APP) with the same stepsize diverges (the iterates of APP are deferred to Figure 6). In this example, it is apparent from the rates, that there is a benefit in replacing the conservative inner update in RAPP with GDA in LA-GDA as explored in Section 7.
This is the average of GDA and  $\mathrm{EG + }$  (when  $\lambda \in (0,1 / 2)$ ). This observation allows us to show convergence under cohypomonotonicity. This positive result for nonmonotone problems partially explains the stabilizing effect of LA-GDA.
Theorem 7.1. Suppose Assumption 3.2 holds. Consider the sequence  $(z^k)_{k\in \mathbb{N}}$  generated by LA-GDA with  $\tau = 2$ ,  $\gamma \leq 1 / L$  and  $\lambda \in (0,1 / 2)$ . Furthermore, suppose that
$$
2 \rho > - (1 - 2 \lambda) \gamma \quad a n d \quad 2 \rho \geq 2 \lambda \gamma - (1 - \gamma^ {2} L ^ {2}) \gamma . \tag {10}
$$
Then, for all  $z^{\star}\in \mathrm{zer}F$
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} \| F \bar {z} ^ {k} \| ^ {2} \leq \frac {\| z ^ {0} - z ^ {\star} \| ^ {2}}{\lambda \gamma ((1 - 2 \lambda) \gamma + 2 \rho) K}. \tag {11}
$$
Remark 7.2. For  $\lambda \to 0$  and  $\gamma = c / L$  where  $c\in (0,\infty)$ , sufficient condition reduces to  $\rho \geq -\gamma (1 - \gamma^2 L^2) / 2 = -c(1 - c^2) / 2L$ , of which the minimum is attained with  $c = 1 / \sqrt{3}$ , leading to the requirement  $\rho \geq -1 / 3\sqrt{3} L$ . A similar statement is possible for  $z^{k}$ . Thus, (LA-GDA) improves on the range of  $\rho$  compared with EG (see Table 1).
For larger  $\tau$ , LA-GDA does not necessarily converge (see Figure 3 for a counterexample). We next ask what we would require of the base optimizer to guarantee convergence for any  $\tau$ . To this end, we replace the inner iteration with some abstract algorithm  $\mathrm{Alg}:\mathbb{R}^d\to \mathbb{R}^d$ , i.e.
$$
w _ {k} ^ {t + 1} = \operatorname {A l g} \left(w _ {k} ^ {t}\right) \quad \forall t = 0, \dots , \tau - 1 \tag {LA}
$$
Convergence follows from quasi-nonexpansiveness.
Theorem 7.3. Suppose  $\operatorname{Alg}:\mathbb{R}^d\to \mathbb{R}^d$  is quasi-nonexpansive. Then  $(z^{k})_{k\in \mathbb{N}}$  generated by (LA) converges to some  $z^{\star}\in$  fix  $\mathrm{Alg}$
Remark 7.4. Even though the base optimizer Alg might not converge (since nonexpansiveness is not sufficient), Theorem 7.3 shows that the outer loop converges. Interestingly, this aligns with the benefit observed in practice of using the outer iteration of Lookahead (see Figure 4).
Cocoercive From Theorem 7.3 we almost immediately get converge of LA-GDA for coercive problems since  $V = \mathrm{id} - \gamma F$  is nonexpansive iff  $\gamma F$  is  $1/2$ -cocoercive.
Corollary 7.5. Suppose  $F$  is  $1 / L$ -cocoercive. Then  $(z^k)_{k \in \mathbb{N}}$  generated by LA-GDA with  $\gamma \leq 2 / L$  converges to some  $z^\star \in \mathrm{zer}F$ .
Remark 7.6. Corollary 7.5 can trivially be extended to the constrained case by observing that also  $V = (\mathrm{id} + \gamma A)^{-1}(\mathrm{id} - \gamma F)$  is nonexpansive when  $A$  is maximally monotone. As a special case this captures constrained convex and gradient Lipschitz minimization problems.
Figure 3: We test the Lookahead variants on Pethick et al. (2022, Ex. 3(iii)) where  $\rho \in (-1/8L, -1/10L)$  (left) and Pethick et al. (2022, Ex. 5) with  $\rho = -1/3$  (right). For the left example LA-GDA (provably) converges for  $\tau = 2$ , but may be nonconvergent for larger  $\tau$  as illustrate. Both variants of LA-GDA diverges in the more difficult example on the right, while LA-CEG+ in contrast provably converges. It seems that LA-CEG+ trades off a constant slowdown in the rate for convergence in a larger class.
Monotone When only monotonicity and Lipschitz holds we may instead consider the following extragradient based version of Lookahead (first empirically investigated in Chavdarova et al. (2020))
$$
w _ {k} ^ {t + 1} = \operatorname {E G} \left(w _ {t} ^ {k}\right) \quad \forall t = 0, \dots , \tau - 1 \tag {LA-EG}
$$
where  $\mathrm{EG}(z) = z - \gamma F(z - \gamma Fz)$ . We show in Theorem F.1 that the EG-operator of the inner loop is quasi-nonexpansive, which implies convergence of LA-EG through Theorem 7.3. Theorem F.1 extends even to cases where  $A\neq 0$  by utilizing the forward-backward-forward construction of Tseng (1991). This providing the first global convergence guarantee for Lookahead beyond bilinear games.
Cohypomonotone For cohypomonotone problems large  $\tau$  may prevent LA-GDA from converging (see Figure 3 for a counterexample). Therefore we propose replacing the inner optimization loop in LA-GDA with the method proposed in (Pethick et al., 2022, Alg. 1). Let  $H = \mathrm{id} - \gamma F$ . We can write one step of the inner update with  $\alpha \in (0,1)$  as
$$
\mathrm {C E G} ^ {+} (w) = w + 2 \alpha (H \bar {w} - H w) \quad \text {w i t h} \quad \bar {w} = (\mathrm {i d} + \gamma A) ^ {- 1} H w. \tag {12}
$$
The usefulness of the operator  $\mathrm{CEG}^+:\mathbb{R}^d\to \mathbb{R}^d$  comes from the fact that it is quasi-nonexpansive under Assumption 3.2 (see Theorem G.1). Thus, Theorem 7.3 applies even when  $F$  is only cohypomonotone if we make the following modification to LA-GDA
$$
w _ {k} ^ {t + 1} = \operatorname {C E G} ^ {+} \left(w _ {k} ^ {t}\right) \quad \forall t = 0, \dots , \tau - 1 \tag {LA-CEG+}
$$
In the unconstrained case  $(A\equiv 0)$  this reduces to using the EG  $^+$  algorithm of Diakonikolas et al. (2021) for the inner loop. We have the following convergence guarantee.
Corollary 7.7. Suppose Assumption 3.2 holds. Then  $(z^k)_{k\in \mathbb{N}}$  generated by LA-CEG+ with  $\lambda \in (0,1)$ ,  $\gamma \in (\lfloor -2\rho \rfloor_+,\frac{1}{L})$  and  $\alpha \in (0,1 + \frac{2\rho}{\gamma})$  converges to some  $z^{\star}\in \mathrm{zer}S$ .
This section demonstrates that linear interpolation can lead to an improvement over common baselines.
Synthetic examples Figures 2 and 3 demonstrate RAPP, LA-GDA and LA-CEG+ on a host of nonmonotone problems (Hsieh et al. (2021, Ex. 5.2), Pethick et al. (2022, Ex. 3(iii)), Pethick et al. (2022, Ex. 5)). See Appendix H.2 for definitions and further details.
Table 2: Adam-based. The combination of Lookahead and extragradient-like methods performs the best.
FID	ISC
Adam	21.04±2.20	7.61±0.15
ExtraAdam	18.23±1.13	7.79±0.08
ExtraAdam+	22.94±1.93	7.65±0.13
LA-Adam	17.63±0.65	7.86±0.07
LA-ExtraAdam	15.88±0.67	7.97±0.12
LA-ExtraAdam+	17.86±1.03	8.08±0.15
Table 3: GDA-based. Both RAPP and Lookahead increases the scores substantially.
FID	ISC
GDA	19.36±0.08	7.84±0.07
EG	18.94±0.60	7.84±0.02
EG+	19.35±4.28	7.74±0.44
LA-GDA	16.87±0.18	8.01±0.08
LA-EG	16.91±0.66	7.97±0.12
LA-EG+	17.20±0.44	7.94±0.11
RAPP	17.76±0.82	7.98±0.08
Image generation We replicate the experimental setup of Chavdarova et al. (2020); Miyato et al. (2018), which uses hinge version of the non-saturated loss and a ResNet with spectral normalization for the discriminator (see Appendix H.2 for details). To evaluate the performance we rely on the commonly used Inception score (ISC) (Salimans et al., 2016) and Fréchet inception distance (FID) (Heusel et al., 2017) and report the best iterate. We demonstrate the methods on the CIFAR10 dataset (Krizhevsky et al., 2009). The aim is not to beat the state-of-the-art, but rather to complement the already exhaustive numerical evidence provided in Chavdarova et al. (2020).
For a fair computational comparison we count the number of gradient computations instead of iterations  $k$  as in Chavdarova et al. (2020). Maybe surprisingly, we find that the extrapolation methods such as EG and RAPP still outperform the baseline, despite having fewer effective iterations. RAPP improves over EG, which suggest that it can be worthwhile to spend more computation on refining the updates at the cost of making fewer updates.
The first experiment we conduct matches the setting of Chavdarova et al. (2020) by relying on the Adam optimizer and using and update ratio of  $5:1$  between the discriminator and generator. We find in Table 2 that LA-ExtraAdam+ has the highest ISC (8.08) while LA-ExtraAdam has the lowest FID (15.88). In contrast, we confirm that Adam is unstable while Lookahead prevents divergence as apparent from Figure 4, which is in agreement with Chavdarova et al. (2020). In addition, the outer loop of Lookahead achieves better empirical performance, which corroborate the theoretical result (cf. Remark 7.4). Notice that ExtraAdam+ has slow convergence (without Lookahead), which is possibly due to the  $1/2$ -smaller stepsize.
Figure 4: Adam eventually diverges on CIFAR10 while Lookahead is stable with the outer iterate enjoying superior performance.
We additionally simplify the setup by using GDA-based optimizers with an update ratio of  $1:1$ , which avoids the complexity of diagonal adaptation, gradient history and multiple steps of the discriminator as in the Adam-based experiments. The results are found in Table 3. The learning rates are tuned for GDA and we use those parameters fixed across all other methods. Despite being tuned on GDA, we find that extragradient methods, Lookahead-based methods and RAPP all still outperform GDA in terms of FID. The biggest improvement comes from the linear interpolation based methods Lookahead and RAPP (see Figure 8 for further discussion on  $\mathrm{EG + }$ ). Interesting, the Lookahead-based methods are roughly comparable with their Adam variants (Table 2) while GDA even performs better than Adam.
9 Conclusion & limitations
We have precisely characterized the stabilizing effect of linear interpolation by analyzing it under cohypomonotonicity. We proved last iterate convergence rates for our proposed method RAPP. The algorithm is double-looped, which introduces a log factor in the rate as mentioned in Remark E.4. It thus remains open whether last iterate is possible using only  $\tau = 2$  inner iterations (for which RAPP reduces to  $\mathrm{EG + }$  in the unconstrained case). By replacing the inner solver we subsequently rediscovered and analyzed Lookahead using nonexpansive operators. In that regard, we have only dealt with compositions of operators. It would be interesting to further extend the idea to understanding and developing both Federated Averaging and the meta-learning algorithm Reptile (of which Lookahead can be seen as a single client and single task instance respectively), which we leave for future work.
Thanks to Leello Dadi and Stratis Skoulakis for helpful discussion. This work was supported by Google. This work was supported by Hasler Foundation Program: Hasler Responsible AI (project number 21043). This work was supported by the Swiss National Science Foundation (SNSF) under grant number 200021_205011. This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement n° 725594 - time-data).
Arjevani, Y., Shalev-Shwartz, S., and Shamir, O. On lower and upper bounds for smooth and strongly convex optimization problems. arXiv preprint arXiv:1503.06833, 2015.
Banach, S. Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales. Fund. math, 3(1):133-181, 1922.
Bauschke, H. H. and Combettes, P. L. Convex analysis and monotone operator theory in Hilbert spaces. CMS Books in Mathematics. Springer, 2017. ISBN 978-3-319-48310-8.
Bauschke, H. H., Moursi, W. M., and Wang, X. Generalized monotone operators and their averaged resolvents. Mathematical Programming, 189(1):55-74, 2021.
Bertsekas, D. P. Incremental proximal methods for large scale convex optimization. Mathematical programming, 129(2):163-195, 2011.
Bianchi, P. On the convergence of a stochastic proximal point algorithm. In CAMSAP, 2015.
Bravo, M. and Cominetti, R. Stochastic fixed-point iterations for nonexpansive maps: Convergence and error bounds. arXiv preprint arXiv:2208.04193, 2022.
Brezis, H. and Lions, P. L. Produits infinis de résolvantes. *Israel Journal of Mathematics*, 29(4): 329-345, 1978.
Cai, Y., Oikonomou, A., and Zheng, W. Accelerated algorithms for monotone inclusions and constrained nonconvex-nonconcave min-max optimization. arXiv preprint arXiv:2206.05248, 2022.
Cevher, V., Piliouras, G., Sim, R., and Skoulakis, S. Min-max optimization made simple: Approximating the proximal point method via contraction maps, 2023. URL https://arxiv.org/abs/2301.03931.
Chavdarova, T., Pagliardini, M., Stich, S. U., Fleuret, F., and Jaggi, M. Taming GANs with Lookahead-Minmax. arXiv preprint arXiv:2006.14567, 2020.
Combettes, P. L. Quasi-Fejerian analysis of some optimization algorithms. In Studies in Computational Mathematics, volume 8, pp. 115-152. Elsevier, 2001.
Combettes, P. L. and Pennanen, T. Proximal methods for cohypomonotone operators. SIAM journal on control and optimization, 43(2):731-742, 2004.
Diakonikolas, J. Halpern iteration for near-optimal and parameter-free monotone inclusion and strong solutions to variational inequalities. In Conference on Learning Theory, pp. 1428-1451. PMLR, 2020.
Diakonikolas, J., Daskalakis, C., and Jordan, M. I. Efficient methods for structured nonconvex-nonconcave min-max optimization. In International Conference on Artificial Intelligence and Statistics, pp. 2746-2754. PMLR, 2021.
Eckstein, J. and Bertsekas, D. P. On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators. Mathematical Programming, 55(1):293-318, 1992.
Gidel, G., Berard, H., Vignoud, G., Vincent, P., and Lacoste-Julien, S. A variational inequality perspective on generative adversarial networks. arXiv preprint arXiv:1802.10551, 2018.
Patrascu, A. and Irofti, P. Stochastic proximal splitting algorithm for composite minimization. Optimization Letters, 15(6):2255-2273, 2021.
Patrascu, A. and Necoara, I. Nonasymptotic convergence of stochastic proximal point methods for constrained convex optimization. The Journal of Machine Learning Research, 18(1):7204-7245, 2017.
Pethick, T., Latafat, P., Patrinos, P., Fercoq, O., and Cevher, V. Escaping limit cycles: Global convergence for constrained nonconvex-nonconcave minimax problems. In International Conference on Learning Representations, 2022.
Pethick, T., Fercoq, O., Latafat, P., Patrinos, P., and Cevher, V. Solving stochastic weak Minty variational inequalities without increasing batch size. In The Eleventh International Conference on Learning Representations, 2023.
Pushkin, D. and Barba, L. Multilayer Lookahead: a nested version of Lookahead. arXiv preprint arXiv:2110.14254, 2021.
Rockafellar, R. T. Monotone operators and the proximal point algorithm. SIAM journal on control and optimization, 14(5):877-898, 1976.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training GANs. Advances in neural information processing systems, 29, 2016.
Toulis, P., Horel, T., and Airoldi, E. M. The proximal Robbins-Monro method. arXiv preprint arXiv:1510.00967, 2015.
Toulis, P., Tran, D., and Airoldi, E. Towards stability and optimality in stochastic gradient descent. In Artificial Intelligence and Statistics, pp. 1290-1298. PMLR, 2016.
Tseng, P. Applications of a splitting algorithm to decomposition in convex programming and variational inequalities. SIAM Journal on Control and Optimization, 29(1):119-138, 1991.
Wang, J., Tantia, V., Ballas, N., and Rabbat, M. Lookahead converges to stationary points of smooth non-convex functions. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8604-8608. IEEE, 2020.
Yoon, T. and Ryu, E. K. Accelerated algorithms for smooth convex-concave minimax problems with o  $(1 / \mathrm{k}^{\wedge}2)$  rate on squared gradient norm. In International Conference on Machine Learning, pp. 12098-12109. PMLR, 2021.
Zhang, M., Lucas, J., Ba, J., and Hinton, G. E. Lookahead optimizer: k steps forward, 1 step back. Advances in neural information processing systems, 32, 2019.
Zhou, P., Yan, H., Yuan, X., Feng, J., and Yan, S. Towards understanding why lookahead generalizes better than SGD and beyond. Advances in Neural Information Processing Systems, 34:27290-27304, 2021.