SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Therefore, the proposed new dual-accelerating step (including the gradient different prediction step in (3), the gradient ascent correction in (4) and momentum acceleration in (5)) is a key accelerated technique for our EGDA algorithm, and can help to improve the complexity bound from  $\mathcal{O}(\epsilon^{-4})$  to  $\mathcal{O}(\epsilon^{-2})$ . In particular, our EGDA algorithm performs both gradient descent and ascent steps to the original function  $f$ . In contrast, many existing algorithms such as [51, 47, 44, 18] optimize their surrogate functions with smoothing terms instead of the original function. In particular, their smoothing parameters need to tune by repeatedly executing the algorithms, which may make them impractical [1]. As in our theoretical guarantees below, the proposed single-loop algorithm is able to significantly improve the best-known gradient complexity,  $\mathcal{O}(\epsilon^{-4})$ , of existing single-loop algorithms such as [27, 51, 44] to  $\mathcal{O}(\epsilon^{-2})$ .
4 Convergence Guarantees
In this section, we provide the convergence guarantees of our EGDA algorithm (i.e., Algorithm 1) for solving constrained NC-NC and NC-C problems. We first present the definitions of the two optimality criteria (i.e., an  $\epsilon$ -stationary point of  $f$  or  $\phi$ ). All the proofs of the lemmas, properties and theorems below are included in the Supplementary Material.
4.1 Optimality Criteria and Key Property
Since finding a global minimum of a nonconvex optimization problem is NP-hard, finding a global saddle point (or Nash equilibrium) of a NC-NC function  $f$  is intractable [30]. As in the literature in the NC-NC setting, we introduce the local surrogates (i.e., the stationary point of  $f$ ) and in the NC-C setting, we introduce the local surrogates (i.e., the stationary point of  $f$  or  $\phi$ ), whose gradient mappings are equal to zero. Below we define the following two optimality measures (i.e., an  $\epsilon$ -stationary points of  $f$  or  $\phi$ ) for our theoretical analysis.
Definition 1 (An  $\epsilon$ -stationary point of  $f$  [27]). A point  $(\overline{x},\overline{y})\in \mathcal{X}\times \mathcal{Y}$  is an  $\epsilon$ -stationary point of  $f(\cdot ,\cdot)$  if  $\| \pi (\overline{x},\overline{y})\| \leq \epsilon$ , where  $\eta_{x}$  and  $\eta_{y}$  are two constants, and
$$
\pi (\bar {x}, \bar {y}) := \left[ \begin{array}{l} \left(1 / \eta_ {x}\right) (\bar {x} - \mathcal {P} _ {\mathcal {X}} (\bar {x} - \eta_ {x} \nabla_ {x} f (\bar {x}, \bar {y})) \\ \left(1 / \eta_ {y}\right) (\bar {y} - \mathcal {P} _ {\mathcal {Y}} (\bar {y} + \eta_ {y} \nabla_ {y} f (\bar {x}, \bar {y})) \end{array} \right]. \tag {6}
$$
If  $\epsilon = 0$ , then  $(\overline{x},\overline{y})$  is a stationary point of  $f$ .
For the NC-C setting, we also present another convergence criterion used in [40, 24]. Let  $\phi(x) := \max_{y \in \mathcal{Y}} f(x, y)$ ,  $\hat{x}$  is called an  $\epsilon$ -stationary point of a smooth primal function  $\phi: \mathbb{R}^m \to \mathbb{R}$ , if  $\|\nabla \phi(\hat{x})\| \leq \epsilon$ . However, the function  $\phi$  is not necessarily differentiable for minimax problems. Following [9, 40, 24], we introduce the Moreau envelope of  $\phi$  for the optimality criterion, especially when  $\phi$  is a weakly convex function, i.e.,  $\phi$  is  $L$ -weakly convex if the function  $\phi(\cdot) + \frac{L}{2} \| \cdot \|^2$  is convex. We refer readers to [9, 24] for the comparison of these two criteria.
Definition 2 (An  $\epsilon$ -stationary point of  $L$ -weakly convex function  $\phi$ ).  $\hat{x}$  is an  $\epsilon$ -stationary point of an  $L$ -weakly convex function  $\phi: \mathbb{R}^m \to \mathbb{R}$ , if  $\|\nabla \phi_{1/(2L)}(\hat{x})\| \leq \epsilon$ , where  $\phi_{1/(2L)}$  is the Moreau envelope of  $\phi$  and is defined as:  $\phi_{\rho}(x) := \min_z \phi(z) + (1/2\rho)\|z - x\|^2$ . If  $\epsilon = 0$ , then  $\hat{x}$  is a stationary point.
Then we give the following important property for the analysis of the proposed algorithm. By leveraging the property, we can obtain the complexity bound of the proposed algorithm.
Property 1 (Quasi-Cocoercivity). Let  $u_{t+1/2}$  be updated in Eq. (3), then
$$
\left\langle \nabla_ {y} f (x _ {t}, u _ {t - 1 / 2}) - \nabla_ {y} f (x _ {t}, y _ {t - 1}), u _ {t + 1 / 2} - y _ {t} \right\rangle = \beta \| \nabla_ {y} f (x _ {t}, u _ {t - 1 / 2}) - \nabla_ {y} f (x _ {t}, y _ {t - 1}) \| ^ {2}.
$$
4.2 Core Lemma
Our theoretical results mainly rely on Lemma 1 below, which plays a key role in the proofs of Theorems 1 and 2. Let  $\{(x_{t},y_{t},u_{t},u_{t - 1 / 2})\}$  be a sequence generated by Algorithm 1, and we define the potential function
$$
G _ {t} := f \left(x _ {t}, y _ {t}\right) + 9 L \left\| u _ {t} - y _ {t - 1} \right\| ^ {2} + 8 L \beta^ {2} \left\| \nabla_ {y} f \left(x _ {t - 1}, u _ {t - 3 / 2}\right) - \nabla_ {y} f \left(x _ {t - 1}, y _ {t - 2}\right) \right\| ^ {2}.
$$
Next we need to prove that our defined potential function can make sufficient decrease at each iteration, i.e.,  $G_{t} - G_{t + 1} > 0$  as in Lemma 1 below. To prove Lemma 1, we will provide and prove the following upper bounds.
Proposition 1 (Upper bound of primal updates). Suppose Assumption 1 holds. Let  $\{(x_{t},y_{t},u_{t})\}$  be generated by Algorithm 1 with  $\eta_y^t = \min \{\frac{\beta\|\nabla_yf(x_t,u_{t - 1 / 2}) - \nabla_yf(x_t,y_{t - 1})\|^2}{4\|\nabla_yf(x_t,u_{t - 1 / 2})\|^2},\frac{1}{28L},\eta_x\} +\delta$  and  $0 < \delta \leq \frac{1}{28L}$  is a constant.
$$
\begin{array}{l} f \left(x _ {t + 1}, u _ {t + 1 / 2}\right) - f \left(x _ {t}, y _ {t}\right) \\ \leq - \left(\frac {1}{\eta_ {x}} - \frac {L}{2}\right) \| x _ {t + 1} - x _ {t} \| ^ {2} + L \| u _ {t + 1 / 2} - y _ {t} \| ^ {2} + L \| y _ {t} - y _ {t - 1} \| ^ {2} + \underbrace {\left\langle \nabla_ {y} f (x _ {t} , y _ {t - 1}) , u _ {t + 1 / 2} - y _ {t} \right\rangle} _ {A _ {1}}. \\ \end{array}
$$
Proposition 2 (Upper bound of dual updates). Suppose Assumption 1 holds. Let  $\{(x_t, y_t, u_t)\}$  be a sequence generated by Algorithm 1, then
$$
\begin{array}{l} f \left(x _ {t + 1}, y _ {t + 1}\right) - f \left(x _ {t + 1}, u _ {t + 1 / 2}\right) \\ \leq \underbrace {\tau \langle \nabla_ {y} f (x _ {t} , u _ {t - 1 / 2}) , y _ {t} - u _ {t + 1} \rangle} _ {A _ {2}} + \underbrace {\langle \nabla_ {y} f (x _ {t} , u _ {t - 1 / 2}) , u _ {t + 1} - u _ {t + 1 / 2} \rangle} _ {A _ {3}} + a _ {t}, \\ \end{array}
$$
where  $a_{t} := 2L\|x_{t+1} - x_{t}\|^{2} + 6L\beta^{2}\|\nabla_{y}f(x_{t},u_{t-1/2}) - \nabla_{y}f(x_{t},y_{t-1})\|^{2} + 8L\|u_{t} - y_{t-1}\|^{2} + 8L\beta^{2}\|\nabla_{y}f(x_{t-1},u_{t-3/2}) - \nabla_{y}f(x_{t-1},y_{t-2})\|^{2} + 3L\|u_{t+1} - y_{t}\|^{2}$ .
Using the optimal condition of Problem (4) and our quasi-cocoercivity in Property 1, we can further bound  $A_{1} + A_{2} + A_{3}$ . The proof sketch of Lemma 1 is listed as follows:
$$
\begin{array}{l} G _ {t + 1} - G _ {t} := \underbrace {f (x _ {t + 1} , y _ {t + 1}) - f (x _ {t + 1} , u _ {t + 1 / 2})} _ {\text {P r o p o s i t i o n 1}} + \underbrace {f (x _ {t + 1} , u _ {t + 1 / 2}) - f (x _ {t} , y _ {t})} _ {\text {P r o p o s i t i o n 2}} + \text {o t h e r t e r m s} \\ = \underbrace {A _ {1} + A _ {2} + A _ {3}} _ {\text {Q u a s i - C o c o e r c i v i t y i n P r o p e r t y 1}} + \text {o t h e r t e r m s}. \\ \end{array}
$$
Combining the above results and the definition of  $G_{t}$ , we can get the descent estimate of the potential function in Lemma 1, which is a main lemma for Theorems 1 and 2 below.
Lemma 1 (Descent estimate of  $G$ ). Suppose Assumption 1 holds. Let  $\{(x_{t},y_{t},u_{t})\}$  be a sequence generated by Algorithm 1, then
$$
\begin{array}{l} G _ {0} - G _ {T} = \sum_ {t = 0} ^ {T - 1} (G _ {t} - G _ {t + 1}) \\ \geq \sum_ {t = 0} ^ {T - 1} \left(L \| u _ {t + 1} - y _ {t} \| ^ {2} + \frac {\beta - 3 0 L \beta^ {2}}{2} \| \nabla_ {y} f (x _ {t}, u _ {t - 1 / 2}) - \nabla_ {y} f (x _ {t}, y _ {t - 1}) \| ^ {2} + \frac {\| x _ {t + 1} - x _ {t} \| ^ {2}}{2 \eta_ {x}}\right). \\ \end{array}
$$
4.3 Convergence Results
In this subsection, by using the optimality measures in Definitions 1 and 2, we present the following theoretical results in Theorems 1 and 2 for the constrained NC-NC setting and NC-C setting, respectively.
Assumption 2.  $\mathcal{V}$  is a closed, convex and compact set with a diameter  $D_{\mathcal{V}} > 0$ , and  $\mathcal{X}$  is a closed and convex set.
Furthermore, using Lemma 1 and the optimality measure in Definition 1, we will study the relation between  $\pi(x_{t}, u_{t})$  and the difference (i.e.,  $G_{t} - G_{t+1}$ ) to obtain the gradient complexity in Theorem 1 by computing the number of iterations to achieve an  $\epsilon$ -stationary point of  $f$ .
Theorem 1 (Stationarity of  $f$  in constrained NC-NC settings). Suppose Assumptions 1 and 2 hold. Let the two stepsizes  $\eta_x \leq \frac{1}{4L}$ ,  $\eta_y^t = \min \left\{\frac{\beta\|\nabla_yf(x_t,u_{t - 1 / 2}) - \nabla_yf(x_t,y_{t - 1})\|^2}{4\|\nabla_yf(x_t,u_{t - 1 / 2})\|^2}, \frac{1}{28L}, \eta_x\right\} + \delta$ ,  $\beta \leq \frac{1}{60L}$ ,  $\tau \geq 3 / 4$  and  $\delta \leq \frac{1}{28L}$ , then the complexity of Algorithm 1 to find an  $\epsilon$ -stationary point of  $f$  is bounded by
$$
\mathcal {O} \Big (\frac {G _ {0} - \underline {{G}} + 2 L D _ {\mathcal {Y}} ^ {2}}{\epsilon^ {2}} \Big),
$$
where  $G_0 \coloneqq G(x_0, y_0)$ , and  $\underline{G} \coloneqq \min_{x \in \mathcal{X}} \phi(x)$ .
Remark 1. For constrained NC-NC problems, the gradient complexity of our EGDA algorithm to find an  $\epsilon$ -stationary point of  $f$  is  $\mathcal{O}(\epsilon^{-2})$ . That is, our EGDA algorithm is first to obtain the gradient complexity in constrained NC-NC setting. In addition, our method can achieve the same complexity,  $\mathcal{O}(\epsilon^{-2})$ , as the algorithm with an additional structured assumption for NC-NC problems. However, different from existing algorithms, our algorithm is more practical. That is, it only requires that  $\mathcal{V}$  is a compact set, while existing algorithms need some stronger structured assumptions in the structured NC-NC setting. The detailed comparison is shown in Table 1. As two special cases of the constrained NC-NC problem, this theoretical result in Theorem 1 can be extended to the constrained NC-C and C-NC settings. For NC-C problems, the gradient complexity of our EGDA algorithm to find an  $\epsilon$ -stationary point of  $f$  is  $\mathcal{O}(\epsilon^{-2})$ , while the best-known result of single-loop algorithms such as [27, 51, 44] is  $\mathcal{O}(\epsilon^{-4})$ , and the best-known result of multi-loop algorithms such as [24] is  $\widetilde{\mathcal{O}}(\epsilon^{-2.5})$ . That is, our EGDA algorithm improves the best-known gradient complexity from  $\widetilde{\mathcal{O}}(\epsilon^{-2.5})$  to  $\mathcal{O}(\epsilon^{-2})$ . Smoothed-GDA [51] can also obtain the complexity  $\mathcal{O}(\epsilon^{-2})$  for a special case of Problem (1) and  $\mathcal{O}(\epsilon^{-4})$  for general NC-C minimax problems, while our algorithm attains the optimal result  $\mathcal{O}(\epsilon^{-2})$  for all NC-C minimax problems. Existing algorithms such as HiBSA [27] and AGP [44] require the compactness of the domain  $\mathcal{X}$  in the NC-C setting, while our EGDA algorithm does not, which significantly extends its applicability. For C-NC minimax problems, the proposed algorithm can obtain the complexity,  $\mathcal{O}(\epsilon^{-2})$ , while the best-known complexity as in [44] is  $\mathcal{O}(\epsilon^{-4})$ . In other words, the proposed algorithm can improve the best-known result from  $\mathcal{O}(\epsilon^{-4})$  to  $\mathcal{O}(\epsilon^{-2})$ .
By using the criterion in Definition 2 as the optimality measure, we use the definition of  $\phi_{1/2L}$  and introduce the property of  $\nabla \phi_{1/2L}$  as in [24]. With a similar setting for varying step sizes as in [24], we study the relation between  $\nabla \phi_{1/2L}$  and the basic descent estimation of the potential function in Lemma 1 by using the property of  $\nabla \phi_{1/2L}$ , and compute the number of iterations to achieve an  $\epsilon$ -stationary point of  $\phi$ . We provide the following theoretical result and its detailed proof in the Supplementary Material.
Theorem 2 (Stationarity of  $\phi$  for constrained NC-C settings). Using the same notation as in Theorem 1, and  $f$  is concave with respect to  $y$ . Let  $\{(x_t,y_t,u_t)\}$  be a sequence generated by Algorithm 1 with the stepsizes  $\eta_y^t = \min \{\frac{\beta\|\nabla_yf(x_t,u_{t - 1 / 2}) - \nabla_yf(x_t,y_{t - 1})\|^2}{4\|\nabla_yf(x_t,u_{t - 1 / 2})\|^2},\frac{1}{28L},\eta_x\} +\delta ,\beta \leq \frac{1}{60L},\tau \geq 3 / 4$  and  $\delta \leq \frac{1}{28L}$ . Then the gradient complexity of Algorithm 1 to find an  $\epsilon$ -stationary point of  $\phi$  is bounded by
$$
\mathcal {O} \Big (\frac {D _ {\mathcal {Y}} \left(G _ {0} - \underline {{G}} + 2 L D _ {\mathcal {Y}} ^ {2}\right)}{\epsilon^ {2}} \Big).
$$
Remark 2. From Theorem 2, it is clear that the gradient complexity of the proposed algorithm is  $\mathcal{O}(\epsilon^{-2})$ . That is, Algorithm 1 can improve the best-known gradient complexity from  $\widetilde{\mathcal{O}} (\epsilon^{-3})$  as in [24] to  $\mathcal{O}(\epsilon^{-2})$ . Therefore, Algorithm 1 is the first algorithm, which attains the gradient complexity  $\mathcal{O}(\epsilon^{-2})$  to find an  $\epsilon$ -stationary point of  $\phi$  for NC-C minimax problems.
Figure 1: Comparison of all the methods for solving the NC-NC problem,  $f(x,y) = x^{2} + 3\sin^2 x\sin^2 y - 4y^2 - 10\sin^2 y$ . Left: Convergence in terms of  $\| x_t - x^*\|^2 + \| y_t - y^*\|^2$ , where  $(x^*, y^*)$  is the global saddle point; Right: Convergence in terms of  $\| \nabla_x f\|^2 + \| \nabla_y f\|^2$ .
5 Numerical Results
We conduct many experiments to illustrate the performance of the proposed algorithm for solving NC-NC, NC-C and convex-concave problems.
NC-NC Problems. We conduct some experiments to illustrate the performance of the proposed algorithm, EGDA, for solving NC-NC problems. Moreover, we compare it against existing methods such as GDA [46], EG [10], EAG [49] and FEG [20].
We compare EGDA with GDA, EG and FEG for solving a simple NC-NC minimax problem (i.e.,  $f(x,y) = x^{2} + 3\sin^{2}x\sin^{2}y - 4y^{2} - 10\sin^{2}y$ ), which satisfies the Polyak-Lojasiewicz condition, as shown in Fig. 1. All the results show that EGDA indeed converges to the global saddle point and is significantly faster than other methods including GDA and FEG in terms of both  $\| x_{t} - x^{*}\|^{2} + \| y_{t} - y^{*}\|^{2}$  and  $\| \nabla_{x}f\|^{2} + \| \nabla_{y}f\|^{2}$ . Although FEG has a fast theoretical rate,  $\mathcal{O}(1 / t^2)$ , with a negative monotone assumption, it converges much slower than EGDA in practice.
Figure 2: Comparison of all the methods for solving the convex-concave problem,  $f(x,y) = \log (1 + e^{x}) + 3xy - \log (1 + e^{y})$ . Left: Trajectories of the three algorithms; Right: Convergence in terms of  $\| x_{t} - x^{*}\|^{2} + \| y_{t} - y^{*}\|^{2}$ .
Convex-Concave Problems. Fig. 2 shows the convergence results of the methods including GDA, EAG [49] and EGDA for solving the convex-concave problem,  $f(x,y) = \log (1 + e^{x}) + 3xy - \log (1 + e^{y})$ . It is clear that EGDA converges much faster than other methods such as EAG. We also observe empirically when the same step-size is used, even if small, GDA may not converge to stationary points, and it is proven to always have bounded iterates.
NC-C Problems. We also apply our EGDA algorithm to train robust neural networks against adversarial attacks on Fashion MNIST and MNIST, and verify our theoretical results. In [15, 28, 31],
Figure 3: Convergence speed of all the algorithms on Fashion MNIST (left) and MNIST (right).
the robust training procedure can be formulated into a NC-C minimax optimization problem. It is clear that the minimax problem is nonconvex in  $x$ , but concave in  $y$ .
FGSM [15] and PGD [19] are two popular methods for generating adversarial examples. To obtain the targeted attack  $\hat{x}_{ij}$ , we use the same procedure as in [31, 51] as follows. The perturbation level  $\varepsilon$  is chosen from  $\{0.0, 0.1, 0.2, 0.3, 0.4\}$ , and the stepsize is 0.01. Note that the number of iterations is set to 40 for FGSM and 10 for PGD, respectively. We compare our EGDA method with GDA [25], MGDA [31] and Smoothed-GDA [51], and illustrate the convergence of all the algorithms on the loss function in Fig. 3. The results show that Smoothed-GDA and EGDA converge significantly faster than other algorithms. This verifies that they have a faster convergence rate,  $\mathcal{O}(\epsilon^{-2})$ . Moreover, EGDA converges much faster than Smoothed-GDA.
In this paper, we proposed a new single-loop accelerated algorithm for solving various constrained NC-NC minimax problems. In the proposed algorithm, we designed a novel extra-gradient difference scheme for dual updates. Moreover, we provided the convergence guarantees for the proposed algorithm, and the theoretical results show that to find an  $\epsilon$ -stationary point of  $f$ , the proposed algorithm obtains the complexity bound  $\mathcal{O}(\epsilon^{-2})$  for constrained NC-NC problems. That is, this is the first time that the proposed algorithm attains the complexity bound  $\mathcal{O}(\epsilon^{-2})$  in constrained NC-NC settings (including constrained NC-C and C-NC). For NC-C problems, we provided the theoretical guarantees of the proposed algorithm under the stationarity of  $\phi$ , which show that our algorithm improves the complexity bound from  $\widetilde{\mathcal{O}}(\epsilon^{-3})$  to  $\mathcal{O}(\epsilon^{-2})$ . Experimental results also verified the theoretical results of the proposed algorithm, which have the factors of  $\epsilon^{-1}$  and  $\epsilon^{-2}$  faster than existing algorithms, respectively. For further work, we will extend our directly accelerating algorithm to stochastic, non-smooth, nonconvex-nonconcave and federated learning settings as in [47, 11, 4, 55, 45, 21, 37].
We want to thank the anonymous reviewers for their valuable suggestions and comments. This work was supported by the National Key R&D Program of China (No. 2022ZD0160302), the National Natural Science Foundation of China (Nos. 62276182, 61976164, 62276004 and 61836009), the National Science Basic Research Plan in Shaanxi Province of China (No. 2022GY-061), the major key project of PCL, China (No. PCL2021A12), and Qualcomm.
[1] Z. Allen-Zhu. Katyusha: The first direct acceleration of stochastic gradient methods. In STOC, 2017.
[2] Z. Allen-Zhu and Y. Yuan. Improved SVRG for non-strongly-convex or sum-of-non-convex objectives. In ICML, pages 1080–1089, 2016.
[3] J. P. Bailey, G. Gidel, and G. Piliouras. Finite regret and cycles with fixed step-size via alternating gradient descent-ascent. In The Annual Conference on Learning Theory (COLT), pages 391-407, 2020.
[46] J. Yang, N. Kiyavash, and N. He. Global convergence and variance reduction for a class of nonconvex-nonconcave minimax problems. In NeurIPS, 2020.
[47] J. Yang, S. Zhang, N. Kiyavash, and N. He. A catalyst framework for minimax optimization. In The Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
[48] T. Yoon and E. K. Ryu. Accelerated algorithms for smooth convex-concave minimax problems with  $\mathrm{o}(1 / \mathrm{k}^2)$  rate on squared gradient norm. In M. Meila and T. Zhang, editors, ICML, pages 12098-12109, 2021.
[49] T. Yoon and E. K. Ryu. Accelerated algorithms for smooth convex-concave minimax problems with  $o(1 / k^2)$  rate on squared gradient norm. In ICML, pages 12098-12109, 2021.
[50] J. Zhang, M. Hong, and S. Zhang. On lower iteration complexity bounds for the saddle point problems. Mathematical Programming, 194:901-935, 2022.
[51] J. Zhang, P. Xiao, R. Sun, and Z. Luo. A single-loop smoothed gradient descent-ascent algorithm for nonconvex-concave min-max problems. In The Annual Conference on Neural Information Processing Systems (NeurIPS), pages 7377-7389, 2020.
[52] S. Zhang, J. Yang, C. Guzman, N. Kiyavash, and N. He. The complexity of nonconvex-strongly-concave minimax optimization. In The Conference on Uncertainty in Artificial Intelligence (UAI), 2021.
[53] W. Zhang and D. Han. A new alternating direction method for co-coercive variational inequality problems. Comput. Math. Appl., 57(7):1168-1178, 2009.
[54] Y. Zhang and L. Xiao. Stochastic primal-dual coordinate method for regularized empirical risk minimization. J. Mach. Learn. Res., 18:1-42, 2017.
[55] R. Zhao. A primal dual smoothing framework for max-structured nonconvex optimization. Mathematics of Operations Research, 2023.