SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Taoli Zheng
tlzheng@se.cuhk.edu.hk
Jose Blanchet
jose.blanchet@stanford.edu
Jiajin Li*
Nonconvex-nonconcave minimax optimization has received intense attention over the last decade due to its broad applications in machine learning. Most existing algorithms rely on one-sided information, such as the convexity (resp. concavity) of the primal (resp. dual) functions, or other specific structures, such as the Polyak-Lojasiewicz (PL) and Kurdyka-Lojasiewicz (KL) conditions. However, verifying these regularity conditions is challenging in practice. To meet this challenge, we propose a novel universally applicable single-loop algorithm, the doubly smoothed gradient descent ascent method (DS-GDA), which naturally balances the primal and dual updates. That is, DS-GDA with the same hyperparameters is able to uniformly solve nonconvex-concave, convex-nonconcave, and nonconvex-nonconcave problems with one-sided KL properties, achieving convergence with  $\mathcal{O}(\epsilon^{-4})$  complexity. Sharper (even optimal) iteration complexity can be obtained when the KL exponent is known. Specifically, under the one-sided KL condition with exponent  $\theta \in (0,1)$ , DS-GDA converges with an iteration complexity of  $\mathcal{O}(\epsilon^{-2\max\{2\theta,1\}})$ . They all match the corresponding best results in the literature. Moreover, we show that DS-GDA is practically applicable to general nonconvex-nonconcave problems even without any regularity conditions, such as the PL condition, KL condition, or weak Minty variational inequalities condition. For various challenging nonconvex-nonconcave examples in the literature, including "Forsaken", "Bilinearly-coupled minimax", "Sixth-order polynomial", and "PolarGame", the proposed DS-GDA can all get rid of limit cycles. To the best of our knowledge, this is the first first-order algorithm to achieve convergence on all of these formidable problems.
In this paper, we are interested in studying nonconvex-nonconcave minimax problems of the form
$$
\min  _ {x \in \mathcal {X}} \max  _ {y \in \mathcal {Y}} f (x, y), \tag {P}
$$
where  $f:\mathbb{R}^n\times \mathbb{R}^d\to \mathbb{R}$  is nonconvex in  $x$  and nonconcave in  $y$ , and  $\mathcal{X}\subseteq \mathbb{R}^n$ ,  $\mathcal{Y}\subseteq \mathbb{R}^d$  are convex compact sets. Such problems have found significant applications in machine learning and operation research, including generative adversarial networks [27, 2], adversarial training [44, 53], multi-agent
reinforcement learning [16, 48], and (distributionally) robust optimization [6, 19, 38, 25, 7], to name a few.
We propose a new algorithm called the Doubly Smoothed Gradient Descent Ascent (DS-GDA) algorithm. DS-GDA builds upon S-GDA by applying the Moreau-Yosida smoothing technique to both the primal and dual variables, which allows a better trade-off between primal and dual updates. All hyperparameters, including the stepsize for gradient descent and ascent steps, and extrapolation parameters are carefully and explicitly controlled to ensure the sufficient descent property of a novel Lyapunov function that we introduce in our paper. The carefully selected variables automatically decide the dominant player, thereby achieving the balance of primal and dual updates by the interaction of parameters. Furthermore, the doubly smoothing technique enables the use of a set of symmetric parameters to achieve universality. This stands in sharp contrast to S-GDA, where only primal/dual function is smoothed. Specifically, the regularized function and the primal-dual updates in DS-GDA are inherently symmetric, which provides the possibility of applying the DS-GDA without prior information on the primal and dual functions.
To validate our idea and demonstrate the universality of DS-GDA, we provide a visual representation of the feasible symmetric parameter selections by relating the regularizer to Lipschitz constants and step sizes. This graphical illustration, depicted in Figure 1a, reveals that numerous parameter settings
can be chosen to guarantee convergence, which showcases the flexibility of DS-GDA. An experiment for a nonconvex-nonconcave problem has also been done to test the efficiency of using symmetric parameters.
We also evaluate the performance of DS-GDA on a range of representative and challenging nonconvex-nonconcave problems from the literature, which violate all known regularity conditions. These include the "Forsaken" example [32, Example 5.2], the "Bilinearly-coupled minimax" example [29], the "Sixth-order polynomial" example [18, 56, 13], and the "PolarGame" example [50]. In all cases, DS-GDA successfully escapes limit cycles and converges to the desired stationary point, while other methods either suffer from the recurrence behavior or diverge. Moreover, our algorithm exhibits robustness to parameter selection (see Section 4.2), offering practical flexibility.
To corroborate its superior performance and have a better understanding of its convergence behaviors, we demonstrate that DS-GDA converges to a stationary point for nonconvex-concave, convex-nonconcave, and nonconvex-nonconcave problems that satisfy a one-sided KL property. By employing a single set of parameters, DS-GDA converges with an iteration complexity of  $\mathcal{O}(\epsilon^{-4})$  across all these scenarios. Remarkably, DS-GDA achieves this without prior verification of these conditions. However, if we have access to a one-sided KL condition or knowledge of the convexity (concavity) of the primal (dual) function, the range of allowable parameters can be expanded. What is more, DS-GDA attains a lower or even optimal iteration complexity of  $\mathcal{O}(\epsilon^{-2\max\{2\theta,1\}})$  when the one-sided KL property with exponent  $\theta \in (0,1)$  is satisfied. Notably, these convergence results match the best results for single-loop algorithms when the dual function is concave [62, 40] or satisfies KL condition [40]. To the best of our knowledge, our work demonstrates, for the first time, the possibility of having a simple and unified single-loop algorithm for solving nonconvex-nonconcave, nonconvex-concave, and convex-nonconcave minimax problems. However, it remains an open question whether convergence results can be derived without any regularity conditions. This question is intriguing and warrants further theoretical investigation of nonconvex-nonconcave minimax problems in the future.
(i) We present DS-GDA, the first universal algorithm for convex-nonconcave, nonconvex-concave, and nonconvex-nonconcave problems with one-sided KL property. A single set of parameters can be applied across all these scenarios, guaranteeing an iteration complexity of  $\mathcal{O}(\epsilon^{-4})$ . With the KL exponent  $\theta \in (0,1)$  of the primal or dual function, we improve the complexity to  $\mathcal{O}(\epsilon^{-2\max\{2\theta,1\}})$ . Our current convergence analysis achieves the best-known results in the literature.
(ii) We demonstrate that DS-GDA converges on various challenging nonconvex-nonconcave problems, even when no regularity conditions are satisfied. This makes DS-GDA the first algorithm capable of escaping limit cycles in all these hard examples.
2 Doubly Smoothed GDA
In this section, we propose our algorithm (i.e., DS-GDA) for solving (P). To start with, we introduce the blanket assumption, which is needed throughout the paper.
Assumption 1 (Lipschitz gradient) The function  $f$  is continuously differentiable and there exist positive constants  $L_{x}, L_{y} > 0$  such that for all  $x, x' \in \mathcal{X}$  and  $y, y' \in \mathcal{Y}$
$$
\left\| \nabla_ {x} f (x, y) - \nabla_ {x} f \left(x ^ {\prime}, y ^ {\prime}\right) \right\| \leq L _ {x} \left(\left\| x - x ^ {\prime} \right\| + \left\| y - y ^ {\prime} \right\|\right),
$$
$$
\left\| \nabla_ {y} f (x, y) - \nabla_ {y} f \left(x ^ {\prime}, y ^ {\prime}\right) \right\| \leq L _ {y} \left(\left\| x - x ^ {\prime} \right\| + \left\| y - y ^ {\prime} \right\|\right).
$$
For simplicity, we assume  $L_{y} = \lambda L_{x} = \lambda L$  with  $\lambda > 0$ .
For general smooth nonconvex-concave problems, a simple and natural algorithm is GDA, which suffers from oscillation even for the bilinear problem  $\min_{x\in [-1,1]}\max_{y\in [-1,1]}xy$ . To address this issue, a smoothed GDA algorithm that uses Moreau-Yosida smoothing techniques is proposed in [62]. Specifically, they introduced an auxiliary variable  $z$  and defined a regularized function as follows:
$$
F (x, y, z) := f (x, y) + \frac {r}{2} \| x - z \| ^ {2}.
$$
The additional quadratic term smooths the primal update. Consequently, the algorithm can achieve a better trade-off between primal and dual updates. We adapt the smoothing technique to the nonconvex-nonconcave setting, where the balance of primal and dual updates is not a trivial task. To tackle this
problem, we also smooth the dual update by subtracting a quadratic term of dual variable and propose a new regularized function  $F: \mathbb{R}^n \times \mathbb{R}^d \times \mathbb{R}^n \times \mathbb{R}^d \to \mathbb{R}$  as
$$
F (x, y, z, v) := f (x, y) + \frac {r _ {1}}{2} \| x - z \| ^ {2} - \frac {r _ {2}}{2} \| y - v \| ^ {2}
$$
with different smoothed parameters  $r_1 > L_x$ ,  $r_2 > L_y$  for  $x$  and  $y$ , respectively. Then, our DS-GDA is formally presented in Algorithm 1.
Algorithm 1: Doubly Smoothed GDA (DS-GDA)
Data: Initial  $x^0, y^0, z^0, v^0$ , stepsizes  $\alpha, c > 0$ , and extrapolation parameters  $0 < \beta, \mu < 1$   
1 for  $t = 0, \ldots, T$  do  
2  $\begin{array}{rl} & x^{t + 1} = \mathrm{proj}_{\mathcal{X}}(x^t - c\nabla_xF(x^t, y^t, z^t, v^t)); \\ & y^{t + 1} = \mathrm{proj}_{\mathcal{Y}}(y^t + \alpha\nabla_yF(x^{t + 1}, y^t, z^t, v^t)); \\ & z^{t + 1} = z^t + \beta (x^{t + 1} - z^t); \\ & v^{t + 1} = v^t + \mu (y^{t + 1} - v^t). \end{array}$   
3  
4  
5  
6 end
The choice of  $r_1$  and  $r_2$  is crucial for the convergence of the algorithm in both theoretical and practical senses. In particular, when  $r_1 = r_2$ , it reduces to the proximal-point mapping proposed in [43] and inexact proximal point method (PPM) is only known to be convergent under certain VI conditions. Even with the exact computation of proximal mapping, PPM will diverge in the absence of regularity conditions [29]. By contrast, with an unbalanced  $r_1$  and  $r_2$ , our algorithm can always converge. The key insight here is to carefully adjust  $r_1$  and  $r_2$  to balance the primal-dual updates, ensuring the sufficient descent property of a novel Lyapunov function introduced in our paper. In fact, as we will show later in Section 3,  $r_1$  and  $r_2$  are typically not equal theoretically and practically. The two auxiliary variables  $z$  and  $v$ , which are updated by averaging steps, are also indispensable in our convergence proof. Intuitively, the exponential averaging applied to proximal variables  $z$  and  $v$  ensures they do not deviate too much from  $x$  and  $y$ , contributing to sequence stability.
We would like to highlight that the way we use the Moreau-Yosida smoothing technique is a notable departure from the usual approach. Smoothing techniques are commonly invoked when solving nonconvex-concave problems to achieve better iteration complexity [62, 40, 59]. However, we target at smoothing both the primal and dual variables with different magnitudes to ensure global convergence.
3 Convergence Analysis
The convergence result of the proposed DS-GDA (i.e., Algorithm 1) will be discussed in this section. To illustrate the main result, we first provide the stationarity measure in Definition 1.
Definition 1 (Stationarity measure) The point  $(\hat{x},\hat{y})\in \mathcal{X}\times \mathcal{Y}$  is said to be an
(i)  $\epsilon$ -game stationary point (GS) if
$$
\operatorname {d i s t} (\mathbf {0}, \nabla_ {x} f (\hat {x}, \hat {y}) + \partial \mathbf {1} _ {\mathcal {X}} (\hat {x})) \leq \epsilon \quad a n d \quad \operatorname {d i s t} (\mathbf {0}, - \nabla_ {y} f (\hat {x}, \hat {y}) + \partial \mathbf {1} _ {\mathcal {Y}} (\hat {y})) \leq \epsilon ;
$$
(ii)  $\epsilon$ -optimization stationary point (OS) if
$$
\left\| \operatorname {p r o x} _ {\max _ {y \in \mathcal {Y}} f (\cdot , \hat {y}) + \mathbf {1} _ {\mathcal {X}}} (\hat {x}) - \hat {x} \right\| \leq \epsilon .
$$
Remark 1 The definition of game stationary point is a natural extension of the first-order stationary point in minimization problems. It is a necessary condition for local minimax point [34] and has been widely used in nonconvex-nonconcave optimization [21, 37]. We have investigated their relationships in Appendix  $K$ .
3.1 Complexity under Nonconvex-(Non)concave Setting
Inspired by [62, 40], we consider a novel Lyapunov function  $\Phi : \mathbb{R}^n \times \mathbb{R}^d \times \mathbb{R}^n \times \mathbb{R}^d \to \mathbb{R}$  defined as follows:
$$
\begin{array}{l} \Phi (x, y, z, v) := \underbrace {F (x , y , z , v) - d (y , z , v)} _ {\text {P r i m a l d e s c e n t}} + \underbrace {p (z , v) - d (y , z , v)} _ {\text {D u a l a s c e n t}} + \underbrace {q (z) - p (z , v)} _ {\text {P r o x i m a l a s c e n t}} + \underbrace {q (z) - \underline {{F}}} _ {\text {P r o x i m a l d e s c e n t}} + \underline {{F}} \\ = F (x, y, z, v) - 2 d (y, z, v) + 2 q (z), \\ \end{array}
$$
where  $d(y,z,v) \coloneqq \min_{x \in \mathcal{X}} F(x,y,z,v)$ ,  $p(z,v) \coloneqq \max_{y \in \mathcal{Y}} d(y,z,v)$ ,  $q(z) \coloneqq \max_{v \in \mathbb{R}^d} p(z,v)$ , and  $\underline{F} \coloneqq \min_{z \in \mathbb{R}^n} q(z)$ . Obviously, this Lyapunov function is lower bounded, that is,  $\Phi \geq \underline{F}$ . To gain a better understanding of the rationale behind the construction of  $\Phi$ , it is observed that the Lyapunov function has a strong connection to the updates of the iterations. The primal update corresponds to "primal descent" and gradient ascent in dual variable corresponds to the "dual ascent". The averaging updates of proximal variables could be understood as an approximate gradient descent of  $p(z,v)$  and an approximate gradient ascent of  $g(v)$ , resulting in the "proximal descent" and "proximal ascent" terms in the Lyapunov function. Compared with that in [62, 40], we have an additional "proximal ascent" term. It is introduced by the regularized term for dual variable in  $F$  and the update of proximal variable  $v$ . Essentially, the "nonconcavity" of  $f(x,\cdot)$  brings the additional term. With this Lyapunov function, we can establish the following basic descent property as our first important result.
Proposition 1 (Basic descent estimate) Suppose that Assumption 1 holds and  $r_1 \geq 2L$ ,  $r_2 \geq 2\lambda L$  with the parameters
$$
0 <   c \leq \min  \left\{\frac {4}{3 (L + r _ {1})}, \frac {1}{6 \lambda L} \right\}, 0 <   \alpha \leq \min  \left\{\frac {2}{3 \lambda L \sigma^ {2}}, \frac {1}{6 L _ {d}}, \frac {1}{5 \lambda \sqrt {\lambda + 5} L} \right\},
$$
$$
0 <   \beta \leq \min  \left\{\frac {2 4 r _ {1}}{3 6 0 r _ {1} + 5 r _ {1} ^ {2} \lambda + (2 \lambda L + 5 r _ {1}) ^ {2}}, \frac {\alpha \lambda^ {2} L ^ {2}}{3 8 4 r _ {1} (\lambda + 5) (\lambda + 1) ^ {2}} \right\},
$$
$$
0 <   \mu \leq \min  \left\{\frac {2 (\lambda + 5)}{2 (\lambda + 5) + \lambda^ {2} L ^ {2}}, \frac {\alpha \lambda^ {2} L ^ {2}}{6 4 r _ {2} (\lambda + 5)} \right\}.
$$
Then for any  $t \geq 0$ ,
$$
\begin{array}{l} \Phi^ {t} - \Phi^ {t + 1} \geq \frac {r _ {1}}{3 2} \| x ^ {t + 1} - x ^ {t} \| ^ {2} + \frac {r _ {2}}{1 5} \| y ^ {t} - y _ {+} ^ {t} (z ^ {t}, v ^ {t}) \| ^ {2} + \frac {r _ {1}}{5 \beta} \| z ^ {t} - z ^ {t + 1} \| ^ {2} + \frac {r _ {2}}{4 \mu} \| v _ {+} ^ {t} (z ^ {t + 1}) - v ^ {t} \| ^ {2} \\ - 4 r _ {1} \beta \| x \left(z ^ {t + 1}, v \left(z ^ {t + 1}\right)\right) - x \left(z ^ {t + 1}, v _ {+} ^ {t} \left(z ^ {t + 1}\right)\right) \| ^ {2}. \tag {1} \\ \end{array}
$$
where  $\sigma := \frac{2cr_1 + 1}{c(r_1 - L)}$  and  $L_d := \left(\frac{\lambda L}{r_1 - L} + 2\right)\lambda L + r_2$ . Moreover, we have  $y_+(z, v) := \operatorname{proj}_{\mathcal{Y}}(y + \alpha \nabla_y F(x(y, z, v), y, z, v))$  and  $v_+(z) := v + \mu(y(x(z, v), z, v) - v)$  with the following definitions: (i)  $x(y, z, v) := \operatorname{argmin}_{x \in \mathcal{X}} F(x, y, z, v)$ , (ii)  $y(x, z, v) := \operatorname{argmax}_{y \in \mathcal{Y}} F(x, y, z, v)$ , (iii)  $x(z, v) := \operatorname{argmin}_{x \in \mathcal{X}} \max_{y \in \mathcal{Y}} F(x, y, z, v)$ , (iv)  $v(z) := \operatorname{argmax}_{v \in \mathbb{R}^d} P(z, v)$ .
The lower bound of  $\Phi$  by  $F$  is established by its construction, so the crux of proving subsequence convergence is to establish the decreasing property of the Lyapunov function. Although Proposition 1 quantifies the variation of the Lyapunov function values between two consecutive iterates, there is a negative error term  $\| x(z^{t + 1},v(z^{t + 1})) - x(z^{t + 1},v_{+}^{t}(z^{t + 1}))\|$  that makes the decreasing property of  $\Phi$  unclear.
Next, we characterize the negative error term in terms of other positive terms and then exhibit the sufficient descent property by bounding the coefficients. Conceptually, the error term is related to  $\| v_{+}^{t}(z^{t + 1}) - v(z^{t + 1})\|$  by the Lipschitz property of the solution mapping  $x(z,\cdot)$ . However,  $\| v_{+}^{t}(z^{t + 1}) - v(z^{t + 1})\|$  may not be a suitable surrogate since it includes the information about the optimal solution  $v(z^{t + 1})$ . Fortunately, with the help of the global KL property or concavity for the dual function (see Assumption 2 and 3), we can bound the negative error term by  $\| v_{+}^{t}(z^{t + 1}) - v(z^{t + 1})\|$  (called the proximal error bound). The explicit form of this bound is provided in the following Propositions 2.
Assumption 2 (KL property with exponent  $\theta$  of the dual function) For any fixed point  $x\in \mathcal{X}$ , the problem  $\max_{y\in \mathcal{Y}}f(x,y)$  has a nonempty solution set and a finite optimal value. Moreover, there
exist  $\tau >0,\theta \in (0,1)$  such that for any  $x\in \mathcal{X},y\in \mathcal{Y}$
$$
\left(\max  _ {y ^ {\prime} \in \mathcal {Y}} f (x, y ^ {\prime}) - f (x, y)\right) ^ {\theta} \leq \frac {1}{\tau} \operatorname {d i s t} (\mathbf {0}, - \nabla_ {y} f (x, y) + \partial \mathbf {1} _ {\mathcal {Y}} (y)).
$$
Assumption 3 (Concavity of the dual function) For any fixed point  $x \in \mathcal{X}$ ,  $f(x, \cdot)$  is concave.
Proposition 2 (Proximal error bound) Under the setting of Proposition 1 with Assumption 2 or 3, for any  $z \in \mathbb{R}^n$ ,  $v \in \mathbb{R}^d$  one has
(i) KL exponent  $\theta \in (0,1)$ :
$$
\left\| x \left(z ^ {t + 1}, v _ {+} ^ {t} \left(z ^ {t + 1}\right)\right) - x \left(z ^ {t + 1}, v \left(z ^ {t + 1}\right)\right) \right\| ^ {2} \leq \omega_ {0} \left\| v _ {+} ^ {t} \left(z ^ {t + 1}\right) - v ^ {t} \right\| ^ {\frac {1}{\theta}};
$$
(ii) Concave:
$$
\left\| x \left(z ^ {t + 1}, v _ {+} ^ {t} \left(z ^ {t + 1}\right)\right) - x \left(z ^ {t + 1}, v \left(z ^ {t + 1}\right)\right) \right\| ^ {2} \leq \omega_ {1} \left\| v _ {+} ^ {t} \left(z ^ {t + 1}\right) - v ^ {t} \right\|,
$$
where  $\omega_0 := \frac{2}{(r_1 - L)\tau}\left(\frac{r_2(1 - \mu)}{\mu} + \frac{r_2^2}{r_2 - \lambda L}\right)^{\frac{1}{\theta}}$  and  $\omega_1 := \frac{4r_2\mathrm{diam}(\mathcal{V})}{r_1 - L}\left(\frac{1 - \mu}{\mu} + \frac{r_2}{r_2 - \lambda L}\right)$ . Here,  $\mathrm{diam}(\mathcal{V})$  denotes the diameter of the set  $\mathcal{V}$ .
Armed with Proposition 1 and Proposition 2, we establish the main theorem concerning the iteration complexity of DS-GDA with respect to the above-mentioned standard stationarity measure for (P).
Theorem 1 (Iteration complexity for nonconvex-(non)concave problems) Under the setting of Theorem 1 and Proposition 2, for any  $T > 0$ , there exists a  $t \in \{1, 2, \dots, T\}$  such that
(i) KL exponent  $\theta \in (\frac{1}{2}, 1)$ :  $(x^{t+1}, y^{t+1})$  is an  $\mathcal{O}(T^{-\frac{1}{4\theta}})$ -GS and  $z^{t+1}$  is an  $\mathcal{O}(T^{-\frac{1}{4\theta}})$ -OS if  $\beta \leq \mathcal{O}(T^{-\frac{2\theta - 1}{2\theta}})$ ;
(ii) KL exponent  $\theta \in (0,\frac{1}{2} ]$  ..  $(x^{t + 1},y^{t + 1})$  is an  $\mathcal{O}(T^{-\frac{1}{2}})$  -GS and  $z^{t + 1}$  is an  $\mathcal{O}(T^{-\frac{1}{2}})$  -OS if  $\beta \leq$ $32r_{1}\mu \omega_{0}(2\mathrm{diam}(\mathcal{V}))^{\frac{1}{\theta} -2};$
(iii) Concave:  $(x^{t + 1},y^{t + 1})$  is an  $\mathcal{O}(T^{-\frac{1}{4}})$ -GS and  $z^{t + 1}$  is an  $\mathcal{O}(T^{-\frac{1}{4}})$ -OS if  $\beta \leq \mathcal{O}(T^{-\frac{1}{2}})$ .
Moreover, when the problem (P) equipped with the widely fulfilled semi-algebraic structure [9, 4, 15] and the dual function satisfies the KL property with  $\theta \in (0,\frac{1}{2} ]$ , we additionally have the following sequential convergence result of the DS-GDA.
Theorem 2 (Last-iterate convergence of DS-GDA) Consider the setting of Theorem 1 and suppose that Assumption 2 holds with  $\theta \in (0, \frac{1}{2}]$ . Suppose that  $f(\cdot, y)$  is semi-algebraic and  $\mathcal{X}, \mathcal{Y}$  are semi-algebraic sets. Then, the sequence  $\{(x^t, y^t, z^t, v^t)\}$  converges to  $(x^*, y^*, z^*, v^*)$ , where  $(x^*, y^*)$  is a GS and  $z^*$  is an OS.
3.2 Universal Results
For convex-nonconcave or nonconvex-nonconcave minimax problems in which the primal function satisfies the KL property, analogous results to Theorem 1 can be established. This can be accomplished by introducing an alternative Lyapunov function  $\Psi : \mathbb{R}^n \times \mathbb{R}^d \times \mathbb{R}^n \times \mathbb{R}^d \to \mathbb{R}$  as follows:
$$
\Psi (x, y, z, v) := \underbrace {h (x , z , v) - F (x , y , z , v)} _ {\text {D u a l a s c e n t}} + \underbrace {h (x , z , v) - p (z , v)} _ {\text {P r i m a l d e s c e n t}} + \underbrace {p (z , v) - g (v)} _ {\text {P r o x i m a l d e s c e n t}} + \underbrace {\overline {{F}} - g (v)} _ {\text {P r o x i m a l a s c e n t}},
$$
where  $h(x,z,v) \coloneqq \max_{y\in \mathcal{Y}}F(x,y,z,v), g(v) \coloneqq \min_{z\in \mathbb{R}^n}p(z,v),$  and  $\overline{F} \coloneqq \max_{v\in \mathbb{R}^d}g(v)$ .
It is worth noting that our Lyapunov function exhibits symmetry with respect to nonconvex (non)concave problems, since the adjustment only entails interchanging the position between  $(d(y,z,v),q(z))$  and  $(h(x,z,v),g(v))$ . Therefore, similar convergence results as Theorem 1 could be derived without any effort. A more detailed proof can be found in Appendix H.
Based on these results, we are ready to show that our DS-GDA is a universal algorithm. By incorporating the choices of parameters, we can identify a consistent set of parameters that ensures the convergence of DS-GDA in nonconvex-nonconcave, nonconvex-concave, and convex-nonconcave problems. The universal convergence rate is stated as follows:
Theorem 3 (Universal convergence of DS-GDA) Without loss of generality, we consider the case where  $\lambda = 1$ , implying  $L_{x} = L_{y} = L$ . For convex-nonconcave, nonconvex-concave, and nonconvex-nonconcave minimax problems that satisfy the one-sided KL property, if we further set  $r_1 = r_2 = r \geq 20L$ , DS-GDA converges provided certain parameter conditions are met:
$$
\frac {- (r - L) \sqrt {L ^ {2} - 1 4 L r + r ^ {2}} + L ^ {2} - 8 L r + r ^ {2}}{1 2 L r ^ {2}} \leq c = \alpha \leq \frac {1}{6 L _ {d}},
$$
$$
0 <   \beta = \mu \leq \min \left\{\frac {2 4 r}{3 6 0 r + 5 r ^ {2} + (2 L + 5 r) ^ {2}}, \frac {c L ^ {2}}{9 2 1 6 r}, \frac {1 2}{1 2 + L ^ {2}} \right\} \leq \mathcal {O} (T ^ {- \frac {1}{2}}).
$$
When  $KL$  exponent  $\theta \in (0,\frac{1}{2} ]$ , we further require
$$
\beta = \mu \leq \frac {1}{4 \sqrt {2 \max \left\{\omega_ {0} (2 \operatorname {d i a m} (\mathcal {Y})) ^ {\frac {1}{\theta} - 2} , \omega_ {2} (2 \operatorname {d i a m} (\mathcal {X})) ^ {\frac {1}{\theta} - 2} \right\}}},
$$
where  $\omega_0$  and  $\omega_{2}$  are the coefficients in Propositions 2 and 6. Then,  $(x^{t + 1},y^{t + 1})$  is an  $\mathcal{O}(T^{-\frac{1}{4}})$ -GS.