SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
- \nabla_ {x} \mathcal {L} (\bar {x}, \bar {y}) \in \partial \mathbb {1} _ {X} (\bar {x}) + u, \quad \text {s . t .} \| u \| _ {\mathcal {X}} \leq \mathcal {O} (\epsilon),
$$
$$
\nabla_ {y} \mathcal {L} (\bar {x}, \bar {y}) \in \partial \mathbb {1} _ {Y} (\bar {y}) + v, \quad \text {s . t .} \| v \| _ {\mathcal {Y}} \leq \mathcal {O} (\epsilon).
$$
Therefore, we consider the above definition as a unified notion of  $\epsilon$ -stationary solution similar to [49]. We refer the reader to [21, 28] for the details of the relation between a game stationary solution and other notations of stationary solution.
Definition 2.5. Let  $(\mathcal{U},\| \cdot \|_{\mathcal{U}})$  be a finite-dimensional normed vector space. A convex set  $\kappa \subset \mathcal{U}$  is  $\alpha$ -strongly convex with respect to  $\| \cdot \|_{\mathcal{U}}$  if for any  $u,v\in \mathcal{K}$ ,  $w\in \mathcal{U}$ , and  $\gamma \in [0,1]$ ,
$$
\gamma u + (1 - \gamma) v + \gamma (1 - \gamma) \frac {\alpha}{2} \| u - v \| _ {\mathcal {U}} ^ {2} w \in \mathcal {K}, \quad \text {s . t .} \quad \| w \| _ {\mathcal {U}} = 1.
$$
We next state our main assumptions considered in the paper.
Assumption 2.6. (I) For any  $y \in Y$ ,  $\mathcal{L}(\cdot, y)$  is continuously differentiable with a Lipschitz continuous gradient, i.e., there exists  $L_{xx} \geq 0$  and  $L_{yx} > 0$  such that for any  $x, \bar{x} \in X$  and  $y, \bar{y} \in Y$  the followings hold
$$
\| \nabla_ {x} \mathcal {L} (x, y) - \nabla_ {x} \mathcal {L} (\bar {x}, \bar {y}) \| _ {\mathcal {X} ^ {*}} \leq L _ {x x} \| x - \bar {x} \| _ {\mathcal {X}} + L _ {y x} \| y - \bar {y} \| _ {\mathcal {Y}}.
$$
(II) For any  $x \in X$ ,  $\mathcal{L}(x, \cdot)$  is concave and continuously differentiable with a Lipschitz continuous gradient, i.e., there exists  $L_{yy} \geq 0$  and  $L_{yx} > 0$  such that for any  $x, \bar{x} \in X$  and  $y, \bar{y} \in Y$  the followings hold
$$
\| \nabla_ {y} \mathcal {L} (x, y) - \nabla_ {y} \mathcal {L} (\bar {x}, \bar {y}) \| _ {\mathcal {Y} ^ {*}} \leq L _ {y x} \| x - \bar {x} \| _ {\mathcal {X}} + L _ {y y} \| y - \bar {y} \| _ {\mathcal {Y}}.
$$
(III)  $X \subseteq \mathcal{X}$  and  $Y \subseteq \mathcal{Y}$  are convex and compact sets with diameters  $D_X$  and  $D_Y$ , respectively, i.e.,  $D_X \triangleq \sup_{x,\bar{x} \in X} \| x - \bar{x} \|_{\mathcal{X}}$  and  $D_Y \triangleq \sup_{y,\bar{y} \in Y} \| y - \bar{y} \|_{\mathcal{Y}}$ .
In the case where LMO is available for both minimization and maximization components of (1), we will consider the following additional assumption.
Assumption 2.7.  $Y \subseteq \mathcal{V}$  is  $\alpha$ -strongly convex for some  $\alpha > 0$ .
Remark 2.8. Strongly convex sets arise in many applications and there have been several studies characterizing various instances [15]. In particular, in many of these examples, the corresponding LMO admits a closed-form solution or can be solved efficiently. Here we present two interesting examples:
(1) Let  $\mathcal{B}_f(r) \triangleq \{x \in \mathcal{X} \mid f(x) \leq r\}$  where  $r > 0$  and  $f: \mathcal{X} \to \mathbb{R}_+$  is a  $\mu$ -strongly convex and  $L$ -smooth function. Then, the set  $\mathcal{B}_f(r)$  is strongly convex with modulus  $\alpha = \mu / \sqrt{2Lr}$ . In particular, this example includes  $\ell_p$ -norm ball when  $f(x) = \|x\|_p^2$  for  $p \in (1,2]$ .
(2) For a given matrix  $A \in \mathbb{R}^{n \times m}$ , let the singular values be denoted by  $\{\sigma_i(A)\}_{i=1}^q$  where  $q = \min(n, m)$ . Schatten  $\ell_p$  ball for  $p \in (1, 2]$ , i.e.,  $\mathcal{B}_{S(p)}(r) \triangleq \{A \in \mathbb{R}^{n \times m} \mid (\sum_{i=1}^q \sigma_i(A)^p)^{1/p} \leq r\}$ , is a strongly convex set with modulus  $\alpha = (p - 1) q^{\frac{1}{2} - \frac{1}{2}} / r$  (For details and more examples see [15]).
In this section, we propose our algorithms based on a primal-dual conditional-gradient approach for addressing problem (1). As previously mentioned in section 1, we assume that the minimization component of problem (1) includes a constraint set  $X$ , which allows for an efficient LMO, whereas the associated PO may involve computationally expensive procedures. However, with regard to the maximization component of the objective function, we consider two main scenarios based on the Oracle assumption:  $(i)$  when an LMO is available; and  $(ii)$  when a PO is available.
Note that problem (1) can be viewed as a minimization problem  $\min_{x\in X}f(x)$  where  $f(x)\triangleq$ $\max_{y\in Y}\mathcal{L}(x,y)$ . A naive implementation of a conditional gradient method, such as the Frank Wolfe method, has two main challenges: Firstly, evaluation of the objective function and/or its first-order information requires exact evaluation of  $y^{*}(x)\in \mathrm{argmax}_{y\in Y}\mathcal{L}(x,y)$  at each iteration given  $x\in X$  which may not be possible. Secondly, since the objective function  $\mathcal{L}(x,\cdot)$  is concave for any  $x\in X$ , it implies that  $f(x)$  is a nonsmooth and nonconvex function. In fact, it can be shown that  $\nabla_x\mathcal{L}(x,y^* (x))\in \partial f(x)$ , for any  $y^{*}(x)\in \mathrm{argmax}_{y\in Y}\mathcal{L}(x,y)$ . To overcome the later challenge, a typical approach is to add a regularization term to the objective function to provide a smooth approximation for the function  $f$ . More specifically, one can add a regularization term  $-\frac{\mu}{2}\| y - y_0\| _Y^2$  for some  $y_0\in Y$  to the objective function leading to the following regularized SP problem
$$
\min_{x\in X}\max_{y\in Y}\mathcal{L}_{\mu}(x,y)\triangleq \mathcal{L}(x,y) - \frac{\mu}{2}\| y - y_{0}\|_{\mathcal{Y}}^{2}.
$$
Note the objective function in the above problem is strongly concave and smooth in  $y$ , hence, for any given  $x \in X$  the corresponding maximizer  $y_{\mu}^{*}(x) = \operatorname{argmax}_{y \in Y} \mathcal{L}_{\mu}(x, y)$  is uniquely defined. Moreover, to bypass the requirement for evaluating an exact solution at each iteration, one can provide an increasingly accurate approximated solution for  $y_{\mu}^{*}(x)$ , however, this leads to a two-loop method that still requires an excessive computational cost at each iteration to solve a subproblem in exactly [53]. Moreover, the performance of such inexact methods is generally sensitive to the choice of the subproblem's parameters. Therefore, to resolve these issues, we propose single-loop and easy-to-implement inexact projection-free primal-dual algorithms.
Algorithm 1 Regularized Primal-dual Conditional Gradient (R-PDCG) method
Input:  $x_0\in X,y_0\in Y,\mu >0,\{\tau_k\}_k\subseteq \mathbb{R}_+$    
for  $k = 0,\ldots ,K - 1$  do  $s_k\gets \mathrm{argmin}_{x\in X}\langle \nabla_x\mathcal{L}(x_k,y_k),x\rangle$ $x_{k + 1}\leftarrow \tau_k s_k + (1 - \tau_k)x_k$ $p_k\gets \operatorname {argmax}_{y\in Y}\langle \nabla_y\mathcal{L}(x_k,y_k) - \mu (y_k - y_0),y\rangle$ $y_{k + 1}\gets \sigma_kp_k + (1 - \sigma_k)y_k$    
end for
In particular, when an LMO is available for both primal and dual steps, we develop an alternating conditional gradient method where at each iteration a Frank Wolfe step is taken with respect to the primal variable via the direction  $\nabla_{x}\mathcal{L}(x_{k},y_{k})$  followed by a Frank Wolfe step for the regularized maximization problem with respect to the dual variable. The outline of our proposed method is presented in Algorithm 1. Moreover, considering the scenario where the projection onto set  $Y$  is efficiently computable, we propose a one-sided projection-free primal-dual method. In particular, at each iteration, similar to the previous algorithm we perform a Frank Wolfe step with respect to the primal variable. Then to update the dual variable, instead of taking an FW-type update, we take a step of projected gradient ascent with respect to the regularized objective function as follows
$$
y _ {k + 1} \leftarrow \mathcal {P} _ {Y} \Big (y _ {k} + \sigma_ {k} \big (\nabla_ {y} \mathcal {L} _ {\mu} (x _ {k}, y _ {k}) \big) \Big).
$$
The outline of these steps is presented in Algorithm 2.
Algorithm 2 Conditional Gradient with Regularized Projected Gradient Ascent (CG-RPGA)
Input:  $x_0\in X,y_0\in Y,\mu >0,\{\tau_k,\sigma_k\} _k\subseteq \mathbb{R}_+$    
for  $k = 0,\ldots ,K - 1$  do  $s_k\gets \mathrm{argmin}_{x\in X}\langle \nabla_x\mathcal{L}(x_k,y_k),x\rangle$ $x_{k + 1}\gets \tau_k s_k + (1 - \tau_k)x_k$ $y_{k + 1}\leftarrow \mathcal{P}_Y\Big(y_k + \sigma_k\big(\nabla_y\mathcal{L}(x_k,y_k) - \mu (y_k - y_0)\big)\Big)$    
end for
4 Convergence Analysis of R-PDCG
In this section, we study the convergence properties of Algorithms 1 and 2. First, in the next lemma, we provide a one-step analysis of Algorithm 1 by providing an upper bound on the reduction of the objective function in terms of the consecutive iterates.
Lemma 4.1. Suppose Assumptions 2.6 and 2.7 hold and let  $\{(x_k, y_k)\}_{k \geq 0}$  be the sequence generated by Algorithm 1 with step-sizes  $\tau_k = \tau > 0$  and  $\{\sigma_k\}_{k \geq 0}$ , and parameter  $\mu > 0$ . Define  $\sigma_k \triangleq \min \{1, \frac{\alpha}{4(L_{yy} + \mu)} \| \nabla_y \mathcal{L}_\mu(x_k, y_k) \|_{\mathcal{Y}^*}\}$  and  $H_k \triangleq \mathcal{L}_\mu(x_k, y_\mu^*(x_k)) - \mathcal{L}_\mu(x_k, y_k)$  for any  $k \geq 0$ . Then for any  $k \geq 0$
$$
H _ {k + 1} \leq \max  \left\{\frac {1}{2}, 1 - \frac {\alpha \sqrt {\mu}}{8 \sqrt {2} \left(L _ {y y} + \mu\right)} \sqrt {H _ {k}} \right\} H _ {k} + \mathcal {E} (\tau), \tag {5}
$$
where  $\mathcal{E}(\tau) \triangleq L_{yx}\tau D_YD_X + 2L_{xx}\tau^2 D_X^2$ .
Note that the above one-step inequality demonstrates that given the iterate sequence  $\{x_{k}\}_{k\geq 0}$ , the generated iterates  $\{y_{k}\}_{k\geq 0}$  reduce the suboptimality of regularized objective function  $\mathcal{L}_{\mu}$  within an error bound  $\mathcal{E}(\tau)$ . In other words, the generated iterates  $\{y_{k}\}$  provides a progressively accurate estimation of the optimal trajectory  $y_{\mu}^{*}(x_{k})$  with an error depending on the primal step-size  $\tau$ . Therefore, the key idea lies in the meticulous selection of  $\tau$  to prevent error growth while ensuring sufficient progress in the primal variable. To this end, in the following theorem we establish bounds on the primal and dual gaps based on the previous lemma that are connected via parameters  $\tau$  and  $\mu$ . Subsequently, in the next corollary by carefully selecting those parameters we demonstrate a convergence rate guarantee for Algorithm 1.
Theorem 4.2. Suppose Assumptions 2.6 and 2.7 hold and let  $\{(x_k, y_k)\}_{k \geq 0}$  be the sequence generated by Algorithm 1 with step-sizes  $\tau_k = \tau > 0$  and  $\{\sigma_k\}_{k \geq 0}$ , and parameter  $\mu > 0$ . Define  $\sigma_k \triangleq \min \{1, \frac{\alpha}{4(L_{yy} + \mu)} \| \nabla_y \mathcal{L}_\mu(x_k, y_k) \|_{\mathcal{Y}^*}\}$ , then for any  $K \geq 1$ , there exists  $t \in \{\lceil K/2 \rceil, \dots, K-1\}$  such that  $(x_t, y_t) \in X \times Y$  satisfy the following bounds
$$
\begin{array}{l} \mathcal {G} _ {X} (x _ {t}, y _ {t}) \leq \frac {2 (f (x _ {0}) - f (x _ {K}))}{\tau K} + \frac {\mu}{\tau K} D _ {Y} ^ {2} + \frac {(L _ {x x} + L _ {y x} ^ {2} / \mu) \tau}{K} D _ {X} ^ {2} + \frac {2 \sqrt {2} L _ {y x}}{\sqrt {\mu}} D _ {X} \left[ \begin{array}{c} 1 \\ 0 \\ 0 \end{array} \right]. \\ \sqrt {\mathcal {E} (\tau)} + \frac {3 \log (K + 1)}{K} \max  \left\{\sqrt {H _ {0}}, \frac {1 6 (L _ {y y} + \mu)}{\alpha \sqrt {\mu}} \right\} + \left(\frac {8 \sqrt {2} (L _ {y y} + \mu) \mathcal {E} (\tau)}{\alpha \sqrt {\mu}}\right) ^ {1 / 3} ], \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {G} _ {Y} (x _ {t}, y _ {t}) \leq \frac {3 6 c _ {\mu}}{(K + 4) ^ {2}} \max \left\{H _ {0}, \frac {2 5 6 (L _ {y y} + \mu) ^ {2}}{\alpha^ {2} \mu} \right\} + c _ {\mu} \mathcal {E} (\tau) \\ + \left(\frac {8 \sqrt {2} (L _ {y y} + \mu) \mathcal {E} (\tau)}{\alpha \sqrt {\mu}}\right) ^ {2 / 3} c _ {\mu} + \mu D _ {Y} ^ {2}. \\ \end{array}
$$
where  $c_{\mu} \triangleq 2 + L_{yy} / \mu$  and  $\mathcal{E}(\tau)$  is defined in Lemma 4.1.
Now, we are ready to state the convergence rate and complexity results of Algorithm 1.
Corollary 4.3. Under the premises of Theorem 4.2, choose  $\mu = \mathcal{O}(\epsilon)$  and  $\tau = \mathcal{O}(\epsilon^5)$ , then for any  $K \geq 1$ , there exists  $t \in \{\lceil K/2 \rceil, \ldots, K-1\}$  such that  $(x_t, y_t) \in X \times Y$  satisfy  $\mathcal{G}_Z(x_t, y_t) \leq \mathcal{O}(1/K^{1/6})$ . Moreover,  $(x_t, y_t)$  satisfy  $\mathcal{G}_Z(x_t, y_t) \leq \epsilon$  and consequently is an  $\epsilon$ -game stationary within  $\mathcal{O}(\epsilon^{-6})$  iterations.
Considering that the proposed method achieves convergence through dual regularization, it becomes essential to address the question of convergence guarantee for Algorithm (1) when the objective function  $\mathcal{L}$  is nonconvex in  $x$  and strongly concave in  $y$ . In such a scenario, where the objective function exhibits strong concavity, regularization can be circumvented by setting  $\mu = 0$ . This adjustment aligns Algorithm 1 with the approach presented in [16], albeit it should be noted that the study in [16] solely focused on convex-concave settings and their analysis does not extend to the nonconvex scenario.
Theorem 4.4. Suppose Assumptions 2.6 and 2.7 hold and function  $\mathcal{L}(x,\cdot)$  is  $\tilde{\mu}$ -strongly concave for any  $x\in X$ . Let  $\{(x_k,y_k)\}_{k\geq 0}$  be the sequence generated by Algorithm 1 with parameter  $\mu = 0$  and step-sizes  $\tau_{k} = \mathcal{O}(1 / K^{3 / 4})$  and  $\{\sigma_k\}_{k\geq 0}$ . Define  $\sigma_{k}\triangleq \min \{1,\frac{\alpha}{4L_{yy}}\| \nabla_{y}\mathcal{L}(x_{k},y_{k})\|_{\mathcal{Y}^{*}}\}$ , then for any  $K\geq 1$ , there exists  $t\in \{\lceil K / 2\rceil ,\ldots ,K - 1\}$  such that  $(x_{t},y_{t})\in X\times Y$  satisfy:
(i)  $\mathcal{G}_X(x_t,y_t)\leq \mathcal{O}(1 / K^{1 / 4})$  and  $\mathcal{G}_Y(x_t,y_t)\leq \mathcal{O}(1 / K^{1 / 2})$
(ii)  $(x_{t},y_{t})$  satisfy  $\epsilon$ -primal gap  $\mathcal{G}_X(x_t,y_t)\leq \epsilon$  within  $\mathcal{O}(\epsilon^{-4})$  iterations and satisfy  $\epsilon$ -dual gap  $\mathcal{G}_Y(x_t,y_t)\leq \epsilon$  within  $\mathcal{O}(\epsilon^{-2})$  iterations.
Remark 4.5. It is important to emphasize that Algorithm 1 is a single-loop method that leverages the gradient of the objective function and relies solely on the use of the LMO for handling constraints. To the best of our knowledge, our results in Corollary 4.3 and Theorem 4.4 represent the first complexity results for finding an  $\epsilon$ -stationary solution in nonconvex-concave and nonconvex-strongly concave SP problems, respectively, without the need for projection onto the constraint sets.
5 Convergence Analysis of CG-RPGA
In this section, we assume that the projection onto set  $Y$  can be computed efficiently and the vector space  $\mathcal{Y}$  is equipped with the Euclidean norm, i.e.,  $\| \cdot \|_{\mathcal{Y}} = \| \cdot \|_2$ . We present the analysis for
convergence of CG-RPGA (Algorithm 2) for solving problem 1. In particular, we first consider the nonconvex-concave setting and in the following theorem and corollary, we demonstrate that the convergence rate can be improved to  $\mathcal{O}(1 / K^{1 / 4})$ . Afterward, we consider a nonconvex-strongly concave setting and we establish the convergence rate results for CG-RPGA in Theorem 5.3.
Theorem 5.1. Suppose Assumptions 2.6 hold and let  $\{(x_k, y_k)\}_{k \geq 0}$  be the sequence generated by Algorithm 2 with step-sizes  $\tau_k = \tau > 0$  and  $\sigma_k = \sigma \leq \frac{2}{L_{yy} + 2\mu}$ , and parameter  $\mu > 0$ . Then for any  $K \geq 1$ , there exists  $t \in \{\lceil K/2 \rceil, \dots, K-1\}$  such that  $(x_t, y_t) \in X \times Y$  satisfy the following bounds
$$
\begin{array}{l} \mathcal {G} _ {X} \left(x _ {t}, y _ {t}\right) \leq \frac {2 \left(f \left(x _ {0}\right) - f \left(x _ {K}\right)\right)}{\tau K} + \frac {\mu}{\tau K} D _ {Y} ^ {2} + \frac {2 L _ {y x}}{(1 - \rho) K} D _ {X} \left\| y _ {0} - y _ {\mu} ^ {*} \left(x _ {0}\right) \right\| _ {2} \\ + \left(\frac {2 L _ {y x} ^ {2} \rho}{\mu (1 - \rho)} + L _ {x x} + \frac {L _ {y x} ^ {2}}{\mu}\right) \tau D _ {X} ^ {2}, \\ \end{array}
$$
$$
\mathcal {G} _ {Y} \left(x _ {t}, y _ {t}\right) \leq \frac {2 \rho^ {K / 2}}{\sigma} \left\| y _ {0} - y _ {\mu} ^ {*} \left(x _ {0}\right) \right\| _ {2} + \frac {2 L _ {y x} \rho}{\sigma \mu (1 - \rho)} \tau D _ {X} + \frac {L _ {y x}}{\sigma \mu} \tau D _ {X} + \mu D _ {Y}.
$$
Corollary 5.2. Under the premises of Theorem 5.1, choose  $\mu = \mathcal{O}(\epsilon)$  and  $\tau = \mathcal{O}(\epsilon^3)$ , then for any  $K \geq 1$ , there exists  $t \in \{\lceil K/2 \rceil, \ldots, K-1\}$  such that  $(x_t, y_t) \in X \times Y$  satisfy  $\mathcal{G}_Z(x_t, y_t) \leq \mathcal{O}(1/K^{1/4})$ . Moreover,  $(x_t, y_t)$  satisfy  $\mathcal{G}_Z(x_t, y_t) \leq \epsilon$  and consequently is an  $\epsilon$ -game stationary within  $\mathcal{O}(\epsilon^{-4})$  iterations.
Similar to section 4, it is important to address the question of convergence guarantee for Algorithm (2) when the objective function is strongly concave in  $y$ . In this scenario, regularization can be avoided by setting  $\mu = 0$ . In the following theorem, we establish the convergence rate of Algorithm 2 for SP problem (1) in this setting.
Theorem 5.3. Suppose Assumptions 2.6 hold and function  $\mathcal{L}(x,\cdot)$  is  $\tilde{\mu}$ -strongly concave for any  $x\in X$ . Let  $\{(x_k,y_k)\}_{k\geq 0}$  be the sequence generated by Algorithm 2 with parameter  $\mu = 0$  and step-sizes  $\tau_{k} = \mathcal{O}(1 / K^{1 / 2})$  and  $\sigma_{k} = \sigma \leq \frac{2}{L_{yy} + \tilde{\mu}}$ . Then for any  $K\geq 1$ , there exists  $t\in \{\lceil K / 2\rceil ,\ldots ,K - 1\}$  such that  $(x_{t},y_{t})\in X\times Y$  satisfy  $\mathcal{G}_Z(x_t,y_t)\leq \mathcal{O}(1 / K^{1 / 2})$ . Moreover,  $(x_{t},y_{t})$  satisfy  $\mathcal{G}_Z(x_t,y_t)\leq \epsilon$  within  $\mathcal{O}(\epsilon^{-2})$  iterations.
6 Numerical Experiment
In this section, we implement our methods to solve Robust Multiclass Classification problem described in Example 1 and Dictionary Learning problem in Example 2. For all the algorithms, the step-sizes are selected as suggested by their theoretical result and scaled to have the best performance. In particular, for R-PDCG we let  $\tau = \frac{10}{K^{5/6}}$  and  $\mu = \frac{10^{-3}}{K^{1/6}}$ ; for CG-RPGA we let  $\tau = \frac{10}{K^{3/4}}$  and  $\mu = \frac{10^{-3}}{K^{1/4}}$ ; for AGP we let the primal step-size  $\frac{1}{\sqrt{k}}$ , dual step-size as 0.2, and the dual regularization parameter as  $\frac{10^{-1}}{k^{1/4}}$ ; for SPFW both primal and dual step-sizes are selected to be diminishing as  $\frac{2}{k+2}$ . Supplementary plots are provided in the appendix.
Robust Multiclass Classification: To assess the performance of our proposed algorithms (R-PDCG and CG-RPGA), we tested them against the Alternating Gradient Projection (AGP) algorithm introduced by [49] and the Saddle Point Frank Wolfe (SPFW) algorithm introduced by [16]. We conduct experiments on rcv1 dataset ( $n = 15564$ ,  $d = 47236$ ,  $k = 53$ ) and news20 dataset ( $n = 15935$ ,  $d = 62061$ ,  $k = 20$ ) from LIBSVM repository<sup>1</sup>. As shown in Figure 1, our algorithms outperform the competing approaches, highlighting the advantage of utilizing a projection-free approach. In this example, the high per-iteration computational cost of the projection operator significantly impacts AGP, with more than three iterations taking over 300 seconds reflecting the benefit of projection-free algorithms for a certain class of problems. For problems with easy-to-project constraints, projection-based algorithms such as AGP may have a better performance, however, this example supports the motivation behind the development of projection-free methods for saddle point problems with hard-to-project constraints, particularly when an LMO is available.
<sup>1</sup>https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets
(a)rcv1
(b) news20
Figure 1: Comparing the performance of our proposed methods R-PDCG (blue) and CG-RPGA (red) with AGP (magenta) and SPFW (green) in the Robust Multiclass Classification problem.
Dictionary Learning: Considering the dictionary learning problem in (4), we compared the performance of our proposed methods, R-PDCG (Algorithm 1) and CG-RPGA (Algorithm 2) with AGP [49] and SPFW [16] although SPFW does not have a theoretical guarantee for nonconvex-concave SP. The datasets are generated randomly from a standard Gaussian distribution with details described in Section F of the Appendix. Notably, CG-RPGA has a faster convergence rate compared to R-PDCG matching our theoretical results (see Table 1). Moreover, AGP which is a fully projection-based algorithm has the slowest convergence behavior in terms of time compared to other methods. Solving a linear optimization problem over the nuclear norm ball requires computing only a single pair of singular vectors corresponding to the largest singular value, whereas computing a projection onto the nuclear norm ball demands a full SVD. The computational cost of latter operation is  $\mathcal{O}(kd\min(k,d))$ , while the computational cost of the former one is  $\mathcal{O}(\nu\ln(k+d)\sqrt{\sigma_1}/\sqrt{\epsilon})$ , where  $\nu \leq kd$  and  $\sigma_1$  are the number of nonzero entries and the top singular value of  $-\nabla_x\mathcal{L}(x,y)$ , respectively, and  $\epsilon$  is the accuracy [10]. Therefore, in this example, LMO is considerably more cost-effective to compute than the projection method. Figure 2 depicts our methods' superior performance compared to other algorithms.
Figure 2: Comparing the performance of our proposed methods R-PDCG (blue) and CG-RPGA (red) with other methods AGP (magenta) and SPFW (green). The plots from left to right are trajectories of gap function and infeasibility for problem (4).
In this paper, we proposed primal-dual projection-free methods for solving a broad class of constrained nonconvex-concave problems. Using a regularization technique we devised a single-loop method relying on LMO for handling constraints. In particular, we show that R-PDCG achieves an  $\epsilon$ -stationary solution within  $\mathcal{O}(\epsilon^{-6})$  iterations assuming that the constraint set is strongly convex. Also, our method achieves  $\epsilon$ -primal and  $\epsilon$ -dual gaps within  $\mathcal{O}(\epsilon^{-4})$  and  $\mathcal{O}(\epsilon^{-2})$  iterations, respectively, for nonconvex-strongly concave problems. To the best of our knowledge, this is the first fully projection-free primal-dual method with a convergence guarantee for nonconvex SP problems. Additionally, when the projection on the maximization constraint is easy to compute we propose a one-sided projection-free primal-dual method called CG-RPGA with iteration complexity of  $\mathcal{O}(\epsilon^{-4})$  matching the best-known results for projection-based primal-dual methods, and improves to  $\mathcal{O}(\epsilon^{-2})$  iterations for nonconvex-strongly concave setting. We acknowledge that the proposed method is currently limited to the deterministic setting and we plan to study such SP problems under uncertainty and distributed settings in future work.