SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 3.3. Under Assumption 3.1 and given an inexact initialization oracle, Algorithm 1 with  $r = \epsilon /D^2$ ,  $\epsilon_r = (\epsilon /2)\min \{1,\delta^2 /(4D^2)\}$  and AGD [61] as base algorithm  $\mathcal{A}$  outputs an  $\epsilon$ -optimal point  $x_{r}$  with  $\tilde{\mathcal{O}} (\sqrt{\ell D^2 / \epsilon})$  gradient complexity, and the reproducibility is  $\| x_r - x_r'\|^2\leq 4\delta^2$ .
This theorem implies that we can simultaneously achieve the near-optimal complexity of  $\tilde{\mathcal{O}} (\sqrt{\ell D^2 / \epsilon})$  and optimal reproducibility of  $\mathcal{O}(\delta^2)$ , which improves over the  $\mathcal{O}(\ell D^2 /\epsilon)$  complexity of GD [1]. In fact, when combined with any base algorithm that solves the auxiliary problem, Algorithm 1 attains optimal reproducibility. However, using AGD as the base algorithm results in the best complexity. To the best of our knowledge, this is the only algorithm capable of achieving the best of both worlds. Previously, Attia and Koren [6] proved that the algorithmic reproducibility (referred to as initialization stability in their study) of Nesterov's AGD is  $\Theta (\delta^{2}e^{1 / \sqrt{\epsilon}})$  when the initialization is  $\delta^2$ -apart.
Remark 1. Adding regularization is a common and useful technique in the optimization literature. Our algorithmic framework solves one auxiliary regularized strongly-convex problem, which is referred to as classical regularization reduction in Allen-Zhu and Hazan [3]. Algorithm 1 is biased and requires the knowledge of  $\epsilon$  and  $D$  to control the biased term introduced by the regularization term. The convergence guarantee also has an additional sub-optimal logarithmic term. Allen-Zhu and Hazan [3] proposed to use a double-loop algorithm, where a sequence of auxiliary regularized strongly-convex problems with decreasing regularization parameters are solved. The vanishing regularization ensures the algorithm is unbiased, and the resulting convergence guarantee requires no knowledge of  $\epsilon$  and does not have an additional logarithmic term. Similar idea could apply to our case as well, and the task of bridging such gaps is deferred to future work.
3.2 Inexact Deterministic Gradient Oracle
We further study the algorithmic reproducibility and gradient complexity of Algorithm 1 under the inexact gradient oracle model that returns an inexact gradient  $G(x) \in \mathbb{R}^d$  such that  $\| G(x) - \nabla F(x) \|^2 \leq \delta^2$  at any query point  $x \in \mathcal{X}$ . From the inexact gradient oracle of  $F$ , we can construct an inexact gradient oracle for the auxiliary problem  $F_r \colon G_r(x) = G(x) + r(x - x_0)$  which satisfies the condition  $\| G_r(x) - \nabla F_r(x) \|^2 = \| G(x) - \nabla F(x) \|^2 \leq \delta^2$ . To solve the auxiliary problem, we consider AGD with an inexact oracle (Inexact-AGD) as proposed by Devolder et al. [27]. The proposition below establishes its convergence behavior.
Proposition 3.4. Consider  $\min_{x\in \mathcal{X}}F_r(x)$ , where  $F_{r}$  is  $r$ -strongly-convex and  $(\ell +r)$ -smooth. Given an inexact gradient oracle that returns  $G_{r}(x)$  such that  $\| G_r(x) - \nabla F_r(x)\| ^2\leq \delta^2$ , starting from  $y_0 = x_0$ , AGD with the following update rule
$$
x _ {t + 1} = \Pi_ {\mathcal {X}} \left(y _ {t} - \frac {1}{2 (\ell + r)} G _ {r} (y _ {t})\right),
$$
$$
y _ {t + 1} = x _ {t + 1} + \frac {2 - \sqrt {r / (\ell + r)}}{2 + \sqrt {r / (\ell + r)}} \left(x _ {t + 1} - x _ {t}\right), \tag {Inexact-AGD}
$$
for  $t = 0,1,\dots ,T - 1$  satisfies that
$$
F _ {r} \left(x _ {T}\right) - F _ {r} \left(x _ {r} ^ {*}\right) \leq \exp \left(- \frac {T}{2} \sqrt {\frac {r}{2 \ell}}\right) \left(F _ {r} \left(x _ {0}\right) - F _ {r} \left(x _ {r} ^ {*}\right) + \frac {r}{4} \| x _ {0} - x _ {r} ^ {*} \| ^ {2}\right) + \sqrt {\frac {2 \ell}{r}} \left(\frac {1}{\ell + r} + \frac {2}{r}\right) \delta^ {2},
$$
where  $x_{r}^{*}$  is the unique minimizer of  $F_{r}(x)$ .
This proposition suggests that Inexact-AGD converges to a neighborhood with a radius of  $\mathcal{O}(\delta^2 /r^{3 / 2})$  around the optimal value. We note that convergence to the exact solution is unattainable for algorithms employing inexact gradients [27, 28], and the size of this neighborhood is important in determining the reproducibility of  $x_{r}$ .
Theorem 3.5. Under Assumption 3.1 with  $0 < \epsilon \leq \ell D^2$  and given an inexact deterministic gradient oracle in Definition 1, Algorithm 1 with  $r = \epsilon /D^2$ ,  $\epsilon_r = 6\delta^2 D^3\sqrt{\ell / (2\epsilon^3)}$  and Inexact-AGD as base algorithm outputs a  $(6\delta^{2}D^{3}\sqrt{\ell / (2\epsilon^{3})} +\epsilon /2)$ -optimal point  $x_{r}$  with  $\tilde{\mathcal{O}} (\sqrt{\ell D^2 / \epsilon})$  gradient complexity, and the reproducibility is  $\| x_r - x_r'\|^2\leq \mathcal{O}(\delta^2 /\epsilon^{5 / 2})$ .
Ahn et al. [1] showed that GD achieves optimal reproducibility of  $\mathcal{O}(\delta^2 /\epsilon^2)$  and a complexity of  $\mathcal{O}(1 / \epsilon)$  when  $\delta \leq \mathcal{O}(\epsilon)$ . Our results indicate that a reproducibility of  $\mathcal{O}(\delta^2 /\epsilon^{5 / 2})$  and a near-optimal complexity of  $\tilde{\mathcal{O}} (1 / \sqrt{\epsilon})$  can be attained when  $\delta \leq \mathcal{O}(\epsilon^{5 / 4})$ . We conjecture that this suboptimal reproducibility bound is inevitable for the proposed framework given the lower bound result in Devolder et al. [27] for algorithms under a  $(\delta ,\ell ,\mu)$ -inexact oracle associated with  $\ell$ -smooth  $\mu$ -strongly-convex functions. Further discussions are provided in Appendix A.2. Moreover, we point out that for minimizing  $\ell$ -smooth and  $\mu$ -strongly-convex functions, Proposition 3.4 already implies that Inexact-AGD attains the optimal reproducibility of  $\mathcal{O}(\min \{\delta^2,\epsilon \})$  and the optimal complexity of  $\tilde{\mathcal{O}} (\sqrt{\ell / \mu})$  when the problem is well-conditioned, improving over the  $\tilde{\mathcal{O}} (\ell /\mu)$  complexity in the previous work [1].
Remark 2. In Appendix D, we demonstrate the effectiveness of Algorithm 1 on a quadratic minimization problem equipped with an inexact gradient oracle. The results are plotted in Figure 1 in the appendix. We observe that the reproducibility can be greatly improved when adding regularization, with only a small degradation in the convergence performance.
4 Deterministic Gradient Oracle for Minimax Problems
In this section, we address the minimax optimization problem of the form
$$
\min  _ {x \in \mathcal {X}} \max  _ {y \in \mathcal {Y}} F (x, y),
$$
where  $\mathcal{X}$  and  $\mathcal{Y}$  are convex compact sets. We focus on the standard smooth and convex-concave setting as detailed in Assumption 4.1. We aim to find an  $\epsilon$ -saddle point  $(\hat{x},\hat{y})$  such that its duality gap satisfies  $\max_{y\in \mathcal{Y}}F(\hat{x},y) - \min_{x\in \mathcal{X}}F(x,\hat{y})\leq \epsilon$ . Here, the assumption that the domains are convex and bounded ensures the existence of the saddle point when the objective is convex-concave [73]. We focus on minimax problems equipped with inexact initialization oracles and inexact deterministic gradient oracles as defined in Definition 4. We first show that two classical algorithms, gradient descent ascent (GDA) and Extragradient (EG) [48, 72], are either sub-optimal in convergence or suboptimal in reproducibility, which mirrors the minimization setting. Based on the same regularization idea, we propose two new frameworks in Algorithm 2 and 3 that successfully attain near-optimal convergence and optimal reproducibility at the same time.
Assumption 4.1. For all  $y \in \mathcal{V}$ ,  $F(\cdot, y)$  is convex, and for all  $x \in \mathcal{X}$ ,  $F(x, \cdot)$  is concave. Furthermore,  $F$  is  $\ell$ -smooth on the domain  $\mathcal{X} \times \mathcal{Y}$ . Additionally, both  $\mathcal{X}$  and  $\mathcal{Y}$  have a diameter of  $D$ . This means that  $\| x_1 - x_2 \|^2 \leq D^2$  and  $\| y_1 - y_2 \|^2 \leq D^2$  for all  $x_1, x_2 \in \mathcal{X}$  and  $y_1, y_2 \in \mathcal{Y}$ .
The optimal gradient complexity to find  $\epsilon$ -saddle point under such assumptions is  $\Theta(1/\epsilon)$  [63]. Since the minimax problem reduces to a minimization problem on  $\mathcal{X}$  when the domain  $\mathcal{Y}$  is restricted to be a singleton, the reproducibility lower-bounds [1] for smooth convex minimization hold as lower-bounds for smooth convex-concave minimax optimization as well. That is,  $\Omega(\delta^2)$  under the inexact initialization oracle, and  $\Omega(\delta^2/\epsilon^2)$  under the inexact gradient oracle (see Lemma B.3). We now present the convergence rate and reproducibility bounds of GDA (see Algorithm 4) and EG (see Algorithm 5).
Theorem 4.2. (GDA) Under Assumption 4.1, the average iterate  $(\bar{x}_T,\bar{y}_T)$  output by GDA with stepsize  $1 / (\ell \sqrt{T})$  after  $T = \mathcal{O}(1 / \epsilon^2)$  iterations is an  $\epsilon$ -saddle point. Furthermore, the reproducibility of the output is (i)  $\mathcal{O}(\delta^2)$  under  $\delta$ -inexact initialization oracle; (ii)  $\mathcal{O}(\delta^2 /\epsilon^2)$  under  $\delta$ -inexact deterministic gradient oracle if  $\delta \leq \mathcal{O}(\epsilon)$ .
Theorem 4.3. (Extragradient) Under Assumption 4.1, the average iterate  $(\bar{x}_{T + 1 / 2},\bar{y}_{T + 1 / 2})$  output by  $EG$  with stepsize  $1 / \ell$  after  $T = \mathcal{O}(1 / \epsilon)$  iterations is an  $\epsilon$ -saddle point. Furthermore, the reproducibility of this output is (i)  $\mathcal{O}(\min \{\delta^2 e^{1 / \epsilon},\delta^2 +1 / \epsilon^2,D^2\})$  under  $\delta$ -inexact initialization oracle; (ii)  $\mathcal{O}(\min \{\delta^2 e^{1 / \epsilon},1 / \epsilon^2,D^2\})$  under  $\delta$ -inexact deterministic gradient oracle if  $\delta \leq \mathcal{O}(\epsilon)$ .
While GDA can achieve optimal reproducibility, it converges with a sub-optimal complexity of  $\mathcal{O}(1 / \epsilon^2)$ . On the other hand, EG achieves an optimal  $\mathcal{O}(1 / \epsilon)$  complexity but is not optimally reproducible. Further details on this are provided in Appendix B. In Appendix B.3.4, we also demonstrate that EG, through an alternative parameter selection, can achieve optimal reproducibility at a sub-optimal rate  $\mathcal{O}(1 / \epsilon^{3 / 2})$ . The question that remains open is how to simultaneously attain both optimal reproducibility and gradient complexity. To address this, we have developed two algorithmic frameworks with near-optimal guarantees, one based on regularization and the other based on proximal point methods [66, 12].
4.1 Regularization Helps!
Algorithm 2 Reproducible Algorithmic Framework for Convex-Concave Minimax Problems
Input: Regularization parameter  $r > 0$ , accuracy  $\epsilon_r > 0$ , base algorithm  $\mathcal{A}$ , initialization  $(x_0, y_0)$ . Apply  $\mathcal{A}$  to in exactly solve the  $r$ -strongly-convex-strongly-concave and  $(\ell + r)$ -smooth problem
$$
\left(x _ {r}, y _ {r}\right) \gets \min  _ {x \in \mathcal {X}} \max  _ {y \in \mathcal {Y}} F _ {r} (x, y) := F (x, y) + \frac {r}{2} \| x - x _ {0} \| ^ {2} - \frac {r}{2} \| y - y _ {0} \| ^ {2}, \qquad (*)
$$
such that  $\forall (x,y)\in \mathcal{X}\times \mathcal{Y},$
$$
\nabla_ {x} F _ {r} \left(x _ {r}, y _ {r}\right) ^ {\top} \left(x _ {r} - x\right) - \nabla_ {y} F _ {r} \left(x _ {r}, y _ {r}\right) ^ {\top} \left(y _ {r} - y\right) \leq \epsilon_ {r}. \tag {2}
$$
Output:  $(x_{r},y_{r})$
We demonstrate that adding regularization is sufficient to achieve near-optimal guarantees for smooth convex-concave minimax problems. The general framework is summarized in Algorithm 2, where a base algorithm  $\mathcal{A}$  is applied to solve a regularized auxiliary problem which is strongly-convex in  $x$  and strongly-concave in  $y$ . For the inexact initialization case, we show that an optimal reproducibility bound of  $\mathcal{O}(\delta^2)$  and a near-optimal convergence rate of  $\tilde{\mathcal{O}}(1/\epsilon)$  can be attained simultaneously.
Theorem 4.4. Under Assumption 4.1 and given an inexact initialization oracle, Algorithm 2 with  $r = \epsilon /D^2$ ,  $\epsilon_r = \epsilon \cdot \min \{1,\delta^2 /(8D^2)\}$  and EG as base algorithm  $\mathcal{A}$  outputs a  $(2\epsilon)$ -saddle point  $(x_{r},y_{r})$  with  $\tilde{\mathcal{O}} (\ell D^2 /\epsilon)$  gradient complexity, and the reproducibility is  $4\delta^{2}$ .
Consider a  $\delta$ -inexact deterministic gradient oracle that returns  $G(x,y) = (G_{x}(x,y), G_{y}(x,y))$ . First note  $G_{r}(x,y) = (G_{x}(x,y) + r(x - x_{0}), G_{y}(x,y) - r(y - y_{0}))$  is a  $\delta$ -inexact gradient for the auxiliary problem  $(*)$ . We now characterize the convergence behavior of EG with this  $\delta$ -inexact gradient oracle, referred to as Inexact-EG, to solve the auxiliary problem.
Lemma 4.5. Consider  $\min_{x\in \mathcal{X}}\max_{y\in \mathcal{Y}}F_r(x,y)$ , where  $F_{r}(x,y)$  is  $r$ -strongly-convex-strongly-concave and  $(\ell +r)$ -smooth. Given an inexact gradient oracle that returns  $G_{r}(x,y)$  such that  $\| G_r(x,y) - \nabla F_r(x,y)\|^2\leq \delta^2$ , Inexact-EG with stepsize  $1 / (2(\ell +r))$  satisfies
$$
\left\| x _ {T} - x _ {r} ^ {*} \right\| ^ {2} + \left\| y _ {T} - y _ {r} ^ {*} \right\| ^ {2} \leq \exp \left(- \frac {T}{8} \frac {r}{\ell + r}\right) \left(\left\| x _ {0} - x _ {r} ^ {*} \right\| ^ {2} + \left\| y _ {0} - y _ {r} ^ {*} \right\| ^ {2}\right) + \frac {8 \delta^ {2}}{r} \left(\frac {2}{\ell + r} + \frac {1}{r}\right).
$$
where  $(x_r^*, y_r^*)$  is the unique saddle point of  $F_r(x, y)$ .
This lemma implies that Inexact-EG converges linearly to a neighborhood of size  $\mathcal{O}(\delta^2 /r^2)$  around the saddle point, which can be translated to the inaccuracy measure in (2) with  $\epsilon_r = \mathcal{O}(\delta /r)$  utilizing Lemma C.5. It is worth emphasizing that the size of this neighborhood is critical for achieving optimal reproducibility, and the dependency on  $r$  in the above convergence rate is key for attaining near-optimal complexity. Stonyakin et al. [70] analyzed Mirror-Prox [59] with restarts for strongly-monotone variational inequalities under a different inexact oracle (see Devolder et al. [27] and [70, Example 6.1] for its relationship with the inexactness notion of ours). Compared to Inexact-EG, their two-loop structure of the restart scheme is more complicated to implement.
Theorem 4.6. Under Assumption 4.1 with  $0 < \epsilon \leq \ell D^2$  and given an inexact gradient oracle, Algorithm 2 with  $r = \epsilon /D^2$ ,  $\epsilon_r = \mathcal{O}(\delta /r)$  and Inexact-EG as base algorithm  $\mathcal{A}$  outputs an  $\mathcal{O}(\epsilon +\delta /\epsilon)$ -saddle point with  $\tilde{\mathcal{O}} (\ell D^2 /\epsilon)$  gradient complexity, and the reproducibility is  $\mathcal{O}(\delta^2 /\epsilon^2)$ .
Remark 3. Some numerical experiments on a bilinear matrix game with inexact gradient information are provided in Appendix D (see Figure 2). With a small degradation in the convergence speed, the regularized framework in Algorithm 2 effectively improves the reproducibility of the base algorithm.
The theorem indicates that optimal reproducibility  $\mathcal{O}(\delta^2 /\epsilon^2)$  and near-optimal gradient complexity  $\tilde{\mathcal{O}} (1 / \epsilon)$  can be achieved when  $\delta \leq \mathcal{O}(\epsilon^2)$ . Note by Theorem 4.2 and 4.3, GDA and EG can find  $\epsilon$ -saddle points when  $\delta \leq \mathcal{O}(\epsilon)$ . Next, we introduce an alternative algorithmic framework that preserves the optimal reproducibility and attains the near-optimal complexity as long as  $\delta \leq \mathcal{O}(\epsilon)$ .
4.2 Inexact Proximal Point Method
We propose a two-loop inexact proximal point framework, presented in Algorithm 3, which can achieve both near-optimal gradient complexity and optimal algorithmic reproducibility. Compared to Algorithm 2, the regularization parameter  $1 / \alpha = \mathcal{O}(\ell)$  does not depend on the target accuracy  $\epsilon$  and the diameter  $D$ , and the center of the regularization term is the last iterate  $(x_{t},y_{t})$  instead of the initial point. Since the auxiliary problem is  $\ell$ -strongly-convex-strongly-concave and  $2\ell$ -smooth with condition number being  $\Theta(1)$ , a wider range of base algorithms can be used to achieve the optimal complexity than solving the problem in Algorithm 2 where the condition number is  $\Theta(1 / \epsilon)$ .
Algorithm 3 Inexact Proximal Point Method for Convex-Concave Minimax Problems
Input: Stepsize  $\alpha > 0$ , accuracy  $\hat{\epsilon} > 0$ , algorithm  $\mathcal{A}$ , initialization  $(x_0, y_0)$ , iteration number  $T$ .
for  $t = 0,1,\dots T - 1$  do
Apply  $\mathcal{A}$  to in exactly solve the smooth strongly-convex-strongly-concave problem
$$
(x_{t + 1},y_{t + 1})\gets \min_{x\in \mathcal{X}}\max_{y\in \mathcal{Y}}\hat{F}_{t}(x,y):= F(x,y) + \frac{1}{2\alpha}\| x - x_{t}\|^{2} - \frac{1}{2\alpha}\| y - y_{t}\|^{2}.
$$
$$
\nabla_ {x} \hat {F} _ {t} (x _ {t + 1}, y _ {t + 1}) ^ {\top} (x _ {t + 1} - x) - \nabla_ {y} \hat {F} _ {t} (x _ {t + 1}, y _ {t + 1}) ^ {\top} (y _ {t + 1} - y) \leq \hat {\epsilon}.
$$
Output:  $(\bar{x}_{T + 1},\bar{y}_{T + 1}) = (1 / T)\sum_{t = 0}^{T - 1}(x_{t + 1},y_{t + 1}).$
Theorem 4.7. Under Assumption 4.1 and given a  $\delta$ -inexact initialization oracle in Definition 4 with  $\delta \leq \mathcal{O}(1 / \sqrt{\epsilon})$ , Algorithm 3 with  $\hat{\epsilon} \leq \delta^2 / (2\alpha T^2)$  and  $\alpha = 1 / \ell$  outputs an  $\mathcal{O}(\epsilon)$ -saddle point after  $T = \mathcal{O}(1 / \epsilon)$  iterations, and the reproducibility is  $9\delta^2$ .
Remark 4. The required accuracy  $\hat{\epsilon}$  for the auxiliary problem is  $\mathcal{O}(\delta^2\epsilon^2)$ . Given that the auxiliary problem is  $\ell$ -strongly-convex-strongly-concave and  $2\ell$ -smooth, various linearly convergent algorithms such as EG, GDA, and Optimistic GDA [35] can find a point that satisfies the stopping criterion within  $\mathcal{O}(\log(1 / (\delta \epsilon)))$  iterations. As a result, the total gradient complexity is  $\tilde{\mathcal{O}}(1/\epsilon)$ . In contrast, using GDA as the base algorithm in Algorithm 2 will lead to a sub-optimal gradient complexity.
Theorem 4.8. Under Assumption 4.1 and given a  $\delta$ -inexact deterministic gradient oracle in Definition 4 with  $\delta \leq \mathcal{O}(\epsilon)$ , Algorithm 3 with  $\hat{\epsilon} \leq \mathcal{O}(\delta)$  and  $\alpha = 1/\ell$  outputs an  $\mathcal{O}(\epsilon)$ -saddle point after  $T = \mathcal{O}(1/\epsilon)$  iterations, and the reproducibility is  $\mathcal{O}(\delta^2/\epsilon^2)$ .
Remark 5. This theorem requires solving the auxiliary problem with a  $\delta$ -inexact gradient oracle. In addition to Inexact-EG presented in Lemma 4.5, we show in Appendix C.1 that GDA with inexact gradients (Inexact-GDA) can also converge linearly to the optimal point up to a  $\mathcal{O}(\delta^2)$  error. Thus the total complexity is  $\mathcal{O}((1/\epsilon)\log(1/\delta))$  using both Inexact-EG and Inexact-GDA.
5 Stochastic Gradient Oracle for Minimax Problems
To provide a complete picture, in this section, we consider the stochastic minimax problem:
$$
\min  _ {x \in \mathcal {X}} \max  _ {y \in \mathcal {Y}} F (x, y) = \mathbb {E} _ {\xi} [ f (x, y; \xi) ], \tag {3}
$$
where the expectation is taken over a random vector  $\xi$ . We have access to a  $\delta$ -inexact stochastic gradient oracle that can return unbiased gradients  $\nabla f(x,y;\xi)$  with a bounded variance  $\delta^2$  at each point  $(x,y)$ . We consider the popular algorithm called stochastic gradient descent ascent (SGDA). The convergence behaviors of SGDA for the stochastic minimax problem (3) are well-known in various settings. However, due to the randomness in the gradient oracle, independent runs of SGDA may lead to different outputs even with the same parameters. Following Definition 6, we further establish the  $(\epsilon,\delta)$ -deviation of SGDA in the theorem below.
Theorem 5.1. Under Assumptions 4.1 and given an inexact stochastic gradient oracle in Definition 4 with  $\delta = \mathcal{O}(1)$ , the average iterates  $(\bar{x}_T, \bar{y}_T) = (1 / T) \sum_{t=0}^{T-1} (x_t, y_t)$  of SGDA with stepsize  $1 / (\ell \epsilon T)$  after  $T = \Omega(1 / \epsilon^2)$  iterations is an  $\mathcal{O}(\epsilon)$ -stationary point and the reproducibility is  $\mathcal{O}\big(\delta^2 / (\epsilon^2 T)\big)$ .
The  $\mathcal{O}(1 / \epsilon^2)$  sample complexity of SGDA is known to be optimal when the objective  $F(x,y)$  is convex-concave [46]. Moreover, our results suggest that SGDA is also optimally reproducible, as the lower-bound of  $\Omega\left(\delta^{2} / (\epsilon^{2}T)\right)$  for convex minimization problems [1] is also valid for minimax optimization according to our discussions in Lemma B.3.
In this work, instead of solely focusing on convergence performance, we investigate another crucial property of machine learning algorithms, i.e., algorithms should be reproducible against slight perturbations. We provide the first algorithms to simultaneously achieve optimal algorithmic reproducibility and near-optimal gradient complexity for both smooth convex minimization and smooth convex-concave minimax problems under various inexact oracle models. We focus on the convex case as a first step since it is the most basic and fundamental setting in optimization. We believe a solid understanding of the reproducibility in convex optimization will shed insights for that of the more challenging nonconvex optimization. Note that some of the analysis and techniques used in this paper can be extended to the smooth nonconvex setting, aligning with the stability analysis for nonconvex objectives [42, 49]. The proposed regularized framework can be applied to nonconvex functions as well using the convergence analysis of regularization or proximal point-based methods [2, 74]. However, the non-expansiveness property in Lemma 3.2 that is essential for the reproducibility analysis will not hold any more without the convexity assumption. One potential way to alleviate it is to impose additional structural assumptions on the gradients such as negative comonotonicity [39]. We leave a detailed study of the reproducibility in nonconvex optimization to future work.
Other possible improvements of our results include deriving optimal reproducibility with an accelerated convergence rate for smooth convex minimization problems under the inexact gradient oracle, removing the additional logarithmic terms in the complexity of our algorithms using techniques in Allen-Zhu and Hazan [3], studying the reproducibility under the presence of mixed inexact oracles, and extending the results to nonsmooth settings. Another interesting direction is to design simpler and more direct methods with both optimal reproducibility and convergence guarantees. A possible way is to directly unwrap the regularized algorithmic framework 1 or 2, leading to Tikhonov regularization [8] or anchoring methods [75].
We thank the anonymous reviewers for their valuable suggestions. Liang Zhang gratefully acknowledges funding from the Max Planck ETH Center for Learning Systems (CLS). Amin Karbasi
acknowledges funding in direct support of this work from NSF (IIS-1845032), ONR (N00014-19-1-2406), and the AI Institute for Learning-Enabled Optimization at Scale (TILOS). Niao He is supported by ETH research grant, NCCR Automation, and Swiss National Science Foundation Project Funding No. 200021-207343.