SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Liang Zhang*
ETH Zurich & Max Planck Institute
liang.zhang@inf.ethz.ch
Amin Karbasi
Yale University & Google Research
amin.karbasi@yale.edu
Junchi Yang*
junchi.yang@inf.ethz.ch
niao.he@inf.ethz.ch
Algorithmic reproducibility measures the deviation in outputs of machine learning algorithms upon minor changes in the training process. Previous work suggests that first-order methods would need to trade-off convergence rate (gradient complexity) for better reproducibility. In this work, we challenge this perception and demonstrate that both optimal reproducibility and near-optimal convergence guarantees can be achieved for smooth convex minimization and smooth convex-concave minimax problems under various error-prone oracle settings. Particularly, given the inexact initialization oracle, our regularization-based algorithms achieve the best of both worlds – optimal reproducibility and near-optimal gradient complexity – for minimization and minimax optimization. With the inexact gradient oracle, the near-optimal guarantees also hold for minimax optimization. Additionally, with the stochastic gradient oracle, we show that stochastic gradient descent ascent is optimal in terms of both reproducibility and gradient complexity. We believe our results contribute to an enhanced understanding of the reproducibility-convergence trade-off in the context of convex optimization.
In the realm of machine learning, improving model performance remains a primary focus; however, this alone falls short when it comes to the practical deployment of algorithms. There has been a growing emphasis on the development of machine learning systems that prioritize trustworthiness and reliability. Central to this pursuit is the concept of reproducibility [38, 64], which requires algorithms to yield consistent outputs, in the face of minor changes to the training environment. Unfortunately, a lack of reproducibility has been reported across various domains [10, 40, 41, 64], posing significant challenges to the integrity and dependability of scientific research. Notably, empirical studies in Henderson et al. [43] have revealed that reproducing baseline algorithms in reinforcement learning is a formidable task due to both inherent sources (e.g., random seeds, environment properties) and external sources (e.g., hyperparameters, codebases) of non-determinism. These findings underscore the criticality of having access to the relevant code and data, as well as sufficient documentation of experimental details, to ensure reproducibility in machine learning algorithms.
Instead of considering the irreproducibility issue solely from an empirical perspective, Ahn et al. [1] initiated the theoretical study of reproducibility in machine learning as an inherent characteristic of the algorithms themselves. They focus on first-order algorithms for convex minimization problems and
Table 1: Algorithmic reproducibility (Def. 3) and gradient complexity for algorithms in the smooth convex minimization setting given inexact deterministic oracles (Def. 1). Here, "LB" stands for lower-bound and  $a \wedge b$  denotes  $\min \{a, b\}$ . For the inexact gradient oracle,  $\delta \leq \mathcal{O}(\epsilon)$  is required for GD to be  $\epsilon$ -optimal and  $\delta \leq \mathcal{O}(\epsilon^{5/4})$  is required for Algo. 1.
Algorithm	Inexact Initialization	Inexact Gradient
Convergence	Reproducibility	Convergence	Reproducibility
GD [1]	O(1/ε)	O(δ2)	O(1/ε)	O(δ2/ε2)
AGD [6]	O(1/√ε)	O(δ2e1/√ε)	-	-
Algo. 1 (Thm. 3.3, 3.5)	O(1/√ε)	O(δ2)	O(1/√ε)	O(δ2/ε2.5)
LB [61, 1]	Ω(1/√ε)	Ω(δ2)	Ω(1/√ε)	Ω(δ2/ε2)
define reproducibility as the deviation in outputs of independent runs of the algorithms, accounting for sources of irreproducibility captured by inexact or noisy oracles. In particular, they consider three practical error-prone operations, including inexact initialization, inexact gradient computation due to numerical errors, and stochastic gradient computation due to sampling or shuffling. When restricting the outputs to be  $\epsilon$ -optimal and assuming the level of inexactness that could cause irreproducibility is bounded by  $\delta$ , they establish both lower and upper reproducibility bounds of (stochastic) gradient descent for all three settings. The lower-bounds indicate the existence of intrinsic irreproducibility for any first-order algorithms, while the matching upper-bounds suggest that (stochastic) gradient descent already achieves optimal reproducibility.
An important question arises regarding whether there is a fundamental trade-off between reproducibility and convergence speed in algorithms. For example, in the case of inexact initialization, the optimally reproducible algorithm [1], gradient descent (GD), is known to be strictly sub-optimal in terms of gradient complexity for smooth convex minimization problems [61]. On the other hand, the optimally convergent algorithm, Nesterov's accelerated gradient descent (AGD) [60], suffers from a worse reproducibility bound [6]. The situation becomes more intricate in the case of inexact gradient computation. A natural question that we aim to address in this paper is: Can we achieve the best of both worlds - optimal convergence and reproducibility?
On another front, while minimization problems can effectively model and explain the behavior of many traditional machine learning systems, recent years have witnessed a surge of applications that are formulated as minimax optimization problems. Important examples include generative adversarial networks (GANs) [37], robust optimization [54], and reinforcement learning [25]. Despite a wealth of convergence theory for various minimax optimization algorithms, extensive empirical evidence suggests that these algorithms can be hard to train in practice [67, 4, 53]: the training procedure can be very unstable [23] and highly sensitive to changes of hyper-parameters. Motivated by such issues, we initiate the theoretical study of algorithmic reproducibility in minimax optimization. The second question that we aim to address in this paper is: What are the fundamental limits of reproducibility for minimax optimization algorithms and their convergence-reproducibility trade-offs? We will focus on smooth convex-concave minimax optimization as a first step, where the irreproducibility issue comes from either inexact initialization, inexact gradient computation, or stochastic gradient computation.
Our main contributions are two-fold:
First, we propose Algorithm 1, which solves a regularized version of the smooth convex minimization problem. This algorithm achieves both optimal algorithmic reproducibility of  $\mathcal{O}(\delta^2)$  and near-optimal gradient complexity of  $\tilde{\mathcal{O}} (1 / \sqrt{\epsilon})^2$  under the  $\delta$ -inexact initialization oracle. Table 1 provides a comparison with GD and AGD. Our results rely on the key observation that solutions to strongly-
2 Throughout the paper,  $\tilde{\mathcal{O}}$  hides additional logarithmic factors. We claim near-optimality of the result when it is optimal up to logarithmic terms.
Table 2: Algorithmic reproducibility (Def. 6) and gradient complexity for algorithms in the smooth convex-concave minimax setting given inexact deterministic oracles (Def. 4). Here, "LB" stands for lower-bound and  $a \wedge b$  denotes  $\min \{a, b\}$ . For the inexact gradient oracle,  $\delta \leq \mathcal{O}(\epsilon)$  is required for GDA, EG, and Algo. 3 to be  $\epsilon$ -optimal, and  $\delta \leq \mathcal{O}(\epsilon^2)$  is required for Algo. 2. The diameter  $D$  in Assumption 4.1 is a trivial upper-bound for reproducibility in all cases.
Algorithm	Inexact Initialization	Inexact Gradient
Convergence	Reproducibility	Convergence	Reproducibility
GDA (Thm. 4.2)	O(1/ε2)	O(δ2)	O(1/ε2)	O(δ2/ε2)
EG (Thm. 4.3)	O(1/ε)	O(δ2e1/ε ∧ (δ2+1/ε2))	O(1/ε)	O(δ2e1/ε ∧ 1/ε2)
Algo. 2 (Thm. 4.4, 4.6)	O(1/ε)	O(δ2)	O(1/ε)	O(δ2/ε2)
Algo. 3 (Thm. 4.7, 4.8)	O(1/ε)	O(δ2)	O(1/ε)	O(δ2/ε2)
LB ([63], Lem. B.3)	Ω(1/ε)	Ω(δ2)	Ω(1/ε)	Ω(δ2/ε2)
convex regularized problems are unique, allowing algorithms that converge close to the minimizers to be reproducible. This highlights the effectiveness of regularization in achieving near-optimal convergence without compromising reproducibility.
Second, we extend the notion of reproducibility to smooth convex-concave minimax optimization (1) under inexact initialization and inexact gradient oracles. We establish the first reproducibility analysis for commonly-used minimax optimization algorithms such as gradient descent ascent (GDA) and Extragradient (EG) [48]. Our results indicate that they are either sub-optimal in terms of convergence or reproducibility. To address this, we propose two new algorithms (Algorithm 2 and 3) which utilize regularization techniques to achieve optimal algorithmic reproducibility and near-optimal gradient complexity. The summarized results are presented in Table 2. Additional numerical experiments showcasing the effectiveness of our algorithms can be found in Appendix D. Although smooth convex-concave minimax optimization is nonsmooth in its primal form, our results indicate an improved reproducibility compared to the result of general nonsmooth convex problems [1] by leveraging the additional minimax structure. Lastly, in the case of stochastic gradient oracle, we show stochastic GDA can simultaneously attain both optimal convergence and optimal reproducibility.
Related Notions. (Reproducibility) Previous works that study reproducibility in machine learning are mostly on the empirical side. They either conduct experiments to report irreproducibility issues in the community [40, 43, 18, 64], or propose practical tricks to improve reproducibility [69, 79, 56, 19]. Ahn et al. [1] initiated the theoretical study of reproducibility in convex minimization problems as a property of the algorithm itself. (Replicability) In an independent work, Impagliazzo et al. [45] proposed the notion of replicability in statistical learning, where an algorithm is replicable if its outputs on two i.i.d. datasets are exactly the same with high probability. Its connection to generalization and differential privacy [29] is established in Bun et al. [21] and Kalavasis et al. [47]. Replicable algorithms are proposed in the context of stochastic bandits [30] and clustering [31]. (Stability) Depending on the context, the term stability may have different meanings. In empirical studies [4, 5, 22], instability often refers to issues such as oscillations or failure to converge during training. In learning theory, algorithmic stability [17] measures the deviation in an algorithm's outputs for finite-sum problems when a single item in the input dataset is replaced by an i.i.d. in-distribution sample. The concept receives increasing attention as it implies dimension-independent generalization bounds of gradient-based methods for both minimization [42, 11, 6] and minimax [33, 49, 16] problems. In the area of differential equations [13] and variational inequalities [32], stability is also examined as a property of the solution set in response to perturbations in the problem conditions.
In this work, we consider the notion of reproducibility that characterizes the behavior of algorithms upon slight perturbations in the training. We defer the task of establishing intrinsic connections
among related notions to future work. The most closely related concept is algorithmic stability, where the analysis is similar to reproducibility under the inexact deterministic gradient oracle. Attia and Koren [6] showed the stability of AGD [60] grows exponentially with the number of iterations. Later, this is improved to quadratic dependence [7] based on a similar idea as ours that leverages stability of solutions to strongly-convex minimization problems [68, 34]. However, since there is no inexactness of the gradients in their setting, it is possible to ensure outputs that are arbitrarily close to the optimal solution. Given the presence of inexact gradients in our case, the convergence is only limited to a neighborhood of the optimal solution, which makes the problem more challenging. The trade-off between stability and convergence was investigated in Chen et al. [24]. Their results suggest that a faster algorithm has to be less stable, and vice versa. However, we show the feasibility of achieving both optimal reproducibility and near-optimal convergence simultaneously in the setting we considered.
Minimax Optimization. Existing literature on minimax optimization primarily focuses on convergence analysis across various settings. For instance, there are studies on the strongly-convex-strongly-concave case [72, 57], convex-concave case [59, 63], and nonconvex-(strongly)-concave case [52, 71]. The lower complexity bounds have also been established for these settings [80, 50, 77]. Our work aims to design reproducible algorithms while maintaining the optimal oracle complexities achieved in these previous works.
Inexact Gradient Oracles. A series of works investigate the convergence properties of first-order methods under deterministic inexact oracles for minimization [26, 27, 28] and minimax [70] problems. However, their inexact oracles differ from ours, and our focus is more on reproducibility. In recent years, there has been increasing interest in studying biased stochastic gradient oracles as well, where the bias arises from various sources such as problem structure [44], compression [14] or Byzantine failure [15] in distributed learning, and gradient-free optimization [62]. These biases can also contribute to irreproducibility, and this direction would be an interesting avenue of research.
Regularization Technique. The central algorithmic insight driving our improvements towards obtaining both optimal convergence and reproducibility is the regularization technique, which is commonly used in the optimization literature. One important use case is to boost convergence by leveraging known and good convergence properties of algorithms on smooth strongly-convex functions for solving convex and nonsmooth problems, see e.g., [51, 3, 74], just to name a few. In addition, the regularization technique has also been demonstrated to be useful in improving stability and generalization [76, 7], enhancing sensitivity and privacy guarantees [34, 78], etc. In this paper, we provide another important use case by showing an improved convergence-reproducibility trade-off.
2 Preliminaries in Algorithmic Reproducibility
Notation. We use  $\| \cdot \|$  to represent the Euclidean norm.  $\Pi_{\mathcal{C}}(x)$  denotes the projection of  $x$  onto the set  $\mathcal{C}$ . A function  $h:S\to \mathbb{R}$  is  $\ell$ -smooth if it is differentiable and its gradient  $\nabla h$  satisfies  $\| \nabla h(x_1) - \nabla h(x_2)\| \leq \ell \| x_1 - x_2\|$  for any  $x_{1},x_{2}$  in the domain  $S\in \mathbb{R}^d$ . A function  $g:S\rightarrow \mathbb{R}$  is convex if  $g(\alpha x_{1} + (1 - \alpha)x_{2})\leq \alpha g(x_{1}) + (1 - \alpha)g(x_{2})$  for any  $\alpha \in [0,1]$  and  $x_{1},x_{2}\in S$ . If  $g$  satisfies  $g(x) - (\mu /2)\| x\|^2$  being convex with  $\mu >0$ , then it is  $\mu$ -strongly-convex. Similarly, a function  $g:S\to \mathbb{R}$  is concave if  $-g$  is convex, and  $\mu$ -strongly-concave if  $-g$  is  $\mu$ -strongly-convex.
Ahn et al. [1] studied the algorithmic reproducibility for convex minimization problems  $\min_{x\in \mathcal{X}}F(x)$  measured by the  $(\epsilon ,\delta)$ -deviation bound of an algorithm  $\mathcal{A}$ . Here,  $\delta$  denotes the size of errors in the oracles that can lead to different outputs in independent runs of the same algorithm. The notion of reproducibility also requires  $\mathcal{A}$  to produce  $\epsilon$ -optimal solutions, avoiding trivial outputs.
Definition 1. Three different inexact oracle models are considered: (i) a  $\delta$ -inexact initialization oracle that returns a starting point  $x_0 \in \mathcal{X}$  such that  $\|x_0 - u_0\|^2 \leq \delta^2/4$  for some reference point  $u_0 \in \mathcal{X}$ , (ii) a  $\delta$ -inexact deterministic gradient oracle that returns an inexact gradient  $G(x)$  such that  $\|\nabla F(x) - G(x)\|^2 \leq \delta^2$  for the true gradient  $\nabla F(x)$ , (iii) a  $\delta$ -inexact stochastic gradient oracle that returns an unbiased gradient estimate  $\nabla f(x; \xi)$  such that  $\mathbb{E} \| \nabla f(x; \xi) - \nabla F(x)\|^2 \leq \delta^2$ .
Definition 2. A point  $\hat{x} \in \mathcal{X}$  is an  $\epsilon$ -optimal solution if  $F(\hat{x}) - \min_{x \in \mathcal{X}} F(x) \leq \epsilon$  in the deterministic setting, or  $\mathbb{E}[F(\hat{x})] - \min_{x \in \mathcal{X}} F(x) \leq \epsilon$  in the stochastic setting, where the expectation is taken over all the randomness in the gradient oracle and in the algorithm that outputs  $\hat{x}$ .
Definition 3. The  $(\epsilon, \delta)$ -deviation  $\| \hat{x} - \hat{x}'\|^2$  is used to measure the reproducibility of an algorithm  $\mathcal{A}$  with  $\epsilon$ -optimal solutions  $\hat{x}$  and  $\hat{x}'$ , where  $\hat{x}$  and  $\hat{x}'$  are outputs of two independent runs of the algorithm  $\mathcal{A}$  given a  $\delta$ -inexact oracle in Definition 1.
We expand the definitions of reproducibility to encompass minimax optimization problems:
$$
\min  _ {x \in \mathcal {X}} \max  _ {y \in \mathcal {Y}} F (x, y). \tag {1}
$$
Our goal is to find the saddle point  $(x^{*},y^{*})$  of the function  $F(x,y)$ , such that  $F(x^{*},y) \leq F(x^{*},y^{*}) \leq F(x,y^{*})$  holds for all  $(x,y) \in \mathcal{X} \times \mathcal{Y}$ . The optimality of a point  $(\hat{x},\hat{y})$  can be assessed by its duality gap, defined as  $\max_{y \in \mathcal{Y}} F(\hat{x},y) - \min_{x \in \mathcal{X}} F(x,\hat{y})$ . In the minimax setting, we analyze reproducibility under the following inexact oracle models.
Definition 4. Three different inexact oracle models are considered: (i) a  $\delta$ -inexact initialization oracle that returns a starting point  $(x_0, y_0) \in \mathcal{X} \times \mathcal{Y}$  such that  $\|x_0 - u_0\|^2 + \|y_0 - v_0\|^2 \leq \delta^2/4$  for some reference point  $(u_0, v_0) \in \mathcal{X} \times \mathcal{Y}$ , (ii) a  $\delta$ -inexact deterministic gradient oracle that returns an inexact gradient  $G(x, y) = (G_x(x, y), G_y(x, y))$  at any querying point  $(x, y) \in \mathcal{X} \times \mathcal{Y}$  such that  $\|\nabla F(x, y) - G(x, y)\|^2 \leq \delta^2$  for the true gradient  $\nabla F(x, y) = (\nabla_x F(x, y), \nabla_y F(x, y))$ , (iii) a  $\delta$ -inexact stochastic gradient oracle that returns an unbiased gradient estimate  $\nabla f(x, y; \xi) = (\nabla_x f(x, y; \xi), \nabla_y f(x, y; \xi))$  such that  $\mathbb{E}_{\xi} \|\nabla f(x, y; \xi) - \nabla F(x, y)\|^2 \leq \delta^2$ .
Definition 5. A point  $(\hat{x},\hat{y})\in \mathcal{X}\times \mathcal{Y}$  is an  $\epsilon$  -saddle point solution if its duality gap satisfies that  $\max_{y\in \mathcal{Y}}F(\hat{x},y) - \min_{x\in \mathcal{X}}F(x,\hat{y})\leq \epsilon$  in the deterministic setting, or its weak duality gap satisfies that  $\max_{y\in \mathcal{Y}}\mathbb{E}[F(\hat{x},y)] - \min_{x\in \mathcal{X}}\mathbb{E}[F(x,\hat{y})]\leq \epsilon$  in the stochastic setting.
Definition 6. The  $(\epsilon, \delta)$ -deviation  $\| \hat{x} - \hat{x}'\|^2 + \| \hat{y} - \hat{y}'\|^2$  is used to measure the reproducibility of an algorithm  $\mathcal{A}$  with  $\epsilon$ -saddle points  $(\hat{x}, \hat{y})$  and  $(\hat{x}', \hat{y}')$ , where  $(\hat{x}, \hat{y})$  and  $(\hat{x}', \hat{y}')$  are outputs of two independent runs of the algorithm  $\mathcal{A}$  given a  $\delta$ -inexact oracle in Definition 4.
The optimal convergence rates are well-understood for the convex optimization problems, including convex minimization [61] and convex-concave minimax optimization [63]. Ahn et al. [1] provided the theoretical lower-bounds of reproducibility for convex minimization problems, which can be extended to convex-concave minimax problems as well (Lemma B.3). We say an algorithm achieves optimal reproducibility if its reproducibility upper-bounds match the established theoretical lower-bounds.
3 Deterministic Gradient Oracle for Minimization Problems
In this section, we consider convex minimization problems of the form
$$
\min _ {x \in \mathcal {X}} F (x),
$$
where  $\mathcal{X}$  is a convex and closed set. We focus on the standard smooth and convex setting as detailed in Assumption 3.1. Our goal is to find an  $\epsilon$ -optimal point as in Definition 2. Ahn et al. [1] showed that the optimal convergence rate and reproducibility can be achieved at the same time using stochastic gradient descent (SGD) for the stochastic gradient oracle model. In the deterministic case, they showed GD achieves the optimal reproducibility, albeit with a sub-optimal convergence rate [60, 61]. Considering the instability of accelerated gradient descent (AGD) [26, 28, 6], Ahn et al. [1] conjectured that  $\Omega(1/\epsilon)$  gradient complexity is necessary to attain the optimal reproducibility.
Assumption 3.1. The function  $F$  is convex and  $\ell$ -smooth. We have access to initial points  $x_0$  that are  $D$ -close to an optimal solution, i.e.,  $\| x^* - x_0 \|^2 \leq D^2$  for some  $x^* \in \arg \min_{x \in \mathcal{X}} F(x)$ .
We introduce a generic algorithmic framework outlined in Algorithm 1, that solves a quadratically regularized auxiliary problem  $(\star)$  using a base algorithm  $\mathcal{A}$  with initialization  $x_0$  until an accuracy of  $\epsilon_r$  is reached. Our key insight is that since the optimal solution for strongly convex problems is unique, the reproducibility of the outputs from the regularized problem can be easily guaranteed. Note that the regularization parameter  $r$  presents a trade-off: as  $r$  increases, the auxiliary problem can be solved more efficiently, but the obtained solution deviates further from the original solution. We will show that Algorithm 1 achieves a near-optimal complexity of  $\tilde{\mathcal{O}}(1/\sqrt{\epsilon})$ , along with optimal reproducibility under an inexact initialization oracle and slightly sub-optimal reproducibility under an inexact deterministic gradient oracle. This finding disproves the conjecture [1] that  $\Omega(1/\epsilon)$  complexity is necessary to achieve optimal reproducibility.
Algorithm 1 Reproducible Algorithmic Framework for Convex Minimization Problems
Input: Regularization parameter  $r > 0$ , accuracy  $\epsilon_r > 0$ , base algorithm  $\mathcal{A}$ , initial point  $x_0 \in \mathcal{X}$ . Apply  $\mathcal{A}$  to approximately solve the  $r$ -strongly-convex and  $(\ell + r)$ -smooth problem
$$
x _ {r} \leftarrow \underset {x \in \mathcal {X}} {\arg \min } F _ {r} (x) := F (x) + \frac {r}{2} \| x - x _ {0} \| ^ {2}, \quad (\star)
$$
such that the optimality gap
$$
F _ {r} (x _ {r}) - \min  _ {x \in \mathcal {X}} F _ {r} (x) \leq \epsilon_ {r}.
$$
Output:  $x_{r}$ .
3.1 Inexact Initialization Oracle
We first examine the behavior of Algorithm 1 with access to exact deterministic gradients but given different initializations. Starting from two distinct initial points  $x_0$  and  $x_0'$  such that  $\| x_0 - x_0' \|^2 \leq \delta^2$ , we want to control the deviation between the final outputs  $x_r$  and  $x_r'$  of the algorithm. The following contraction property is essential to attain optimal reproducibility.
Lemma 3.2. Let  $x_{r}^{*} = \arg \min_{x\in \mathcal{X}}\{F(x) + (r / 2)\| x - x_{0}\|^{2}\}$  and  $(x_r^*)' = \arg \min_{x\in \mathcal{X}}\{F(x) + (r / 2)\| x - x_0'\| ^2\}$ . When  $F$  is convex, it holds that  $\| x_r^* -(x_r^*)'\|^2\leq \| x_0 - x_0'\|^2$  for any  $r > 0$ .
This indicates the optimal solutions are reproducible up to  $\delta^2$ . Consequently, if we can solve the auxiliary problem  $(\star)$  to a high accuracy  $\epsilon_r$ , we can ensure the final output  $x_{r}$  is reproducible. The selection of  $\epsilon_r$  exhibits a trade-off: a smaller value increases complexity, yet brings the output closer to the reproducible  $x_{r}^{*}$ . We characterize the complexity and reproducibility of Algorithm 1 by carefully choosing the parameters  $r$  and  $\epsilon_r$ .