Title: Optimal Guarantees for Algorithmic Reproducibility and Gradient Complexity in Convex Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reproducibility

Convex Optimization

Minimax Optimization

Saddle-Point Problem
Abstract: Algorithmic reproducibility measures the deviation in outputs of machine learning algorithms upon minor changes in the training process. Previous work suggests that first-order methods would need to trade-off convergence rate (gradient complexity) for better reproducibility. In this work, we challenge this perception and demonstrate that both optimal reproducibility and near-optimal convergence guarantees can be achieved for smooth convex minimization and smooth convex-concave minimax problems under various error-prone oracle settings. Particularly, given the inexact initialization oracle, our regularization-based algorithms achieve the best of both worlds -- optimal reproducibility and near-optimal gradient complexity -- for minimization and minimax optimization. With the inexact gradient oracle, the near-optimal guarantees also hold for minimax optimization. Additionally, with the stochastic gradient oracle, we show that stochastic gradient descent ascent is optimal in terms of both reproducibility and gradient complexity. We believe our results contribute to an enhanced understanding of the reproducibility-convergence trade-off in the context of convex optimization.

Optimal Guarantees for Algorithmic Reproducibility and Gradient Complexity in Convex Optimization

Abstract

Algorithmic reproducibility measures the deviation in outputs of machine learning algorithms upon minor changes in the training process. Previous work suggests that first-order methods would need to trade-off convergence rate (gradient complexity) for better reproducibility. In this work, we challenge this perception and demonstrate that both optimal reproducibility and near-optimal convergence guarantees can be achieved for smooth convex minimization and smooth convex-concave minimax problems under various error-prone oracle settings. Particularly, given the inexact initialization oracle, our regularization-based algorithms achieve the best of both worlds - optimal reproducibility and near-optimal gradient complexity - for minimization and minimax optimization. With the inexact gradient oracle, the near-optimal guarantees also hold for minimax optimization. Additionally, with the stochastic gradient oracle, we show that stochastic gradient descent ascent is optimal in terms of both reproducibility and gradient complexity. We believe our results contribute to an enhanced understanding of the reproducibility-convergence trade-off in the context of convex optimization.

1 Introduction

In the realm of machine learning, improving model performance remains a primary focus; however, this alone falls short when it comes to the practical deployment of algorithms. There has been a growing emphasis on the development of machine learning systems that prioritize trustworthiness and reliability. Central to this pursuit is the concept of reproducibility [38, 64], which requires algorithms to yield consistent outputs, in the face of minor changes to the training environment. Unfortunately, a lack of reproducibility has been reported across various domains [10, 40, 41, 64], posing significant challenges to the integrity and dependability of scientific research. Notably, empirical studies in Henderson et al. [43] have revealed that reproducing baseline algorithms in reinforcement learning is a formidable task due to both inherent sources (e.g., random seeds, environment properties) and external sources (e.g., hyperparameters, codebases) of non-determinism. These findings underscore the criticality of having access to the relevant code and data, as well as sufficient documentation of experimental details, to ensure reproducibility in machine learning algorithms.

Instead of considering the irreproducibility issue solely from an empirical perspective, Ahn et al. [1] initiated the theoretical study of reproducibility in machine learning as an inherent characteristic of the algorithms themselves. They focus on first-order algorithms for convex minimization problems and

Junchi Yang ∗

define reproducibility as the deviation in outputs of independent runs of the algorithms, accounting for sources of irreproducibility captured by inexact or noisy oracles. In particular, they consider three practical error-prone operations, including inexact initialization, inexact gradient computation due to numerical errors, and stochastic gradient computation due to sampling or shuffling. When restricting the outputs to be ϵ -optimal and assuming the level of inexactness that could cause irreproducibility is bounded by δ , they establish both lower and upper reproducibility bounds of (stochastic) gradient descent for all three settings. The lower-bounds indicate the existence of intrinsic irreproducibility for any first-order algorithms, while the matching upper-bounds suggest that (stochastic) gradient descent already achieves optimal reproducibility.

An important question arises regarding whether there is a fundamental trade-off between reproducibility and convergence speed in algorithms. For example, in the case of inexact initialization, the optimally reproducible algorithm [1], gradient descent (GD), is known to be strictly sub-optimal in terms of gradient complexity for smooth convex minimization problems [61]. On the other hand, the optimally convergent algorithm, Nesterov's accelerated gradient descent (AGD) [60], suffers from a worse reproducibility bound [6]. The situation becomes more intricate in the case of inexact gradient computation. A natural question that we aim to address in this paper is: Can we achieve the best of both worlds - optimal convergence and reproducibility?

On another front, while minimization problems can effectively model and explain the behavior of many traditional machine learning systems, recent years have witnessed a surge of applications that are formulated as minimax optimization problems. Important examples include generative adversarial networks (GANs) [37], robust optimization [54], and reinforcement learning [25]. Despite a wealth of convergence theory for various minimax optimization algorithms, extensive empirical evidence suggests that these algorithms can be hard to train in practice [67, 4, 53]: the training procedure can be very unstable [23] and highly sensitive to changes of hyper-parameters. Motivated by such issues, we initiate the theoretical study of algorithmic reproducibility in minimax optimization. The second question that we aim to address in this paper is: What are the fundamental limits of reproducibility for minimax optimization algorithms and their convergence-reproducibility trade-offs? We will focus on smooth convex-concave minimax optimization as a first step, where the irreproducibility issue comes from either inexact initialization, inexact gradient computation, or stochastic gradient computation.

1.1 Our Contributions

Our main contributions are two-fold:

First, we propose Algorithm 1, which solves a regularized version of the smooth convex minimization problem. This algorithm achieves both optimal algorithmic reproducibility of O ( δ 2 ) and near-optimal gradient complexity of ˜ O (1 / √ ϵ ) 2 under the δ -inexact initialization oracle. Table 1 provides a comparison with GD and AGD. Our results rely on the key observation that solutions to strongly-

convex regularized problems are unique, allowing algorithms that converge close to the minimizers to be reproducible. This highlights the effectiveness of regularization in achieving near-optimal convergence without compromising reproducibility.

Second, we extend the notion of reproducibility to smooth convex-concave minimax optimization (1) under inexact initialization and inexact gradient oracles. We establish the first reproducibility analysis for commonly-used minimax optimization algorithms such as gradient descent ascent (GDA) and Extragradient (EG) [48]. Our results indicate that they are either sub-optimal in terms of convergence or reproducibility. To address this, we propose two new algorithms (Algorithm 2 and 3) which utilize regularization techniques to achieve optimal algorithmic reproducibility and near-optimal gradient complexity. The summarized results are presented in Table 2. Additional numerical experiments showcasing the effectiveness of our algorithms can be found in Appendix D. Although smooth convexconcave minimax optimization is nonsmooth in its primal form, our results indicate an improved reproducibility compared to the result of general nonsmooth convex problems [1] by leveraging the additional minimax structure. Lastly, in the case of stochastic gradient oracle, we show stochastic GDA can simultaneously attain both optimal convergence and optimal reproducibility.

1.2 Related Works

Related Notions. (Reproducibility) Previous works that study reproducibility in machine learning are mostly on the empirical side. They either conduct experiments to report irreproducibility issues in the community [40, 43, 18, 64], or propose practical tricks to improve reproducibility [69, 79, 56, 19]. Ahn et al. [1] initiated the theoretical study of reproducibility in convex minimization problems as a property of the algorithm itself. (Replicability) In an independent work, Impagliazzo et al. [45] proposed the notion of replicability in statistical learning, where an algorithm is replicable if its outputs on two i.i.d. datasets are exactly the same with high probability. Its connection to generalization and differential privacy [29] is established in Bun et al. [21] and Kalavasis et al. [47]. Replicable algorithms are proposed in the context of stochastic bandits [30] and clustering [31]. (Stability) Depending on the context, the term stability may have different meanings. In empirical studies [4, 5, 22], instability often refers to issues such as oscillations or failure to converge during training. In learning theory, algorithmic stability [17] measures the deviation in an algorithm's outputs for finite-sum problems when a single item in the input dataset is replaced by an i.i.d. in-distribution sample. The concept receives increasing attention as it implies dimension-independent generalization bounds of gradient-based methods for both minimization [42, 11, 6] and minimax [33, 49, 16] problems. In the area of differential equations [13] and variational inequalities [32], stability is also examined as a property of the solution set in response to perturbations in the problem conditions.

In this work, we consider the notion of reproducibility that characterizes the behavior of algorithms upon slight perturbations in the training. We defer the task of establishing intrinsic connections

among related notions to future work. The most closely related concept is algorithmic stability, where the analysis is similar to reproducibility under the inexact deterministic gradient oracle. Attia and Koren [6] showed the stability of AGD [60] grows exponentially with the number of iterations. Later, this is improved to quadratic dependence [7] based on a similar idea as ours that leverages stability of solutions to strongly-convex minimization problems [68, 34]. However, since there is no inexactness of the gradients in their setting, it is possible to ensure outputs that are arbitrarily close to the optimal solution. Given the presence of inexact gradients in our case, the convergence is only limited to a neighborhood of the optimal solution, which makes the problem more challenging. The trade-off between stability and convergence was investigated in Chen et al. [24]. Their results suggest that a faster algorithm has to be less stable, and vice versa. However, we show the feasibility of achieving both optimal reproducibility and near-optimal convergence simultaneously in the setting we considered.

Minimax Optimization. Existing literature on minimax optimization primarily focuses on convergence analysis across various settings. For instance, there are studies on the strongly-convexstrongly-concave case [72, 57], convex-concave case [59, 63], and nonconvex-(strongly)-concave case [52, 71]. The lower complexity bounds have also been established for these settings [80, 50, 77]. Our work aims to design reproducible algorithms while maintaining the optimal oracle complexities achieved in these previous works.

Inexact Gradient Oracles. A series of works investigate the convergence properties of first-order methods under deterministic inexact oracles for minimization [26, 27, 28] and minimax [70] problems. However, their inexact oracles differ from ours, and our focus is more on reproducibility. In recent years, there has been increasing interest in studying biased stochastic gradient oracles as well, where the bias arises from various sources such as problem structure [44], compression [14] or Byzantine failure [15] in distributed learning, and gradient-free optimization [62]. These biases can also contribute to irreproducibility, and this direction would be an interesting avenue of research.

Regularization Technique. The central algorithmic insight driving our improvements towards obtaining both optimal convergence and reproducibility is the regularization technique, which is commonly used in the optimization literature. One important use case is to boost convergence by leveraging known and good convergence properties of algorithms on smooth strongly-convex functions for solving convex and nonsmooth problems, see e.g., [51, 3, 74], just to name a few. In addition, the regularization technique has also been demonstrated to be useful in improving stability and generalization [76, 7], enhancing sensitivity and privacy guarantees [34, 78], etc. In this paper, we provide another important use case by showing an improved convergence-reproducibility trade-off.

2 Preliminaries in Algorithmic Reproducibility

Notation. We use ∥·∥ to represent the Euclidean norm. Π C ( x ) denotes the projection of x onto the set C . A function h : S → R is ℓ -smooth if it is differentiable and its gradient ∇ h satisfies ∥∇ h ( x 1 ) -∇ h ( x 2 ) ∥ ≤ ℓ ∥ x 1 -x 2 ∥ for any x 1 , x 2 in the domain S ∈ R d . A function g : S → R is convex if g ( αx 1 +(1 -α ) x 2 ) ≤ αg ( x 1 ) + (1 -α ) g ( x 2 ) for any α ∈ [0 , 1] and x 1 , x 2 ∈ S . If g satisfies g ( x ) -( µ/ 2) ∥ x ∥ 2 being convex with µ > 0 , then it is µ -strongly-convex. Similarly, a function g : S → R is concave if -g is convex, and µ -strongly-concave if -g is µ -strongly-convex.

Ahn et al. [1] studied the algorithmic reproducibility for convex minimization problems min x ∈X F ( x ) , measured by the ( ϵ, δ ) -deviation bound of an algorithm A . Here, δ denotes the size of errors in the oracles that can lead to different outputs in independent runs of the same algorithm. The notion of reproducibility also requires A to produce ϵ -optimal solutions, avoiding trivial outputs.

Definition 1. Three different inexact oracle models are considered: ( i ) a δ -inexact initialization oracle that returns a starting point x 0 ∈ X such that ∥ x 0 -u 0 ∥ 2 ≤ δ 2 / 4 for some reference point u 0 ∈ X , ( ii ) a δ -inexact deterministic gradient oracle that returns an inexact gradient G ( x ) such that ∥∇ F ( x ) -G ( x ) ∥ 2 ≤ δ 2 for the true gradient ∇ F ( x ) , ( iii ) a δ -inexact stochastic gradient oracle that returns an unbiased gradient estimate ∇ f ( x ; ξ ) such that E ∥∇ f ( x ; ξ ) -∇ F ( x ) ∥ 2 ≤ δ 2 .

Definition 2. Apoint ˆ x ∈ X is an ϵ -optimal solution if F (ˆ x ) -min x ∈X F ( x ) ≤ ϵ in the deterministic setting, or E [ F (ˆ x )] -min x ∈X F ( x ) ≤ ϵ in the stochastic setting, where the expectation is taken over all the randomness in the gradient oracle and in the algorithm that outputs ˆ x .

Definition 3. The ( ϵ, δ ) -deviation ∥ ˆ x -ˆ x ′ ∥ 2 is used to measure the reproducibility of an algorithm A with ϵ -optimal solutions ˆ x and ˆ x ′ , where ˆ x and ˆ x ′ are outputs of two independent runs of the algorithm A given a δ -inexact oracle in Definition 1.

We expand the definitions of reproducibility to encompass minimax optimization problems:

Our goal is to find the saddle point ( x ∗ , y ∗ ) of the function F ( x, y ) , such that F ( x ∗ , y ) ≤ F ( x ∗ , y ∗ ) ≤ F ( x, y ∗ ) holds for all ( x, y ) ∈ X × Y . The optimality of a point (ˆ x, ˆ y ) can be assessed by its duality gap , defined as max y ∈Y F (ˆ x, y ) -min x ∈X F ( x, ˆ y ) . In the minimax setting, we analyze reproducibility under the following inexact oracle models.

Definition 4. Three different inexact oracle models are considered: ( i ) a δ -inexact initialization oracle that returns a starting point ( x 0 , y 0 ) ∈ X × Y such that ∥ x 0 -u 0 ∥ 2 + ∥ y 0 -v 0 ∥ 2 ≤ δ 2 / 4 for some reference point ( u 0 , v 0 ) ∈ X × Y , ( ii ) a δ -inexact deterministic gradient oracle that returns an inexact gradient G ( x, y ) = ( G x ( x, y ) , G y ( x, y )) at any querying point ( x, y ) ∈ X × Y such that ∥∇ F ( x, y ) -G ( x, y ) ∥ 2 ≤ δ 2 for the true gradient ∇ F ( x, y ) = ( ∇ x F ( x, y ) , ∇ y F ( x, y )) , ( iii ) a δ -inexact stochastic gradient oracle that returns an unbiased gradient estimate ∇ f ( x, y ; ξ ) = ( ∇ x f ( x, y ; ξ ) , ∇ y f ( x, y ; ξ )) such that E ξ ∥∇ f ( x, y ; ξ ) -∇ F ( x, y ) ∥ 2 ≤ δ 2 .

Definition 5. A point (ˆ x, ˆ y ) ∈ X × Y is an ϵ -saddle point solution if its duality gap satisfies that max y ∈Y F (ˆ x, y ) -min x ∈X F ( x, ˆ y ) ≤ ϵ in the deterministic setting, or its weak duality gap satisfies that max y ∈Y E [ F (ˆ x, y )] -min x ∈X E [ F ( x, ˆ y )] ≤ ϵ in the stochastic setting.

Definition 6. The ( ϵ, δ ) -deviation ∥ ˆ x -ˆ x ′ ∥ 2 + ∥ ˆ y -ˆ y ′ ∥ 2 is used to measure the reproducibility of an algorithm A with ϵ -saddle points (ˆ x, ˆ y ) and (ˆ x ′ , ˆ y ′ ) , where (ˆ x, ˆ y ) and (ˆ x ′ , ˆ y ′ ) are outputs of two independent runs of the algorithm A given a δ -inexact oracle in Definition 4.

The optimal convergence rates are well-understood for the convex optimization problems, including convex minimization [61] and convex-concave minimax optimization [63]. Ahn et al. [1] provided the theoretical lower-bounds of reproducibility for convex minimization problems, which can be extended to convex-concave minimax problems as well (Lemma B.3). We say an algorithm achieves optimal reproducibility if its reproducibility upper-bounds match the established theoretical lower-bounds.

3 Deterministic Gradient Oracle for Minimization Problems

In this section, we consider convex minimization problems of the form

where X is a convex and closed set. We focus on the standard smooth and convex setting as detailed in Assumption 3.1. Our goal is to find an ϵ -optimal point as in Definition 2. Ahn et al. [1] showed that the optimal convergence rate and reproducibility can be achieved at the same time using stochastic gradient descent (SGD) for the stochastic gradient oracle model. In the deterministic case, they showed GD achieves the optimal reproducibility, albeit with a sub-optimal convergence rate [60, 61]. Considering the instability of accelerated gradient descent (AGD) [26, 28, 6], Ahn et al. [1] conjectured that Ω(1 /ϵ ) gradient complexity is necessary to attain the optimal reproducibility.

Assumption 3.1. The function F is convex and ℓ -smooth. We have access to initial points x 0 that are D -close to an optimal solution, i.e., ∥ x ∗ -x 0 ∥ 2 ≤ D 2 for some x ∗ ∈ arg min x ∈X F ( x ) .

We introduce a generic algorithmic framework outlined in Algorithm 1, that solves a quadratically regularized auxiliary problem ( ⋆ ) using a base algorithm A with initialization x 0 until an accuracy of ϵ r is reached. Our key insight is that since the optimal solution for strongly convex problems is unique, the reproducibility of the outputs from the regularized problem can be easily guaranteed. Note that the regularization parameter r presents a trade-off: as r increases, the auxiliary problem can be solved more efficiently, but the obtained solution deviates further from the original solution. We will show that Algorithm 1 achieves a near-optimal complexity of ˜ O (1 / √ ϵ ) , along with optimal reproducibility under an inexact initialization oracle and slightly sub-optimal reproducibility under an inexact deterministic gradient oracle. This finding disproves the conjecture [1] that Ω(1 /ϵ ) complexity is necessary to achieve optimal reproducibility.

Algorithm 1 Reproducible Algorithmic Framework for Convex Minimization Problems

Input: Regularization parameter r > 0 , accuracy ϵ r > 0 , base algorithm A , initial point x 0 ∈ X . Apply A to approximately solve the r -strongly-convex and ( ℓ + r ) -smooth problem

such that the optimality gap

Output: x r .

3.1 Inexact Initialization Oracle

We first examine the behavior of Algorithm 1 with access to exact deterministic gradients but given different initializations. Starting from two distinct initial points x 0 and x ′ 0 such that ∥ x 0 -x ′ 0 ∥ 2 ≤ δ 2 , we want to control the deviation between the final outputs x r and x ′ r of the algorithm. The following contraction property is essential to attain optimal reproducibility.

Lemma 3.2. Let x ∗ r = arg min x ∈X { F ( x ) + ( r/ 2) ∥ x -x 0 ∥ 2 } and ( x ∗ r ) ′ = arg min x ∈X { F ( x ) + ( r/ 2) ∥ x -x ′ 0 ∥ 2 } . When F is convex, it holds that ∥ x ∗ r -( x ∗ r ) ′ ∥ 2 ≤ ∥ x 0 -x ′ 0 ∥ 2 for any r > 0 .

This indicates the optimal solutions are reproducible up to δ 2 . Consequently, if we can solve the auxiliary problem ( ⋆ ) to a high accuracy ϵ r , we can ensure the final output x r is reproducible. The selection of ϵ r exhibits a trade-off: a smaller value increases complexity, yet brings the output closer to the reproducible x ∗ r . We characterize the complexity and reproducibility of Algorithm 1 by carefully choosing the parameters r and ϵ r .

Theorem 3.3. Under Assumption 3.1 and given an inexact initialization oracle, Algorithm 1 with r = ϵ/D 2 , ϵ r = ( ϵ/ 2) min { 1 , δ 2 / (4 D 2 ) } and AGD [61] as base algorithm A outputs an ϵ -optimal point x r with ˜ O ( √ ℓD 2 /ϵ ) gradient complexity, and the reproducibility is ∥ x r -x ′ r ∥ 2 ≤ 4 δ 2 .

This theorem implies that we can simultaneously achieve the near-optimal complexity of ˜ O ( √ ℓD 2 /ϵ ) and optimal reproducibility of O ( δ 2 ) , which improves over the O ( ℓD 2 /ϵ ) complexity of GD [1]. In fact, when combined with any base algorithm that solves the auxiliary problem, Algorithm 1 attains optimal reproducibility. However, using AGD as the base algorithm results in the best complexity. To the best of our knowledge, this is the only algorithm capable of achieving the best of both worlds. Previously, Attia and Koren [6] proved that the algorithmic reproducibility (referred to as initialization stability in their study) of Nesterov's AGD is Θ( δ 2 e 1 / √ ϵ ) when the initialization is δ 2 -apart.

Remark 1. Adding regularization is a common and useful technique in the optimization literature. Our algorithmic framework solves one auxiliary regularized strongly-convex problem, which is referred to as classical regularization reduction in Allen-Zhu and Hazan [3]. Algorithm 1 is biased and requires the knowledge of ϵ and D to control the biased term introduced by the regularization term. The convergence guarantee also has an additional sub-optimal logarithmic term. Allen-Zhu and Hazan [3] proposed to use a double-loop algorithm, where a sequence of auxiliary regularized strongly-convex problems with decreasing regularization parameters are solved. The vanishing regularization ensures the algorithm is unbiased, and the resulting convergence guarantee requires no knowledge of ϵ and does not have an additional logarithmic term. Similar idea could apply to our case as well, and the task of bridging such gaps is deferred to future work.

3.2 Inexact Deterministic Gradient Oracle

We further study the algorithmic reproducibility and gradient complexity of Algorithm 1 under the inexact gradient oracle model that returns an inexact gradient G ( x ) ∈ R d such that ∥ G ( x ) -∇ F ( x ) ∥ 2 ≤ δ 2 at any query point x ∈ X . From the inexact gradient oracle of F , we can construct an inexact gradient oracle for the auxiliary problem F r : G r ( x ) = G ( x ) + r ( x -x 0 ) which satisfies the condition ∥ G r ( x ) - ∇ F r ( x ) ∥ 2 = ∥ G ( x ) - ∇ F ( x ) ∥ 2 ≤ δ 2 . To solve the auxiliary problem, we consider AGD with an inexact oracle (Inexact-AGD) as proposed by Devolder et al. [27]. The proposition below establishes its convergence behavior.

Proposition 3.4. Consider min x ∈X F r ( x ) , where F r is r -strongly-convex and ( ℓ + r ) -smooth. Given an inexact gradient oracle that returns G r ( x ) such that ∥ G r ( x ) -∇ F r ( x ) ∥ 2 ≤ δ 2 , starting from y 0 = x 0 , AGD with the following update rule

for t = 0 , 1 , · · · , T -1 , satisfies that

This proposition suggests that Inexact-AGD converges to a neighborhood with a radius of O ( δ 2 /r 3 / 2 ) around the optimal value. We note that convergence to the exact solution is unattainable for algorithms employing inexact gradients [27, 28], and the size of this neighborhood is important in determining the reproducibility of x r .

Theorem 3.5. Under Assumption 3.1 with 0 < ϵ ≤ ℓD 2 and given an inexact deterministic gradient oracle in Definition 1, Algorithm 1 with r = ϵ/D 2 , ϵ r = 6 δ 2 D 3 √ ℓ/ (2 ϵ 3 ) and Inexact-AGD as base algorithm outputs a (6 δ 2 D 3 √ ℓ/ (2 ϵ 3 ) + ϵ/ 2) -optimal point x r with ˜ O ( √ ℓD 2 /ϵ ) gradient complexity, and the reproducibility is ∥ x r -x ′ r ∥ 2 ≤ O ( δ 2 /ϵ 5 / 2 ) .

Ahn et al. [1] showed that GD achieves optimal reproducibility of O ( δ 2 /ϵ 2 ) and a complexity of O (1 /ϵ ) when δ ≤ O ( ϵ ) . Our results indicate that a reproducibility of O ( δ 2 /ϵ 5 / 2 ) and a near-optimal complexity of ˜ O (1 / √ ϵ ) can be attained when δ ≤ O ( ϵ 5 / 4 ) . We conjecture that this suboptimal reproducibility bound is inevitable for the proposed framework given the lower bound result in Devolder et al. [27] for algorithms under a ( δ, ℓ, µ ) -inexact oracle associated with ℓ -smooth µ -strongly-convex functions. Further discussions are provided in Appendix A.2. Moreover, we point out that for minimizing ℓ -smooth and µ -strongly-convex functions, Proposition 3.4 already implies that Inexact-AGD attains the optimal reproducibility of O (min { δ 2 , ϵ } ) and the optimal complexity of ˜ O ( √ ℓ/µ ) when the problem is well-conditioned, improving over the ˜ O ( ℓ/µ ) complexity in the previous work [1].

Remark 2. In Appendix D, we demonstrate the effectiveness of Algorithm 1 on a quadratic minimization problem equipped with an inexact gradient oracle. The results are plotted in Figure 1 in the appendix. We observe that the reproducibility can be greatly improved when adding regularization, with only a small degradation in the convergence performance.

4 Deterministic Gradient Oracle for Minimax Problems

In this section, we address the minimax optimization problem of the form

where X and Y are convex compact sets. We focus on the standard smooth and convex-concave setting as detailed in Assumption 4.1. We aim to find an ϵ -saddle point (ˆ x, ˆ y ) such that its duality gap satisfies max y ∈Y F (ˆ x, y ) -min x ∈X F ( x, ˆ y ) ≤ ϵ . Here, the assumption that the domains are convex and bounded ensures the existence of the saddle point when the objective is convex-concave [73]. We focus on minimax problems equipped with inexact initialization oracles and inexact deterministic gradient oracles as defined in Definition 4. We first show that two classical algorithms, gradient descent ascent (GDA) and Extragradient (EG) [48, 72], are either sub-optimal in convergence or suboptimal in reproducibility, which mirrors the minimization setting. Based on the same regularization idea, we propose two new frameworks in Algorithm 2 and 3 that successfully attain near-optimal convergence and optimal reproducibility at the same time.

Assumption 4.1. For all y ∈ Y , F ( · , y ) is convex, and for all x ∈ X , F ( x, · ) is concave. Furthermore, F is ℓ -smooth on the domain X × Y . Additionally, both X and Y have a diameter of D . This means that ∥ x 1 -x 2 ∥ 2 ≤ D 2 and ∥ y 1 -y 2 ∥ 2 ≤ D 2 for all x 1 , x 2 ∈ X and y 1 , y 2 ∈ Y .

The optimal gradient complexity to find ϵ -saddle point under such assumptions is Θ(1 /ϵ ) [63]. Since the minimax problem reduces to a minimization problem on X when the domain Y is restricted to be a singleton, the reproducibility lower-bounds [1] for smooth convex minimization hold as lower-bounds for smooth convex-concave minimax optimization as well. That is, Ω( δ 2 ) under the inexact initialization oracle, and Ω( δ 2 /ϵ 2 ) under the inexact gradient oracle (see Lemma B.3). We now present the convergence rate and reproducibility bounds of GDA (see Algorithm 4) and EG (see Algorithm 5).

Theorem 4.2. (GDA) Under Assumption 4.1, the average iterate (¯ x T , ¯ y T ) output by GDA with stepsize 1 / ( ℓ √ T ) after T = O (1 /ϵ 2 ) iterations is an ϵ -saddle point . Furthermore, the reproducibility of the output is ( i ) O ( δ 2 ) under δ -inexact initialization oracle ; ( ii ) O ( δ 2 /ϵ 2 ) under δ -inexact deterministic gradient oracle if δ ≤ O ( ϵ ) .

Theorem 4.3. (Extragradient) Under Assumption 4.1, the average iterate (¯ x T +1 / 2 , ¯ y T +1 / 2 ) output by EG with stepsize 1 /ℓ after T = O (1 /ϵ ) iterations is an ϵ -saddle point. Furthermore, the reproducibility of this output is ( i ) O (min { δ 2 e 1 / ϵ , δ 2 + 1 /ϵ 2 , D 2 } ) under δ -inexact initialization oracle ; ( ii ) O (min { δ 2 e 1 / ϵ , 1 /ϵ 2 , D 2 } ) under δ -inexact deterministic gradient oracle if δ ≤ O ( ϵ ) .

While GDA can achieve optimal reproducibility, it converges with a sub-optimal complexity of O (1 /ϵ 2 ) . On the other hand, EG achieves an optimal O (1 /ϵ ) complexity but is not optimally reproducible. Further details on this are provided in Appendix B. In Appendix B.3.4, we also demonstrate that EG, through an alternative parameter selection, can achieve optimal reproducibility at a sub-optimal rate O (1 /ϵ 3 / 2 ) . The question that remains open is how to simultaneously attain both optimal reproducibility and gradient complexity. To address this, we have developed two algorithmic frameworks with near-optimal guarantees, one based on regularization and the other based on proximal point methods [66, 12].

4.1 Regularization Helps!

Algorithm 2 Reproducible Algorithmic Framework for Convex-Concave Minimax Problems

Input: Regularization parameter r > 0 , accuracy ϵ r > 0 , base algorithm A , initialization ( x 0 , y 0 ) . Apply A to inexactly solve the r -strongly-convex-strongly-concave and ( ℓ + r ) -smooth problem

such that ∀ ( x, y ) ∈ X × Y ,

Output: ( x r , y r ) .

We demonstrate that adding regularization is sufficient to achieve near-optimal guarantees for smooth convex-concave minimax problems. The general framework is summarized in Algorithm 2, where a base algorithm A is applied to solve a regularized auxiliary problem which is strongly-convex in x and strongly-concave in y . For the inexact initialization case, we show that an optimal reproducibility bound of O ( δ 2 ) and a near-optimal convergence rate of ˜ O (1 /ϵ ) can be attained simultaneously.

Theorem 4.4. Under Assumption 4.1 and given an inexact initialization oracle, Algorithm 2 with r = ϵ/D 2 , ϵ r = ϵ · min { 1 , δ 2 / (8 D 2 ) } and EG as base algorithm A outputs a (2 ϵ ) -saddle point ( x r , y r ) with ˜ O ( ℓD 2 /ϵ ) gradient complexity, and the reproducibility is 4 δ 2 .

Consider a δ -inexact deterministic gradient oracle that returns G ( x, y ) = ( G x ( x, y ) , G y ( x, y )) . First note G r ( x, y ) = ( G x ( x, y )+ r ( x -x 0 ) , G y ( x, y ) -r ( y -y 0 )) is a δ -inexact gradient for the auxiliary problem ( ∗ ). We now characterize the convergence behavior of EG with this δ -inexact gradient oracle, referred to as Inexact-EG, to solve the auxiliary problem.

Lemma 4.5. Consider min x ∈X max y ∈Y F r ( x, y ) , where F r ( x, y ) is r -strongly-convex-stronglyconcave and ( ℓ + r ) -smooth. Given an inexact gradient oracle that returns G r ( x, y ) such that ∥ G r ( x, y ) -∇ F r ( x, y ) ∥ 2 ≤ δ 2 , Inexact-EG with stepsize 1 / (2( ℓ + r )) satisfies

where ( x ∗ r , y ∗ r ) is the unique saddle point of F r ( x, y ) .

This lemma implies that Inexact-EG converges linearly to a neighborhood of size O ( δ 2 /r 2 ) around the saddle point, which can be translated to the inaccuracy measure in (2) with ϵ r = O ( δ/r ) utilizing Lemma C.5. It is worth emphasizing that the size of this neighborhood is critical for achieving optimal reproducibility, and the dependency on r in the above convergence rate is key for attaining near-optimal complexity. Stonyakin et al. [70] analyzed Mirror-Prox [59] with restarts for stronglymonotone variational inequalities under a different inexact oracle (see Devolder et al. [27] and [70, Example 6.1] for its relationship with the inexactness notion of ours). Compared to Inexact-EG, their two-loop structure of the restart scheme is more complicated to implement.

Theorem 4.6. Under Assumption 4.1 with 0 < ϵ ≤ ℓD 2 and given an inexact gradient oracle, Algorithm 2 with r = ϵ/D 2 , ϵ r = O ( δ/r ) and Inexact-EG as base algorithm A outputs an O ( ϵ + δ/ϵ ) -saddle point with ˜ O ( ℓD 2 /ϵ ) gradient complexity, and the reproducibility is O ( δ 2 /ϵ 2 ) .

Remark 3. Some numerical experiments on a bilinear matrix game with inexact gradient information are provided in Appendix D (see Figure 2). With a small degradation in the convergence speed, the regularized framework in Algorithm 2 effectively improves the reproducibility of the base algorithm.

The theorem indicates that optimal reproducibility O ( δ 2 /ϵ 2 ) and near-optimal gradient complexity ˜ O (1 /ϵ ) can be achieved when δ ≤ O ( ϵ 2 ) . Note by Theorem 4.2 and 4.3, GDA and EG can find ϵ -saddle points when δ ≤ O ( ϵ ) . Next, we introduce an alternative algorithmic framework that preserves the optimal reproducibility and attains the near-optimal complexity as long as δ ≤ O ( ϵ ) .

4.2 Inexact Proximal Point Method

We propose a two-loop inexact proximal point framework, presented in Algorithm 3, which can achieve both near-optimal gradient complexity and optimal algorithmic reproducibility. Compared to Algorithm 2, the regularization parameter 1 /α = O ( ℓ ) does not depend on the target accuracy ϵ and the diameter D , and the center of the regularization term is the last iterate ( x t , y t ) instead of the initial point. Since the auxiliary problem is ℓ -strongly-convex-strongly-concave and 2 ℓ -smooth with condition number being Θ(1) , a wider range of base algorithms can be used to achieve the optimal complexity than solving the problem in Algorithm 2 where the condition number is Θ(1 /ϵ ) .

Algorithm 3 Inexact Proximal Point Method for Convex-Concave Minimax Problems

Input: Stepsize α > 0 , accuracy ˆ ϵ > 0 , algorithm A , initialization ( x 0 , y 0 ) , iteration number T .

Apply A to inexactly solve the smooth strongly-convex-strongly-concave problem

such that ∀ ( x, y ) ∈ X × Y ,

Theorem 4.7. Under Assumption 4.1and given a δ -inexact initialization oracle in Definition 4 with δ ≤ O (1 / √ ϵ ) , Algorithm 3 with ˆ ϵ ≤ δ 2 / (2 αT 2 ) and α = 1 /ℓ outputs an O ( ϵ ) -saddle point after T = O (1 /ϵ ) iterations, and the reproducibility is 9 δ 2 .

Remark 4. The required accuracy ˆ ϵ for the auxiliary problem is O ( δ 2 ϵ 2 ) . Given that the auxiliary problem is ℓ -strongly-convex-strongly-concave and 2 ℓ -smooth, various linearly convergent algorithms such as EG, GDA, and Optimistic GDA [35] can find a point that satisfies the stopping criterion within O (log(1 / ( δϵ ))) iterations. As a result, the total gradient complexity is ˜ O (1 /ϵ ) . In contrast, using GDA as the base algorithm in Algorithm 2 will lead to a sub-optimal gradient complexity.

Theorem 4.8. Under Assumption 4.1 and given a δ -inexact deterministic gradient oracle in Definition 4 with δ ≤ O ( ϵ ) , Algorithm 3 with ˆ ϵ ≤ O ( δ ) and α = 1 /ℓ outputs an O ( ϵ ) -saddle point after T = O (1 /ϵ ) iterations, and the reproducibility is O ( δ 2 /ϵ 2 ) .

Remark 5. This theorem requires solving the auxiliary problem with a δ -inexact gradient oracle. In addition to Inexact-EG presented in Lemma 4.5, we show in Appendix C.1 that GDA with inexact gradients (Inexact-GDA) can also converge linearly to the optimal point up to a O ( δ 2 ) error. Thus the total complexity is O ((1 /ϵ ) log(1 /δ )) using both Inexact-EG and Inexact-GDA.

5 Stochastic Gradient Oracle for Minimax Problems

To provide a complete picture, in this section, we consider the stochastic minimax problem:

where the expectation is taken over a random vector ξ . We have access to a δ -inexact stochastic gradient oracle that can return unbiased gradients ∇ f ( x, y ; ξ ) with a bounded variance δ 2 at each point ( x, y ) . We consider the popular algorithm called stochastic gradient descent ascent (SGDA). The convergence behaviors of SGDA for the stochastic minimax problem (3) are well-known in various settings. However, due to the randomness in the gradient oracle, independent runs of SGDA may lead to different outputs even with the same parameters. Following Definition 6, we further establish the ( ϵ, δ ) -deviation of SGDA in the theorem below.

Theorem 5.1. Under Assumptions 4.1 and given an inexact stochastic gradient oracle in Definition 4 with δ = O (1) , the average iterates (¯ x T , ¯ y T ) = (1 /T ) ∑ T -1 t =0 ( x t , y t ) of SGDA with stepsize 1 / ( ℓϵT ) after T = Ω(1 /ϵ 2 ) iterations is an O ( ϵ ) -stationary point and the reproducibility is O ( δ 2 / ( ϵ 2 T ) ) .

The O (1 /ϵ 2 ) sample complexity of SGDA is known to be optimal when the objective F ( x, y ) is convex-concave [46]. Moreover, our results suggest that SGDA is also optimally reproducible, as the lower-bound of Ω ( δ 2 / ( ϵ 2 T ) ) for convex minimization problems [1] is also valid for minimax optimization according to our discussions in Lemma B.3.

6 Conclusion

In this work, instead of solely focusing on convergence performance, we investigate another crucial property of machine learning algorithms, i.e., algorithms should be reproducible against slight perturbations. We provide the first algorithms to simultaneously achieve optimal algorithmic reproducibility and near-optimal gradient complexity for both smooth convex minimization and smooth convexconcave minimax problems under various inexact oracle models. We focus on the convex case as a first step since it is the most basic and fundamental setting in optimization. We believe a solid understanding of the reproducibility in convex optimization will shed insights for that of the more challenging nonconvex optimization. Note that some of the analysis and techniques used in this paper can be extended to the smooth nonconvex setting, aligning with the stability analysis for nonconvex objectives [42, 49]. The proposed regularized framework can be applied to nonconvex functions as well using the convergence analysis of regularization or proximal point-based methods [2, 74]. However, the non-expansiveness property in Lemma 3.2 that is essential for the reproducibility analysis will not hold any more without the convexity assumption. One potential way to alleviate it is to impose additional structural assumptions on the gradients such as negative comonotonicity [39]. We leave a detailed study of the reproducibility in nonconvex optimization to future work.

Other possible improvements of our results include deriving optimal reproducibility with an accelerated convergence rate for smooth convex minimization problems under the inexact gradient oracle, removing the additional logarithmic terms in the complexity of our algorithms using techniques in Allen-Zhu and Hazan [3], studying the reproducibility under the presence of mixed inexact oracles, and extending the results to nonsmooth settings. Another interesting direction is to design simpler and more direct methods with both optimal reproducibility and convergence guarantees. A possible way is to directly unwrap the regularized algorithmic framework 1 or 2, leading to Tikhonov regularization [8] or anchoring methods [75].

Acknowledgments and Disclosure of Funding

We thank the anonymous reviewers for their valuable suggestions. Liang Zhang gratefully acknowledges funding from the Max Planck ETH Center for Learning Systems (CLS). Amin Karbasi

acknowledges funding in direct support of this work from NSF (IIS-1845032), ONR (N0001419-1-2406), and the AI Institute for Learning-Enabled Optimization at Scale (TILOS). Niao He is supported by ETH research grant, NCCR Automation, and Swiss National Science Foundation Project Funding No. 200021-207343.

References

Kwangjun Ahn, Prateek Jain, Ziwei Ji, Satyen Kale, Praneeth Netrapalli, and Gil I Shamir. Reproducibility in optimization: Theoretical framework and limits. Advances in Neural Information Processing Systems , 35, 2022.

Zeyuan Allen-Zhu. How to make the gradients small stochastically: Even faster convex and nonconvex sgd. Advances in Neural Information Processing Systems , 31, 2018.

Zeyuan Allen-Zhu and Elad Hazan. Optimal black-box reductions between optimization objectives. Advances in Neural Information Processing Systems , 29, 2016.

Martin Arjovsky and Leon Bottou. Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations , 2017.

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning , pages 214-223. PMLR, 2017.

Amit Attia and Tomer Koren. Algorithmic instabilities of accelerated gradient descent. Advances in Neural Information Processing Systems , 34:1204-1214, 2021.

Amit Attia and Tomer Koren. Uniform stability for first-order empirical risk minimization. In Conference on Learning Theory , pages 3313-3332. PMLR, 2022.

Hedy Attouch, Zaki Chbani, and Hassan Riahi. Combining fast inertial dynamics for convex optimization with Tikhonov regularization. Journal of Mathematical Analysis and Applications , 457(2):1065-1094, 2018.

James P Bailey and Georgios Piliouras. Multiplicative weights update in zero-sum games. In Proceedings of the ACM Conference on Economics and Computation , pages 321-338, 2018.

Monya Baker. 1,500 scientists lift the lid on reproducibility. Nature , 533:452-454, 2016.

Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems , 33:4381-4391, 2020.

Heinz H Bauschke, Patrick L Combettes, et al. Convex analysis and monotone operator theory in Hilbert spaces , volume 408. Springer, 2011.

Richard Bellman. Stability theory of differential equations . Courier Corporation, 2008.

Aleksandr Beznosikov, Samuel Horváth, Peter Richtárik, and Mher Safaryan. On biased compression for distributed learning. arXiv preprint arXiv:2002.12410 , 2020.

Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in Neural Information Processing Systems , 30, 2017.

Digvijay Boob and Cristóbal Guzmán. Optimal algorithms for differentially private stochastic monotone variational inequalities and saddle-point problems. Mathematical Programming , pages 1-43, 2023.

Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research , 2:499-526, 2002.

Xavier Bouthillier, César Laurent, and Pascal Vincent. Unreproducible research is reproducible. In International Conference on Machine Learning , pages 725-734. PMLR, 2019.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations , 2019.

Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning , 8(3-4):231-357, 2015.

Mark Bun, Marco Gaboardi, Max Hopkins, Russell Impagliazzo, Rex Lei, Toniann Pitassi, Satchit Sivakumar, and Jessica Sorrell. Stability is stable: Connections between replicability, privacy, and adaptive generalization. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing , pages 520-527, 2023.

Tatjana Chavdarova, Matteo Pagliardini, Sebastian U Stich, François Fleuret, and Martin Jaggi. Taming GANs with lookahead-minmax. In International Conference on Learning Representations , 2021.

Tong Che, Yanran Li, Athul Jacob, Yoshua Bengio, and Wenjie Li. Mode regularized generative adversarial networks. In International Conference on Learning Representations , 2017.

Yuansi Chen, Chi Jin, and Bin Yu. Stability and convergence trade-off of iterative optimization algorithms. arXiv preprint arXiv:1804.01619 , 2018.

Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. SBEED: Convergent reinforcement learning with nonlinear function approximation. In International Conference on Machine Learning , pages 1125-1134. PMLR, 2018.

Alexandre d'Aspremont. Smooth optimization with approximate gradient. SIAM Journal on Optimization , 19(3):1171-1183, 2008.

Olivier Devolder, François Glineur, Yurii Nesterov, et al. First-order methods with inexact oracle: the strongly-convex case. CORE Discussion Papers , 2013016:47, 2013.

Olivier Devolder, François Glineur, and Yurii Nesterov. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming , 146:37-75, 2014.

Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407, 2014.

Hossein Esfandiari, Alkis Kalavasis, Amin Karbasi, Andreas Krause, Vahab Mirrokni, and Grigoris Velegkas. Replicable bandits. In International Conference on Learning Representations , 2023.

Hossein Esfandiari, Amin Karbasi, Vahab Mirrokni, Grigoris Velegkas, and Felix Zhou. Replicable clustering. arXiv preprint arXiv:2302.10359 , 2023.

Francisco Facchinei and Jong-Shi Pang. Finite-dimensional variational inequalities and complementarity problems . Springer, 2003.

Farzan Farnia and Asuman Ozdaglar. Train simultaneously, generalize better: Stability of gradient-based minimax learners. In International Conference on Machine Learning , pages 3174-3185. PMLR, 2021.

Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: optimal rates in linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , pages 439-449, 2020.

Gauthier Gidel, Hugo Berard, Gaëtan Vignoud, Pascal Vincent, and Simon Lacoste-Julien. A variational inequality perspective on generative adversarial networks. In International Conference on Learning Representations , 2019.

Gauthier Gidel, Reyhane Askari Hemmat, Mohammad Pezeshki, Rémi Le Priol, Gabriel Huang, Simon Lacoste-Julien, and Ioannis Mitliagkas. Negative momentum for improved game dynamics. In International Conference on Artificial Intelligence and Statistics , pages 1802-1811. PMLR, 2019.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems , 27, 2014.

Steven N Goodman, Daniele Fanelli, and John PA Ioannidis. What does research reproducibility mean? Science Translational Medicine , 8(341):341ps12-341ps12, 2016.

Eduard Gorbunov, Adrien Taylor, Samuel Horváth, and Gauthier Gidel. Convergence of proximal point and extragradient-based methods beyond monotonicity: the case of negative comonotonicity. In International Conference on Machine Learning , pages 11614-11641. PMLR, 2023.

Odd Erik Gundersen and Sigbjørn Kjensmo. State of the art: Reproducibility in artificial intelligence. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Benjamin Haibe-Kains, George Alexandru Adam, Ahmed Hosny, Farnoosh Khodakarami, Levi Waldron, Bo Wang, Chris McIntosh, Anna Goldenberg, Anshul Kundaje, Casey S Greene, et al. Transparency and reproducibility in artificial intelligence. Nature , 586(7829):14-16, 2020.

Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning , pages 12251234. PMLR, 2016.

Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Yifan Hu, Siqi Zhang, Xin Chen, and Niao He. Biased stochastic first-order methods for conditional stochastic optimization and applications in meta learning. Advances in Neural Information Processing Systems , 33:2759-2770, 2020.

Russell Impagliazzo, Rex Lei, Toniann Pitassi, and Jessica Sorrell. Reproducibility in learning. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 818-831, 2022.

Anatoli Juditsky, Arkadi Nemirovski, and Claire Tauvel. Solving variational inequalities with stochastic mirror-prox algorithm. Stochastic Systems , 1(1):17-58, 2011.

Alkis Kalavasis, Amin Karbasi, Shay Moran, and Grigoris Velegkas. Statistical indistinguishability of learning algorithms. In International Conference on Machine Learning , volume 202, pages 15586-15622. PMLR, 2023.

Galina M Korpelevich. The extragradient method for finding saddle points and other problems. Matecon , 12:747-756, 1976.

Yunwen Lei, Zhenhuan Yang, Tianbao Yang, and Yiming Ying. Stability and generalization of stochastic gradient methods for minimax problems. In International Conference on Machine Learning , pages 6175-6186. PMLR, 2021.

Haochuan Li, Yi Tian, Jingzhao Zhang, and Ali Jadbabaie. Complexity lower bounds for nonconvex-strongly-concave min-max optimization. Advances in Neural Information Processing Systems , 34:1792-1804, 2021.

Hongzhou Lin, Julien Mairal, and Zaid Harchaoui. A universal catalyst for first-order optimization. Advances in Neural Information Processing Systems , 28, 2015.

Tianyi Lin, Chi Jin, and Michael Jordan. On gradient descent ascent for nonconvex-concave minimax problems. In International Conference on Machine Learning , pages 6083-6093. PMLR, 2020.

Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are GANs created equal? A large-scale study. Advances in Neural Information Processing Systems , 31, 2018.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations , 2018.

Panayotis Mertikopoulos, Christos Papadimitriou, and Georgios Piliouras. Cycles in adversarial regularized learning. In Proceedings of the 29th Annual ACM-SIAM Symposium on Discrete Algorithms , pages 2703-2717. SIAM, 2018.

Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of GANs. Advances in Neural Information Processing Systems , 30, 2017.

Aryan Mokhtari, Asuman Ozdaglar, and Sarath Pattathil. A unified analysis of extra-gradient and optimistic gradient methods for saddle point problems: Proximal point approach. In International Conference on Artificial Intelligence and Statistics , pages 1497-1507. PMLR, 2020.

Aryan Mokhtari, Asuman E Ozdaglar, and Sarath Pattathil. Convergence rate of O (1 /k ) for optimistic gradient and extragradient methods in smooth convex-concave saddle point problems. SIAM Journal on Optimization , 30(4):3230-3251, 2020.

Arkadi Nemirovski. Prox-method with rate of convergence O (1 /t ) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization , 15(1):229-251, 2004.

Yurii Nesterov. A method for solving a convex programming problem with convergence rate O (1 /k 2 ) . In Soviet Mathematics. Doklady , volume 27, pages 367-372, 1983.

Yurii Nesterov. Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media, 2003.

Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics , 17:527-566, 2017.

Yuyuan Ouyang and Yangyang Xu. Lower complexity bounds of first-order methods for convex-concave bilinear saddle-point problems. Mathematical Programming , 185(1-2):1-35, 2021.

Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d'Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research (A report from the NeurIPS 2019 reproducibility program). The Journal of Machine Learning Research , 22(1):7459-7478, 2021.

R Tyrrell Rockafellar. Monotone operators associated with saddle-functions and minimax problems. Nonlinear Functional Analysis , 18(part 1):397-407, 1970.

R Tyrrell Rockafellar. Monotone operators and the proximal point algorithm. SIAM Journal on Control and Optimization , 14(5):877-898, 1976.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. Advances in Neural Information Processing Systems , 29, 2016.

Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Stochastic convex optimization. In Conference on Learning Theory , volume 2, page 5, 2009.

Gil I Shamir and Lorenzo Coviello. Anti-distillation: Improving reproducibility of deep networks. arXiv preprint arXiv:2010.09923 , 2020.

Fedor Stonyakin, Alexander Gasnikov, Pavel Dvurechensky, Alexander Titov, and Mohammad Alkousa. Generalized mirror prox algorithm for monotone variational inequalities: Universality and inexact oracle. Journal of Optimization Theory and Applications , 194(3):988-1013, 2022.

Kiran K Thekumparampil, Prateek Jain, Praneeth Netrapalli, and Sewoong Oh. Efficient algorithms for smooth minimax optimization. Advances in Neural Information Processing Systems , 32, 2019.

Paul Tseng. On linear convergence of iterative methods for the variational inequality problem. Journal of Computational and Applied Mathematics , 60(1-2):237-252, 1995.

John Von Neumann. On the theory of games of strategy. Contributions to the Theory of Games , 4:13-42, 1959.

Junchi Yang, Siqi Zhang, Negar Kiyavash, and Niao He. A catalyst framework for minimax optimization. Advances in Neural Information Processing Systems , 33:5667-5678, 2020.

TaeHo Yoon and Ernest K Ryu. Accelerated algorithms for smooth convex-concave minimax problems with O (1 /k 2 ) rate on squared gradient norm. In International Conference on Machine Learning , pages 12098-12109. PMLR, 2021.

Junyu Zhang, Mingyi Hong, Mengdi Wang, and Shuzhong Zhang. Generalization bounds for stochastic saddle point problems. In International Conference on Artificial Intelligence and Statistics , pages 568-576. PMLR, 2021.

Junyu Zhang, Mingyi Hong, and Shuzhong Zhang. On lower iteration complexity bounds for the convex concave saddle point problems. Mathematical Programming , 194(1-2):901-935, 2022.

Liang Zhang, Kiran K Thekumparampil, Sewoong Oh, and Niao He. Bring your own algorithm for optimal differentially private stochastic minimax optimization. Advances in Neural Information Processing Systems , 35:35174-35187, 2022.

Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton. Lookahead optimizer: k steps forward, 1 step back. Advances in Neural Information Processing Systems , 32, 2019.

Siqi Zhang, Junchi Yang, Cristóbal Guzmán, Negar Kiyavash, and Niao He. The complexity of nonconvex-strongly-concave minimax optimization. In Uncertainty in Artificial Intelligence , pages 482-492. PMLR, 2021.

Contents

D Numerical Experiments

A Near-optimal Guarantees in the Minimization Case

This section provides proof for the near-optimal guarantees of Algorithm 1 in the minimization case. We start with some commonly-used facts that follow from basic algebraic calculations. See Bauschke et al. [12] for an example.

Lemma A.1. The following facts will be used in the analysis. For any vectors a, b ∈ R d , it holds that

A.1 Inexact Initialization Oracle

This section contains proof of Lemma 3.2 and Theorem 3.3 for the near-optimal guarantees of Algorithm 1 in the inexact initialization case.

Proof of Lemma 3.2. By the optimality conditions of x ∗ r and ( x ∗ r ) ′ , we have that for any x, x ′ ∈ X ,

Taking x ′ = x ∗ r and x = ( x ∗ r ) ′ in the above equation, we obtain that

Since ∇ F is monotone when F is convex, rearranging terms, we get

Given r > 0 , this means

38

≤ ∥

x

∗

r

-

(

x

∗

r

)

∥∥

Dividing both sides by ∥ x ∗ r -( x ∗ r ) ′ ∥ , the proof is complete.

By converging sufficiently close to the optimal solution, we can ensure Algorithm 1 is reproducible. The near-optimal convergence rate is achieved using AGD [60] as the base algorithm.

Proof of Theorem 3.3. We first analyze the convergence guarantee. Let x ∗ ∈ arg min x ∈X F ( x ) be one minimizer of F ( x ) , and x ∗ r = arg min x ∈X F r ( x ) be the unique minimizer of F r ( x ) . By the definition of F r ( x ) , we have that

′

0

′

x

0

-

x

∥

.

ϵ r and r will be selected later. For reproducibility, we proceed as

where we use the optimality condition of x ∗ r by r -strong-convexity of F r ( x ) :

the optimality condition of ( x ∗ r ) ′ and Lemma 3.2. Setting

we guarantee that F ( x r ) -F ( x ∗ ) ≤ ϵ and ∥ x r -x ′ r ∥ ≤ 2 δ . The gradient complexity of AGD to achieve ϵ r approximation error on the function value gap of an ℓ -smooth and ( ℓ + r ) -strongly convex function is O ( √ ( ℓ + r ) /r log(1 /ϵ r )) = ˜ O ( √ ℓD 2 /ϵ ) , where ˜ O hides logarithmic terms.

A.2 Inexact Deterministic Gradient Oracle

This section contains proof of Lemma 3.4 and Theorem 3.5 for the guarantees in the inexact deterministic gradient case. We first study the convergence behavior of AGD [60] for smooth and strongly-convex functions under the inexact gradient oracle. For the sake of simplicity and to enable a general analysis, we slightly abuse notation here to consider the optimization problem

where f : X → R satisfies the following assumption.

Assumption A.2. f ( x ) is ℓ -smooth and µ -strongly convex on the closed convex domain X .

We consider the inexact gradient oracle defined below (referred to as δ -oracle in this section).

Definition 7. ( δ -oracle) At any querying point x ∈ X , the δ -oracle returns a vector g ( x ) ∈ R d such that ∥ g ( x ) -∇ f ( x ) ∥ 2 ≤ δ 2 , where ∇ f ( x ) is the true gradient of f ( x ) .

In previous work, Devolder et al. [27] define a different inexact oracle that is motivated by the exact first-order oracle and study the convergence behavior of first-order algorithms including AGD.

Definition 8. ( ( δ, ℓ, µ ) -oracle [27]) At any querying point x ∈ X , the ( δ, ℓ, µ ) -oracle returns approximate first-order information ( f δ,ℓ,µ ( x ) , g δ,ℓ,µ ( x )) such that for any y ∈ X ,

The lemma below characterizes that the two oracles can be transformed into each other (adapted from Devolder et al. [27, 28]).

Lemma A.3. Under Assumption A.2. A δ -oracle can be transformed to a ( δ ′ , ℓ ′ , µ ′ ) -oracle with δ ′ = (1 / (2 ℓ ) + 1 /µ ) δ 2 , ℓ ′ = 2 ℓ , and µ ′ = µ/ 2 . A ( δ, ℓ, µ ) -oracle can be transformed to a δ ′ -oracle for δ ′ defined in (7) .

Proof. Given a δ -oracle that returns g ( x ) at any point x ∈ X , we construct a ( δ ′ , ℓ ′ , µ ′ ) -oracle as

By ℓ -smoothness of f ( x ) and fact ( iii ) in Lemma A.1, we have that

Similarly by µ -strong convexity of f ( x ) and fact ( iii ) in Lemma A.1, we have that

Combined the above two equations together, we obtain that

This concludes the proof of the first part. For the second part, given a ( δ, ℓ, µ ) -oracle in Definition 8, we construct a δ ′ -oracle as follows: g ( x ) = g δ,ℓ,µ ( x ) . Taking y = x in Definition 8, we obtain ∀ x ,

Therefore, by strong-convexity of f ( x ) , we have that ∀ x, y ,

Combined with the second part of Definition 8, we obtain that ∀ x, y ,

Then by a similar proof as for the convex case in Devolder et al. [28]. Let ∆( x ) = ∇ f ( x ) -g δ,ℓ,µ ( x ) and y = x + min { √ 2 δ/ ( ℓ -µ ) , r ( x ) } ∆( x ) / ∥ ∆( x ) ∥ for r ( x ) = max { r ∈ R | ( x + r ∆( x ) / ∥ ∆( x ) ∥ ) ∈ X} . We have that

Since we use g ( x ) = g δ,ℓ,µ ( x ) , the proof is complete.

Devolder et al. [27] prove that AGD equipped with ( δ, ℓ, µ ) -oracle in Definition 8 converges to a O ( δ √ ℓ/µ ) -neighborhood of the optimal solution with accelerated rate T = ˜ O ( √ ℓ/µ ) :

where x T is the output of T -step AGD and f ∗ is the optimal value. They further establish a lowerbound showing tightness of the O ( δ √ ℓ/µ ) error for any first-order methods with accelerated rate.

Here, we are interested in the performance of AGD under the δ -oracle in Definition 7. Motivated by the transformation in Lemma A.3, we choose the parameters in AGD as follows:

The results can be implied by Devolder et al. [27] together with Lemma A.3. We provide detailed proof in the following for completeness of the paper.

Lemma A.4. Under Assumption A.2. Let x ∗ be the unique minimizer of f ( x ) and κ = ℓ/µ be the condition number. Given an inexact δ -oracle in Definition 7. Starting from y 0 = x 0 , AGD with updates (8) for t = 0 , 1 , · · · , T -1 converges with

Proof. By (5) in the proof of Lemma A.3, we have that

Similarly by (6), we know for any x ∈ X ,

Combing the above two results, for any x ∈ X , we have

where in the last inequality we use the optimality condition of the projection step such that ∀ x ∈ X ,

Let θ := 1 / (2 √ κ ) = √ µ/ (4 ℓ ) . Setting x = x t and x = x ∗ in the above equation, we get

Let ∆ t := f ( x t ) -f ( x ∗ ) ≥ 0 . Summing the above two up, by fact ( i ) in Lemma A.1, we obtain

Let θu t := x t -(1 -θ ) x t -1 for t ≥ 1 . From the update (8) of AGD, we observe

Rearranging terms, we can get x t = (1 + θ ) y t -θu t and thus

It is easy to verify that the above also holds when u 0 = x 0 = y 0 . Since ℓθ 2 = µ/ 4 , we have that

where we use fact ( iv ) in Lemma A.1. Rearranging terms, we then obtain

Unrolling the recursion, we have that

where we use the fact that 1 + η ≤ e , ∀ η ∈ .

Lemma 3.4 immediately follows from Lemma A.4. With the above results at hand, we are ready to show proof of Theorem 3.5 below.

Proof of Theorem 3.5. For the convergence guarantee, similarly to the perturbed initialization case in (4), for x ∗ ∈ arg min x ∈X F ( x ) and x ∗ r = arg min x ∈X F r ( x ) , we have that

For the reproducibility guarantee, using r -strong-convexity of F r ( x ) , we can obtain that

Applying Lemma 3.4, if (Inexact-AGD) is used as the base algorithm A and x r is the output given initialization y 0 = x 0 after T iterations, since r = ϵ/D 2 ≤ ℓ , we know that

When setting T = O ( √ ℓ/r log( r 3 / 2 /δ 2 )) , this means the algorithm converges to F r ( x r ) -F r ( x ∗ r ) ≤ 6 δ 2 √ ℓ/ (2 r 3 ) and ∥ x r -x ∗ r ∥ 2 ≤ 12 δ 2 √ ℓ/ (2 r 5 ) . Therefore, since r = ϵ/D 2 , we have that

and the reproducibility is ∥ x -x ′ ∥ 2 ≤ O ( δ 2 /ϵ 5 / 2 ) .

The results suggest that to achieve ϵ -approximation error on the function value gap, we need to set δ ≤ O ( ϵ 5 / 4 ) , which is a smaller regime compared to δ ≤ O ( ϵ ) in the previous work [1] when ϵ ≤ 1 . Furthermore, optimal reproducibility O ( δ 2 /ϵ 2 ) is not attained. We observe from the proof that the additional O ( √ κ ) -factor in the last term of the error bound in Lemma A.4 leads to this degradation. Since we set r = O ( ϵ ) to balance the convergence rate and approximation error introduced through regularization, this factor can be O ( √ 1 /ϵ ) . Based on the lower-bound in Devolder et al. [27] for ( δ, ℓ, µ ) -oracle such that this O ( √ κ ) -factor is unavoidable for an accelerated convergence rate and the transformation between the two inexact oracles in Lemma A.3, we thus make the conjecture here that the above results cannot be further improved. Algorithms that achieve optimal convergence and reproducibility under this setting require better designs and we leave it for future work.

B Preliminary Results in the Minimax Case

In this section, we provide proof of some preliminary results in the minimax setting. We start with a proof of the lower-bounds in Lemma B.3. Sub-optimal guarantees of gradient descent ascent (GDA) in the deterministic case, as well as optimal guarantees of stochastic gradient descent ascent (SGDA), are provided in Section B.2. Sub-optimal results of Extragradient (EG) are proved in Section B.3.

Before that, we introduce some notations and helpful lemmas that will be used in the analysis. We let z = ( x, y ) and ˜ ∇ F ( z ) = ( ∇ x F ( x, y ) , -∇ y F ( x, y )) for simplicity of the notation in the remaining of the paper. The following results will be frequently used.

LemmaB.1. Under Assumption 4.1, the operator ˜ ∇ F is monotone and ℓ -Lipschitz. That is, ∀ z 1 , z 2 ∈ X × Y , ∥ ˜ ∇ F ( z 1 ) -˜ ∇ F ( z 2 ) ∥ ≤ ℓ ∥ z 1 -z 2 ∥ and ( ˜ ∇ F ( z 1 ) -˜ ∇ F ( z 2 )) ⊤ ( z 1 -z 2 ) ≥ 0 . Moreover, ∀ z ∈ X × Y , ∥ ˜ ∇ F ( z ) ∥ ≤ L where we define L := min ∥ ˜ ∇ F ( z ∗ ) ∥ + √ 2 ℓD for minimum taking w.r.t. any saddle point z ∗ = ( x ∗ , y ∗ ) of F ( x, y ) .

Proof. Lipschitzness of ˜ ∇ F directly follows from ℓ -smoothness of F ( x, y ) . The fact that ˜ ∇ F is monotone when F ( x, y ) is convex-concave is well-known in the literature (e.g., see Theorem 1 in Rockafellar [65]). For the last statement, taking any saddle point z ∗ , we have that ∀ z ∈ X × Y ,

The proof is complete since the domain X and Y have a diameter of D .

LemmaB.2. Under Assumption 4.1. For some integer T ≥ 1 , let z t = ( x t , y t ) for t = 0 , 1 , · · · , T -1 and ¯ z T = (¯ x T , ¯ y T ) = (1 /T ) ∑ T -1 t =0 ( x t , y t ) . If ∀ z ∈ X × Y , (1 /T ) ∑ T -1 t =0 ˜ ∇ F ( z t ) ⊤ ( z t -z ) ≤ ϵ , then it satisfies that max y ∈Y F (¯ x T , y ) -min x ∈X F ( x, ¯ y T ) ≤ ϵ .

Proof. Since F ( x, y ) is convex-concave, we get that ∀ z = ( x, y ) ∈ X × Y ,

Summing up from t = 0 to T -1 and dividing both sides by T , by Jensen's inequality, we have that

Taking y = arg max v ∈Y F (¯ x T , v ) and x = arg min u ∈X F ( u, ¯ y T ) , we conclude the proof.

B.1 Lower-bounds for Reproducibility

The lower-bounds follow from the minimization setting [1].

Lemma B.3. For smooth convex-concave minimax optimization under Assumption 4.1, the reproducibility, i.e., ( ϵ, δ ) -deviation, of any algorithm A is at least ( i ) Ω( δ 2 ) for the inexact initialization oracle; ( ii ) Ω( δ 2 /ϵ 2 ) for the deterministic inexact gradient oracle; ( iii ) Ω( δ 2 / ( Tϵ 2 )) for the stochastic gradient oracle, where T is the total number of iterations of the algorithm.

Proof. The lower-bound of reproducibility in Ahn et al. [1] for smooth convex minimization problems is also a valid lower-bound for smooth convex-concave minimax problems. To show this, we consider a special case of the minimax problem (1) where the domain Y is a singleton, i.e., Y = { y 0 } for some y 0 . Then the original smooth convex-concave minimax problem min x ∈X max y ∈Y F ( x, y ) is equivalent to the smooth convex minimization problem min x ∈X F ( x, y 0 ) . For all three inexact oracles, let (ˆ x, ˆ y ) and (ˆ x ′ , ˆ y ′ ) be the ϵ -approximate outputs of independent two runs of the same algorithm, i.e., the duality gap can be upper-bounded by ϵ , then the reproducibility ∥ ˆ x -ˆ x ′ ∥ 2 + ∥ ˆ y -ˆ y ′ ∥ 2 = ∥ ˆ x -ˆ x ′ ∥ 2 since ˆ y = ˆ y ′ = y 0 . Moreover, ˆ x and ˆ x ′ are also ϵ -approximate solutions of the function F ( x, y 0 ) based on the definition of duality gap. Thus the lower-bound in the minimization setting [1] directly implies the lower-bound in the minimax setting. To be specific, the lower-bound is: ( i ) Ω( δ 2 ) for the inexact initialization case; ( ii ) Ω( δ 2 /ϵ 2 ) for the inexact deterministic gradient case; and ( iii ) Ω( δ 2 / ( Tϵ 2 )) for the stochastic gradient case.

B.2 Guarantees of Gradient Descent Ascent

This section provides proof of Theorem 4.2 for the sub-optimal guarantees of GDA in the deterministic setting and Theorem 5.1 for the optimal guarantees of SGDA in the stochastic setting. We first provide a general analysis and then expand it for three different inexact oracles in subsequent sections.

B.2.1 General Analysis

We consider (stochastic) gradient descent ascent (GDA/SGDA) outlined in Algorithm 4 for solving minimax problems (1) or (3). The algorithm iteratively updates the variables x t and y t using exact gradients ∇ F ( x t , y t ) , or inexact gradients G ( x t , y t ) , or stochastic gradients ∇ f ( x t , y t ; ξ t ) based on different types of the inexact oracles in Definition 4.

We first analyze the behavior of GDA with access to exact gradients. It is well-known that the last iterate of GDA can diverge even for bilinear functions [55, 9, 36], and the average iterates converge with a sub-optimal rate O (1 / √ T ) . We provide proof for completeness.

Lemma B.4. Under Assumption 4.1. When setting the stepsize to α = 1 / ( ℓ T ) , the average iterates (¯ x T , ¯ y T ) of GDA converges with

√

This suggests O (1 /ϵ 2 ) gradient complexity is required to achieve ϵ -saddle point.

Proof. Recall z t = ( x t , y t ) and ˜ ∇ F ( z t ) = ( ∇ x F ( x t , y t ) , -∇ y F ( x t , y t )) . The GDA updates in Algorithm 4 can be simplified to

Since the projection step is nonexpansive [12], we have that ∀ z = ( x, y ) ∈ X × Y ,

Rearranging terms and using Lemma B.1, we can obtain that

Taking summation from t = 0 to T -1 and dividing both sides by T , by Lemma B.2, we thus have

When setting α = 1 / ( ℓ √ T ) , this means the complexity is required to be T ≥ ( ℓD 2 + L 2 / (2 ℓ )) 2 /ϵ 2 to achieve an ϵ -saddle point such that max y ∈Y F (¯ x T , y ) -min x ∈X F ( x, ¯ y T ) ≤ ϵ .

Lemma B.5. Under Assumption 4.1, the GDA update (9) is (1 + α 2 ℓ 2 ) -expansive. That is, if ( x t +1 , y t +1 ) is obtained through 1-step of the update given ( x t , y t ) , and ( x ′ t +1 , y ′ t +1 ) is obtained given ( x ′ t , y ′ t ) , we have that

Proof. Recall z t = ( x t , y t ) and z ′ t = ( x ′ t , y ′ t ) . By the updates of GDA (9), we get that

where we use the fact that the projection step is nonexpansive and Lemma B.1.

B.2.2 Inexact Initialization Oracle

Theorem B.6 (Restate Theorem 4.2, part ( i ) ) . Under Assumptions 4.1. The average iterate (¯ x T , ¯ y T ) of GDA satisfies max y ∈Y F (¯ x T , y ) -min x ∈X F ( x, ¯ y T ) ≤ O ( ϵ ) with complexity T = O (1 /ϵ 2 ) if setting stepsize α = 1 / ( ℓ √ T ) . The reproducibility, i.e., ( ϵ, δ ) -deviation between outputs (¯ x T , ¯ y T ) and (¯ x ′ T , ¯ y ′ T ) of two independent runs given different initialization is ∥ ¯ x T -¯ x ′ T ∥ 2 + ∥ ¯ y T -¯ y ′ T ∥ 2 ≤ O ( δ 2 ) .

Proof. The convergence analysis directly follows from Lemma B.4. For the reproducibility analysis, by Lemma B.5 and the choice that α = 1 / ( ℓ √ T ) , we have that for t = 1 , 2 , · · · , T -1 ,

The above also trivially holds for t = 0 . Therefore, by Jensen's inequality, we can obtain that

The choice of α is to avoid exponential dependence on ℓ in the reproducibility bound.

B.2.3 Inexact Deterministic Gradient Oracle

When only given an inexact gradient oracle in Definition 4, the updates of GDA become

where we let ˜ G ( z t ) = ( G x ( x t , y t ) , -G y ( x t , y t )) for the inexact gradients.

Theorem B.7 (Restate Theorem 4.2, part ( ii ) ) . Under Assumptions 4.1. Given an inexact deterministic gradient oracle in Definition 4 with δ ≤ O ( ϵ ) . The average iterate (¯ x T , ¯ y T ) of GDA satisfies max y ∈Y F (¯ x T , y ) -min x ∈X F ( x, ¯ y T ) ≤ O ( ϵ ) with complexity T = O (1 /ϵ 2 ) if setting stepsize α = 1 / ( ℓ √ T ) . Furthermore, the reproducibility is ∥ ¯ x T -¯ x ′ T ∥ 2 + ∥ ¯ y T -¯ y ′ T ∥ 2 ≤ O ( δ 2 /ϵ 2 ) .

Proof. We first show the optimization guarantee. By the GDA updates in (10) and Definition 4 such that ∥ ˜ ∇ F ( z t ) -˜ G ( z t ) ∥ 2 ≤ δ 2 , we have that for any z = ( x, y ) ∈ X × Y ,

Taking summation from t = 0 to T -1 , we obtain that

Supposing δ ≤ ϵ/ (2 √ 2 D ) and setting α = 1 / ( ℓ √ T ) , by Lemma B.2, this means

ϵ -saddle point is guaranteed when T = c/ϵ 2 for some constant c ≥ 4( ℓD 2 +( L 2 + δ 2 ) /ℓ ) 2 .

We then prove the reproducibility guarantee. Let { z t } T t =1 and { z ′ t } T t =1 be the trajectories of two independent runs of GDA with the same initial point z 0 ∈ X × Y and stepsize α > 0 . By the GDA updates (10) and Lemma B.5, we have that

Since the initialization z 0 = z ′ 0 is the same, we obtain that for any t = 1 , 2 , · · · , T -1 ,

The above also holds for t = 0 denoting ∑ -1 i =0 = 0 . Setting α = 1 / ( ℓ √ T ) , the reproducibility is

which is O ( δ 2 /ϵ 2 ) when T = c/ϵ 2 as required in the convergence analysis of GDA.

B.2.4 Stochastic Gradient Oracle

For the stochastic minimax problem (3), with access to a stochastic gradient oracle in Definition 4, SGDA updates for t = 0 , 1 , · · · , T -1 ,

where ˜ ∇ f ( z t ; ξ t ) = ( ∇ x f ( x t , y t ; ξ t ) , -∇ y f ( x t , y t ; ξ t )) and { ξ t } T -1 t =0 are i.i.d. samples.

Proof of Theorem 5.1. We first show the convergence guarantee. By the SGDA updates in (13), given all the information up to iteration t and taking expectation with respect to ξ t , we have ∀ z ∈ X × Y ,

Taking full expectation, rearranging terms, and summing up from t = 0 to T -1 , we have that

Therefore, by slightly modifying the proof of Lemma B.2 through taking expectations, and then setting x = arg min u ∈X E [ F ( u, ¯ y T )] and y = arg max v ∈Y E [ F (¯ x T , v )] , we get

We obtain that max y ∈Y E [ F (¯ x T , y )] -min x ∈X E [ F ( x, ¯ y T )] ≤ ( ℓD 2 + ( L 2 + δ 2 ) / (2 ℓ )) ϵ if the inexactness δ = O (1) , and we set α = 1 / ( ℓϵT ) , T ≥ 1 /ϵ 2 .

We then show the reproducibility guarantee. For two independent runs of SGDA (13) with output { z t } T t =1 and { z ′ t } T t =1 , by Lemma B.5, we have that for any t = 0 , 1 , · · · , T -1 ,

Unrolling the recursion, noticing z 0 = z ′ 0 , we have that for any t = 0 , 1 , · · · , T -1 ,

Since T ≥ 1 /ϵ 2 , we know α = 1 / ( ℓϵT ) ≤ 1 / ( ℓ √ T ) . The reproducibility is thus

The last step uses the choice of α such that α 2 T = 1 / ( ℓ 2 ϵ 2 T ) .

B.3 Guarantees of Extragradient

This section provides proof of Theorem 4.3 for the sub-optimal guarantees of Extragradient (EG).

B.3.1 General Analysis

Algorithm 5 Extragradient

For deterministic smooth convex-concave minimax optimization, Extragradient [48, 72] (EG), summarized in Algorithm 5, achieves the optimal O (1 /ϵ ) convergence rate. When only given inexact gradients or stochastic gradients, the true gradients are just replaced by G ( x t , y t ) or ∇ f ( x t , y t ; ξ t ) .

We provide proof of its O (1 /ϵ ) convergence for completeness. The proof is standard in the literature, e.g., see Nemirovski [59] or Section 4.5 of Bubeck [20].

Lemma B.8. Under Assumption 4.1. When setting the stepsize to α = 1 /ℓ , the average iterates (¯ x T +1 / 2 , ¯ y T +1 / 2 ) of EG converges with

This suggests O (1 /ϵ ) gradient complexity is required to achieve ϵ -saddle point.

Proof. Recall z t = ( x t , y t ) and ˜ ∇ F ( z t ) = ( ∇ x F ( x t , y t ) , -∇ y F ( x t , y t )) . The EG updates in Algorithm 5 can be simplified to

By fact ( i ) in Lemma A.1, we have that for any z ∈ X × Y ,

where we use the optimality condition of the projection step such that (Π C ( u ) -u ) ⊤ ( v -Π C ( u )) ≥ 0 , ∀ v ∈ C . For the same reason, we can obtain that

Summing up the above two inequalities, we get

According to Lemma B.1, we can obtain

Therefore, rearranging terms, by the choice of stepsize α ≤ 1 /ℓ , we have ∀ z ∈ X × Y ,

Taking summation from t = 0 to T -1 , by Lemma B.2, we have

Since ∥ z -z ∥ 2 ≤ 2 D 2 and α = 1 /ℓ

The following results are motivated from Boob and Guzmán [16].

Lemma B.9. Under Assumption 4.1. Let z t +1 = ( x t +1 , y t +1 ) be obtained through 1-step of EG update (14) given z t = ( x t , y t ) , and z ′ t +1 is obtained given z ′ t . Setting α ≤ 1 /ℓ , then we have

Proof. For any z = ( x, y ) ∈ X × Y , we define an operator P z t ( · ) : X × Y → X × Y as P z t ( z ) = Π X×Y ( z t -α ˜ ∇ F ( z )) , and the EG updates can be written as z t +1 = P z t ( P z t ( z t )) . When the stepsize α ≤ 1 /ℓ , the operator P z t ( · ) is nonexpansive, i.e., ∀ z 1 , z 2 ∈ X × Y ,

Since the domain X×Y is a nonempty bounded closed convex set, by Theorem 4.19 in Bauschke et al. [12], the nonexpansive operator P z t ( · ) admits fixed points. Denote one fixed point as u t ∈ X × Y such that u t = Π X×Y ( z t -α ˜ ∇ F ( u t )) = P z t ( u t ) . The nonexpansiveness of P z t ( · ) implies

The same holds true for z ′ t +1 and u ′ t = P z ′ t ( u ′ t ) defined for z ′ t . As a result, we can obtain that

By optimality conditions of u t = Π X×Y ( z t -α ˜ ∇ F ( u t )) and u ′ t = Π X×Y ( z ′ t -α ˜ ∇ F ( u ′ t )) , we obtain that for any z, z ′ ∈ X × Y ,

Taking z = u ′ t and z ′ = u t and using the fact that ˜ ∇ F is monotone by Lemma B.1, we obtain that

Combined with (18), the proof is complete since ∥ u t -u ′ t ∥ ≤ ∥ z t -z ′ t ∥ .

Remark 6. We can alternatively derive the relation between ∥ z t +1 -z ′ t +1 ∥ and ∥ z t -z ′ t ∥ as follows:

Here, we use Lemma B.1 and B.5. The above results will lead to reproducibility that grows with O ( e T ) , which is similar to the results of AGD for the minimization setting [6].

B.3.2 Inexact Initialization Oracle

Theorem B.10 (Restate Theorem 4.3, part ( i ) ) . Under Assumptions 4.1. The average iterate (¯ x T +1 / 2 , ¯ y T +1 / 2 ) of EG satisfies max y ∈Y F (¯ x T +1 / 2 , y ) -min x ∈X F ( x, ¯ y T +1 / 2 ) ≤ O ( ϵ ) with complexity T = O (1 /ϵ ) if setting stepsize α = 1 /ℓ . Furthermore, the reproducibility, i.e., ( ϵ, δ ) -deviation between outputs of two independent runs of EG given different initialization is ∥ ¯ x T +1 / 2 -¯ x ′ T +1 / 2 ∥ 2 + ∥ ¯ y T +1 / 2 -¯ y ′ T +1 / 2 ∥ 2 ≤ O (min { δ 2 e 1 / ϵ , δ 2 +1 /ϵ 2 , D 2 } ) .

Proof. The convergence part directly follows from Lemma B.8 with T = c/ϵ for some constant c ≥ ℓD 2 . For reproducibility, by Lemma B.5, B.9 and the stepsize α = 1 /ℓ , we have that for t = 1 , 2 , · · · , T -1 ,

The above also holds for t = 0 . Therefore, by Jensen's inequality, we obtain

Alternatively, by Remark 6, we know that ∥ z t +1 / 2 -z ′ t +1 / 2 ∥ 2 ≤ 2(1 + √ 2) 2 t δ 2 , and thus the reproducibility is ∥ ¯ x T +1 / 2 -¯ x ′ T +1 / 2 ∥ 2 + ∥ ¯ y T +1 / 2 -¯ y ′ T +1 / 2 ∥ 2 ≤ O ( e T δ 2 ) . The proof is complete by taking the minimum between the two results and replacing T with c/ϵ .

B.3.3 Inexact Deterministic Gradient Oracle

When only given inexact gradient ( G x ( x t , y t ) , G y ( x t , y t )) , the updates of EG becomes

where exact gradients ˜ ∇ F ( z t ) in (14) are replaced by ˜ G ( z t ) = ( G x ( x t , y t ) , -G y ( x t , y t )) .

Theorem B.11 (Restate Theorem 4.3, part ( ii ) ) . Under Assumptions 4.1. Given an inexact deterministic gradient oracle in Definition 4 with δ ≤ O ( ϵ ) . The average iterate (¯ x T +1 / 2 , ¯ y T +1 / 2 ) of EG satisfies max y ∈Y F (¯ x T +1 / 2 , y ) -min x ∈X F ( x, ¯ y T +1 / 2 ) ≤ O ( ϵ ) with complexity T = O (1 /ϵ ) if setting stepsize α = 1 /ℓ . Furthermore, the reproducibility is O (min { δ 2 e 1 / ϵ , 1 /ϵ 2 , D 2 } ) .

Proof. Let ∆( z t ) = ˜ G ( z t ) -˜ ∇ F ( z t ) . We know ∥ ∆( z t ) ∥ ≤ δ by Definition 4. Using (15) in the proof of Lemma B.8, we have that ∀ z ∈ X × Y ,

The above is the same as (15) up to an additional error in O ( δ ) . Following the same proof after (15), with α = 1 /ℓ , we obtain that

When δ ≤ ϵ/ (6 √ 2 D ) and T = c/ϵ for some constant c ≥ 2 ℓD 2 /ϵ , we get ϵ -saddle point.

We then show the reproducibility guarantee. Let u t = Π X×Y ( z t -α ˜ ∇ F ( u t )) be the same as in the proof of Lemma B.9. Similarly to (17), we have that

As a result, the same as (18), since α = 1 /ℓ , we can obtain that ∀ t = 0 , 1 , · · · , T -1 ,

Therefore, by Jensen's inequality and (12) in Section B.2.3 for the guarantee of GDA, we know

Note that T = c/ϵ and δ ≤ O ( ϵ ) . Thus the reproducibility is O (1 /ϵ 2 ) . Alternatively, by Remark 6 and similarly to (12), we have that

Thus ∥ z t +1 / 2 -z ′ t +1 / 2 ∥ ≤ √ 2 ∥ z t -z ′ t ∥ +2 δ/ℓ ≤ O ( e T δ/ℓ ) and the reproducibility is O ( δ 2 e 1 / ϵ ) . The proof is complete by taking the minimum between the two results.

B.3.4 More Discussions

In this section, we show that Extragradient can also be optimally reproducible by a different selection of parameters. Although it will suffer from a sub-optimal convergence rate O (1 /ϵ 3 / 2 ) instead of O (1 /ϵ ) , this is still an improvement on the O (1 /ϵ 2 ) rate of GDA.

Theorem B.12. Under Assumptions 4.1. The average iterate (¯ x T +1 / 2 , ¯ y T +1 / 2 ) of EG satisfies max y ∈Y F (¯ x T +1 / 2 , y ) -min x ∈X F ( x, ¯ y T +1 / 2 ) ≤ O ( ϵ ) with complexity T = O (1 / ( δ 1 / 2 ϵ 3 / 2 )) if setting stepsize α = min { 1 /ℓ, ( δ/ (2 ℓ 2 T )) 1 / 3 } . The reproducibility is O ( δ 2 ) .

Proof. The same as Section B.3.2, by the choice of stepsize α such that α 3 T ≤ δ/ 2 ℓ 2 , we obtain

By Lemma B.8, when the stepsize α ≤ 1 /ℓ , we have that

This means a O (1 / ( δ 1 / 2 ϵ 3 / 2 )) convergence rate with reproducibility O ( δ 2 ) . In the case δ = O (1) , the gradient complexity is O (1 /ϵ 3 / 2 ) .

Theorem B.13. Under Assumptions 4.1. Given an inexact deterministic gradient oracle in Definition 4 with δ ≤ O ( ϵ ) . The average iterate (¯ x T +1 / 2 , ¯ y T +1 / 2 ) of EG satisfies max y ∈Y F (¯ x T +1 / 2 , y ) -min x ∈X F ( x, ¯ y T +1 / 2 ) ≤ O ( ϵ ) with complexity T = O (1 / ( ϵ √ δ )) if setting stepsize α = min { 1 /ℓ, ( δ/ (2 ℓ 2 )) 1 / 2 } . The reproducibility is O ( δ 2 /ϵ 2 ) .

Proof. The same as Section B.3.3, since αℓ ≤ 1 and α 2 ≤ δ/ (2 ℓ 2 ) , we have that

When the stepsize α ≤ 1 /ℓ , we also have

To guarantee O ( ϵ ) -saddle point, we need to ensure δ ≤ O ( ϵ ) and αT = c/ϵ for some constant c . This means a O (1 / ( ϵ √ δ )) convergence rate with reproducibility O ( δ 2 /ϵ 2 ) . Since δ ≤ O ( ϵ ) , the gradient complexity is O (1 /ϵ 3 / 2 ) .

Finally, we want to mention that the analysis can also be extended to reproducibility under stochastic gradient oracle and stability of Extragradient [16] that matches with SGDA. We will not provide all details here. The key is to select stepsize α to balance the convergence O (1 / ( αT )) in Lemma B.8 and the error term O ( α 3 T ) that appears according to Lemma B.9. Moreover, we also acknowledge that it is unclear whether the analysis of EG is tight since the specific lower-bound is unknown. We leave this problem for future exploration.

C Near-optimal Guarantees in the Minimax Case

This section discusses near-optimal guarantees for algorithmic reproducibility and gradient complexity in smooth convex-concave minimax optimization.

C.1 Useful Lemmas

We first establish the convergence behavior of gradient descent ascent (GDA) and Extragradient (EG) [48] for smooth and strongly-convex-strongly-concave (SC-SC) functions under the inexact gradient oracle in Definition 4. For the sake of simplicity and to enable a general analysis, we slightly abuse notation here to consider the minimax optimization problem

where f : X × Y → R satisfies the following assumption.

Assumption C.1. The function f ( x, y ) is ℓ -smooth and µ -strongly-convex-strongly-concave on the closed convex domain X × Y .

Assumption C.2. We assume the existence of an inexact gradient oracle that returns a vector g ( x, y ) = ( g x ( x, y ) , g y ( x, y )) at any querying point ( x, y ) ∈ X × Y such that ∥∇ f ( x, y ) -g ( x, y ) ∥ 2 ≤ δ 2 where ∇ f ( x, y ) = ( ∇ x f ( x, y ) , ∇ y f ( x, y )) is the true gradient at ( x, y ) .

The lemma below shows the convergence behavior of GDA under the inexact gradient oracle presented above, also referred to as Inexact-GDA.

Lemma C.3. Under Assumption C.1. Let z ∗ = ( x ∗ , y ∗ ) ∈ X × Y be the unique saddle point of f ( x, y ) and κ := ℓ/µ be the condition number. Given an inexact gradient oracle in Assumption C.2. Denote z t = ( x t , y t ) and ˜ g ( z t ) = ( g x ( x t , y t ) , -g y ( x t , y t )) . Starting from z 0 ∈ X × Y , GDA that updates for t = 0 , 1 , · · · , T -1 ,

with stepsize α = µ/ (4 ℓ 2 ) converges with

Proof. Let ˜ ∇ f ( z t ) = ( ∇ x f ( x t , y t ) , -∇ y f ( x t , y t )) . It holds that z ∗ = Π X×Y ( z ∗ -α ˜ ∇ f ( z ∗ )) since the saddle point problem and the projection problem share the same optimality condition when f ( x, y ) is convex-concave (see Proposition 1.4.2 in Facchinei and Pang [32]) such that

Therefore, similarly to (11), by the GDA updates, we have

Since ˜ ∇ f is µ -strongly-monotone if f ( x, y ) is µ -strongly-convex-strongly-concave [66, 35], i.e., ∀ z 1 , z 2 ∈ X × Y , ( ˜ ∇ f ( z 1 ) -˜ ∇ f ( z 2 )) ⊤ ( z 1 -z 2 ) ≥ µ ∥ z 1 -z 2 ∥ 2 , we have that

where we use Assumption C.2 such that ∥ ˜ g ( z t ) -˜ ∇ f ( z t ) ∥ ≤ δ and fact ( iii ) in Lemma A.1. Then by ℓ -smoothness of f ( x, y ) , we can obtain that

Combining all three results together, when choosing the stepsize α = µ/ (4 ℓ 2 ) , we get that

Unrolling the recursion, we thus obtain

This means a O ( κ 2 ) convergence rate to a O ( δ 2 ) neighborhood, where κ = ℓ/µ is the condition number.

The lemma below establishes the convergence performance of EG under Assumption C.1 and C.2.

Lemma C.4. Under Assumption C.1. Let z ∗ = ( x ∗ , y ∗ ) ∈ X × Y be the unique saddle point of f ( x, y ) and κ := ℓ/µ be the condition number. Given an inexact gradient oracle in Assumption C.2. Denote z t = ( x t , y t ) and ˜ g ( z t ) = ( g x ( x t , y t ) , -g y ( x t , y t )) . Starting from z 0 ∈ X × Y , Extragradient that updates for t = 0 , 1 , · · · , T -1 ,

with stepsize α = 1 / (2 ℓ ) converges with

Proof. Let ˜ ∇ f ( z t ) = ( ∇ x f ( x t , y t ) , -∇ y f ( x t , y t )) and ∆( z t ) = ˜ g ( z t ) -˜ ∇ f ( z t ) . By (15) in the proof of Lemma B.8, setting z = z ∗ , we have that,

By strong-convexity-strong-concavity of the function f ( x, y ) , we know that

Summing up the above two inequalities, using the definition of saddle points, we have

where we use fact ( iii ) in Lemma A.1 and ∥ z t -z ∗ ∥ 2 ≤ 2 ∥ z t -z t +1 / 2 ∥ 2 +2 ∥ z t +1 / 2 -z ∗ ∥ 2 . By smoothness of f ( x, y ) and fact ( iii ) in Lemma A.1, we also have that

Plugging (22) and (23) back into (21), choosing α = 1 / (2 ℓ ) , we obtain that

Unrolling the recursion, since 1 + η ≤ e η , ∀ η ∈ R , we get that

This means a O ( κ ) convergence rate to a O ( δ 2 ) neighborhood, where κ = ℓ/µ is the condition number.

Lemma 4.5 directly follows from Lemma C.4 observing that G ( x, y )+ r ( x -x 0 , y 0 -y ) is a δ -inexact gradient of F r ( x, y ) . Next, we provide a useful lemma showing how to satisfy the stopping criteria for the auxiliary smooth SC-SC sub-problem in Algorithm 2 when presented with inexact gradients. The results are motivated from Yang et al. [74].

Lemma C.5. Under Assumption C.1 and C.2. Suppose the domain X and Y have a diameter of D . Denote z ∗ = ( x ∗ , y ∗ ) be the unique saddle point of f ( x, y ) . For any ˆ z = (ˆ x, ˆ y ) ∈ X × Y , we let ˜ g (ˆ z ) = ( g x (ˆ x, ˆ y ) , -g y (ˆ x, ˆ y )) and define [ˆ z ] β = ([ˆ x ] β , [ˆ y ] β ) for β ≥ 2 ℓ to be

which is obtained through one step of GDA starting from ˆ z with inexact gradients. Denote the true gradient as ˜ ∇ f ([ˆ z ] β ) = ( ∇ x f ([ˆ x ] β , [ˆ y ] β ) , -∇ y f ([ˆ x ] β , [ˆ y ] β )) . Then we have that ∀ z = ( x, y ) ∈ X × Y ,

Moreover, it also holds that ∥ [ˆ z ] β -z ∗ ∥ ≤ (1 + √ 2 ℓ/β ) ∥ ˆ z -z ∗ ∥ + δ (1 / √ βµ + √ 2 /β ) .

Proof. We construct a 'ghost' point ˆ z 1 = (ˆ x 1 , ˆ y 1 ) ∈ X × Y to be

ˆ z 1 can be regarded as performing one update of inexact-EG with stepsize 1 /β starting from ˆ z . Therefore, by (16) and (19) in the convergence analysis of EG, since 1 /β ≤ 1 /ℓ , we obtain that ∀ z = ( x, y ) ∈ X × Y ,

By (24) in the proof of Lemma C.4, since β ≥ 2 ℓ and µ ≤ ℓ , we have that

Therefore, we can obtain that ∥ ˆ z 1 -z ∗ ∥ ≤ ∥ ˆ z -z ∗ ∥ +2 δ/ βℓ + 2 δ/ βµ , and thus,

For the last statement, since [ˆ z ] β is obtained through 1-step of GDA with inexact gradients, by (20) in the proof of GDA for SC-SC problems before, we have that

The above lemma also applies to the case when exact gradients are available setting δ = 0 and [ˆ z ] β = Π X×Y ( ˆ z -1 β ˜ ∇ f (ˆ z ) ) for the true gradients ˜ ∇ f (ˆ z ) . This implies the stopping criteria ˜ ∇ f (ˆ z ) ⊤ (ˆ z -z ) ≤ ˆ ϵ, ∀ z ∈ X × Y in Algorithm 2 and 3 can be translated to ∥ ˆ z -z ∗ ∥ 2 ≤ O (ˆ ϵ 2 ) , which can be satisfied within O (log(1 / ˆ ϵ )) complexity using Lemma C.3 and C.4 with δ = 0 (or existing results in Tseng [72] or Facchinei and Pang [32]).

C.2 Regularization Helps!

Proof of Theorem 4.4 and 4.6 for the near-optimal guarantees of Algorithm 2 is provided here.

C.2.1 Inexact Initialization Oracle

We also use ( x 0 , y 0 ) as the initialization point when solving the auxiliary strongly-convex problem. Note that the gradient steps starting from ( x 0 , y 0 ) remain the same on F ( x, y ) and F r ( x, y ) .

Proof of Theorem 4.4. We first show the convergence guarantee. Let z r = ( x r , y r ) . By fact ( i ) in Lemma A.1, we have that ∀ z = ( x, y ) ∈ X × Y ,

According to Lemma B.2, this means max y ∈Y F ( x r , y ) -min x ∈X F ( x, y r ) ≤ ϵ r + rD 2 .

We then show the reproducibility guarantee. Denote the saddle point of F r ( x, y ) given ( x 0 , y 0 ) as ( x ∗ r , y ∗ r ) , and the saddle point of F ′ r ( x, y ) = F ( x, y ) + ( r/ 2) ∥ x -x ′ 0 ∥ 2 -( r/ 2) ∥ y -y ′ 0 ∥ 2 given ( x ′ 0 , y ′ 0 ) as (( x ∗ r ) ′ , ( y ∗ r ) ′ ) . By Lemma B.4 in Appendix B.3 of Zhang et al. [78], we have that

Let z r = ( x r , y r ) , z ∗ r = ( x ∗ r , y ∗ r ) and z 0 = ( x 0 , y 0 ) for simplicity of the notation. z ′ r , ( z ∗ r ) ′ and z ′ 0 can be defined in the same way. Similarly to the minimization case, we have

where we use ∥ z ∗ r -( z ∗ r ) ′ ∥ ≤ ∥ z 0 -z ′ 0 ∥ ≤ δ and optimality of z ∗ r by r strong-convexity-strongconcavity (SC-SC) of F r ( x, y ) (the same holds true for z ′ r and ( z ∗ r ) ′ as well):

Thus setting r = ϵ/D 2 and ϵ r = ϵ · min { 1 , δ 2 / (8 D 2 ) } , we guarantee that max y ∈Y F ( x r , y ) -min x ∈X F ( x, y r ) ≤ 2 ϵ and ∥ x r -x ′ r ∥ 2 + ∥ y r -y ′ r ∥ 2 ≤ 4 δ 2 . Applying Lemma C.5 with δ = 0 , the complexity using Extragradient (EG) [72, 57] to achieve ϵ r -error on r -SC-SC ( ℓ + r ) -smooth minimax optimization is O (( ℓ/r +1)log(1 /ϵ r )) = ˜ O ( ℓD 2 /ϵ ) , where ˜ O hides logarithmic terms.

C.2.2 Inexact Deterministic Gradient Oracle

This section contains proof of Theorem 4.6 for the near-optimal guarantees in the inexact deterministic gradient case. The proof is based on Lemma 4.5 (restated and proved as Lemma C.4 in Section C.1) and Lemma C.5.

Proof of Theorem 4.6. For the convergence guarantee, the same as (25), we have that

For the reproducibility guarantee, we can obtain that

Let z T be the output of T -step Extragradient with initialization z 0 . By Lemma 4.5, we have that

Setting T ≥ (32 ℓ/r ) log( rD/δ ) and r = ϵ/D 2 , this means the algorithm converges to ∥ z T -z ∗ r ∥ ≤ 3 √ 2 D 2 ( δ/ϵ ) . Therefore, according to Lemma C.5, if we choose z r = [ z T ] 2 ℓ , since 1 ≤ ℓD 2 /ϵ , we can guarantee that ∥ z r -z ∗ r ∥ ≤ 3(2 √ 2 + 1) D 2 ( δ/ϵ ) and that

The reproducibility is ∥ z r -z ′ r ∥ 2 ≤ 36(9 + 4 √ 2) D 4 ( δ 2 /ϵ 2 ) .

C.3 Inexact Proximal Point Method

Proof of Theorem 4.7 and 4.8 for the guarantees of Algorithm 3 is provided in this section.

C.3.1 General Analysis

We first analyze the convergence of the inexact proximal point method (Inexact-PPM). Given initialization ( x 0 , y 0 ) and α > 0 , for t = 0 , 1 , · · · , T -1 , each step of Inexact-PPM is

Lemma C.6. If we run Inexact-PPM and make sure that for each sub-problem ˜ ∇ ˆ F t ( z t +1 ) ⊤ ( z t +1 -z ) ≤ ˆ ϵ for all z = ( x, y ) ∈ X × Y , where z t +1 = ( x t +1 , y t +1 ) and ˜ ∇ ˆ F t ( z t +1 ) = ( ∇ x ˆ F t ( x t +1 , y t +1 ) , -∇ y ˆ F t ( x t +1 , y t +1 )) , then we have ∀ z ∈ X × Y ,

Proof. The proof is similar to Proposition 7 in Mokhtari et al. [58]. The same as (25), for any z = ( x, y ) ∈ X × Y and any t = 0 , 1 , · · · , T -1 , we have that

Taking summation from t = 0 to T -1 and dividing both sides by T , we conclude that

The proof is completed by Lemma B.2.

C.3.2 Inexact Initialization Oracle

This section provides proof of Theorem 4.7.

Proof of Theorem 4.7. Let ¯ z T +1 = (¯ x T +1 , ¯ y T +1 ) = (1 /T ) ∑ T -1 t =0 ( x t +1 , y t +1 ) . By Lemma C.6 and the choice that α = 1 /ℓ , ˆ ϵ = δ 2 / (2 αT 2 ) , we immediately have

O (1 /T ) convergence rate is guaranteed for δ ≤ O ( √ T ) . Note that the condition number of ˆ F t ( x, y ) is O (1) when α = 1 /ℓ . Therefore, to guarantee an ϵ -saddle point of F ( x, y ) , a total complexity of O ( T log(1 / ˆ ϵ )) = O ((1 /ϵ ) log(1 / ( ϵδ ))) is sufficient for various algorithms including GDA [32] and EG [72] applying Lemma C.5 with δ = 0 .

Let z ∗ t = ( x ∗ t , y ∗ t ) be the unique saddle point of ˆ F t ( x, y ) with proximal center z t , and ( z ∗ t ) ′ be the saddle point when the proximal center is z ′ t . For the reproducibility guarantee, similarly to Section C.2.1, we can obtain that

where we use Lemma B.4 in Zhang et al. [78] and (1 /α ) -SC-SC of ˆ F t ( x, y ) :

Therefore, by induction, we have that for any t = 1 , 2 , · · · , T ,

The reproducibility is then ∥ ¯ z T +1 -¯ z ′ T +1 ∥ 2 ≤ 9 δ 2 using Jensen's inequality.

C.3.3 Inexact Deterministic Gradient Oracle

For Theorem 4.8, we provide proof when using GDA as the base algorithm. According to Lemma C.4, EG can also be applied here with a similar argument.

Proof of Theorem 4.8. When setting α = 1 /ℓ , the auxiliary problem is ℓ -strongly-convex-stronglyconcave and 2 ℓ -smooth. Let z t K be the output of K -step GDA with initialization z 0 t on the minimax problem min x ∈X max y ∈Y ˆ F t ( x, y ) at iteration t . Denote its saddle point as z ∗ t . By Lemma C.3, if K ≥ 32 log(8 ℓ 2 D 2 / (3 δ 2 )) , we have that

By Lemma C.5, we can thus set z t +1 = [ z t K ] 2 ℓ and guarantee that

According to Lemma C.6, we then have

When δ ≤ ϵ/ (42 D ) , T ≥ 2 ℓD 2 /ϵ is required to obtain an ϵ -saddle point, and the total gradient complexity is TK = (64 ℓD 2 /ϵ ) log(8 ℓ 2 D 2 / (3 δ 2 )) = ˜ O (1 /ϵ ) with ˜ O hiding logarithmic terms.

We then show the reproducibility guarantee. From Lemma C.5, we know that ∥ z t +1 -z ∗ t ∥ ≤ (1 + √ 2 / 2) ∥ z t K -z ∗ t ∥ + √ 2 δ/ℓ ≤ 4 . 5 δ/ℓ . By (26), we have that

By induction, we conclude that ∥ z t -z ′ t ∥ ≤ 9 t ( δ/ℓ ) , and thus the reproducibility is ∥ ¯ z T +1 -¯ z ′ T +1 ∥ 2 ≤ 81 δ 2 T 2 /ℓ 2 = 324 D 4 ( δ 2 /ϵ 2 ) .

D Numerical Experiments

Some numerical experiments that demonstrate the effectiveness of regularization to improve reproducibility are provided in this section. We test the algorithms on two problems: a minimization problem with a quadratic objective and a minimax problem with a bilinear objective. The experiments are conducted on a single local machine.

Minimization. We first compare the performance of gradient descent (GD), accelerated gradient descent (AGD), Algorithm 1 with GD as the base algorithm (Reg-GD), and Algorithm 1 with AGD as the base algorithm (Reg-AGD) on a quadratic minimization problem

Here, b ∈ R d with each entry sampled from the Gaussian distribution with mean 0 and standard deviation 10 and A ∈ R d × d is a random positive semi-definite matrix with rank d -1 that makes

Function Value

Deviation in Trajectory

104

AGD

AGD

103

103

Reg-AGD

Reg-AGD

Reg-GD

102

Reg-GD

101

10°

102

10-1

10-2

100

101

102

103

104

100

101

102

103

104

# Iterations

# Iterations

sure the problem is convex but not strongly-convex. To be specific, we let A = U Σ U ⊤ where U is a random orthogonal matrix drawn from the Haar distribution, and Σ is a diagonal matrix with 1 entry being 0 and the others uniformly sampled from [0 . 1 , 10] . This ensures that the problem is smooth with a parameter smaller than 100 .

We implement an inexact gradient oracle that returns A ⊤ ( Ax -b ) + δe where e ∈ R d is an allone vector and δ ∈ R controls the inexactness level. We test the aforementioned four algorithms with this inexact gradient oracle on both convergence performance measured by function value and reproducibility performance measured by the deviation compared to the trajectory obtained from using the true gradient when δ = 0 . In the experiments, we let d = 100 and δ = 0 . 1 . For all four algorithms, we set the number of iterations to be T = 10000 , and the stepsize to be 0.01 based on the fact that the smoothness parameter is at most 100. For the regularization-based methods, we set the regularization parameter of the auxiliary problem to 0.05. All other parameters are set according to the theoretically suggested values. The results are illustrated in Figure 1.

In Figure 1, we see AGD converges faster than GD, but the deviation in iterates is much larger. When introducing regularization, i.e., Reg-AGD, the reproducibility guarantee is greatly improved with only a small degradation in the convergence performance. It is worth mentioning that Reg-GD also has a smaller deviation bound compared to GD. All the results align with our theoretical analysis. Changing the inexactness level δ or the random seed for sampling the matrix A and the vector b does not influence the phenomenon too much, so we do not report the results with different selections.

Minimax. We also test the performance of gradient descent ascent (GDA), Extragradient (EG), and their regularized counterparts (Reg-GDA and Reg-EG) in Algorithm 2 on a bilinear matrix game

Here, A ∈ R d × d is generated the same as in the quadratic minimization example, X = { x ∈ R d | ∥ x ∥ ≤ D } and Y = { y ∈ R d | ∥ y ∥ ≤ D } are d -dimensional balls with diameter 2 D measured by the Euclidean norm. The projection onto these balls can be easily achieved. We implement an inexact gradient oracle that returns Ay + δe and A ⊤ x + δe for the partial gradients w.r.t. x and y respectively, where e ∈ R d is an all-one vector and δ ∈ R controls the inexactness level.

We test the aforementioned four algorithms with this inexact gradient oracle on both convergence performance measured by the duality gap (computable due to bounded domain) and reproducibility performance measured by the deviation compared to the trajectory obtained from using the true

Duality Gap

Deviation in Trajectory

102

102

101

10°

10-1,

EG

EG

101

GDA

10-2

GDA

Reg-EG

Reg-EG

10-3

Reg-GDA

Reg-GDA

100

101

102

103

104

100

101

102

103

104

# Iterations

# Iterations

gradient when δ = 0 . In the experiments, we let d = 500 and δ = 0 . 1 . For all four algorithms, we set the number of iterations to be T = 10000 , and the stepsize is 0.1 for EG, 0.05 for Reg-EG, 0.001 for GDA, and 0.0001 for Reg-GDA. The choice of stepsizes here adheres to our theoretical analysis noticing that the smoothness parameter is no larger than 10. Larger stepsize for GDA will make the trajectory easily diverge. For the regularization-based methods, we set the regularization parameter of the auxiliary problem to 0.05. The results are plotted in Figure 2. We see again the effectiveness of regularization to improve the reproducibility of the algorithms. Reg-EG largely reduces the deviation of EG even with respect to magnitude (note the figure is logarithmically scaled), with only a small degradation in terms of the convergence speed.

Peer reviews:
comments: SUMMARY:
This work proposes optimization algorithms that achieve optimal convergence rate and reproducibility for the settings of convex optimization and convex-concave minimax settings. This work settled some of the open questions from the previous work, and extend the results to the minimax setting.

STRENGTHS:
- Clear presentation.
- Novel technical contributions.
- Strong theoretical results and cleanly written proofs.

WEAKNESSES:
- Overall, the presentation is very clear, and the results are rigorous, and I didn't see any major weaknesses.
- Some details about Inexact-EG would have been helpful since it seems like a new method developed in this work. In particular, is it a direct extension of Devolder et al. to the minimax setting?

QUESTIONS:
- Regarding the O(delta^2/\eps^{2/5}) bound on the reproducibility, do you have some intuitions as to why improving upon this would be difficult? Even if it's a heuristic argument, it would help readers understand the main challenge.

LIMITATIONS:
- They discussed them well in the conclusion section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-01

comments: SUMMARY:
The authors proposed and studied optimization algorithms in both the convex and the convex-concave minimax setting, where both the criteria of convergence and reproducibility are measured. The authors provided upper bounds on these criteria for the proposed algorithms that matches nearly all of the lower bounds. 

While I'm not an expert on this subject (and I leave the judgement of correctness to other experts in this area), I believe this work has reached satisfying results, and I would recommend accept.

STRENGTHS:
1. The upper bounds on convergence and reproducibility matches nearly all of the known lower bounds. 
2. The presentation of the work is quite clean and readable.

WEAKNESSES:
N/A

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
The authors investigate the reproducibility problem of algorithms that solve convex and convex-concave minimax optimization problems.  They propose new methods with better reproducibility guarantees while maintaining the theoretical state-of-the-art convergence rates.

STRENGTHS:
*I want to acknowledge that I got this paper for review **after the deadline**, so I didn't have much time to check every detail, especially the proof.*

The idea of using regularization is good. It leads to new theoretical state-of-the-art convergence rates and reproducibility guarantees. This is a solid contribution to the NeurIPS community.

WEAKNESSES:
1. The new method requires $\epsilon$ and the distance $D$ between the starting point $x_0$ and $x^*.$ (e.g. Theorem 3.3.) I am not sure that the previous methods in Table 1 need these parameters. Do the authors discuss these important limitations?
2. Algorithm 1 is a well-known method in the optimization community. For instance, see https://arxiv.org/pdf/1603.05642.pdf. It is called "regularization technique" or "regularization reduction." I believe that the authors should cite the previous works that consider Algorithm 1.
3. Wrong citation [55] in Theorem 3.3. The paper [55] doesn't provide an analysis of AGD for strongly convex functions. It is better to cite any of Nesterov's books.

QUESTIONS:
.

LIMITATIONS:
.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of ensuring reproducibility in convex optimization. It builds on a recent framework for understanding reproducibility initiated by Ahn et al. The paper considers both minimization and minimax optimization (the latter being a new setting investigated in this work). The key results of the paper are the following:\n\n1. Minimization problems: The paper shows an improvement in the convergence-reproducibility tradeoffs under inexact initialization and inexact gradients compared to the results of Ahn et al. For inexact initialization, the paper shows that an L2 regularized version of AGD simultaneously obtains optimal convergence and optimal reproducibility. For inexact gradients, the same algorithm obtains sub-optimal reproducibility but with optimal convergence.\n\n2. Minimax optimization: L2 regularized versions of existing algorithms achieve optimal reproducibility and near-optimal gradient complexity. Similar to Ahn et al., SGD attains optimal reproducibility and convergence under a stochastic gradient oracle.\n\nSTRENGTHS:\n1. The problem of developing reproducible optimization algorithms is well-motivated and relevant. Various empirical studies have shown that randomness in initialization, training, data augmentation and numerical instabilities can lead to models which make significantly different predictions on test points.\n\n2. The paper demonstrates a valuable algorithmic principle of using L2 regularization to ensure reproducibility. All the results in the paper (apart from those for a stochastic gradient oracle) are obtained by incorporating L2 regularization into prior algorithms. The results strongly suggest that there could be deeper connections between stability and reproducibility, since L2 regularization is also a similarly useful techniques for ensuring algorithmic stability. Investigating this could be an interesting direction of future work.\n\n3. The paper improves stronger bounds across a number of settings compared to prior work. It also demonstrates that suspected instability issues of AGD are not a barrier to obtaining reproducibility guarantees for it. The paper also broadens the study of reproducibility to minimax optimization, with a similar message for it.\n\nWEAKNESSES:\n1. The writing of the paper is decent overall, but could do with some improvements. The paper does not motivate reproducibility adequately on its own. Some of the comments seem to concern reproducibility in science broadly rather than the particular issues which concern reproducibility in modern ML. Since the paper has a lot of different results, some more intuition behind the specific bounds that are obtained could be useful. For example, the paper could comment on the reproducibility obtained in different settings and why these bounds arise from the algorithm.\n\n2. Though I can understand that there is not too much space given the number of results, some intuition for the technical ideas which go into the bounds would be good as well. In particular, what are the main ideas behind extending reproducibility to minimax optimization? Does the intuition for why L2 regularization works for reproducibility in minimization mostly carry over and give the bounds for minimax optimization?\n\nQUESTIONS:\nOverall, this paper makes a good contribution on a relevant problem and I don't have any major concerns or questions. Some other suggestions are included above.\n\nLIMITATIONS:\nThese are discussed adequately.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of reproducibility in convex optimization. The notion of reproducibility, borrowed from prior work, measures the \"stability\" of the output of a procedure under noisy initialization or gradient computation. For the smooth convex setting, they design an algorithm based on running accelerated gradient descent on a regularized objective, which achieves optimal reproducibility and convergence rate. This answers an open question from prior work. The authors further extend their results to the minmax optimization setting deriving many new results.\n\nSTRENGTHS:\n1. Reproducibility has become an important topic in modern machine learning. Since (convex) optimization is the dominant algorithmic paradigm for modern ML, it is imporant to formulate and study reproducibility in optimization. The topic of the paper thus is important and timely.\n\n2. The paper obtains the optimal bounds on convergence and reproducibility for the smooth convex setting, something which prior work conjectured to be unattainable. This is an important contribution. Further, they managed to also get optimal, and non-trivial rates for the minmax optimization, a setting which has received considerable attention lately.\n\n 3. The paper is well written. Granted that it covers a lot of algorithms and results, the writing is to the point and the flow of ideas is natural.\n\nWEAKNESSES:\n1. About the definition of reproducibility: since this is a new field, I presume that the community has not yet agreed upon a definition. However, it seems to me that the only paper using the definition in this paper is the prior work of Ahn et al. Does adhering to this definition indeed reflect reprodubility in practice, in some sense? Even in optimization settings, there are other sources of instability not accounted for in the analysis, for instance, truncation and rounding due to the finite precision. Is adhering to reproducility (say with respect to initialization) and disregarding potential numerical instability arising in other steps give some meaningful in practice?\n\n2. Related to the above, some experiments demonstrating usefulness of the framework would strengthen the paper. In the current version, there are no experiments.\n\n3. The underlying idea is very simple and has been used many a times in (related) prior works -- regularization makes the problem strongly convex and thus aids leasds to (various forms of) stability. Nonetheless, the authors build on this to provide non-trivial bounds for many settings.\n\n4. Some technical details, which I presume are in prior work,  are not covered in the main text. Something that confused me is how to define  \"optimal reroducibility\", which is referred many a times in the paper. Some text explaining it, perhaps in the preliminaries will be helpful.\n\n5. What if we consider an inexact initialization as well as inexact grdient, with the same $\\delta$ say -- is it possible to say something about this from the algorithms proposed?\n\n6. The authors analyze a number of algorithms in the minmax setting, as a result this part of the paper looks rather dense with Thm statements.  Some organization of what is to come will help the reader. From my understanding, Alg3, Inexact Proximal Point Method, strictly improves over all others? If yes, this should be conveyed early on in this section.\n\nQUESTIONS:\nPlease answer the questions posed in the weekaneses.\n\nLIMITATIONS:\nThe work is limited to a certain notion of reproducility, used only in one prior work, and thus which may or may not be the definition as the area matures. Further, even though the authors identify two sources of instability: initialization and gradient computation, they are studied separately. A unified analysis can perhaps reflect more about the practical aspects.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce a novel algorithmic framework that can achieve near optimal convergence while preserving optimal reproducibility, for minimizing smooth convex objectives and minimax optimization of convex-concave objectives. Here, reproducibility under inexact initialization oracles, inexact deterministic gradient oracles, and inexact stochastic gradient oracles are considered. The key idea is to optimize a regularized strongly convex objective ( or Strongly convex - strongly concave objective for minimax optimization) using a given base algorithm, and bounding the error introduced by the regularization. The authors derive convergence and reproducibility guarantees that match or improve existing guarantees for different base algorithms applied to the proposed framework.\n\nSTRENGTHS:\n* The paper discuss about theoretically improving the trade-off between optimal convergence and optimal reproducibility of algorithms, which is an important emerging research area.\n\n* The paper is fairly easy to read, and the methods and results are presented in a clear manner.\n\n* Using the proposed algorithmic framework, the authors show, contrary to what was previously believed, that accelerated gradient descent (AGD) method can achieve near optimal convergence preserving optimal reproducibility, which seems like a non-trivial and interesting result.\n\nWEAKNESSES:\n* The paper covers only convex (convex-concave) objective minimization (minimax optimization), which prevents the results of this paper being applied to many applications where reproducibility is a challenge (e.g. reinforcement learning) as mentioned in the introduction of the paper.\n\n* The paper considers a constrained optimization setting for minimax optimization, while most applications of optimization, such as machine learning, will be deployed in an unconstrained setting. This might again prevent these results being applied to many practical applications.\n\n* The dependence of the convergence bounds on the diameter of $\\mathcal{X}$ and $\\mathcal{Y}$ in Theorems 4.4. and 4.6 makes the corresponding bounds too loose when $\\mathcal{X}$ and $\\mathcal{Y}$ are significantly large.\n\nMinor comments\n\n* Using $x^*_{r’}$ to denote  $\\underset{x\\in\\mathcal{X}}{\\operatorname{argmax}}\\\\{ F(x) + (r/2) \\Vert x - x_0'  \\Vert^2\\\\}$, it might suggest $x^*_{r’}$ corresponds to the optimum when $r’$ is used as the regularization parameter, which can be a bit confusing.\n\n* Introduction of Assumptions 3.1 and 4.1 seems abrupt, and some discussion on the assumption (e.g. their implications and how these assumptions compare with prior work) seems missing.\n\nQUESTIONS:\n* What kind of modifications to the proposed algorithmic framework or the assumptions on the objectives will allow this framework (or similar framework) being applied to the non-convex setting?\n\n* Intuitively, why it is necessary to consider the constrained optimization setting for minimax optimization when obtaining these results given using the proposed framework?\n\nLIMITATIONS:\nThis work only considers convex (convex-concave) setting, and some convergence results contain logarithmic factors, which are recognized by the authors in the conclusion. In addition to these, this work considers a constrained minimax optimization setting, which might limit the applicability of the corresponding results in practical applications.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}