Title: Achieving Near-Optimal Convergence for Distributed Minimax Optimization with Adaptive Stepsizes
Venue: NeurIPS 2024
Accepted: yes
Keywords: Minimax Optimization

Distributed Learning

Nonconvex Optimization

Convergence Analysis

Stepsize Inconsistency
Abstract: In this paper, we show that applying adaptive methods directly to distributed minimax problems can result in non-convergence due to inconsistency in locally computed adaptive stepsizes. To address this challenge, we propose D-AdaST, a Distributed Adaptive minimax method with Stepsize Tracking. The key strategy is to employ an adaptive stepsize tracking protocol involving the transmission of two extra (scalar) variables. This protocol ensures the consistency among stepsizes of nodes, eliminating the steady-state error due to the lack of coordination of stepsizes among nodes that commonly exists in vanilla distributed adaptive methods, and thus guarantees exact convergence. For nonconvex-strongly-concave distributed minimax problems, we characterize the specific transient times that ensure time-scale separation of stepsizes and quasi-independence of networks, leading to a near-optimal convergence rate of $\tilde{\mathcal{O}} \left( \epsilon ^{-\left( 4+\delta \right)} \right)$ for any small $\delta > 0$, matching that of the centralized counterpart. To our best knowledge, D-AdaST is the *first* distributed adaptive method achieving near-optimal convergence without knowing any problem-dependent parameters for nonconvex minimax problems. Extensive experiments are conducted to validate our theoretical results.

Achieving Near-Optimal Convergence for Distributed Minimax Optimization with Adaptive Stepsizes

Abstract

In this paper, we show that applying adaptive methods directly to distributed minimax problems can result in non-convergence due to inconsistency in locally computed adaptive stepsizes. To address this challenge, we propose D-AdaST, a Distributed Adaptive minimax method with Stepsize Tracking. The key strategy is to employ an adaptive stepsize tracking protocol involving the transmission of two extra (scalar) variables. This protocol ensures the consistency among stepsizes of nodes, eliminating the steady-state error due to the lack of coordination of stepsizes among nodes that commonly exists in vanilla distributed adaptive methods, and thus guarantees exact convergence. For nonconvex-strongly-concave distributed minimax problems, we characterize the specific transient times that ensure timescale separation of stepsizes and quasi-independence of networks, leading to a near-optimal convergence rate of ˜ O ( ϵ -(4+ δ ) ) for any small δ > 0 , matching that of the centralized counterpart. To our best knowledge, D-AdaST is the first distributed adaptive method achieving near-optimal convergence without knowing any problem-dependent parameters for nonconvex minimax problems. Extensive experiments are conducted to validate our theoretical results.

1 Introduction

Distributed optimization has seen significant research progress over the last decade, resulting in numerous algorithms (Nedic and Ozdaglar, 2009; Yuan et al., 2016; Lian et al., 2017; Pu and Nedi´ c, 2021). However, the traditional focus of distributed optimization has primarily been on minimization tasks. With the rapid growth of machine learning research, various applications have emerged that go beyond simple minimization, such as Generative Adversarial Networks (GANs) (Goodfellow et al., 2014; Gulrajani et al., 2017), robust optimization (Mohri et al., 2019; Sinha et al., 2017), adversary training of neural networks (Wang et al., 2021), fair machine learning (Madras et al., 2018), and just

Xiang Li

Niao He

to name a few. These tasks typically involve a minimax structure as follows:

where X ⊆ R p , Y ⊆ R d , and x, y are the primal and dual variables to be learned, respectively. One of the simplest yet effective methods for solving the above minimax problem is Gradient Descent Ascent (GDA) (Dem'yanov and Pevnyi, 1972; Nemirovski et al., 2009) which alternately performs stochastic gradient descent for the primal variable and stochastic gradient ascent for the dual variable. This approach has demonstrated its effectiveness in solving minimax problems, especially for convexconcave objectives (Hsieh et al., 2021; Daskalakis et al., 2021; Antonakopoulos et al., 2021), i.e., the function f ( · , y ) is convex for any y ∈ Y , and f ( x, · ) is concave for any x ∈ X .

Adaptive gradient methods, such as AdaGrad (Duchi et al., 2011), Adam (Kingma and Ba, 2014), and AMSGrad (Reddi et al., 2018), are often integrated with GDA to effectively solve minimax problems with theoretical guarantees in convex-concave settings (Diakonikolas, 2020; Antonakopoulos et al., 2021; Ene and Lê Nguyen, 2022). These adaptive methods are capable of adjusting stepsizes based on historical gradient information, making it robust to hyper-parameters tuning and can converge without requiring to know problem-dependent parameters (a characteristic often referred to as being 'parameter-agnostic"). However, in the nonconvex regime, it has been shown by Lin et al. (2020); Yang et al. (2022b) that it is necessary to have a time-scale separation in stepsizes between the minimization and maximization processes to ensure the convergence of GDA and GDA-based adaptive algorithms. In particular, the stepsize ratio between primal and dual variables needs to be smaller than a threshold depending on the properties of the problem such as the smoothness and strong-concavity parameters (Li et al., 2022; Guo et al., 2021; Huang et al., 2021), which are often unknown or difficult to estimate in real-world tasks, such as training deep neural networks.

Applying GDA-based adaptive methods into decentralized settings poses additional challenges due to the presence of inconsistency in locally computed adaptive stepsizes. In particular, it has been shown that the inconsistency of stepsizes can result in non-convergence in federated learning with heterogeneous computation speeds (Wang et al., 2020; Sharma et al., 2023). This is mainly due to the lack of a central node coordinating the stepsizes of nodes in distributed settings, making it difficult to converge, as observed in minimization problems (Liggett, 2022; Chen et al., 2023b). As a result, the following question arises naturally:

'Can we design an adaptive minimax method that ensures the time-scale separation and consistency of stepsizes with provable convergence in fully distributed settings?"

Contributions. In this paper, we aim to propose a distributed adaptive method for efficiently solving nonconvex-strongly-concave (NC-SC) minimax problems. The contributions are threefold:

We construct counterexamples showing that directly applying adaptive methods designed for centralized problems will lead to inconsistencies in locally computed adaptive stepsizes, resulting in non-convergence in distributed settings. To tackle this issue, we propose the first distributed adaptive minimax method, named D-AdaST, that incorporates an efficient stepsize tracking mechanism to maintain consistency across local stepsizes, which involves transmission of merely two extra (scalar) variables. The proposed algorithm exhibits timescale separation in stepsizes and parameter-agnostic capability in fully distributed settings.

Theoretically, we prove that D-AdaST is able to achieve a near-optimal convergence rate of ˜ O ( ϵ -(4+ δ ) ) with arbitrarily small δ > 0 to find an ϵ -stationary point for distributed NC-SC minimax problems. In contrast, we also prove the existence of a constant steady-state error in both the lower and upper bounds for GDA-based distributed minimax algorithms when being directly integrated with the adaptive stepsize rule without the stepsize tracking mechanism. Moreover, we explicitly characterize the transient times that ensure time-scale separation and quasi-independence of network, respectively.

We conduct extensive experiments on real-world datasets to verify our theoretical findings and the effectiveness of D-AdaST on a variety of tasks, including robust training of neural networks and optimizing Wasserstein GANs. In all tasks, we demonstrate the superiority of D-AdaST over several vanilla distributed adaptive methods across various graphs, initial stepsizes and data distributions (see also additional experiments in Appendix A).

1.1 Related Works

Distributed nonconvex minimax methods. In the realm of federated learning, Deng and Mahdavi (2021) introduce Local SGDA algorithm combining FedAvg/Local SGD with stochastic GDA and show an ˜ O ( ϵ -6 ) sample complexity for NC-SC objective functions. Sharma et al. (2022) provide improved complexity result of ˜ O ( ϵ -4 ) matching that of the lower bound of first-order algorithms for both NC-SC and nonconvex-Polyak-Lojasiewicz (NC-PL) settings (Li et al., 2021; Zhang et al., 2021a) . Yang et al. (2022a) integrate Local SGDA with stochastic gradient estimators to eliminate the data heterogeneity. More recently, Zhang et al. (2023) adopt compressed momentum methods with Local SGD to increase the communication efficiency of the algorithm. For decentralized nonconvex minimax problems, Liu et al. (2020) study the training of GANs using decentralized optimistic stochastic gradient and provide non-asymptotic convergence with fixed stepsizes. Tsaknakis et al. (2020) propose a double-loop decentralized SGDA algorithm with gradient tracking techniques (Pu and Nedi´ c, 2021) and achieve ˜ O ( ϵ -4 ) sample complexity. With a stronger assumption of average smoothness, some studies employ variance reduction techniques to accelerate convergence (Zhang et al., 2021b; Chen et al., 2022; Xian et al., 2021; Tarzanagh et al., 2022; Wu et al., 2023; Chen et al., 2024; Zhang et al., 2024), which require more memory and computational resources due to the need for larger batch-sizes or full gradient evaluations. However, all the above-mentioned methods use a fixed or uniformly decaying stepsize, requiring the prior knowledge of smoothness and concavity.

(Distributed) adaptive minimax methods. For centralized nonconvex minimax problems, Yang et al. (2022b) show that, even in deterministic settings, GDA-based methods necessitate the timescale separation of the stepsizes for primal and dual updates. Many attempts have been made for ensuring the time-scale separation requirement (Lin et al., 2020; Yang et al., 2022c; Bo¸ t and Böhm, 2023; Huang et al., 2023). However, these methods typically come with the prerequisite of having knowledge about problem-dependent parameters, which can be a significant drawback in practical scenarios. To this end, Yang et al. (2022b) introduce a nested adaptive algorithm named NeAda that achieves parameter-agnosticism by incorporating an inner loop to effectively maximize the dual variable, which can obtain an optimal sample complexity of ˜ O ( ϵ -4 ) when the strong-concavity parameter is known. More recently, Li et al. (2023) introduce TiAda, a single-loop parameter-agnostic adaptive algorithm for nonconvex minimax optimization which employs separated exponential factors on the adaptive primal and dual stepsizes, improving upon NeAda on the noise-adaptivity. There has been few works dedicated to adaptive minimax optimization in federated learning settings. For instance, Huang et al. (2024) introduces a federated adaptive algorithm that integrates the stepsize rule of Adam with full-client participation, resembling the centralized counterpart. Ju et al. (2023) study a federated Adam algorithm for fair federated learning where the objective function is properly weighted to account for heterogeneous updates among nodes. To the best of our knowledge, it is still unknown how one can design an adaptive minimax method capable of fulfilling the time-scale separation requirement and being parameter-agnostic in fully distributed settings .

Notations. Throughout this paper, we denote by E [ · ] the expectation of a random variable, ∥·∥ the Frobenius norm, ⟨· , ·⟩ the inner product of two vectors, ⊙ the Hadamard product (entry wise), ⊗ the Kronecker product. We denote by 1 the all-ones vector, I the identity matrix and J = 11 T /n the averaging matrix with n dimension. For a vector or matrix A and constant α , we denote A α the entry-wise exponential operations. We denote Φ ( x ) := f ( x, y ∗ ( x )) as the primal function where y ∗ ( x ) = argmax y ∈Y f ( x, y ) , and P Y ( · ) as the projection operation onto set Y .

2 Distributed Adaptive Minimax Methods

We consider the distributed minimax problem collaboratively solved by a set of agents over a network. The overall objective of the agents is to solve the following finite-sum problem:

where f i : R p + d → R is the local private loss function accessible only by the associated node i ∈ N = { 1 , 2 , · · · , n } , Y ⊂ R d is closed and convex, and ξ i ∼ D i denotes the data sample locally stored at node i ∈ N with distribution D i . We consider a graph G = ( V , E ) , here, V = { 1 , 2 , ..., n } represents the set of agents, and E ⊆ V × V denotes the set of edges consisting of ordered pairs ( i, j )

y

0.0

0.5

1.0

1.5

2.0

x

0

1

2

3

4

stationary points

D-SGDA

D-TiAda

D-AdaST

(a) trajectory

0

2000

4000

6000

8000

10000

Iterations

10 -5

10 -3

10 -1

10 1

10 3

||∇

x

f(x, y)||

2

D-SGDA

D-TiAda

D-AdaST

(b) convergence of

∥∇

x

f

(

x, y

)

∥

2

0

2000

4000

6000

8000

10000

Iterations

10 -8

10 -6

10 -4

10 -2

10 0

Inconsistency ζ 2

v

D-SGDA

D-TiAda

D-AdaST

(c) convergence of

ζ

2

v

representing the communication link from node j to node i . For node i , we define N i = { j | ( i, j ) ∈ E} as the set of its neighboring nodes. Before proceeding to the discussion of distributed algorithms, we first introduce the following notations for brevity:

where x i,k ∈ R p , y i,k ∈ Y denote the primal and dual variable of node i at each iteration k , and

are the corresponding partial stochastic gradients with i.i.d. samples ξ x k , ξ y k in a compact form.

Next, we will first explain the pitfalls of directly applying centralized adaptive stepsize rules to decentralized settings, and then introduce our newly proposed solution to address the challenge.

2.1 Non-Convergence of Direct Extensions

For the distributed minimax optimization problem as depicted in (1) involving NC-SC objective functions, we will show shortly that the Distributed Stochastic Gradient Descent Ascent (D-SGDA) method may not converge due to the inability of time-scale separation with constant stepsizes (c.f., Figure 1), which is also observed in centralized settings (Lin et al., 2020; Yang et al., 2022b). To address this issue, one can adopt the adaptive stepsize rule used in centralized TiAda (Li et al., 2023) for each individual node, which is renowned for its ability to adaptively fulfill the time-scale separation requirements. As a result, we arrive at the following Distributed TiAda (D-TiAda) algorithm.

where γ x and γ y are the stepsizes, W is a doubly-stochastic weight matrix induced by graph G (Xiao et al., 2006) (c.f., Assumption 4), and

with v i,k +1 = max { m x i,k +1 , m y i,k +1 } , u i,k +1 = m y i,k +1 , and

are the local accumulated gradient norm. Note that we impose a maximum operator in the preconditioner v i,k , and employ different stepsize decaying rates, i.e., 0 < β < α < 1 , for the primal and

dual variables, respectively. Such design allows to balance the updates of x and y , and achieves the desired time-scale separation without requiring any knowledge of parameters (Li et al., 2023).

However, in the distributed setting, such direct extension may fail to converge to a stationary point because v i,k and u i,k can be inconsistent due to the difference of local objective functions f i , In particular, we can rewrite the above vanilla distributed optimization algorithm (2) in the sense of average system of primal variables as below,

It is evident that, in comparison to centralized adaptive methods, an unexpected term (i.e., ˜ v k ) on the right-hand side (RHS) arises due to inconsistencies. This term introduces inaccuracies in the directions of gradient descent, degrading the optimization performance. The theorem presented below reveals a gap near the stationary points in a properly designed counterexample, indicating the non-convergence of D-TiAda. The proof is available in Appendix B.3.

Theorem 1. There exists a distributed minimax problem in the form of Problem (1) and certain initialization such that after running D-TiAda with any 0 < β < 0 . 5 < α < 1 and γ x , γ y > 0 , it holds that for any t = 0 , 1 , 2 , . . . , we have,

where ∥∇ x f ( x 0 , y 0 ) ∥ and ∥∇ y f ( x 0 , y 0 ) ∥ can be arbitrarily large depending on the initialization. Remark 1. The counterexample we constructed consists of three nodes, forming a complete graph. Without the stepsize tracking, D-TiAda will remain stationary, and the iterates will not progress if initiated along a specific line. In this counterexample, the only stationary point is at (0 , 0) , but initial points along the line (c.f., Eq. (72)) can be positioned arbitrarily far away from this stationary point, implying the non-convergence of D-TiAda with certain initialization.

Apart from the counterexample discussed in Theorem 1, we also experimentally observe the divergence of of D-SGDA and D-TiAda even in a simple scenario involving only two connected agents. This phenomenon is illustrated in Figure 1 and the functions are depicted as follows:

It is not difficult to verify that the points on the line 3 y = 5 x +2 are stationary points of f ( x, y ) = 1 / 2 ( f 1 ( x, y ) + f 2 ( x, y )) . It follows from Figure 1(a) and 1(b) that D-SGDA does not converge to a stationary point because of the lack of time-scale separation, and D-TiAda also fails to converge due to stepsize inconsistency, as shown in Figure 1(c). In contrast, the utilization of the stepsize tracking protocol in D-AdaST ensures convergence to a stationary point, with the inconsistency in stepsizes gradually diminishing (c.f., Lemma 9). These two motivating examples effectively highlight the challenges associated with applying adaptive minimax algorithms to distributed settings.

2.2 The Proposed D-AdaST Algorithm

To address the issue of stepsize inconsistency across different nodes, we propose the following Distributed Adaptive minimax optimization algorithm with Stepsize Tracking protocol, termed DAdaST, which allows us to asymptotically eliminate the stepsize inconsistency in a decentralized manner over networks. The pseudo-code for the algorithm is summarized in Algorithm 1, and can be rewritten in a compact form as follows:

Algorithm 1 Distributed Adaptive Minimax Method with Stepsize Tracking (D-AdaST)

Initialization: x i, 0 ∈ R p , y i, 0 ∈ Y , buffers m x i, 0 = m y i, 0 = c > 0 , stepsizes γ x , γ y > 0 , exponential factors 0 < β < α < 1 and weight matrix W .

1: for iteration k = 0 , 1 , · · · , each node i ∈ [ n ] , do

3: Accumulate the gradient norm:

4: Compute the ratio:

5: Update primal and dual variables locally:

6: Communicate adaptive stepsizes and decision variables with neighbors:

7: Projection of dual variable on the set Y : y i,k +1 ←P Y ( y i,k +1 ) .

8: end for

where m x k = [ · · · , m x i,k , · · · ] T , m y k = [ · · · , m y i,k , · · · ] T denote the tracking variables for the accumulated global gradient norm, i.e., for z ∈ { x, y } ,

while h z k = [ · · · , ∥ g z i,k ∥ 2 , · · · ] T , and V k , U k are diagonal matrices with v i,k = max { m x i,k , m y i,k } and u i,k = m x i,k . Note that we also provide a variant of D-AdaST with coordinate-wise adaptive stepsizes in Algorithm 2, along with its convergence analysis in Appendix B.5.

3 Convergence Analysis

In this section, we present the main convergence results for the proposed D-AdaST algorithm and compare it with D-TiAda to show the effectiveness of the proposed stepsize tracking protocol.

To this end, letting ¯ u k := 1 /n ∑ n i =1 u i,k , we define the following metrics to evaluate the level of inconsistency of stepsizes among nodes, which are ensured to be bounded by Assumption 3.

3.1 Assumptions

We consider the NC-SC setting of Problem (1) with the following assumptions that are commonly used in the existing works (c.f., Remark 2 and Remark 3). Notably, for the function and algorithm class determined by the assumptions of this work, Li et al. (2021) derived a lower complexity bound of Ω ( ϵ -4 ) and proved that such a dependency on ϵ is optimal (c.f., Remark 2).

Assumption 1 ( µ -strong concavity in y ) . Each objective function f i ( x, y ) is µ -strongly concave in y , i.e., ∀ x ∈ R p , ∀ y, y ′ ∈ Y and µ > 0 ,

Assumption 2 (Joint smoothness) . Each objective function f i ( x, y ) is L -smooth in x and y , i.e., ∀ x, x ′ ∈ R p and ∀ y, y ′ ∈ Y , there exists a constant L such that for z ∈ { x, y } ,

Furthermore, f i is second-order Lipschitz continuous for y , i.e., for z ∈ { x, y } ,

Remark 2. Assumption 1 does not require the convexity in x and the objective function thus can be nonconvex. Assumption 1 and 2 ensure that y ∗ ( · ) is smooth (c.f., Lemma 2), which is essential for achieving (near) optimal convergence rate (Chen et al., 2021; Li et al., 2023). Besides, it can be verified that the constructed 'hard' examples for obtaining the lower complexity bound in Li et al. (2021) satisfy the above second-order Lipschitz continuity (11) on y , implying that the achievable optimal complexity for the function and algorithm class considered in this work is O ( ϵ -4 ) .

Assumption 3 (Stochastic gradient) . For i.i.d. sample ξ i , the stochastic gradient of each i is unbiased, i.e., ∀ x ∈ R p , y ∈ Y , E ξ i [ ∇ z F i ( x, y ; ξ i )] = ∇ z f i ( x, y ) , for z ∈ { x, y } , and there is a constant C > 0 such that ∥∇ z F i ( x, y ; ξ i ) ∥ ⩽ C .

Remark 3. Assumption 3 on unbiased stochastic gradient is widely used for establishing convergence rates of both minimization and minimax optimization methods with AdaGrad (Kavis et al., 2022; Li et al., 2023) or Adam (Zou et al., 2019; Chen et al., 2023a; Huang et al., 2024) adaptive stepsize. We note that under Assumption 2, this assumption can be easily satisfied in many real-world tasks by imposing constraints on the compact domain of f , e.g., neural networks with rectified activation (Dinh et al., 2017) and GANs with projections on the critic (Gulrajani et al., 2017).

Next, we make the following assumption on the underlying graph to ensure its connectivity.

Assumption 4 (Graph connectivity) . The weight matrix W induced by graph G is doubly stochastic, i.e., W 1 = 1 , 1 T W = 1 T and ρ W := ∥ W -J ∥ 2 2 < 1 .

Note that one can always find a proper weight matrix W compliant to the graph that satisfies Assumption 4 once the underlying graph is undirected and connected. For instance, the weight matrix can be easily determined based on the Metropolis-Hastings protocol (Xiao et al., 2006). Moreover, this assumption is more general than that in Lian et al. (2017); Borodich et al. (2021) in the sense that W is not required to be symmetric, implying that certain directed graphs can be included in this assumption, e.g., directed ring and exponential graphs (Ying et al., 2021).

3.2 Main Results

We are now ready to present the key convergence results in terms of the primal function Φ ( x ) := f ( x, y ∗ ( x )) with y ∗ ( x ) = argmax y ∈Y f ( x, y ) , whose proofs can be found in Appendix B.4.

Theorem 2. Suppose Assumption 1-4 hold. Let 0 < β < α < 1 and the total iteration K satisfy

with κ := L/µ to ensure time-scale separation and quasi-independence of the network. For D-AdaST, we have 1

Remark 4 (Near-optimal convergence) . Theorem 2 implies that if the total number of iterations satisfies the conditions (12), the proposed D-AdaST algorithm converges to a stationary point exactly for Problem (1) with an ˜ O ( ϵ -(4+ δ ) ) sample complexity for arbitrarily small δ > 0 , e.g., letting

α = 0 . 5 + δ/ (8 + 2 δ ) and β = 0 . 5 -δ/ (8 + 2 δ ) . It is worth noting that this rate is near-optimal compared to the existing lower bound of Ω ( ϵ -4 ) (Li et al., 2021) for a class of smooth NC-SC functions. Moreover, this result recovers the centralized TiAda algorithm (Li et al., 2023) as a special case, i.e., setting ρ W = 0 , without assuming the existence of interior optimal point (c.f., Assumption 3.3 Li et al. (2023)). To the best of our knowledge, there is no existing fully parameter-agnostic method that achieves a convergence rate of ˜ O ( ϵ -4 ) , even in a centralized setting.

Remark 5 (Parameter-agnostic property and transient times) . The above results show that D-AdaST converges without requiring to know any problem-dependent parameters, i.e., L , µ and ρ W , or tuning the initial stepsize γ x and γ y , and is thus parameter-agnostic. Moreover, we explicitly characterize the transient times (c.f., Eq. (12)) that ensure time-scale separation and quasi-independence of the network, respectively. Indeed, we can see that if α and β are close to each other, the time required for time-scale separation to occur increases significantly, which has been observed in (Li et al., 2023). On the other hand, if α and β are relatively large, then ˜ O ( 1 /K 1 -α +1 /K 1 -β ) dominates the other terms, indicating independence on the network. These observations highlight the trade-offs between the convergence rate and the required duration of the transition phase.

For proper comparison, we also derive an upper bound for D-TiAda as follows. Together with the lower bound in Theorem 1, we demonstrate that without the stepsize tracking mechanism, the inconsistency among local stepsizes prevents D-TiAda from converging in the distributed setting.

Corollary 1. Under the same conditions of Theorem 2. For the proposed D-TiAda, we have

4 Experiments

In this section, we conduct experiments to validate the theoretical findings and demonstrate the effectiveness of the proposed algorithm on real-world machine learning tasks. We compare the proposed D-AdaST with the distributed variants of AdaGrad (Duchi et al., 2011), TiAda (Li et al., 2023) and NeAda (Yang et al., 2022b), namely D-AdaGrad, D-TiAda and D-NeAda, respectively. These experiments run across multiple nodes with different networks, and we consider heterogeneous distributions of local objective functions/datasets. For example, each node can only access samples with a subset of labels on MNIST and CIFAR-10 datasets, which is a common scenario in decentralized and federated learning tasks (Sharma et al., 2023; Huang et al., 2022). The experiments cover three main tasks: synthetic function, robust training of the neural network, and training of Wasserstein GANs (Heusel et al., 2017). For the exponential factors of stepsize, we set α = 0 . 6 and β = 0 . 4 for both D-TiAda and D-AdaST. More detailed settings and additional experiments with different initial stepsizes, data distributions and choices of α and β can be found in Appendix A.

Synthetic example. We consider a distributed minimax problem with the following NC-SC local objective functions over exponential networks with n = 50 ( ρ W = 0 . 71 ) and n = 100 ( ρ W = 0 . 75 ).

where L i ∼ U (1 . 5 , 2 . 5) . The local gradient of each node is computed with an additive N (0 , 0 . 1) Gaussian noise. It follows from Figure 2 (a) and 2 (b) that the proposed D-AdaST algorithm outperforms other distributed adaptive methods for both initial stepsize settings, especially in cases with a favorable initial stepsize ratio, as illustrated in plots (b) and (d) where γ x /γ y = 0 . 2 . Similar observation can be found in Figure 2 (c) and 2 (d), demonstrating the effectiveness of D-AdaST.

Robust training of neural networks. Next, we consider the task of robust training of neural networks, in the presence of adversarial perturbations on data samples (Sharma et al., 2022; Deng and Mahdavi, 2021). The problem can be formulated as min x max y 1 /n ∑ n i =1 f i ( x ; ξ i + y ) -η ∥ y ∥ 2 , where x denotes the parameters of the model, y denotes the perturbation and ξ i denotes the data sample of node i . Note that if η is large enough, the problem is NC-SC. We conduct experiments on

0

1000

2000

3000

4000

# Gradient calls

10 -4

10 -3

10 -2

10 -1

10 0

10 1

10 2

||∇

x

f(x, y)||

2

D-AdaGrad

D-NeAda

D-TiAda

D-AdaST

(a)

γ

x

= 0

.

1

, n

= 50

0

1000

2000

3000

4000

# Gradient calls

10 -4

10 -3

10 -2

10 -1

10 0

10 1

10 2

||∇

x

f(x, y)||

2

D-AdaGrad

D-NeAda

D-TiAda

D-AdaST

(b)

γ

x

= 0

.

02

, n

= 50

0

1000

2000

3000

4000

# Gradient calls

10 -4

10 -3

10 -2

10 -1

10 0

10 1

10 2

||∇

x

f(x, y)||

2

D-AdaGrad

D-NeAda

D-TiAda

D-AdaST

(c)

γ

x

= 0

.

1

, n

= 100

0

1000

2000

3000

4000

# Gradient calls

10 -4

10 -3

10 -2

10 -1

10 0

10 1

10 2

||∇

x

f(x, y)||

2

D-AdaGrad

D-NeAda

D-TiAda

D-AdaST

(d)

γ

x

= 0

.

02

, n

= 100

0

1000

2000

3000

4000

# Gradient calls

10 -1

10 0

10 1

||∇

x

f(x, y)||

2

D-AdaGrad

D-NeAda

D-TiAda

D-AdaST

(a) ring,

ρ

W

= 0

.

97

0

1000

2000

3000

4000

# Gradient calls

10 -1

10 0

10 1

||∇

x

f(x, y)||

2

D-AdaGrad

D-NeAda

D-TiAda

D-AdaST

(b) exp.,

ρ

W

= 0

.

67

0

1000

2000

3000

4000

# Gradient calls

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-AdaGrad

D-NeAda

D-TiAda

D-AdaST

(c) dense,

ρ

W

= 0

.

55

0

4000

8000

12000

16000

# Gradient calls

10 -1

10 0

10 1

||∇

x

f(x, y)||

2

D-AdaST, n = 2

D-AdaST, n = 8

D-AdaST, n = 32

(d) scalability

0

1000

2000

3000

4000

5000

6000

# Gradient calls

10 -2

10 -1

10 0

10 1

∇

x

f(x, y)

2

D-Adam

D-NeAda-Adam

D-TiAda-Adam

D-AdaST-Adam

(e) ring,

ρ

W

= 0

.

97

0

1000

2000

3000

4000

5000

6000

# Gradient calls

10 -2

10 -1

10 0

10 1

∇

x

f(x, y)

2

D-Adam

D-NeAda-Adam

D-TiAda-Adam

D-AdaST-Adam

(f) exp.,

ρ

W

= 0

.

67

0

1000

2000

3000

4000

5000

6000

# Gradient calls

10 -2

10 -1

10 0

10 1

∇

x

f(x, y)

2

D-Adam

D-NeAda-Adam

D-TiAda-Adam

D-AdaST-Adam

(g) dense,

ρ

W

= 0

.

55

0

2000

4000

6000

8000

# Gradient calls

10 -1

10 0

10 1

||∇

x

f(x, y)||

2

D-AdaST-Adam, n = 4

D-AdaST-Adam, n = 16

D-AdaST-Adam, n = 64

(h) scalability

MNIST dataset over different networks, e.g., ring graph, exponential (exp.) graph (Ying et al., 2021) and dense graph with n/ 2 edges for each node. We consider a heterogeneous scenario in which each node possesses only two distinct classes of labeled samples, resulting in heterogeneity among the local datasets across nodes, while the data is i.i.d within each node.

In Figure 3, we compare D-AdaST with D-AdaGrad, D-TiAda and D-NeAda, using adaptive stepsizes in AdaGrad (first row) and Adam (second row, name suffixed with Adam) respectively, it can be observed from the first three columns that the proposed D-AdaST outperforms the others on three different graphs and it is not very sensitive to the graph connectivity (i.e., ρ W ), demonstrating the quasi-independence of network as indicated in Theorem 2. It should be noted that Adam-like algorithms exhibit more fluctuations in the later stages of optimization as the gradient norm vanishes, leading to an inevitable increase in the Adam stepsize as the optimization process converges (Kingma and Ba, 2014). In plots (d) and (h), we further demonstrate that D-AdaST can scale efficiently with respect to the number of nodes, while keeping a constant batch-size of 64 for each node. This showcases the algorithm's ability to handle large-scale distributed scenarios effectively.

Generative Adversarial Networks. We further illustrate the effectiveness of D-AdaST on another popular task of training GANs, which has a generator and a discriminator used to generate and distinguish samples respectively (Goodfellow et al., 2014). In this experiment, we train Wasserstein GANs(Gulrajani et al., 2017) on CIFAR-10 dataset in a decentralized setting where each discriminator is 1-Lipschitz and has access to only two classes of samples. We compare the inception score of D-AdaST with D-Adam and D-TiAda adopting Adam-like stepsizes in Figure 4. It can be observed from the figure that D-AdaST achieves higher inception scores in three cases with different initial stepsizes, and has a small score loss as the initial step size changes. We believe that this example shows the great potential of D-AdaST in solving real-world problems.

0

20000

40000

60000

80000

# Gradient calls

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Inception score

D-Adam

D-TiAda-Adam

D-AdaST-Adam

(a)

γ

x

=

γ

y

= 0

.

001

0

20000

40000

60000

80000

# Gradient calls

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Inception score

D-Adam

D-TiAda-Adam

D-AdaST-Adam

(b)

γ

x

=

γ

y

= 0

.

01

0

20000

40000

60000

80000

# Gradient calls

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Inception score

D-Adam

D-TiAda-Adam

D-AdaST-Adam

(c)

γ

x

=

γ

y

= 0

.

05

5 Conclusion

We introduced a new distributed adaptive minimax method, D-AdaST, designed to tackle the issue of non-convergence in nonconvex-strongly-concave minimax problems caused by the inconsistencies among locally computed adaptive stepsizes. Vanilla distributed adaptive methods could suffer from such inconsistencies, as highlighted by the carefully designed counterexamples for demonstrating their potential non-convergence. In contrast, our proposed method employs an efficient adaptive stepsize tracking protocol that not only ensures the time-scale separation, but also guarantees stepsize consistency among nodes and thus effectively eliminates steady-state errors. Theoretically, we showed that D-AdaST can achieve a near-optimal convergence rate of ˜ O ( ϵ -(4+ δ ) ) with any arbitrarily small δ > 0 . Extensive experiments on both real-world and synthetic datasets have been conducted to validate our theoretical findings across various scenarios.

Acknowledgments

The work of Huang, Shen and Xu has been supported by the National Key R&D Program of China under Grant No. 2022YFB3102100, and in parts by National Natural Science Foundation of China under Grants 62373323, 62088101. The work of Li and He has been supported by the ETH research grant and Swiss National Science Foundation (SNSF) Starting Grant.

References

Antonakopoulos, K., Belmega, V. E., and Mertikopoulos, P. (2021). Adaptive extra-gradient methods for min-max optimization and games. In ICLR 2021-9th International Conference on Learning Representations , pages 1-28.

Borodich, E., Beznosikov, A., Sadiev, A., Sushko, V., Savelyev, N., Takáˇ c, M., and Gasnikov, A. (2021). Decentralized personalized federated min-max problems. arXiv preprint arXiv:2106.07289 .

Bo¸ t, R. I. and Böhm, A. (2023). Alternating proximal-gradient steps for (stochastic) nonconvexconcave minimax problems. SIAM Journal on Optimization , 33(3):1884-1913.

Chen, C., Shen, L., Liu, W., and Luo, Z.-Q. (2023a). Efficient-adam: Communication-efficient distributed adam. IEEE Transactions on Signal Processing .

Chen, L., Ye, H., and Luo, L. (2022). A simple and efficient stochastic algorithm for decentralized nonconvex-strongly-concave minimax optimization. arXiv preprint arXiv:2212.02387 .

Chen, L., Ye, H., and Luo, L. (2024). An efficient stochastic algorithm for decentralized nonconvexstrongly-concave minimax optimization. In International Conference on Artificial Intelligence and Statistics , pages 1990-1998. PMLR.

Chen, T., Sun, Y., and Yin, W. (2021). Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. Advances in Neural Information Processing Systems , 34:25294-25307.

Chen, X., Karimi, B., Zhao, W., and Li, P. (2023b). On the convergence of decentralized adaptive gradient methods. In Asian Conference on Machine Learning , pages 217-232. PMLR.

Daskalakis, C., Skoulakis, S., and Zampetakis, M. (2021). The complexity of constrained minmax optimization. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 1466-1478.

Dem'yanov, V. F. and Pevnyi, A. B. (1972). Numerical methods for finding saddle points. USSR Computational Mathematics and Mathematical Physics , 12(5):11-52.

Deng, Y. and Mahdavi, M. (2021). Local stochastic gradient descent ascent: Convergence analysis and communication efficiency. In International Conference on Artificial Intelligence and Statistics , pages 1387-1395. PMLR.

Diakonikolas, J. (2020). Halpern iteration for near-optimal and parameter-free monotone inclusion and strong solutions to variational inequalities. In Conference on Learning Theory , pages 14281451. PMLR.

Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. (2017). Sharp minima can generalize for deep nets. In International Conference on Machine Learning , pages 1019-1028. PMLR.

Duchi, J., Hazan, E., and Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research , 12(7).

Ene, A. and Lê Nguyen, H. (2022). Adaptive and universal algorithms for variational inequalities with optimal convergence. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 6559-6567.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances in neural information processing systems , 27.

Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. (2017). Improved training of wasserstein gans. Advances in neural information processing systems , 30.

Guo, Z., Xu, Y., Yin, W., Jin, R., and Yang, T. (2021). A novel convergence analysis for algorithms of the adam family. arXiv preprint arXiv:2112.03459 .

Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017). Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30.

Hsieh, Y.-P., Mertikopoulos, P., and Cevher, V. (2021). The limits of min-max optimization algorithms: Convergence to spurious non-critical sets. In International Conference on Machine Learning , pages 4337-4348. PMLR.

Huang, F., Wang, X., Li, J., and Chen, S. (2024). Adaptive federated minimax optimization with lower complexities. In International Conference on Artificial Intelligence and Statistics , pages 4663-4671. PMLR.

Huang, F., Wu, X., and Hu, Z. (2023). Adagda: Faster adaptive gradient descent ascent methods for minimax optimization. In International Conference on Artificial Intelligence and Statistics , pages 2365-2389. PMLR.

Huang, F., Wu, X., and Huang, H. (2021). Efficient mirror descent ascent methods for nonsmooth minimax problems. Advances in Neural Information Processing Systems , 34:10431-10443.

Huang, Y., Sun, Y., Zhu, Z., Yan, C., and Xu, J. (2022). Tackling data heterogeneity: A new unified framework for decentralized SGD with sample-induced topology. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 9310-9345. PMLR.

Ju, L., Zhang, T., Toor, S., and Hellander, A. (2023). Accelerating fair federated learning: Adaptive federated adam. arXiv preprint arXiv:2301.09357 .

Kavis, A., Levy, K. Y., and Cevher, V. (2022). High probability bounds for a class of nonconvex algorithms with adagrad stepsize. In International Conference on Learning Representations .

Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 .

Li, H., Farnia, F., Das, S., and Jadbabaie, A. (2022). On convergence of gradient descent ascent: A tight local analysis. In International Conference on Machine Learning , pages 12717-12740. PMLR.

Li, H., Tian, Y., Zhang, J., and Jadbabaie, A. (2021). Complexity lower bounds for nonconvexstrongly-concave min-max optimization. Advances in Neural Information Processing Systems , 34:1792-1804.

Li, X., YANG, J., and He, N. (2023). Tiada: A time-scale adaptive algorithm for nonconvex minimax optimization. In The Eleventh International Conference on Learning Representations .

Lian, X., Zhang, C., Zhang, H., Hsieh, C.-J., Zhang, W., and Liu, J. (2017). Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in Neural Information Processing Systems , 30.

Liggett, B. (2022). Distributed learning with automated stepsizes.

Lin, T., Jin, C., and Jordan, M. (2020). On gradient descent ascent for nonconvex-concave minimax problems. In International Conference on Machine Learning , pages 6083-6093. PMLR.

Liu, M., Zhang, W., Mroueh, Y., Cui, X., Ross, J., Yang, T., and Das, P. (2020). A decentralized parallel algorithm for training generative adversarial nets. Advances in Neural Information Processing Systems , 33:11056-11070.

Madras, D., Creager, E., Pitassi, T., and Zemel, R. (2018). Learning adversarially fair and transferable representations. In International Conference on Machine Learning , pages 3384-3393. PMLR.

Mohri, M., Sivek, G., and Suresh, A. T. (2019). Agnostic federated learning. In International Conference on Machine Learning , pages 4615-4625. PMLR.

Nedic, A. and Ozdaglar, A. (2009). Distributed subgradient methods for multi-agent optimization. IEEE Transactions on Automatic Control , 54(1):48-61.

Nedic, A., Ozdaglar, A., and Parrilo, P. A. (2010). Constrained consensus and optimization in multi-agent networks. IEEE Transactions on Automatic Control , 55(4):922-938.

Nemirovski, A., Juditsky, A., Lan, G., and Shapiro, A. (2009). Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization , 19(4):1574-1609.

Pu, S. and Nedi´ c, A. (2021). Distributed stochastic gradient tracking methods. Mathematical Programming , 187(1):409-457.

Reddi, S. J., Kale, S., and Kumar, S. (2018). On the convergence of adam and beyond. In International Conference on Learning Representations .

Sharma, P., Panda, R., and Joshi, G. (2023). Federated minimax optimization with client heterogeneity. arXiv preprint arXiv:2302.04249 .

Sharma, P., Panda, R., Joshi, G., and Varshney, P. (2022). Federated minimax optimization: Improved convergence analyses and algorithms. In International Conference on Machine Learning , pages 19683-19730. PMLR.

Sinha, A., Namkoong, H., Volpi, R., and Duchi, J. (2017). Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv:1710.10571 .

Tarzanagh, D. A., Li, M., Thrampoulidis, C., and Oymak, S. (2022). Fednest: Federated bilevel, minimax, and compositional optimization. In International Conference on Machine Learning , pages 21146-21179. PMLR.

Tsaknakis, I., Hong, M., and Liu, S. (2020). Decentralized min-max optimization: Formulations, algorithms and applications in network poisoning attack. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5755-5759. IEEE.

Wang, J., Liu, Q., Liang, H., Joshi, G., and Poor, H. V. (2020). Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems , 33:7611-7623.

Wang, J., Zhang, T., Liu, S., Chen, P.-Y., Xu, J., Fardad, M., and Li, B. (2021). Adversarial attack generation empowered by min-max optimization. Advances in Neural Information Processing Systems , 34:16020-16033.

Wu, X., Sun, J., Hu, Z., Zhang, A., and Huang, H. (2023). Solving a class of non-convex minimax optimization in federated learning. In Thirty-seventh Conference on Neural Information Processing Systems .

Xian, W., Huang, F., Zhang, Y., and Huang, H. (2021). A faster decentralized algorithm for nonconvex minimax problems. Advances in Neural Information Processing Systems , 34:25865-25877.

Xiao, L., Boyd, S., and Lall, S. (2006). Distributed average consensus with time-varying metropolis weights. Automatica , 1:1-4.

Yang, H., Liu, Z., Zhang, X., and Liu, J. (2022a). Sagda: Achieving O ( ε -2 ) communication complexity in federated min-max learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Advances in Neural Information Processing Systems , volume 35, pages 7142-7154. Curran Associates, Inc.

Yang, J., Li, X., and He, N. (2022b). Nest your adaptive algorithm for parameter-agnostic nonconvex minimax optimization. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K., editors, Advances in Neural Information Processing Systems .

Yang, J., Orvieto, A., Lucchi, A., and He, N. (2022c). Faster single-loop algorithms for minimax optimization without strong concavity. In International Conference on Artificial Intelligence and Statistics , pages 5485-5517. PMLR.

Ying, B., Yuan, K., Chen, Y., Hu, H., Pan, P., and Yin, W. (2021). Exponential graph is provably efficient for decentralized deep training. Advances in Neural Information Processing Systems , 34:13975-13987.

Yuan, K., Ling, Q., and Yin, W. (2016). On the convergence of decentralized gradient descent. SIAM Journal on Optimization , 26(3):1835-1854.

Zhang, S., Choudhury, S., Stich, S. U., and Loizou, N. (2023). Communication-efficient gradient descent-accent methods for distributed variational inequalities: Unified analysis and local updates. arXiv preprint arXiv:2306.05100 .

Zhang, S., Yang, J., Guzmán, C., Kiyavash, N., and He, N. (2021a). The complexity of nonconvexstrongly-concave minimax optimization. In Uncertainty in Artificial Intelligence , pages 482-492. PMLR.

Zhang, X., Liu, Z., Liu, J., Zhu, Z., and Lu, S. (2021b). Taming communication and sample complexities in decentralized policy evaluation for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems , 34:18825-18838.

Zhang, X., Mancino-Ball, G., Aybat, N. S., and Xu, Y. (2024). Jointly improving the sample and communication complexities in decentralized stochastic minimax optimization. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 20865-20873.

Zhou, D., Chen, J., Cao, Y., Tang, Y., Yang, Z., and Gu, Q. (2018). On the convergence of adaptive gradient methods for nonconvex optimization. arXiv preprint arXiv:1808.05671 .

Zou, F., Shen, L., Jie, Z., Zhang, W., and Liu, W. (2019). A sufficient condition for convergences of adam and rmsprop. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages 11127-11135.

A Additional Experiments

In this section, we provide detailed experimental settings and perform additional experiments on the task of training robust neural networks with different choices of hyper-parameters. All experiments are deployed in a server with Intel Xeon E5-2680 v4 CPU @ 2.40GHz and 8 Nvidia RTX 3090 GPUs, and implemented using distributed communication package torch.distributed in PyTorch 2.0, where each process serves as a node, and we use inter-process communication to mimic communication between nodes. For the AdaGrad-like algorithms considered in the experiments of training neural networks, similar to the Adam-like stepsize, we adopt a coordinate-wise adaptive stepsize rule as commonly used in existing centralized adaptive methods (Yang et al., 2022b; Li et al., 2023). Moreover, since we attempt to develop a parameter-agnostic algorithm that does not need much effort in tuning hyper-parameters, we set α = 0 . 6 and β = 0 . 4 for all tasks in the main text, and evaluate the effect of the choices of α and β on the performance of D-AdaST individually in an additional experiment on the synthetic objective function as shown in Appendix A.4.

A.1 Experimental details

Communication topology. For the experiments in the main text, we utilize three commonly used communication topologies: indirect ring, exponential graph and dense graph. An indirect ring is a sparse graph in which each node is sequentially connected to form a ring, with only two neighbors per node. Exponential graph (Ying et al., 2021) is a directed graph where each node is connected to nodes at distances of 2 0 , 2 1 ..., 2 log n . Exponential graphs achieve a good balance between the degree and connectivity of the graph. A dense graph is an indirect graph where each node is connected to nodes at distances of 1 , 2 , 4 , ..., n . We also consider directed ring and fully connected graphs, which are more sparsely and densely connected, respectively, in the additional experiments.

Robust training of neural network. In this task, we train CNNs with three convolutional layers and one fully connected layer on MNIST dataset containing images of 10 classes. Each layer adopts batch normalization and ELU activation. The total batch-size is 1280, and the batch-size of each node during training is 1280 /n . For Adam-like algorithms, we set the first and second moment parameters as β 1 = 0 . 9 , β 2 = 0 . 999 respectively. Since NeAda is a double-loop algorithm, for fair comparison, we imply D-AdaGrad and D-Adam using 15 iterations of inner loop in this task.

Generative Adversarial Networks. In this task, we train Wasserstein GANs on CIFAR-10 dataset, where the model used for discriminator is a four layer CNNs, and for generator is a four layer CNNs with transpose convolution layers. The total batch-size is 1280, and the batch-size of each node during training is 128 with 10 nodes. For Adam-like algorithms, we use β 1 = 0 . 5 , β 2 = 0 . 9 . To obtain the inception score, we use 8000 artificially generated samples to feed the previously trained inception network.

A.2 Additional experiments on robust training of neural network.

In this part, we conduct additional experiments on robust training of CNNs on MNIST dataset considering a variety of settings. We compare the convergence performance of D-AdaST with D-AdaGrad, D-TiAda and D-NeAda using adaptive stepsizes of AdaGrad and Adam. Unless otherwise specified, the total batch-size is set to 1280; the initial stepsizes for x and y are assigned as γ x = 0 . 01 , γ y = 0 . 1 for AdaGrad-like algorithms, and γ x = γ y = 0 . 1 for Adam-like algorithms. Specifically, we consider two extra graphs that are more sparse and more dense, respectively in Figure 5, e.g., directed ring and fully-connected (fc) graphs. We consider more initial stepsizes settings for x and y respectively in Figure 6. Further, we also consider different data distributions where each node has samples from 4 of the 10 classes in Figure 7. Finally, we perform a comparison experiment with 40 nodes in Figure 8. Under all settings, the proposed D-AdaST outperforms the others, demonstrating the superiority of D-AdaST.

A.3 Additional experiments on training GANs

We provide additional experiments of training GANs on a more complicated dataset CIFAR-100 to further illustrate the effectiveness of the proposed D-AdaST, as shown in Figure 9. We use the entire training set of CIFAR-100 with coarse labels (20 classes) to train GANs over networks, where each node is assigned with four distinct classes of labeled samples. Under the same settings as in

0

1000

2000

3000

4000

# Gradient calls

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-AdaGrad

D-NeAda

D-TiAda

D-AdaST

(a) directed-ring

0

1000

2000

3000

4000

# Gradient calls

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-AdaGrad

D-NeAda

D-TiAda

D-AdaST

(b) fc

0

1000

2000

3000

4000

5000

6000

10 -2

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-Adam

D-NeAda-Adam

D-TiAda-Adam

D-AdaST-Adam

(c) directed-ring

0

1000

2000

3000

4000

5000

6000

10 -2

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-Adam

D-NeAda-Adam

D-TiAda-Adam

D-AdaST-Adam

(d) fc

0

1000

2000

3000

4000

# Gradient calls

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-AdaGrad

D-NeAda

D-TiAda

D-AdaST

(a) 0.01, 0.01

0

1000

2000

3000

4000

# Gradient calls

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-AdaGrad

D-NeAda

D-TiAda

D-AdaST

(b) 0.001, 0.01

0

1000

2000

3000

4000

5000

6000

10 -3

10 -2

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-Adam

D-NeAda-Adam

D-TiAda-Adam

D-AdaST-Adam

(c) 0.01, 0.01

0

1000

2000

3000

4000

5000

6000

10 -3

10 -2

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-Adam

D-NeAda-Adam

D-TiAda-Adam

D-AdaST-Adam

(d) 0.01, 0.1

0

1000

2000

3000

4000

# Gradient calls

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-AdaGrad

D-NeAda

D-TiAda

D-AdaST

(a) exp.

0

1000

2000

3000

4000

# Gradient calls

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-AdaGrad

D-NeAda

D-TiAda

D-AdaST

(b) dense

0

1000

2000

3000

4000

5000

6000

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-Adam

D-NeAda-Adam

D-TiAda-Adam

D-AdaST-Adam

(c) exp.

0

1000

2000

3000

4000

5000

6000

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-Adam

D-NeAda-Adam

D-TiAda-Adam

D-AdaST-Adam

(d) dense

0

1000

2000

3000

4000

# Gradient calls

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-AdaGrad

D-NeAda

D-TiAda

D-AdaST

(a) exp.

0

1000

2000

3000

4000

# Gradient calls

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-AdaGrad

D-NeAda

D-TiAda

D-AdaST

(b) dense

0

1000

2000

3000

4000

5000

6000

10 -2

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-Adam

D-NeAda-Adam

D-TiAda-Adam

D-AdaST-Adam

(c) exp.

0

1000

2000

3000

4000

5000

6000

10 -2

10 -1

10 0

10 1

||∇ x f(x, y)|| 2

D-Adam

D-NeAda-Adam

D-TiAda-Adam

D-AdaST-Adam

(d) dense

Figure 4 (a), it can be observed that D-AdaST outperforms the others in terms of the inception score. Together with other experimental results in the main text, we believe that we have demonstrated the effectiveness of the proposed D-AdaST method and its potential for further real-world applications.

0

7500

15000

22500

30000

# Gradient calls

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Inception score

D-Adam

D-TiAda-Adam

D-AdaST-Adam

0

2000

4000

6000

8000

# Gradient calls

0

20

40

60

80

100

∇

x

f(x, y)

2

D-AdaST: α = 0.6, β = 0.4

D-AdaST: α = 0.62, β = 0.38

D-AdaST: α = 0.64, β = 0.36

D-AdaST: α = 0.66, β = 0.34

A.4 Additional experiments with different choices of α and β

In this part, we evaluate the effect of the choices of α and β on the performance of D-AdaST. In particular, we provide an additional experimental result on the synthetic quadratic objective functions (15) with a larger ratio of initial stepsizes, i.e., γ x /γ y = 20 (indicating faster minimization and slower maximization processes at the beginning). As shown in Figure 10, it can be observed that the transient time (iteration before the inflection point) becomes longer as α -β decreases, while the convergence rate is relatively faster, which is consistent with Theorem 2 and the result in the centralized TiAda algorithm (c.f., Figure 5, Li et al., 2023).

B Proof of the main results

We recall here some definitions used in the main text. The averaged variables and the inconsistency are defined as follows:

The inconsistency of stepsizes of the primal and dual variables is defined as follows:

Proof Sketch. The convergence analysis of the main results in Theorem 2 is mainly based on carefully analyzing the average system as shown in (5), and the difference between the distributed system and the averaged system. In general, under Assumption 1-4, we first give a telescoped descent lemma from 0 to K -1 iterations in Lemma 3, which is upper bounded by the following key error terms:

S 1 := 1 nK ∑ K -1 k =0 E [ ¯ v -α k +1 ∥∇ x F ( x k , y k ; ξ x k ) ∥ 2 ] : The asymptotically decaying terms by adopting adaptive stepsize;

S 2 := 1 nK ∑ K -1 k =0 E [ ∥ x k -1 ¯ x k ∥ 2 + ∥ y k -1 ¯ y k ∥ 2 ] : The consensus error of x and y between the distributed system and the average system;

S 3 := 1 K ∑ K -1 k =0 E [ f (¯ x k , y ∗ (¯ x k )) -f (¯ x k , ¯ y k )] : The optimality gap in dual variable y ;

Next, we prove the contraction properties of these terms in Lemma 4-8 and Lemma 9 respectively. Finally, these results are integrated into the descent lemma to complete the proof. We note that the proof is not trivial in the sense that these terms are coupled and therefore are needed to be carefully analyzed. This proof can also be adapted to analyze the coordinate-wise adaptive stepsize variant of D-AdaST as explained in Appendix B.5, which is of independent interest.

B.1 Supporting lemmas

In this part, we provide several supporting lemmas that have been shown in the existing literature, which are essential to the subsequent convergence analysis.

Lemma 1 (Lemma A.2 in Yang et al. (2022b)) . Let { x t } T -1 t =0 be a sequence of non-negative real numbers, x 0 > 0 and α ∈ (0 , 1) . Then we have,

When α = 0 , we have

Lemma 2. Suppose Assumption 1 and 2 hold. Define Φ ( x ) := f ( x, y ∗ ( x )) as the envelope function and y ∗ ( x ) = argmax y ∈Y f ( x, y ) . Then, we have,

Φ ( · ) is L Φ -smooth with L Φ = L (1 + κ ) , and ∇ Φ ( x ) = ∇ x f ( x, y ∗ ( x )) (c.f., Lemma 4.3 in Lin et al. (2020));

y ∗ ( · ) is κ -Lipschitz and ˆ L -smooth with ˆ L = κ (1 + κ ) 2 (c.f., Lemma 2 in Chen et al. (2021)).

B.2 Key Lemmas

In this subsection, we give the key lemmas to help the analysis of the main results. For simplicity, we define ∆ k := ∥ x k -1 ¯ x k ∥ 2 + ∥ y k -1 ¯ y k ∥ 2 as the consensus error for primal and dual variables. Then, we have the following lemmas.

Lemma 3 (Descent lemma) . Suppose Assumption 1-4 hold. Then, we have

where κ := L/µ is the condition number of the function in y , Φ max = max x Φ ( x ) , Φ ∗ = min x Φ ( x ) .

Proof. By the smoothness of Φ given in Lemma 2, i.e.,

and noticing that the scalar ¯ v k , ¯ u k are random variables, we have

where we have used the definition of ¯ x k +1 as presented in (5). Then, we bound the inner-product terms on the RHS. Firstly,

wherein the last inequality we have used the smoothness of the objective functions. Then, for the second inner-product in (19), using Young's inequality we get

Then, for the last term on the RHS of (18), recalling the definition of stepsize inconsistency in (8), we have

Plugging the obtained inequalities into (18) and telescoping the terms, we get

Now it remains to bound the first term on the RHS of the above inequality. With the help of Assumption 3, we have

Next, we need to bound the last four terms S 1 -S 4 in (18) respectively. For S 1 , we have the asymptotic convergence for both primal and dual variables in the following lemma.

Lemma 4. Suppose Assumption 1-4 hold. Then, we have

and

Proof. With the help of Lemma 1 and Assumption 3, taking the primal variable x as an example, and noticing that v i, 0 > 0 , i ∈ [ n ] , we have

The similar result can be obtained for dual variable y and we thus complete the proof.

Next, we bound the the consensus error term S 2 in the following lemma.

Lemma 5. Suppose Assumption 1-4 hold. Then, we have

where I [ · ] ∈ { 0 , 1 } is the indicator for specific condition, and the initial consensus error ∆ 0 can be set to 0 with proper initialization.

Proof. By the updating rule of the primal variable, we have

where we have used Young's inequality. Then, by the definition of ζ v in (8), we have

and thus

Then, we bound the last term on the RHS of the above inequality by Lemma 4. For the case α < 1 / 2 , by Assumption 3 we have

For the case α ⩾ 1 / 2 , with the help of Lemma 1, we have

For the dual variable, we have

where

Then, using Young's inequality with parameter λ , we have

Noticing that W y k = P Y ( W y k ) holds for convex set Y , we get

where we have used the non-expansiveness of projection operator. Then, we have

Similar to the primal variable, we can bound the last term above, which completes the proof.

Next, we need to bound the term S 3 i.e., the optimality gap in dual variable. The intuition of the proof relies on the adaptive two time-scale protocol, that is, for given α and β , we try to find the threshold of the iterations k 0 , after which the inner sub-problem can be well solved (faster) to ensure that the computation of outer sub-problem can be solved accurately (slower). In specific, we suppose that there is a constant G such that ¯ u k ⩽ G hold for k = 0 , 1 , · · · , k 0 -1 , then the analysis is divided into two phases.

Lemma 6 (First phase) . Suppose Assumption 1-4 hold. If ¯ u k ⩽ G,k = 0 , 1 , · · · , k 0 -1 , then we have where

Proof. Using Young's inequality with parameter λ k , we get

Recalling that

Then, for the first term on the RHS of (35), by the non-expansiveness property of projection operator P Y ( · ) (c.f., Lemma 1 in (Nedic et al., 2010)), we have

wherein the last inequality we have used the fact ∥ W ∥ 2 2 ⩽ 1 . Then, multiplying by 1 / ( γ y ¯ u -β k +1 ) on both sides of (35) we get

For the inner-product terms on the RHS, taking expectation on both sides, we have

where we have used Young's inequality and strong-concavity of f i , and

For the consensus error of dual variable on the objective function, using strong-concavity of f i and Jensen's inequality, we have

Letting λ k = µγ y ¯ u -β k +1 / 2 , we get

By the κ -smoothness of y ∗ , we have

Telescoping the obtained terms from 0 to k 0 -1 and noticing that ¯ u k ⩽ G for k ⩽ k 0 -1 we complete the proof.

For the second phase, i.e., k ⩾ k 0 , we have the following lemma.

Lemma 7 (Second phase) . Suppose Assumption 1-4 hold. If ¯ u k ⩽ G,k = 0 , 1 , · · · , k 0 -1 , then we have

Proof. Firstly, by the non-expansiveness of projection operator, we have

Then, for the first inner-product term on the RHS, letting ∇ x ˜ F k = ∇ x F ( x k , y k ; ξ k ) -∇ x F ( x k , y k ) , we get

wherein the last inequality we have used the fact that y ∗ is κ -Lipschitz. Then, using Young's inequality with parameter λ k , we get

For the second inner-product term on the RHS, noticing that y ∗ is ˆ L = κ (1 + κ ) 2 smooth given in Lemma 2, we have

wherein the last inequality we have used Young's inequality with parameter τ . Plugging the obtained inequalities into (44), we get

Setting the parameters for Young's inequalities we used as follows,

then we get

Recalling that

and multiplying by 2 ( 2+ µγ y ¯ u -β k +1 ) γ y ¯ u -β k +1 on both sides of (50), we obtain that

Telescoping the terms from t 0 to K -1 , we get

Next we need to further bound the running sums of E [ E 2 ,k ] , E [ E 3 ,k ] and E [ E 4 ,k ] respectively. For E [ E 2 ,k ] , with the help of Assumption 2 and noticing that ¯ u k ⩽ G,k = 0 , 1 , · · · , k 0 -1 , we get

Then, for the term E [ E 3 ,k ] , noticing that ¯ u k +1 ⩽ ¯ v k +1 and ¯ v k +1 ⩾ ¯ v 1 , we have

For the term E 4 ,k , we denote

then we have

where we have used the Lipschitz continuity of y ∗ given in Lemma 2 and Assumption 3. Then, noticing that E [ ∇ x ˜ F k ] = 0 , we obtain

Therefore, combining the obtained inequalities, we complete the proof.

Now, it remains to bound the term E 1 ,k .

Lemma 8. Suppose Assumption 1-4 hold. Then, we have

Proof. Recalling the definition of E 1 ,k as given in (34), we have

Next, we show that the term 1 2 γ y ¯ u -β k +1 -1 2 γ y ¯ u -β k -µ 8 is positive for only a constant number of iterations. If the term is positive at iteration k , then we have

wherein the last inequality we used Bernoulli's inequality. Then we have the following two conditions,

which implies that we have at most

constant number of iterations when the term is positive. Furthermore, when the term is positive, by the inequality (59), we have

where we have used the concavity of f i in y and Assumption 3. Then, we have

which completes the proof.

Next, we show in the following lemma that the inconsistency terms, as described in (5), exhibit asymptotic convergence for the proposed D-AdaST algorithm.

Lemma 9 (Convergence of inconsistency terms) . Suppose Assumption 1-4 hold. For the proposed D-AdaST in Algorithm 1, we have

and

Proof. By the definition of v i,k in (3), we have

Next, for the term of inconsistency of the stepsize ∥ v k -1 ¯ v k ∥ 2 , we consider two cases due to the max operator we used. At iteration k , for the case m x k ⩾ m y k with ∥ m x 0 -1 ¯ m x 0 ∥ 2 = 0 , we have

For the case m x k < m y k , with ∥ m y 0 -1 ¯ m y 0 ∥ 2 = 0

Combining these two cases, and using Lemma 4 and the fact ∥ v α k -1 ¯ v α k ∥ 2 ⩽ ∥ v k -1 ¯ v k ∥ 2 α for α ∈ (0 , 1) , we obtain the result for primal decision variable. Following the same proof, we can also derive the result for dual decision variable. We thus complete the proof.

We further give the following lemma to show that the inconsistency of stepsize remains uniformly bounded for the vanilla D-TiAda algorithm as given in (2).

Lemma 10 (Inconsistency for D-TiAda) . Suppose Assumption 1-4 hold. Then, for D-TiAda, we have

Proof. By the definition of inconsistency of stepsizes in (8) and Assumption 3 on bounded gradient, we immediately get the result.

B.3 Proof of Theorem 1

Proof of Theorem 1. Consider a complete graph with 3 nodes where the functions corresponding to the nodes are as follows:

where a = 2 -1 2 α -1 and b = 2 -1 2 β -1 .

Notice that the only stationary point of f ( x, y ) = ( f 1 ( x, y ) + f 2 ( x, y ) + f 3 ( x, y )) / 3 is (0 , 0) . We denote g x i,k = ∇ x f i ( x k , y k ) and g y i,k = ∇ y f i ( x k , y k ) .

Now we consider points initialized in line

where we have

Note that by our assumptions of the range of α and β , we have a < b . Thus, we have

which means g x 2 , 0 would be chosen in the maximum operator in the denominator of TiAda stepsize for x . Therefore, after one step, we have

Next, we will use induction to show that x and y will stay in x 0 and y 0 for any iteration. Assuming for all iterations k in 1 , . . . , t , x k = x 0 and y k = y 0 , then we have in next step

Note that g = -a · g . Then, we get

Similarly, we can show that y t +1 = y t . Therefore all iterates will stay at ( x 0 , y 0 ) if initialized at line y = -ab + b ab + a x , which implies that the initial gradient norm can be arbitrarily large by picking x 0 to be large.

B.4 Proof of Theorem 2 and Corollary 1

Proof of Theorem 2. Combining the results obtained in Lemma 6, 7 and 8, we get

Letting the separation point between the two phases discussed in Lemma 6 and 7 satisfy

then, plugging above inequality into (18), with the help of Lemma 4-8 and Lemma 9, we get

where ˆ L = κ (1 + κ ) 2 , L Φ = L (1 + κ ) , and

Letting the total iteration K satisfy the conditions given in (12) such that the terms E 0 , E G and E W are dominated by the others, we thus complete the proof.

Proof of Corollary 1. With the help of Lemma 10, we can directly adapt the proof of Theorem 2 to get the result in (14).

B.5 Extend the proof to coordinate-wise stepsize

In this subsection, we show how to extend our convergence analysis of D-AdaST to the coordinatewise adaptive stepsize (Zhou et al., 2018) variant. We first present this variant in Algorithm 2, which can be rewritten in a compact form with the Hadamard product denoted by ⊙ .

Algorithm 2 D-AdaST with coordinate-wise adaptive stepsize

Initialization: x i, 0 ∈ R p , y i, 0 ∈ Y , buffers m x i, 0 , m y i, 0 > 0 , stepsizes γ x , γ y > 0 and 0 < β < α < 1 .

1: for iteration k = 0 , 1 , · · · , each node i ∈ [ n ] , do

2: Sample i.i.d ξ x i,k and ξ y i,k , compute:

3: Accumulate the gradient with Hadamard product:

4: Compute the ratio:

5: Update primal and dual variables locally:

6: Communicate parameters with neighbors:

7: Projection of dual variable on to set Y : y i,k +1 ←P Y ( y i,k +1 ) . 8: end for

where

and the matrices U α k and V β k are redefined as follows:

where [ · ] j denotes the j -th element of a vector.

Recalling the definitions of inconsistency of stepsize in (8), we give the following notations:

and

Building upon the established definitions of coordinate-wise stepsize inconsistency, the subsequent lemma is presented to show the non-convergence of the inconsistency term compared to Lemma 9.

Lemma 11 (Inconsistency, coordinate-wise) . Suppose Assumption 1-4 hold. For the proposed D-AdaST algorithm, we have

and

In contrast, for D-TiAda, we have

Proof. For the coordinate-wise adaptive stepsize, with the definitions of Frobenius norm and Hadamard product, we have

For the first term on the RHS, according to the definitions given in (78), we have

Then, for the second part, we have

where the term ˆ ζ 2 v is not guaranteed to be convergent because the stepsizes between the different dimensions of each node are not consistent. Then, similar to the proof of Lemma 9, we can obtain the result presented in (79).

Next, noticing that for D-TiAda,

and using Lemma 9, we complete the proof.

Theorem 3. Suppose Assumption 1-4 hold. Let 0 < β < α < 1 and the total iteration satisfy

to ensure time-scale separation and quasi-independence of network. For D-AdaST with coordinatewise adaptive stepsize, we have

Proof. With the help of Lemma 11 and the obtained result (75) in the proof of Theorem 2, we can derive the convergence results for D-AdaST with coordinate-wise adaptive stepsize.

Remark 6. In Theorem 3, we show that the coordinate-wise variant of D-AdaST exhibits a steadystate error in its upper bound. This error depends on the number of nodes and the dimension of the problem, which stems from the stepsize inconsistency in each dimension of the local decision variables for each node (c.f., Line 3 of Algorithm 2).

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contributions and scope of this work have been accurately discussed.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] ,

Justification: We have carefully discussed the limitations of this work in terms of assumptions and main results.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided a full set of assumptions and complete proof for the theoretical results. See Section 3 and Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided detailed experimental settings and reproducibility information for the experiments of this work. See Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code of this work is included in the supplementary.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided detailed experimental settings in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Multiple runs with averaging are used to produce the experimental curves in this work.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided sufficient information on the computer resources in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This work conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel are negative.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The license/copyright information of the code and dataset in this paper is clear.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code of this paper is included in the supplementary.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper presents D-AdaST, a distributed adaptive minimax method designed to address non-convergence issues in nonconvex-strongly-concave (NC-SC) minimax problems caused by inconsistent locally computed adaptive stepsizes in distributed settings. The method incorporates a stepsize tracking mechanism, which ensures consistency across local stepsizes and maintains time-scale separation, achieving near-optimal convergence rates comparable to centralized counterparts.

STRENGTHS:
The paper addresses a significant challenge in distributed minimax optimization by introducing a method that achieves near-optimal convergence rates. The theoretical contributions are well-founded, and the experimental validation is comprehensive.

WEAKNESSES:
Please consider my questions for this part.

QUESTIONS:
1- How does D-AdaST compare to other existing adaptive minimax methods in terms of computational time efficiency?

2-The current analysis focuses on nonconvex-strongly-concave minimax problems. How would the theoretical framework extend to other problem classes, such as nonconvex-nonconcave or convex-concave problems? What modifications or additional assumptions would be necessary?

3-In a distributed setting, communication delays and asynchrony can pose significant challenges. How does the theoretical analysis account for these factors? Are there any bounds or guarantees provided for performance in asynchronous or delayed communication settings?

LIMITATIONS:
They discussed the limitations of their work in terms of assumptions and main results.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

comments: SUMMARY:
This paper proposed a decentralized stochastic first-order method for nonconvex minimax optimization. For nonconvex-strongly-concave setting, the proposed D-AdaST has the convergence rate of $O(\epsilon^{-4+\delta})$ for any small $\delta>0$.

STRENGTHS:
see questions

WEAKNESSES:
see questions

QUESTIONS:
The proposed method does not require the knowledge of the parameters $L$, $\mu$ and total iterations number, which is nice in  empirical. 

I have some questions as follows:

1.	This paper states D-AdaST achieves the near-optimal convergence rate, while the definition of optimality is unclear, e.g., problem setting and algorithm class. I’m not sure whether this statement is correct. Specifically, Assumption 2 requires the second-order Lipschitz continuous, while the existing lower for solving nonconvex-strongly-concave minimax problem by first-order methods only considers the first-order Lipschitz continuous. The author should clarify how to fill this gap and explain how to achieve the lower bound in the setting of D-AdaST more clearly. 
2.	Assumption 3 introduce the upper bound $C$ for stochastic gradient, which should be included in the final convergence rate.
3.	Can you provide some explanation why the small $\delta$ cannot be avoided in result? What happens if we try to take $\delta\to0$?
4.	The section of introduction describes the problem setting with constraints on both $x$ and $y$, while the algorithm design and convergence analysis looks only consider the constraint on $y$.
5.	Is it possible to increase the batch-size of stochastic gradient to reduce the total communication rounds? Typically, the communication rounds in decentralized optimization can be smaller than the computation rounds, e.g., Chen et al., 2024.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
The authors introduced a new distributed adaptive minimax method, D-AdaST, to address the issue of non-convergence in nonconvex-strongly-concave minimax problems caused by inconsistencies in locally computed adaptive stepsizes. D-AdaST employs an efficient adaptive stepsize tracking protocol that ensures time-scale separation and consistency of stepsizes among nodes, thereby eliminating steady-state errors. Extensive experiments on both real-world and synthetic datasets validate their theoretical findings across various scenarios.

STRENGTHS:
Originality and Significance:
The paper introduces a novel Distributed Adaptive Minimax Optimization Algorithm with a Stepsize Tracking Protocol. This approach is significant in addressing the non-convergence issues that arise when transitioning adaptive single-machine algorithms to a distributed environment. The authors provide a thorough theoretical analysis, which could facilitate the development of similar methods or modifications to existing single-machine algorithms.

Quality  and Clarity:
The paper is well-structured and clear, with thoroughly explained assumptions and convergence rates, making it easy to understand.

WEAKNESSES:
The experimental section has some deficiencies, as it only includes a GAN experiment on the CIFAR-10 dataset. I believe it would be beneficial to supplement the paper with results from additional datasets or other minimax optimization problems to provide a more comprehensive evaluation.

QUESTIONS:
Please see weakness.

LIMITATIONS:
The paper employs general assumptions for distributed minimax optimization problems; therefore, it is unnecessary to discuss the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a method for distributed minimax optimization, for the scenario that various \"agents\" each hold part of a data set locally, and aim to coordinate to find the minimax solution of some criterion. Here, the criterion consists of the average of local cost functions, which are assumed to be smooth but non-convex-strongly-concave. Each agent is assumed to able to compute a stochastic gradient at each iteration, which can be communicated across the neighboring agents, who collectively communicate according to a given, known graph structure. \n\nIn the nondistributed setting, known gradient-based methods depend on hyper-parameter tuning, in particular in choosing the stepsizes. When such a method is properly adaptive, it finds the right stepsize without apriori knowledge based on continuous adjustment per iteration.\n\nThe authors construct counterexamples showing that directly applying adaptive methods designed for centralized problems results in non-convergence in distributed setting. The authors then propose an adaptive stepsize tracking protocol that involves transmitting two extra scalar variables to ensure consistency among stepsizes of different nodes. The authors give theoretical guarantees, matching (nearly) the convergence rate in the non-distributed setting. Furthermore, the authors exhibit their method on various real-world datasets to further underline their theoretical findings.\n\nSTRENGTHS:\n* The article is well written: the authors explain the problem clearly and the algorithm is clearly outlined. \n* The result and formulation of Theorem 1 are insightful, and the synthetic example of (6) is simple yet very instructive. Together they outline the problem to be overcome in Section 2.2 nicely.\n* The authors provide an (almost) rate optimal theoretical guarantee for their proposed method.\n* The chosen real-world settings are both interesting and illustrative.\n* The proposed method is attractive in terms of communication complexity, with only a couple of scalars being required on top of the gradients.\n\nWEAKNESSES:\n* Whilst I think the problem that the authors address is interesting, and within their scope they provide satisfying answers, I also think that this scope is rather limited. There are many things to investigate here in terms of e.g. network topology, communication efficiency, the influence of various noise sources. The authors stay within a set of rather stringent assumptions and do not capture the effect of these problem characteristics fully.\n\n* In high-dimensional settings, which the authors aim to address, achieving zero bias for stochastic gradients is not always feasible. Additionally, confirming the absence of bias in the gradient estimator is challenging to verify in practice. Does the method still work without this assumption? What if the bias is very small? \n\n* Similarly, the uniform bound on the gradient seems rather strong. Is this really required by the analysis?\n\nQUESTIONS:\nSome of the figures are rather difficult to read when printed on paper as they are rather small, but were readable digitally.\n\n* Is there a reason the constant $C$ does not appear in e.g. (13)?\n\n* I do not see the point of Corollary 1. As I understand it, this is an upper bound for D-TiAda. The authors say that this result is provided for \"proper comparison\". How does Corollary 1 provide any comparison if it is only an upper bound? Isn't Theorem 1 to provide the comparison that the authors are after?\n\nLIMITATIONS:\nWithin the scope of the article, I think limitations are appropriately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}