Title: Ordered Momentum for Asynchronous SGD
Venue: NeurIPS 2024
Accepted: yes
Keywords: optimization

deep learning

distributed learning

asynchronous SGD
Abstract: Distributed learning is essential for training large-scale deep models.
Asynchronous SGD (ASGD) and its variants are commonly used distributed learning methods, particularly in scenarios where the computing capabilities of workers in the cluster are heterogeneous.
Momentum has been acknowledged for its benefits in both optimization and generalization in deep model training. However, existing works have found that naively incorporating momentum into ASGD can impede the convergence.
In this paper, we propose a novel method called ordered momentum (OrMo) for ASGD. In OrMo, momentum is incorporated into ASGD by organizing the gradients in order based on their iteration indexes. We theoretically prove the convergence of OrMo with both constant and delay-adaptive learning rates for non-convex problems. To the best of our knowledge, this is the first work to establish the convergence analysis of ASGD with momentum without dependence on the maximum delay. Empirical results demonstrate that OrMo can achieve better convergence performance compared with ASGD and other asynchronous methods with momentum.

Ordered Momentum for Asynchronous SGD

Abstract

Distributed learning is essential for training large-scale deep models. Asynchronous SGD (ASGD) and its variants are commonly used distributed learning methods, particularly in scenarios where the computing capabilities of workers in the cluster are heterogeneous. Momentum has been acknowledged for its benefits in both optimization and generalization in deep model training. However, existing works have found that naively incorporating momentum into ASGD can impede the convergence. In this paper, we propose a novel method called ordered momentum (OrMo) for ASGD. In OrMo, momentum is incorporated into ASGD by organizing the gradients in order based on their iteration indexes. We theoretically prove the convergence of OrMo with both constant and delay-adaptive learning rates for non-convex problems. To the best of our knowledge, this is the first work to establish the convergence analysis of ASGD with momentum without dependence on the maximum delay. Empirical results demonstrate that OrMo can achieve better convergence performance compared with ASGD and other asynchronous methods with momentum.

1 Introduction

Many machine learning problems can be formulated as optimization problems of the following form:

where w denotes the model parameter, d is the dimension of the parameter, D represents the distribution of the training instances and f ( w ; ξ ) denotes the loss on the training instance ξ .

Stochastic gradient descent (SGD) [28] and its variants [6, 12] are widely employed to solve the problem in (1). At each iteration, SGD uses one stochastic gradient or a mini-batch of stochastic gradients as an estimate of the full gradient to update the model parameter. In practice, momentum [26, 36, 24, 33] is often incorporated into SGD as a crucial technique for faster convergence and better generalization performance. Many popular machine learning libraries, such as TensorFlow [1] and PyTorch [25], include SGD with momentum (SGDm) as one of the optimizers.

Due to the rapid increase in the sizes of both models and datasets in recent years, a single machine is often insufficient to complete the training task of machine learning models within a reasonable time. Distributed learning [42, 37] aims to distribute the computations across multiple machines (workers) to accelerate the training process. Because of its necessity for training large-scale machine learning models, distributed learning has become a hot research topic in recent years. Existing distributed learning methods can be categorized into two main types: synchronous distributed learning (SDL) methods [19, 35, 29, 38, 44, 39] and asynchronous distributed learning (ADL) methods [2, 27, 8, 47, 17, 43]. In SDL methods, faster workers that have completed the computation must wait idly for

the other slower workers in each communication round. Hence, the speed of SDL methods is often hindered by slow workers. In contrast, faster workers do not necessarily wait idly for the other slower workers in ADL methods, because ADL methods require aggregating information from only one worker or a subset of workers in each communication round. Representative ADL methods include asynchronous SGD (ASGD) and its variants [2, 8, 46, 30, 49, 48, 7, 3, 31, 22, 13]. In ASGD, once a worker finishes its gradient computation, the parameter (typically on the server) is immediately updated using this gradient through an SGD step, without waiting for other workers.

Momentum has been acknowledged for its benefits in both optimization and generalization in deep model training [33]. In SDL methods, momentum is extensively utilized across various domains, including decentralized algorithms [18, 45], communication compression algorithms [19, 29, 38, 40, 34, 41], infrequent communication algorithms [44, 39, 40], and federated learning algorithms [21, 32]. However, in ADL methods, some works [23, 9] have found that naively incorporating momentum into ASGD may decrease the convergence rate or even result in divergence. To tackle this challenge, some more sophisticated methods have been proposed to incorporate momentum into ASGD. The works in [23, 9] recommend tuning the momentum coefficient to enhance convergence performance when naively incorporating momentum into ASGD. The work in [9] proposes shifted momentum, which maintains local momentum on each worker. Inspired by Nesterov's accelerated gradient, the work in [4] proposes SMEGA 2 , which leverages the momentum to estimate the future parameter. However, the process of tuning the momentum coefficient in [23, 9] is time-consuming and yields limited improvement in practice. Although shifted momentum and SMEGA 2 can achieve better empirical convergence performance than the method which naively incorporates momentum into ASGD, both of them lack theoretical convergence analysis.

In this paper, we propose a novel method, called ordered momentum (OrMo), for asynchronous SGD. The main contributions of this paper are outlined as follows:

OrMo incorporates momentum into ASGD by organizing the gradients in order based on their iteration indexes.

We theoretically prove the convergence of OrMo with both constant and delay-adaptive learning rates for non-convex problems. To the best of our knowledge, this is the first work to establish the convergence analysis for ASGD with momentum without dependence on the maximum delay.

Empirical results demonstrate that OrMo can achieve better convergence performance compared with ASGD and other asynchronous methods with momentum.

2 Preliminary

In this paper, we use ∥ · ∥ to denote the L 2 norm. For a positive integer n , we use [ n ] to denote the set { 0 , 1 , 2 , . . . , n -1 } . ∇ f ( w ; ξ ) denotes the stochastic gradient computed over the training instance ξ and model parameter w . In this paper, we focus on the widely used Parameter Server framework [15], where the server is responsible for storing and updating the model parameter and the workers are responsible for sampling training instances and computing stochastic gradients. For simplicity, we assume that each worker samples one training instance for gradient computation each time. The analysis of mini-batch sampling on each worker follows a similar approach.

One of the most representative methods for distributing SGD across multiple workers is Synchronous SGD (SSGD) [20, 10]. Distributed SGD (DSGD), as presented in Algorithm 1, unifies SSGD and ASGD within a single framework [13]. The waiting set C in Algorithm 1 is a collection of workers (indexes) that are awaiting the server to send the latest parameter. The only difference between SSGD and ASGD is the communication scheduler associated with the waiting set. SSGD corresponds to DSGD with a synchronous communication scheduler, while ASGD corresponds to DSGD with an asynchronous communication scheduler. We use g k t ite ( k t ,t ) to denote the stochastic gradient ∇ f ( w ite ( k t ,t ) ; ξ k t ) , where k t is the index of the worker whose gradient participates in the parameter update at iteration t and ξ k t denotes a training instance sampled on worker k t . The function ite ( k, t ) denotes the iteration index of the latest parameter sent to worker k before iteration t , where k ∈ [ K ] and t ∈ [ T ] . The delay of the gradient g k t ite ( k t ,t ) is defined as τ t = t -ite ( k t , t ) . When K = 1 , DSGD degenerates to vanilla SGD, i.e., ite ( k t , t ) ≡ t .

Algorithm 1 Distributed SGD

1: Server:

2: Input : number of workers K , number of iterations T , learning rate η ;

3: Initialization : initial parameter w 0 , waiting set C = ∅ ;

4: Send the initial parameter w 0 to all workers;

5: for t = 0 to T -1 do

6: Receive a stochastic gradient g k t ite ( k t ,t ) from some worker k t ;

7: Update the parameter w t +1 = w t -η g k t ite ( k t ,t ) ;

8: Add the worker k t to the waiting set C = C ∪ { k t } ;

9: Execute the communication scheduler: Option I: (Synchronous) only when all the workers are in the waiting set, i.e., C = [ K ] , send the parameter w t +1 to the workers in C and set C to ∅ ; Option II: (Asynchronous) once the waiting set is not empty, i.e., C ̸ = ∅ , immediately send the parameter w t +1 to the worker in C and set C to ∅ ;

10: end for

11: Notify all workers to stop;

12: Worker k : ( k ∈ [ K ])

13: repeat

14: Wait until receiving the parameter w from the server;

15: Randomly sample ξ k ∼ D and then compute the stochastic gradient g k = ∇ f ( w ; ξ k ) ;

16: Send the stochastic gradient g k to the server;

17: until receive server's notification to stop

In ASGD, the latest parameter w t +1 will be immediately sent back to the worker after the server updates the parameter at each iteration. The function ite ( k, t ) in ASGD can be formulated as follows:

̸

where k ∈ [ K ] , t ∈ [ T ] .

In SSGD, there is a barrier in the synchronous communication scheduler since the latest parameter w t +1 will be sent back to the workers only when all the workers are in the waiting set. The function ite ( k, t ) in SSGD can be formulated as ite ( k, t ) = ⌊ t K ⌋ K , where k ∈ [ K ] , t ∈ [ T ] and ⌊·⌋ is the floor function.

Remark 1. In existing works [10, 22], SSGD is often presented in the form of mini-batch SGD:

where s ∈ [ S ] and S denotes the number of iterations. Here, all workers aggregate their stochastic gradients to obtain the mini-batch gradient 1 K ∑ k ∈ [ K ] ∇ f ( ˜ w s ; ξ k ) , which is then used to update the parameter. To unify SSGD and ASGD into a single framework in Algorithm 1, we reformulate SSGD in the form of mini-batch SGD in (2). Specifically, one update using a mini-batch gradient computed over K training instances in (2) is split into K updates, each using a stochastic gradient over a single training instance. Letting η = ˜ η K , T = KS and w 0 = ˜ w 0 , the sequence { w sK } s ∈ [ S ] in SSGD in Algorithm 1 matches { ˜ w s } s ∈ [ S ] in (2).

3 Ordered Momentum

In this section, we first propose a new reformulation of SSGD with momentum, which inspires the design of ordered momentum (OrMo) for ASGD. Then, we present the details of OrMo, including the algorithm and convergence analysis.

3.1 Reformulation of SSGD with Momentum

The widely used SGD with momentum (SGDm) [26] can be expressed as follows:

where ˜ u 0 = 0 , β ∈ [0 , 1) , s ∈ [ S ] and S denotes the number of iterations. β is the momentum coefficient. ˜ u s represents the Polyak's momentum. 1 |B s | ∑ ξ ∈B s ∇ f ( ˜ w s ; ξ ) denotes the stochastic gradient computed over the sampled training instance set B s , which contains either a single training instance or a mini-batch of training instances sampled from D . (3) denotes the parameter update step and (4) denotes the momentum update step. When β = 0 , SGDm degenerates to (mini-batch) SGD.

Since SSGD can be presented in the form of mini-batch SGD as depicted in Remark 1, it's straightforward to implement SSGD with momentum (SSGDm) as follows:

where ˜ u 0 = 0 , β ∈ [0 , 1) , s ∈ [ S ] and S denotes the number of iterations. Here, the server aggregates the stochastic gradients from all the workers to obtain the mini-batch gradient 1 K ∑ k ∈ [ K ] ∇ f ( ˜ w s ; ξ k ) , which is then used to update both the parameter and the momentum in (5).

To gain insights from SSGDm on incorporating momentum into ASGD, we reformulate SSGDm in (5) to fit into the framework of Algorithm 1. Similar to the reformulation in Remark 1, the updates using a mini-batch gradient computed over K training instances in (5) are split into K updates, each using a stochastic gradient over a single training instance. The corresponding implementation details of SSGDm are presented in Algorithm 3 in Appendix B. In this way, the update rules of SSGDm in (5) can be reformulated as follows:

where u 0 = 0 , g k t ⌊ t K ⌋ K = ∇ f ( w ⌊ t K ⌋ K ; ξ k t ) and t ∈ [ T ] . We give the following proposition about the relationship between the sequences in (5) and those in (6). The proof details can be found in Appendix C.1.1.

Proposition 1. Letting η = ˜ η K , T = KS and w 0 = ˜ w 0 , the sequences { w sK } s ∈ [ S ] and { u sK } s ∈ [ S ] in (6) are equivalent to { ˜ w s } s ∈ [ S ] and { ˜ u s } s ∈ [ S ] in (5), respectively.

We investigate how the momentum term u t +1 evolves during the iterations in (6). For t ≥ K and t ∈ [ T ] , u t +1 can be formulated as:

where the superscript of the scalar β indicates the exponent. For t < K , u t +1 = β 0 × ∑ t j =0 η g k j 0 . Figure 1 shows u 10 as an example when K = 4 . We define { η g 0 iK , η g 1 iK , · · · , η g K -1 iK } as the i -th

𝜂𝒈0

0

𝜂𝒈0

1

𝜂𝒈0

2

𝜂𝒈0

3

𝛽 2 ×

+

𝜂𝒈4

0

𝜂𝒈4

1

𝜂𝒈4

2

𝜂𝒈4

3

𝛽 1 ×

+

𝜂𝒈8

0

𝜂𝒈8

1

𝜂𝒈8

2

𝜂𝒈8

3

𝛽 0 ×

(scaled) gradient group, which contains K gradients scaled by the learning rate η . The order of the gradient groups is based on the iteration indexes of their corresponding gradients. Though some gradients may be missing because they have not yet arrived at the server, the momentum is a weighted sum of the gradients from the first several gradient groups. Hence, the momentum in SSGDm is referred to as an ordered momentum . Specifically, the gradients in the i -th gradient group are weighted by β ⌊ t K ⌋-i in the momentum u t +1 , where i ∈ [ ⌊ t K ⌋ + 1] . We refer to the gradient group whose gradients are weighted by β 0 as the latest gradient group, which contains the latest gradients. For u t +1 in SSGDm, the latest gradient group corresponds to the ⌊ t K ⌋ -th gradient group.

Due to the barrier in the synchronous communication scheduler in SSGDm as presented in Algorithm 3, the gradients in SSGDm consistently arrive at the server in the order of their iteration indexes. The arriving gradient always belongs to the latest gradient group at each iteration. Thus, maintaining such an ordered momentum in SSGDm is straightforward. As shown in line 13 of Algorithm 3, the scaled gradient η g k t ite ( k t ,t ) is always added to the momentum with a weight of β 0 at each iteration. However, for ASGD, since the gradients arrive at the server out of order, it's not trivial to incorporate such an ordered momentum. To address this problem, we propose a solution in the following subsection.

3.2 OrMo for ASGD

In this subsection, we introduce our novel method called ordered momentum (OrMo) for ASGD, and present it in Algorithm 2.

Firstly, we define the (scaled) gradient groups in OrMo for ASGD. Due to the differences in the communication scheduler, the iteration indexes of the parameters used to compute the gradients in ASGD differ from those in SSGD (SSGDm). Specifically, the sequence of gradients computed in SSGD (SSGDm) can be formulated as:

In contrast, the sequence of gradients computed in ASGD is given by:

Thus, the i -th (scaled) gradient group in OrMo for ASGD is defined as:

where i ≥ 1 . The 0 -th (scaled) gradient group in OrMo is { η g 0 0 , η g 1 0 , · · · , η g K -1 0 } . Despite the difference in the gradients' iteration indexes, each gradient group in OrMo for ASGD also contains K gradients scaled by the learning rate η , similar to that in SSGDm as discussed in Subsection 3.1.

We use I t +1 to denote the index of the latest gradient group of u t +1 in OrMo. The iteration index of the latest gradient in u t +1 can be t at most. Since the gradient with iteration index t belongs to the ⌈ t K ⌉ -th gradient group, the latest gradient group for u t +1 should be the ⌈ t K ⌉ -th gradient group, i.e., I t +1 ≡ ⌈ t K ⌉ , ∀ t ∈ [ T ] . u t +1 is the weighted sum of the gradients from the first I t +1 +1 gradient groups, where some gradients may be missing because they have not yet arrived at the server. The gradients in the i -th gradient group are weighted by β ⌈ t K ⌉-i in the momentum u t +1 , where i ∈ [ I t +1 +1] and the superscript of the scalar β indicates the exponent. Figure 2 shows an example of u 10 in OrMo when K = 4 .

For the t -th iteration in OrMo, the server performs the following operations:

Algorithm 2 OrMo

1: Server:

2: Input : number of workers K , number of iterations T , learning rate η , momentum coefficient β ∈ [0 , 1) ;

3: Initialization : initial parameter w 0 , momentum u 0 = 0 , index of the latest gradient group I 0 = 0 , waiting set C = ∅ ;

4: Send the initial parameter w 0 and its iteration index 0 to all workers;

5: for t = 0 to T -1 do

6: if the waiting set C is empty and ⌈ t K ⌉ > I t then

7: w t + 1 2 = w t -β u t , u t + 1 2 = β u t , I t +1 = I t +1 ;

8: else

9: w t + 1 2 = w t , u t + 1 2 = u t , I t +1 = I t ;

10: end if

11: Receive a stochastic gradient g k t ite ( k t ,t ) and its iteration index ite ( k t , t ) from some worker k t and then calculate ⌈ ite ( k t ,t ) K ⌉ (i.e., the index of the gradient group that g k t ite ( k t ,t ) belongs to);

12: Update the momentum u t +1 = u t + 1 2 + β I t +1 -⌈ ite ( k t ,t ) K ⌉ × ( η g k t ite ( k t ,t ) ) ;

13: Update the parameter w t +1 = w t + 1 2 -1 -β I t +1 -⌈ ite ( k t ,t ) K ⌉ +1 1 -β × ( η g k t ite ( k t ,t ) ) ;

14: Add the worker k t to the waiting set C = C ∪ { k t } ;

15: Execute the asynchronous communication scheduler: once the waiting set is not empty, i.e., C ̸ = ∅ , immediately send the parameter w t +1 and its iteration index t +1 to the worker in C and set C to ∅ ;

16: end for

17: Notify all workers to stop;

18: Worker k : ( k ∈ [ K ])

19: repeat

20: Wait until receiving the parameter w t ′ and its iteration index t ′ from the server;

22: Send the stochastic gradient g k t ′ and its iteration index t ′ to the server;

21: Randomly sample ξ k ∼ D and then compute the stochastic gradient g k t ′ = ∇ f ( w t ′ ; ξ k ) ;

23: until receive server's notification to stop

𝜂𝒈0

0

𝜂𝒈0

1

𝜂𝒈0

2

𝜂𝒈0

3

𝛽 3 ×

+

𝜂𝒈 1

𝑘0

𝜂𝒈 2

𝑘1

𝜂𝒈 3

𝑘2

𝜂𝒈 4

𝑘3

𝛽 2 ×

+

𝜂𝒈 5

𝑘4

𝜂𝒈 6

𝑘5

𝜂𝒈 7

𝑘6

𝜂𝒈 8

𝑘7

𝛽 1 ×

+

𝜂𝒈 9

𝑘8

𝜂𝒈 10

𝑘9

𝜂𝒈 11

𝑘10

𝜂𝒈 12

𝑘11

𝛽 0 ×

If the parameter with iteration index t that satisfies ⌈ t K ⌉ > I t has been sent to some worker, update the parameter using the momentum and multiply the momentum with β : w t + 1 2 = w t -β u t , u t + 1 2 = β u t , I t +1 = I t +1 .

In this way, the momentum changes the index of its latest gradient group to ⌈ t K ⌉ and gets ready to accommodate the new gradient with iteration index t .

Receive a stochastic gradient g k t ite ( k t ,t ) and its iteration index ite ( k t , t ) from some worker k t and calculate ⌈ ite ( k t ,t ) K ⌉ , which is the index of the gradient group that g k t ite ( k t ,t ) belongs to.

Update the momentum: u t +1 = u t + 1 2 + β I t +1 -⌈ ite ( k t ,t ) K ⌉ × ( η g k t ite ( k t ,t ) ) .

Since the weight of the scaled gradients from the latest gradient group in the momentum is β 0 , the weight of the gradients from the ⌈ ite ( k t ,t ) K ⌉ -th gradient group should be β I t +1 -⌈ ite ( k t ,t ) K ⌉ .

OrMo updates the momentum by adding the scaled gradient η g k t ite ( k t ,t ) into the momentum with a weight of β I t +1 -⌈ ite ( k t ,t ) K ⌉ .

The update rule of the parameter in OrMo is motivated by that in SSGDm, as presented in Algorithm 3. In SSGDm, the scaled gradient η g k t ite ( k t ,t ) is always added to the momentum with a weight of β 0 . At the current iteration, this scaled gradient updates the parameter with a coefficient -β 0 . In subsequent iterations, this scaled gradient in the momentum updates the parameter with the coefficients -β, -β 2 , -β 3 , · · · . By the time this scaled gradient is weighted by β I t +1 -⌈ ite ( k t ,t ) K ⌉ in the momentum of SSGDm, it has already updated the parameter for I t +1 -⌈ ite ( k t ,t ) K ⌉ +1 steps, with a total coefficient -∑ I t +1 -⌈ ite ( k t ,t ) K ⌉ j =0 β j . In OrMo, for the scaled gradient η g k t ite ( k t ,t ) which is added to the momentum with a weight of β I t +1 -⌈ ite ( k t ,t ) K ⌉ , we compensate for the missed I t +1 -⌈ ite ( k t ,t ) K ⌉ +1 steps compared with SSGDm and update the parameter with the coefficient -1 -β I t +1 -⌈ ite ( k t ,t ) K ⌉ +1 1 -β at the current iteration. The design of the parameter update rule is crucial for the derivation of Lemma 2, which is further supported by the ablation study in Appendix A.3.

Add the worker k t to the waiting set C = C ∪ { k t } and execute the asynchronous communication scheduler.

Remark 2. Compared to ASGD, the additional communication overhead introduced by the iteration index in OrMo is negligible since the iteration index is only a scalar.

Remark 3. When the momentum coefficient β is set to 0 , OrMo degenerates to ASGD in Algorithm 1. If the asynchronous communication scheduler in line 15 of Algorithm 2 is replaced by a synchronous communication scheduler: only when all the workers are in the waiting set, i.e., C = [ K ] , send the parameter w t +1 and the iteration index t +1 to the workers in C and set C to ∅ , OrMo degenerates to SSGDm in Algorithm 3.

3.3 Convergence Analysis

In this section, we prove the convergence of OrMo in Algorithm 2 for non-convex problems. We only present the main results here. The proof details can be found in Appendix C.

We make the following assumptions, which are widely used in distributed learning [47, 43, 41, 22].

Assumption 1. For any stochastic gradient ∇ f ( w ; ξ ) , we assume that it satisfies:

Assumption 2. For any stochastic gradient ∇ f ( w ; ξ ) , we assume that it satisfies:

Assumption 3. F ( w ) is L -smooth ( L > 0 ):

Assumption 4. The objective function F ( w ) is lower bounded by F ∗ : F ( w ) ≥ F ∗ , ∀ w ∈ R d .

Firstly, we define the auxiliary sequence { ˆ u t } t ≥ 1 for the momentum: ˆ u 1 = ∑ k ∈ [ K ] η g k 0 , and

for t ≥ 1 .

Lemma 1. For any t ≥ 0 , the gap between u t +1 and ˆ u t +1 can be formulated as follows:

̸

Then, we define the auxiliary sequence { ˆ w t } t ≥ 1 for the parameter: ˆ w 1 = w 0 -∑ k ∈ [ K ] η g k 0 , and

for t ≥ 1 .

Lemma 2. For any t ≥ 0 , the gap between w t +1 and ˆ w t +1 can be formulated as follows:

̸

Then, we define another auxiliary sequence { ˆ y t } t ≥ 1 : ˆ y 1 = ˆ w 1 -β w 0 1 -β , and ˆ y t +1 = ˆ y t -η 1 -β g k t -1 t , for t ≥ 1 .

Lemma 3. For any t ≥ 1 , the gap between ˆ y t and ˆ w t can be formulated as follows:

Theorem 1. With Assumptions 1, 2, 3 and 4, letting η = min { 1 -β 2 KL , (1 -β )∆ 1 2 ( LT ) 1 2 σ , (1 -β ) 5 3 ∆ 1 3 ( LKG ) 2 3 T 1 3 } , Algorithm 2 has the following convergence rate:

Many works [46, 30, 7, 13, 22] consider delay-adaptive methods for ASGD. The key insight of these methods is to penalize the gradients with large delays and reduce their contribution to the parameter update. OrMo is orthogonal to these delay-adaptive methods. Concretely, we can replace the constant learning rate η in Algorithm 2 with a delay-adaptive learning rate η t , which is dependent on the delay of the gradient τ t . Inspired by [13], we adopt the following delay-adaptive learning rate η t :

The convergence of OrMo with the above delay-adaptive learning rate (called OrMo-DA) is guaranteed by Theorem 2.

Theorem 2. With Assumptions 1, 3 and 4, letting η = min { (1 -β ) 2 8 KL , √ (1 -β ) 3 ∆ TLσ 2 } , OrMo-DA has the following convergence rate:

where ∆ = F ( w 0 ) -F ∗ and ¯ w T is randomly chosen from { w 0 , w 1 , · · · , w T -1 } according to a probability distribution which is related to the delay-adaptive learning rates.

The proof details can be found in Appendix C.3. Compared with Theorem 1, Theorem 2 removes the dependence on Assumption 2 (bounded gradient) and provides a better convergence bound.

Remark 4. We focus on the scenario where the training instances across all workers are independent and identically distributed (i.i.d.) from D . This scenario commonly appears in the data-center setup for distributed training [5], where all workers have access to the full training dataset. Our analysis for the i.i.d. scenario can also provide insights into the analysis in a non-i.i.d. scenario [22], which will be studied in future work.

Remark 5. Most existing theoretical analyses of ASGD [16, 49, 3, 20] rely on the maximum delay τ max (e.g., O ( √ Lσ 2 T + τ max L T ) in [20]), where τ max = max t ∈ [ T ] τ t . However, since ASGD can still perform well even when the maximum delay is extremely large ( τ max ≫ K ) in practice, these theoretical analyses don't accurately reflect the true behavior of ASGD. The most closely related works to this work are [13, 22], which analyze ASGD without relying on the maximum delay. But the works in [13, 22] do not consider momentum. To the best of our knowledge, this is the first work to establish the convergence guarantee of ASGD with momentum without relying on the maximum delay.

4 Experiments

In this section, we evaluate the performance of OrMo, OrMo-DA and other baseline methods. All the experiments are implemented based on the Parameter Server framework [15]. Our distributed platform is conducted with Docker. Each Docker container corresponds to either a server or a worker. All the methods are implemented with PyTorch 1.3.

The baseline methods include ASGD, naive ASGDm which naively incorporates momentum into ASGD [23], shifted momentum [9] and SMEGA 2 [4]. The details of naive ASGDm are shown in Algorithm 4. In OrMo-DA, for a gradient with a large delay satisfying τ t > 2 K , its corresponding learning rate will be multiplied by 1 τ t . We evaluate these methods by training ResNet20 model [11] on CIFAR10 and CIFAR100 datasets [14]. The number of workers is set to 16 and 64. The batch size on each worker is set to 64. The momentum coefficient is set to 0.9. Each experiment is repeated 5 times. The experiments are conducted under two settings:

Setting I [homogeneous (hom.)]: each worker has similar computing capabilities, which ensures comparable average time for gradient computations.

Setting II [heterogeneous (het.)]: some workers ( 1 16 of all) are designated as slow workers, with an average computation time that is 10 times longer than that of the others.

For the CIFAR10 dataset, the weight decay is set to 0.0001 and the model is trained with 160 epochs. The learning rate is multiplied by 0.1 at the 80-th and 120-th epoch, as suggested in [11]. The experiments for the CIFAR10 dataset are conducted on NVIDIA RTX 2080 Ti GPUs. For the CIFAR100 dataset, the weight decay is set to 0.0005 and the model is trained with 200 epochs. The learning rate is multiplied by 0.2 at the 60-th, 120-th and 160-th epoch, as suggested in [40]. The experiments for the CIFAR100 dataset are conducted on NVIDIA V100 GPUs.

Table 1 and Table 2 show the empirical results of different methods. Figure 3 and Figure 4 show the test accuracy curves of different methods. We also present the test accuracy curves of SSGDm for reference in Figure 3 and Figure 4. Training loss curves can be found in Appendix A.1. Compared with other asynchronous methods, OrMo and OrMo-DA can achieve better training loss and test accuracy. As shown in Figure 3, naive ASGDm occasionally fails to converge under Setting II. We can find that naively incorporating momentum into ASGD will impede its convergence. Due to the existence of slow workers, the maximum delay under Setting II is far greater than that under Setting I. For example, when training on the CIFAR10 dataset with K = 64 , the maximum delay under Setting II is about 30000 , while it's around 300 under Setting I. OrMo and OrMo-DA perform well under both settings, which aligns with our theoretical results without dependence on the maximum delay.

Figure 5 presents the training curves of OrMo and SSGDm with respect to wall-clock time. As shown in Figure 5(b), OrMo can be 8 times faster than SSGDm under Setting II since the training speed of SSGDm is hindered by slow workers. In contrast, slow workers have a limited impact on OrMo's training speed. Even under Setting I where each worker possesses similar computing













































2





(a)

K

= 16

(hom.)













































2





(b)

K

= 64

(hom.)















































2





(c)

K

= 16

(het.)















































2





(d)

K

= 64

(het.)















































2





(a)

K

= 16

(hom.)















































2





(b)

K

= 64

(hom.)















































2





(c)

K

= 16

(het.)















































2





(d)

K

= 64

(het.)











































































(a) homogeneous







 

































 





















(b) heterogeneous

capability, OrMo can still be more than twice as fast as SSGDm, as shown in Figure 5(a). This advantage arises because the computation time of each worker varies within a certain range even under the homogeneous setting and some workers must wait for others to finish gradient computations in SSGDm.

5 Conclusion

In this paper, we propose a novel method named ordered momentum (OrMo) for asynchronous SGD. We theoretically prove the convergence of OrMo with both constant and delay-adaptive learning rates for non-convex problems. To the best of our knowledge, this is the first work to establish the convergence analysis of ASGD with momentum without dependence on the maximum delay. Empirical results demonstrate that OrMo can achieve state-of-the-art performance.

Acknowledgment

This work is supported by National Key R&D Program of China (No. 2020YFA0713900), NSFC Project (No. 12326615), Major Key Project of Pengcheng Laboratory (No. PCL2024A06) and Key R&D Project of Jiangsu Province (No. BE2023652).

References

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning. In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation , pages 265-283, 2016.

Alekh Agarwal and John C. Duchi. Distributed delayed stochastic optimization. In Advances in Neural Information Processing Systems , pages 873-881, 2011.

Yossi Arjevani, Ohad Shamir, and Nathan Srebro. A tight convergence analysis for stochastic gradient descent with delayed updates. In Proceedings of the International Conference on Algorithmic Learning Theory , pages 111-132, 2020.

Refael Cohen, Ido Hakimi, and Assaf Schuster. SMEGA2: distributed asynchronous deep neural network training with a single momentum buffer. In Proceedings of the International Conference on Parallel Processing , pages 1-10, 2022.

Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc'Aurelio Ranzato, Andrew W. Senior, Paul A. Tucker, Ke Yang, and Andrew Y. Ng. Large scale distributed deep networks. In Advances in Neural Information Processing Systems , pages 1232-1240, 2012.

Aaron Defazio, Francis R. Bach, and Simon Lacoste-Julien. SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. In Advances in Neural Information Processing Systems , pages 1646-1654, 2014.

Sanghamitra Dutta, Gauri Joshi, Soumyadip Ghosh, Parijat Dube, and Priya Nagpurkar. Slow and stale gradients can win the race: Error-runtime trade-offs in distributed SGD. In Proceedings of the International Conference on Artificial Intelligence and Statistics , pages 803-812, 2018.

Hamid Reza Feyzmahdavian, Arda Aytekin, and Mikael Johansson. An asynchronous minibatch algorithm for regularized stochastic optimization. IEEE Transactions on Automatic Control , 61(12):3740-3754, 2016.

Niv Giladi, Mor Shpigel Nacson, Elad Hoffer, and Daniel Soudry. At stability's edge: How to adjust hyperparameters to preserve minima selection in asynchronous training of neural networks? In Proceedings of the International Conference on Learning Representations , 2020.

Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 , 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems , pages 315-323, 2013.

Anastasia Koloskova, Sebastian U. Stich, and Martin Jaggi. Sharper convergence guarantees for asynchronous SGD for distributed and federated learning. In Advances in Neural Information Processing Systems , pages 17202-17215, 2022.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report , 2009.

Mu Li, David G. Andersen, Alexander J. Smola, and Kai Yu. Communication efficient distributed machine learning with the parameter server. In Advances in Neural Information Processing Systems , pages 19-27, 2014.

Xiangru Lian, Yijun Huang, Yuncheng Li, and Ji Liu. Asynchronous parallel stochastic gradient for nonconvex optimization. In Advances in Neural Information Processing Systems , pages 2737-2745, 2015.

Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. Asynchronous decentralized parallel stochastic gradient descent. In Proceedings of the International Conference on Machine Learning , pages 3043-3052, 2018.

Tao Lin, Sai Praneeth Karimireddy, Sebastian U. Stich, and Martin Jaggi. Quasi-global momentum: Accelerating decentralized deep learning on heterogeneous data. In Proceedings of the International Conference on Machine Learning , pages 6654-6665, 2021.

Yujun Lin, Song Han, Huizi Mao, Yu Wang, and Bill Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. In Proceedings of the International Conference on Learning Representations , 2018.

Ji Liu and Ce Zhang. Distributed learning systems with first-order methods. Foundations and Trends® in Databases , 9(1):1-100, 2020.

Wei Liu, Li Chen, Yunfei Chen, and Wenyi Zhang. Accelerating federated learning via momentum gradient descent. IEEE Transactions on Parallel and Distributed Systems , 31(8):1754-1766, 2020.

Konstantin Mishchenko, Francis R. Bach, Mathieu Even, and Blake E. Woodworth. Asynchronous SGD beats minibatch SGD under arbitrary delays. In Advances in Neural Information Processing Systems , pages 420-433, 2022.

Ioannis Mitliagkas, Ce Zhang, Stefan Hadjis, and Christopher Ré. Asynchrony begets momentum, with an application to deep learning. In Proceedings of the Annual Allerton Conference on Communication, Control, and Computing , pages 997-1004, 2016.

Yurii Nesterov. Introductory lectures on convex optimization: A basic course . Springer Science &Business Media, 2013.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , pages 8024-8035, 2019.

Boris Polyak. Some methods of speeding up the convergence of iteration methods. Ussr Computational Mathematics and Mathematical Physics , 4:1-17, 12 1964.

Benjamin Recht, Christopher Ré, Stephen J. Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems , pages 693-701, 2011.

Herbert Robbins and Sutton Monro. A stochastic approximation method. Annals of Mathematical Statistics , 22:400-407, 1951.

Chang-Wei Shi, Shen-Yi Zhao, Yin-Peng Xie, Hao Gao, and Wu-Jun Li. Global momentum compression for sparse communication in distributed learning. arXiv preprint arXiv:1905.12948 , 2019.

Suvrit Sra, Adams Wei Yu, Mu Li, and Alex Smola. Adadelay: Delay adaptive distributed stochastic optimization. In Proceedings of the International Conference on Artificial Intelligence and Statistics , pages 957-965, 2016.

Sebastian U. Stich, Amirkeivan Mohtashami, and Martin Jaggi. Critical parameters for scalable distributed learning with large batches and asynchronous updates. In Proceedings of the International Conference on Artificial Intelligence and Statistics , pages 4042-4050, 2021.

Jianhui Sun, Xidong Wu, Heng Huang, and Aidong Zhang. On the role of server momentum in federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 15164-15172, 2024.

Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. In Proceedings of the International Conference on Machine Learning , pages 1139-1147, 2013.

Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, and Yuxiong He. 1-bit adam: Communication efficient large-scale training with adam's convergence speed. In Proceedings of the International Conference on Machine Learning , pages 10118-10129, 2021.

Hanlin Tang, Shaoduo Gan, Ce Zhang, Tong Zhang, and Ji Liu. Communication compression for decentralized training. In Advances in Neural Information Processing Systems , pages 7663-7673, 2018.

Paul Tseng. An incremental gradient(-projection) method with momentum term and adaptive stepsize rule. SIAM Journal on Optimization , 8(2):506-531, 1998.

Joost Verbraeken, Matthijs Wolting, Jonathan Katzy, Jeroen Kloppenburg, Tim Verbelen, and Jan S Rellermeyer. A survey on distributed machine learning. ACM Computing Surveys , 53(2):1-33, 2020.

Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. In Advances in Neural Information Processing Systems , pages 14236-14245, 2019.

Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael G. Rabbat. Slowmo: Improving communication-efficient distributed SGD with slow momentum. In Proceedings of the International Conference on Learning Representations , 2020.

Cong Xie, Shuai Zheng, Oluwasanmi Koyejo, Indranil Gupta, Mu Li, and Haibin Lin. CSER: Communication-efficient SGD with error reset. In Advances in Neural Information Processing Systems , pages 12593-12603, 2020.

An Xu and Heng Huang. Detached error feedback for distributed SGD with random sparsification. In Proceedings of the International Conference on Machine Learning , pages 24550-24575, 2022.

Tao Yang, Xinlei Yi, Junfeng Wu, Ye Yuan, Di Wu, Ziyang Meng, Yiguang Hong, Hong Wang, Zongli Lin, and Karl H Johansson. A survey of distributed optimization. Annual Reviews in Control , 47:278-305, 2019.

Yi-Rui Yang and Wu-Jun Li. BASGD: buffered asynchronous SGD for byzantine learning. In Proceedings of the International Conference on Machine Learning , pages 11751-11761, 2021.

Hao Yu, Rong Jin, and Sen Yang. On the linear speedup analysis of communication efficient momentum SGD for distributed non-convex optimization. In Proceedings of the International Conference on Machine Learning , pages 7184-7193, 2019.

Kun Yuan, Yiming Chen, Xinmeng Huang, Yingya Zhang, Pan Pan, Yinghui Xu, and Wotao Yin. Decentlam: Decentralized momentum sgd for large-batch deep training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3029-3039, 2021.

Wei Zhang, Suyog Gupta, Xiangru Lian, and Ji Liu. Staleness-aware async-sgd for distributed deep learning. In Proceedings of the International Joint Conference on Artificial Intelligence , pages 2350-2356, 2016.

Shen-Yi Zhao and Wu-Jun Li. Fast asynchronous parallel stochastic gradient descent: A lockfree approach with convergence guarantee. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 2379-2385, 2016.

Shen-Yi Zhao, Gong-Duo Zhang, and Wu-Jun Li. Lock-free optimization for non-convex problems. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 2935-2941, 2017.

Shuxin Zheng, Qi Meng, Taifeng Wang, Wei Chen, Nenghai Yu, Zhiming Ma, and Tie-Yan Liu. Asynchronous stochastic gradient descent with delay compensation. In Proceedings of the International Conference on Machine Learning , pages 4120-4129, 2017.















































2





(a)

K

= 16

(hom.)















































2





(b)

K

= 64

(hom.)















































2





(c)

K

= 16

(het.)















































2





(d)

K

= 64

(het.)

















































2





(a)

K

= 16

(hom.)

















































2





(b)

K

= 64

(hom.)

















































2





(c)

K

= 16

(het.)

















































2





(d)

K

= 64

(het.)

A More Experimental Results

A.1 Loss Curves

Figure 6 and Figure 7 show the training loss curves of ResNet20 model.

A.2 Tuning β for Naive ASGDm

Following the suggestion in [23, 9], we conduct experiments to tune the momentum coefficient β for naive ASGDm and present the results when training ResNet20 on CIFAR10 in Table 3. While tuning the momentum coefficient can enhance the performance of naive ASGDm, hyperparameter tuning is quite time-consuming and costly. In contrast, OrMo achieves better performance using the commonly used momentum value of 0.9, without requiring extensive tuning.

A.3 Ablation Study

An ablation study is also conducted to justify the parameter update rule in line 13 of Algorithm 2. We replace the update rule in line 13 of Algorithm 2 with a vanilla SGD step, w t +1 = w t + 1 2 -η g k t ite ( k t ,t ) , and name it OrMo (vanilla SGD step). The comparison between the experimental results of OrMo and OrMo (vanilla SGD step) are presented in Table 4 and Figure 8.







































(a)

K

= 16

(hom.)







































(b)

K

= 64

(hom.)







































(c)

K

= 16

(het.)







































(d)

K

= 64

(het.)



















































2





(a) homogeneous



















































2





(b) heterogeneous

A.4 Experimental Results on ResNet18 Model

Table 5 and Figure 9 show the performance of different methods when training the ResNet18 model on the CIFAR10 dataset. The number of workers is set to 8 . In the homogeneous setting, each worker has similar computing capabilities. In the heterogeneous setting, one worker is designated as the slow worker, whose average computation time is 10 times longer than that of the others.

B Algorithm Details

Algorithm 3 and Algorithm 4 show the details of SSGDm and naive ASGDm.

Algorithm 3 SSGDm

1: Server:

2: Input : number of workers K , number of iterations T , learning rate η , momentum coefficient β ∈ [0 , 1) ;

3: Initialization : initial parameter w 0 , momentum u 0 = 0 , waiting set C = ∅ ;

4: Send the initial parameter w 0 to all workers;

5: for t = 0 to T -1 do

6: if the waiting set C is empty then

7: w t + 1 2 = w t -β u t , u t + 1 2 = β u t ;

8: else

9: w t + 1 2 = w t , u t + 1 2 = u t ;

10: end if

11: Receive a stochastic gradient g k t ite ( k t ,t ) from some worker k t ;

12: Update the parameter w t +1 = w t + 1 2 -η g k t ite ( k t ,t ) ;

13: Update the momentum u t +1 = u t + 1 2 + η g k t ite ( k t ,t ) ;

14: Add the worker k t to the waiting set C = C ∪ { k t } ;

15: Execute the synchronous communication scheduler: only when all the workers are in the waiting set, i.e., C = [ K ] , send the parameter w t +1 to the workers in C and set C to ∅ ;

16: end for

17: Notify all workers to stop;

18: Worker k : ( k ∈ [ K ])

19: repeat

20: Wait until receiving the parameter w from the server;

21: Randomly sample ξ k ∼ D and then compute the stochastic gradient g k = ∇ f ( w ; ξ k ) ;

22: Send the stochastic gradient g k to the server;

23: until receive server's notification to stop

Algorithm 4 naive ASGDm

1: Server:

2: Input : number of workers K , number of iterations T , learning rate η , momentum coefficient β ∈ [0 , 1) ;

3: Initialization : initial parameter w 0 , momentum u 0 = 0 , waiting set C = ∅ ;

4: Send the initial parameter w 0 to all workers;

5: for t = 0 to T -1 do

6: Receive a stochastic gradient g k t ite ( k t ,t ) from some worker k t ;

7: Update the momentum u t +1 = β u t + η g k t ite ( k t ,t )

8: Update the parameter w t +1 = w t -u t +1

9: Add the worker k t to the waiting set C = C ∪ { k t } ;

10: Execute the asynchronous communication scheduler: once the waiting set is not empty, i.e., C ̸ = ∅ , immediately send the parameter w t +1 to the worker in C and set C to ∅ ;

11: end for

12: Notify all workers to stop;

13: Worker k : ( k ∈ [ K ])

14: repeat

15: Wait until receiving the parameter w from the server;

16: Randomly sample ξ k ∼ D and then compute the stochastic gradient g k = ∇ f ( w ; ξ k ) ;

17: Send the stochastic gradient g k to the server;

18: until receive server's notification to stop

C Proof Details

C.1 Reformulation of SSGDm

C.1.1 Proof of Proposition 1

Proof. It's easy to verify that { k t , k t +1 , · · · , k t + K -1 } = [ K ] in (6), where K | t .

Base case: for s = 0, ˜ w 0 = w 0 and ˜ u 0 = u 0 .

Inductive hypothesis: for some arbitrary integer s ′ ≥ 0 , assume that ˜ w s ′ = w s ′ K , ˜ u s ′ = u s ′ K . Inductive step:

We can conclude that ˜ w s = w sK and ˜ u s = u sK for any s ∈ [ S ] .

C.2 OrMo

C.2.1 Proof of Lemma 1

Lemma 1 can be viewed as a special case of Lemma 5, and its proof is completed by substituting each delay-adaptive learning rate ˆ η k,t in the proof of Lemma 5 with η for all k ∈ [ K ] and t ∈ [ T ] .

C.2.2 Proof of Lemma 2

Lemma 2 can be viewed as a special case of Lemma 6, and its proof is completed by substituting each delay-adaptive learning rate ˆ η k,t in the proof of Lemma 6 with η for all k ∈ [ K ] and t ∈ [ T ] .

C.2.3 Proof of Lemma 3

Lemma 3 can be viewed as a special case of Lemma 7, and its proof is completed by substituting each delay-adaptive learning rate ˆ η k,t in the proof of Lemma 7 with η for all k ∈ [ K ] and t ∈ [ T ] .

Lemma 4. With Assumption 2, the gap between ˆ y t and ˆ w t can be bounded:

Proof. For any t ≥ 1 , ˆ u t can be formulated as follows:

C.2.4 Proof of Theorem 1

Proof.

Summing up the above equation from t = 1 to t = T , we can get that

Since ˆ y 1 -w 0 = 1 1 -β ( ˆ w 1 -w 0 ) = -η 1 -β ∑ k ∈ [ K ] g k 0 , we have

The last inequality above holds because η ≤ 1 -β 2 KL .

Combining the above equations, we can get that

Let η = min { 1 -β 2 KL , (1 -β )[ F ( w 0 ) -F ∗ ] 1 2 ( LT ) 1 2 σ , (1 -β ) 5 3 [ F ( w 0 ) -F ∗ ] 1 3 ( LKG ) 2 3 T 1 3 } , then we can get that

C.3 OrMo with Delay-Adaptive Learning Rate

C.3.1 Algorithm

The details of OrMo with delay-adaptive learning rate (OrMo-DA) are presented in Algorithm 5.

C.3.2 Notation

For a positive integer n , [ n ] = { 0 , 1 , 2 , · · · , n -1 } . [0] is defined as ∅ .

The function ite ( k, t ) denotes the iteration index of the latest parameter sent to worker k before iteration t , which can be formulated as

̸

where k ∈ [ K ] , t ∈ [ T +1] .

Algorithm 5 OrMo-DA

1: Server:

2: Input : number of workers K , number of iterations T , momentum coefficient β ∈ [0 , 1) ;

3: Initialization : initial parameter w 0 , momentum u 0 = 0 , index of the latest gradient group I 0 = 0 , waiting set C = ∅ ;

4: Send the initial parameter w 0 and its iteration index 0 to all workers;

5: for t = 0 to T -1 do

6: if the waiting set C is empty and ⌈ t K ⌉ > I t then

7: w t + 1 2 = w t -β u t , u t + 1 2 = β u t , I t +1 = I t +1 ;

8: else

9: w t + 1 2 = w t , u t + 1 2 = u t , I t +1 = I t ;

10: end if

11: Receive a stochastic gradient g k t ite ( k t ,t ) and its iteration index ite ( k t , t ) from some worker k t and then calculate ⌈ ite ( k t ,t ) K ⌉ (i.e., the index of the gradient group that g k t ite ( k t ,t ) belongs to);

12: Update the momentum u t +1 = u t + 1 2 + β I t +1 -⌈ ite ( k t ,t ) K ⌉ × ( η t g k t ite ( k t ,t ) ) ;

13: Update the parameter w t +1 = w t + 1 2 -1 -β I t +1 -⌈ ite ( k t ,t ) K ⌉ +1 1 -β × ( η t g k t ite ( k t ,t ) ) ;

14: Add the worker k t to the waiting set C = C ∪ { k t } ;

15: Execute the asynchronous communication scheduler: once the waiting set is not empty, i.e., C ̸ = ∅ , immediately send the parameter w t +1 and its iteration index t +1 to the worker in C and set C to ∅ ;

16: end for

17: Notify all workers to stop;

18: Worker k : ( k ∈ [ K ])

19: repeat

20: Wait until receiving the parameter w t ′ and its iteration index t ′ from the server;

22: Send the stochastic gradient g k t ′ and its iteration index t ′ to the server;

21: Randomly sample ξ k ∼ D and then compute the stochastic gradient g k t ′ = ∇ f ( w t ′ ; ξ k ) ;

23: until receive server's notification to stop

η t is the learning rate at iteration t and satisfies that

where t ∈ [ T ] .

The function next ( k, t ) denotes the index of the next iteration that the gradient from worker k will participate in the parameter update after iteration t (including iteration t ), which can be formulated as

̸

where k ∈ [ K ] , t ∈ [ T ] . It's easy to verify that ite ( k, next ( k, t )) = ite ( k, t ) , next ( k, ite ( k, t )) = next ( k, t ) , where k ∈ [ K ] , t ∈ [ T ] .

We define ˆ τ k,t = t -ite ( k, t ) , where k ∈ [ K ] and t ∈ [ T +1] . ˆ τ k,t denotes the current delay of the gradient g k ite ( k,t ) at iteration t , which is the number of iterations that have happened since ite ( k, t ) . It's easy to verify that ˆ τ k t ,t = t -ite ( k t , t ) = τ t , where t ∈ [ T ] .

We also define an auxiliary sequence ˆ η k,t for the adaptive learning rates. ˆ η k,t denotes the learning rate corresponding to the gradient g k ite ( k,t ) .

where k ∈ [ K ] and t ∈ [ T ] . Since next ( k, t ) = next ( k, ite ( k, t )) , we can have that ˆ η k,t = ˆ η k,ite ( k,t ) , where k ∈ [ K ] , t ∈ [ T ] .

If next ( k, t ) ∈ [ T ] , ˆ τ k,next ( k,t ) = ˆ τ k next ( k,t ) ,next ( k,t ) = τ next ( k,t ) and then we have that

It's easy to verify that ˆ η k ,t = η next ( k ,t ) = η t , where t ∈ [ T ] .

t t If next ( k, t ) = T ,

C.3.3 Convergence Analysis for OrMo-DA

Firstly, we define one auxiliary sequence { ˆ u t } t ≥ 1 for the momentum: ˆ u 1 = ∑ k ∈ [ K ] ˆ η k, 0 g k 0 , and

for t ≥ 1 .

Lemma 5. For any t ≥ 0 , the gap between u t +1 and ˆ u t +1 can be formulated as follows:

̸

Proof. Base case: for t = 0 , ˆ u 1 = ∑ k ∈ [ K ] ˆ η k, 0 g k 0 , u 1 = ˆ η k 0 , 0 g k 0 0 , then we have

̸

′ ′

̸

Inductive hypothesis: for some arbitrary integer t -1 ≥ 0 , assume that (14) is true for t = t -1 . Inductive step: We will prove that (14) is true for t = t ′ . Firstly, we divide our discussion into two cases based on whether t ′ -1 is divisible by K and prove that

̸

Case 1: K | ( t ′ -1)

̸

The second equation above holds because ⌈ t ′ K ⌉ > ⌈ t ′ -1 K ⌉ = I t ′ when K | ( t ′ -1) , I t ′ +1 = ⌈ t ′ K ⌉ and ˆ η k t ′ ,ite ( k t ′ ,t ′ ) = η next ( k t ′ ,ite ( k t ′ ,t ′ )) = η t ′ . The last equation above holds because ⌈ t ′ K ⌉ = ⌈ t ′ -1 K ⌉ +1 .

Case 2: K ∤ ( t ′ -1)

̸

The last equation above holds because ⌈ t ′ K ⌉ = ⌈ t ′ -1 K ⌉ .

̸

Since ite ( k, t ′ ) = ite ( k, t ′ -1) , ∀ k = k t ′ -1 , we can get the following equation for both cases above:

̸

If k t ′ = k t ′ -1 , then we have ite ( k t ′ , t ′ ) = t ′ and

̸

̸

̸

̸

If k t ′ = k t ′ -1 , then we have

̸

̸

We can conclude that ˆ u t +1 -u t +1 = ∑ k ∈ [ K ] ,k = k t β ⌈ t K ⌉-⌈ ite ( k,t ) K ⌉ ˆ η k,ite ( k,t ) g k ite ( k,t ) is true for any t ≥ 0 .

Then, we define one auxiliary sequence { ˆ w t } t ≥ 1 for the parameter: ˆ w 1 = w 0 -∑ k ∈ [ K ] ˆ η k, 0 g k 0 , and

for t ≥ 1 .

Lemma 6. For any t ≥ 0 , the gap between w t +1 and ˆ w t +1 can be formulated as follows:

̸

Proof. Base case: for t = 0 , ˆ w 1 = w 0 -∑ k ∈ [ K ] ˆ η k, 0 g k 0 , w 1 = w 0 -ˆ η k 0 , 0 g k 0 0 , then we have

̸

Inductive hypothesis: for some arbitrary integer t ′ -1 ≥ 0 , assume that (15) is true for t = t ′ -1 . Inductive step: We will prove that (15) is true for t = t ′ . Firstly, we divide our discussion into two cases based on whether t ′ -1 is divisible by K and prove that

̸

̸

̸

̸

̸

̸

Case 1: K | ( t ′ -1)

̸

̸

̸

̸

̸

̸

The last equation above holds because ⌈ t ′ K ⌉ = ⌈ t ′ -1 K ⌉ .

̸

Since ite ( k, t ′ ) = ite ( k, t ′ -1) , ∀ k = k t ′ -1 , we can get the following equation for both cases above:

̸

If k t ′ = k t ′ -1 , then we have ite ( k t ′ , t ′ ) = t ′ and

̸

̸

If k t ′ = k t ′ -1 , then we have

̸

̸

̸

We can conclude that ˆ w t +1 -w t +1 = -∑ k ∈ [ K ] ,k = k t 1 -β ⌈ t K ⌉-⌈ ite ( k,t ) K ⌉ +1 1 -β ˆ η k,ite ( k,t ) g k ite ( k,t ) is true for any t ≥ 0 .

Then, we define another auxiliary sequence { ˆ y t } t ≥ 1 : ˆ y 1 = w 0 -1 1 -β ∑ k ∈ [ K ] ˆ η k, 0 g k 0 , and

for t ≥ 1 .

Lemma 7. For any t ≥ 1 , the gap between ˆ y t and ˆ w t can be formulated as follows:

Proof. Base case: For t = 1 , we have that

Inductive hypothesis: for some arbitrary integer t ′ ≥ 1 , assume that ˆ y t -ˆ w t = -β 1 -β ˆ u t is true for t = t ′ .

Inductive step: We will prove that ˆ y t -ˆ w t = -β 1 -β ˆ u t is true for t = t ′ +1 . We divide our discussion into two cases based on whether t ′ -1 is divisible by K .

̸

̸

Case 1: K | ( t ′ -1)

Case 2: K ∤ ( t ′ -1)

We can conclude that ˆ y t -ˆ w t = -β 1 -β ˆ u t is true for any t ≥ 1 .

Lemma 8. For any t ≥ 1 , ˆ u t can be formulated as follows:

Proof. It's straightforward to get this conclusion from the definition of the sequence ˆ u t in (13).

Lemma 9. (descent lemma) With Assumptions 1 and 3, we have the following descent lemma for t ≥ 1 ,:

Proof.

Lemma 10. With Assumption 1, the gap between ˆ y t and ˆ w t in OrMo-DA can be bounded as follows:

Proof.

Let q t = ∑ k ∈ [ K ] β ⌊ t + K -2 K ⌋ + ∑ ⌊ t + K -2 K ⌋ s =1 ∑ min { sK,t -1 } j =( s -1) K +1 β ⌊ t + K -2 K ⌋-s , then we have

Thus, we get that

̸

̸

̸

̸

̸

̸

̸

Theorem 3. With Assumptions 1, 3 and 4, letting

and η = min { (1 -β ) 2 8 KL , √ (1 -β ) 3 ∆ TLσ 2 } , Algorithm 5 has the following convergence rate:

where ∆ = F ( w 0 ) -F ∗ and ¯ w T is randomly chosen from { w 0 , w 1 , · · · , w T -1 } according to a probability distribution which is related to the delay-adaptive learning rates as shown in (17).

Proof. According to Lemma 9, we can get that

Thus, we can get that

where ¯ η 0 = ∑ k ∈ [ K ] ˆ η k, 0 and ¯ η t = ˆ η k t -1 ,t ( t ≥ 1) .

Then, we analyse the lower bound of ∑ T -1 t =0 ¯ η t .

It's easy to verify that

̸

̸

Since ∑ k ∈ [ K ] ˆ τ k, 0 = 0 , we have ∑ k ∈ [ K ] ˆ τ k,T + ∑ T -1 t =0 τ t = ( K -1) T . Moreover, ˆ τ k T -1 ,T = T -ite ( k T -1 , T ) = 0 . We get that ∑ k ∈ [ K ] ,k = k T -1 ˆ τ k,T + ∑ T -1 t =0 τ t = ( K -1) T . Thus, at least T 2 delays are smaller than 2 K . ∑ T t =0 ¯ η t = ∑ T -1 t =1 ˆ η k t -1 ,t + ∑ k ∈ [ K ] ˆ η k, 0 ≥ Tη 2 .

If we choose an output ¯ w T from { w 0 , w 1 , · · · , w T -1 } according to

where t ∈ [ T ] and let η = min { (1 -β ) 2 8 KL , √ (1 -β ) 3 ( F ( w 0 ) -F ∗ ) TLσ 2 } , we have that

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. We propose OrMo for ASGD in Subsection 3.2. We prove the convergence of OrMo theoretically in Subsection 3.3 and verify its superiority empirically in Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the assumptions of OrMo in Subsection 3.3.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The full set of assumptions and the proof sketch can be found in Section 3.3. Please refer to Appendix C for the complete proof details.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: This paper fully discloses all the information needed to reproduce the main experimental results. The implementation details can be found in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the code. The dataset is public.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The implementation details can be found in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Each experiment is repeated 5 times as reported in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The type of compute workers and the time of execution are presented in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms with the NeurIPS Code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original papers.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn distributed learning environments, asynchronous SGD (ASGD) and its variants are often used to deal with computing nodes with uneven computing power. Momentum methods are beneficial for both optimization and generalization in deep model training, but directly applying momentum to ASGD may hinder its convergence. The paper proposes the OrMo method to integrate momentum into ASGD by sorting gradients based on iteration index. The paper provides a theoretical proof of the convergence of OrMo on non-convex problems.\n\nSTRENGTHS:\n1. A new asynchronous stochastic gradient descent optimization method (OrMo) is proposed, and a theoretical proof of convergence on non-convex problems is provided, which is the first time that it does not rely on the bounded delay assumption.\n2. Experiments show the advantages of the OrMo method in convergence performance.\n3. The paper provides detailed algorithm implementation details and open access links to the experimental code, which enhances the reproducibility.\n4. The experiment setting is clear, and the wall-clock time shows that the OrMo outperforms baseline methods significantly.\n\nWEAKNESSES:\n1. Although experimental validation was performed on the CIFAR dataset, the generalization ability of the OrMo method may need to be further tested in more types of datasets and different application scenarios.\n2. The paper does not discuss in detail the computational resource requirements of the OrMo method on data and models of different scales. Specifically, the ordered momentum need to be cached on the server, the extra storage costs should be considerred.\n3. The proposed re-ordering method inherently is like the staleness control. The difference is that the OrMo proposes using the exponentioal term on \\beta to control the staleness of the incoming gradients.\n\nQUESTIONS:\nFigure 3 shows that the curve is very unstable. So, on which epoch, the model is used as the final model for testing?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new ordered momentum for asynchronous SGD based on the delayed update characteristic of asynchronous training, which weights the momentum according to the actual iteration index. The authors proved the convergence of the algorithm both theoretically and experimentally.\n\nSTRENGTHS:\n* This paper proposes a new momentum algorithm for ASGD and proves that the algorithm is convergent.\n\n* Experiments are conducted in this paper to verify the effectiveness of the OrMo algorithm.\n\nWEAKNESSES:\n* This paper studies asynchronous training algorithms but does not explore the effect of asynchronous delays on convergence.\n\n* The theoretical results in this paper are limited by the number of distributed workers $K$, i.e., the algorithm does not scale well to large-scale distributed training systems.\n\nQUESTIONS:\n1. The theoretical results in this paper do not contain asynchronous delay information and fail to reveal the effect of asynchronous delay on the convergence of the algorithm. This may be due to rough gradient upper bounds that obscure the delay gradient information. However, the reviewer believes that analyzing the effect of asynchronous delay is essential in a work that studies ASGD algorithms. Also, the experimental section did not test the empirical effect of different delays on the algorithm.\n\n2. Theorem 1 shows that the convergence of the algorithm is linearly dependent on the number of distributed workers $K$, which implies that the algorithm will fail to converge when $K$ is large. In light of this theoretical result, can't the algorithm proposed in this paper be applied to large-scale distributed training systems?\n\n3. The algorithm design and theoretical analysis in this paper relies on a fixed constant learning rate. Can this be extended to adaptive learning rates or decreasing learning rates that are more common in practice?\n\n4. The result of Lemma 4 in the main text is inconsistent with the proof in the Appendix.\n\nLIMITATIONS:\nPlease refer to Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Ordered Momentum (OrMo), a novel method enhancing the performance of ASGD by systematically incorporating momentum based on the iteration indexes of gradients. The authors provide theoretical proofs demonstrating the convergence of OrMo for non-convex problems, marking an advancement as this is purportedly the first such convergence analysis for ASGD with momentum that does not rely on the bounded delay assumption. Empirical results further validate that OrMo outperforms standard ASGD and other asynchronous variants with momentum in terms of convergence rates.\n\nSTRENGTHS:\n1. The proposed OrMo algorithm addresses the common problem of momentum integration in asynchronous settings, which is often associated with convergence issues.\n2. The paper provides a convergence analysis of OrMo in non-convex settings without the need for a bounded delay assumption.\n\nWEAKNESSES:\n1. The writing and presentation of the paper seems to require substantial improvement. Especially for Section 3, it is hard to fully follow the algorithmic and theoretical details, making it laborious to check the correctness.\n2. The paper considers only scenarios with homogeneous data distributions, where the workers access to a shared dataset $\\mathcal{D}$.\n3. Assumption 1 includes a bounded gradient assumption, which is rather demanding.\n4. The experiment setup does not quite match the theoretical analysis. In the experiment, the learning rate is multiplied by a factor after a specific number of epochs, while Theorem 1 is based on a constant learning rate. There also lacks discussion on how the multiplication factors and the intervals are tuned.\n\nQUESTIONS:\n1. What is the impact of the size of the waiting set $\\mathcal{C}$? How to choose the size of $\\mathcal{C}$?\n2. What are the key insights that the convergence can be guaranteed without bounded delay?\n3. What is the complexity and cost in terms of the ordering of gradients?\n\nLIMITATIONS:\nThe authors have partially addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces OrMo, a novel method to weight stale gradients received on the server in asynchronous SGD with momentum. The algorithm is based on the idea of organizing the sequence of gradients received on the server into \"buckets\" to approximate standard minibatch SGD with momentum. The method is shown to converge under arbitrary (possibly unbounded) delays, with experiments on CIFAR-10 and CIFAR-100 validating the strength of OrMo compared to some baseline.\n\nSTRENGTHS:\n* **Convergence without bounded delay assumption:** Convergence is proved without relying on the bounded delay assumption, extending recent analysis of ASGD [[Mishchenko et al., 2022]](https://arxiv.org/pdf/2206.07638 ), [[Koloskova et al., 2022]](https://arxiv.org/pdf/2206.08307 ) to ASGD with momentum.\n* **Clear explanation of the \"momentum as a sum of weighted buckets\" idea:** Time is spent to clearly lay down the ideas of \"buckets\" and re-writing the momentum updates so that OrMo comes naturally, which is helped by Fig.1 \\& 2.\n* **Promising experimental results:** Compared to naive implementation (e.g., plain ASGD with momentum) and some chosen baselines, OrMo is shown to perform better in practice, especially in cases where the workers speed is heterogeneous.\n\nWEAKNESSES:\n* **Lack of discussion between OrMo and its approximation of standard synchronous SGD with momentum:** While the idea of OrMo may seem natural when we visualize standard momentum as a \"sum of weighted buckets of gradients\", the update performed by OrMo simply *approximate* synchronous momentum. Indeed, as the parameters hosted on the server are updated at each stochastic gradient in the asynchronous setting, the *points* $w_t$ on which the gradients are computed all differ even inside a given \"bucket\", which is not the case for synchronous SGDm. Yet, this is not discussed.\n* **Standard baselines are lacking in the experiments:** In Table 1\\&2, and in Fig. 3\\& 4, a comparison *(for reference only)* to synchronous SGDm could be interesting to have (to see whether or not there is still a gap to fill with synchronous methods in terms of \"performance per iteration\"). Moreover, it seems that the baselines \"tuning the momentum value\" for asynchronous SGD with momentum described in [[Zhang and Mitliagkas, 2018]]( https://arxiv.org/pdf/1706.03471 ) and [[Mitliagkas et al., 2016]]( https://arxiv.org/pdf/1605.09774 ) are lacking. Finally, we can see OrMo as a method to penalize stale gradients received by the server, using some sort of \"exponential weight\". Empirically, there have been other \"penalty functions\" that have been tried in Asynchronous Federated Learning, such as the ones described Part 5.2 of [[Xie et al., 2020]]( https://www.opt-ml.org/papers/2020/paper_28.pdf ). Have you compared OrMo to them?\n\nQUESTIONS:\n* In Theorem 1, the quantities $L,G$ do not appear in the convergence rate (contrary to what is displayed in Theorem 2 of [[Mishchenko et al., 2022]](https://arxiv.org/pdf/2206.07638 ) ). Is it normal?\n* Fig.5: As the Parameter Server is known to become a bottleneck for communications at scale, how does OrMo fair compared to synchronous SGDm at larger scale than $K=16$? (for instance, $K=64$)\n\nLIMITATIONS:\n.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}