EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models
Authors: Jialiang Cheng
Ning Gao
Yun Yue
Zhiling Ye
Jiadi Jiang
Jian Sha
Venue: ICLR 2025 Poster (2025)
Keywords: Distributed Training
Large Language Models
Local SGD
Training Acceleration
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: We propose a novel Local SGD-based distributed training method for training LLMs effectively and efficiently, and we provide a large-scale verification of asynchronous pre-training for LLMs.

Abstract: Distributed training methods are crucial for large language models (LLMs). However, existing distributed training methods often suffer from communication bottlenecks, stragglers, and limited elasticity, particularly in heterogeneous or large-scale environments. Local SGD methods have been proposed to address these issues, but their effectiveness remains limited to small-scale training due to additional memory overhead and lack of concerns on efficiency and stability. To tackle these issues, we propose EDiT, an innovative Efficient Distributed Training method that combines a tailored Local SGD approach with model sharding techniques to enhance large-scale training efficiency. EDiT performs layer-wise parameter synchronization during forward pass, reducing communication and memory overhead and enabling overlap. Besides, EDiT employs a pseudo gradient penalty strategy to suppress loss spikes, which ensures training stability and improves performance. Additionally, we introduce A-EDiT, a fully asynchronous variant of EDiT that accommodates heterogeneous clusters. Building on EDiT/A-EDiT, we conduct a series of experiments to validate large-scale asynchronous training for LLMs, accompanied by comprehensive analyses. Experimental results demonstrate the superior performance of EDiT/A-EDiT, establishing them as robust solutions for distributed LLM training in diverse computational ecosystems. The code is available at Atorch codebase: https://github.com/intelligent-machine-learning/atorch/tree/main/atorch/local_sgd.

Published as a conference paper at ICLR 2025
EDIT: A LOCAL-SGD-BASED EFFICIENT DIS- TRIBUTED TRAINING METHOD FOR LARGE LAN- GUAGE MODELS
Jialiang Cheng, Ning Gao, Yun Yue, Zhiling Ye, Jiadi Jiang, Jian Sha
Ant Group
bjichen.cil yunsheng...... gn, yueyun. aaeeyu.yyy@aatteeeee eantgroup.. com (yezhiling yzl, farl,jiadd jiadi, jidd jian. sha Bantgroup. com
ABSTRACT
Distributed training methods are crucial for large language models (LLMs). How- ever, existing distributed training methods often suffer from communication bot- tlenecks, stragglers, and limited elasticity, particularly in heterogeneous or large- scale environments. Local SGD methods have been proposed to address these issues, but their effectiveness remains limited to small-scale training due to ad- ditional memory overhead and lack of concerns tackle these issues, we propose EDiT, an innovative Efficient Distributed Training method that combines tailored Local SGD approach with model sharding tech- niques to enhance large-scale training efficiency. EDiT performs layer-wise pa- rameter synchronization during forward pass, reducing communication and mem- ory overhead and enabling overlap. Besides, EDiT employs a pscudo gradient penalty strategy to suppress loss spikes, which ensures training stability and im- proves performance. Additionally, we introduce A-EDiT, a fully asynchronous variant of EDIT that accommodates heterogeneous clusters. Building on EDIT/A- EDIT, we conduct a series of experiments to validate large-scale asynchronous training for LLMs, accompanied by comprehensive analyses. Experimental re- sults demonstrate the superior performance of EDiT/A-EDiT, establishing them as robust solutions for distributed LLM training in diverse computational ecosys- tems. The code is available at Atorch codebase |
efficiency and stability. To
INTRODUCTION
With the explosive growth of model scale and data volume (Touvron al. 2023 Bai et al. 2023) distributed methods (Rajbhandari et al.] 2020 Narayanan et al.) 2021 Dean et al. 2012) become increasingly critical for training deep neural networks. These approaches rely synchronous paradigm, which introduces significant communication overhead during the training process (Douil- lard al. 2023). Besides, the synchronous paradigm also introduces the straggler problem, where Taster workers are idle waiting for the slower ones to catch up. This issue is particularly prevalent in large/heterogemeous clusters (Lian al. 2018).. Lastly,
clusters, there is a compelling need for clastic training (Li et al.) a (023). However, synchronous training paradigms struggle in elastic settings, where dynamic scaling of resources disrupts optimal hyperparameters.
ausource-constraing
These challenges have spurred significant research into distributed optimization methods. A typi- cal method is Local Stochastic Gradient Descent (a.k.a Local SGD or Local-Update SGD) (Zhang et al., 2016), where each worker independently executes multiple local optimization steps in parallel before averaging model parameters across all workers. Subsequent studies have improved upon this foundational paradigm to improve the performance (Lin et al.) 2019 Wang et al. 2019 Douillad ct al. 2023 However, existing Local SGD methods are not casily applicable to the training of
large language models (LLMs). These methods do not handle model sharding well, preventing their application to models larger than billions of parameters. Moreover, they have focused on small-
"Corresponding author. https://github..co weeeeellggggneeeee meent-machinn ne- leaachiinn-eemmmm/tttttrr

---

Published as a conference paper at ICLR 2025
Forward
Backward
Local Update
Worker
Worker
Masade Bruet
Macale
Worker
Module
Upolate
Dulteer
Forward
Parare
Outther
world
Seatter
Rockase
Medida Hoar Maitt
Modkae Hout (eact
Params (Periodic)
Syno Grads (Warmup)
Worber
Module!
Module!
Worker C
Worker B
Redace
Gather
Ferward
Pauarr
Gather
word
Scattrr
Reddcee
Paraam
Parame
Mesaaa
Parave
Mockil
Paresne
Gather Params
Gather
Grads
Best
Wonker
Battwrr
Module I
Fonward
Module
Backoe
Rull Pazase
Updatt Params
Params
Held
Released
Paroms
Operation
Operation (Options)
(Model Sync Group)
Communication
Communication
(Modal Shard Group)
Figure I: The schematic illustration of our proposed EDiT method with 4 workers and a 2 x 2 device mesh example. The left part shows the communication groups and parameter sharding, and the right part presents the detailed computation and communication flows within worker B.
scale, highly curated datasets (Zhang et al., [0166] Douillard al. 2023 making their results less transferable to LLM training that relics on vast, noisy datasets where instability may be introduced during training. Besides, although current Local SGD methods diminish the impact of random strag- glers, they still struggle with the presence of consistent stragglers within heterogeneous devices (Liu et al.) 2024). Additionally, in most existing Local SGD methods, parameter synchronization oper- ations will introduce non-overlapped communication overhead (Sun et al. 2023). Lastly, current Local SGD methods predominantly employ uniform averaging strategy to synchronize the pa- rameters, failing to fully capitalize on the inherent differences in training progress across diverse workers (Douillard et al.) (023).
To address these challenges, we propose a novel Efficient Distributed Training (EDiT) method for large language models. As illustrated in Figure EDiT employs a hierarchical distribution strategy on two-dimensional device mesh, where all workers are data parallel. Model parameters are fully sharded along the model shard dimension and synchronized along the model sync dimension. With the efficient communication links within the model shard groups and the low-frequency periodic synchronization strategy within the model sync groups, the impact of communication overhead and random stragglers is effectively alleviated. When synchronizing parameters, EDiT operates layer by layer during the forward pass and makes use of a prefetch strategy to overlap computation and communication, thereby reducing the additional communication and GPU memory overhead intro- penalty
duced by parameter synchronization. Additionally, EDiT employs a novel method, which addresses the instability problem caused by diverse large-scale corpus and leverages the differences among workers to improve performance. Furthermore, we propose an asynchronous variant of the EDiT method named A-EDiT deal with the consistent stragglers in heterogeneous clusters. We conducted comprehensive evaluation of our proposed methods on LLM tasks, demon- strating its effectiveness compared to state-of of-te-of-thh-art thet-art methods
Our primary contributions can be summarized as follows:
Engineering Innovation: We introduce EDiT, an efficient large-scale distributed training method that integrates Local SGD with the model sharding strategy. EDIT reduces the impact of stragglers and communication overhead and supports elastic training.
Algorithmic Novelty: EDiT performs layer-wise parameter syne during forward pass to reduce communication and memory overhead. With prefetch strategy, the parameter-sync communication can be further overlapped with computation. Besides, we propose new pseudo gradient penalty method to improve the training stability and model performance. We also provide fully asynchronous variant of EDiT, called A-EDiT, address the chal- lenges of consistent stragglers.
Practical Contributions: We provide a large-scale verification of asynchronous pre-training for LLMs, along with extensive analysis of convergence, generalization, acceleration, scalability, and stability. This work offers critical insights into optimizing asynchronous distributed L.M training at scale.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
One of the early works that proposed the concept of Local SGD Zhang et al. (2016). establishing the paradigm of parallel multi-step training followed by periodic averaging. [in et al. (2019) intro-
duced the Post Local SGD method, which starts with standard synchronized training for warm-up before switching to the Local SGD mode. SlowMo (Wang et al., al., 2019) utilizes a slow momentum to transform model averaging into moving average. DiLoCo (Douillard et al. 2023) demonstrates that the Nesterov optimizer (Nesterov) 1983) is suitable as an outer optimizer. Multi-Level Local SGD (Castiglia et al. 2020) partition the network into disjoint sub-networks and hierarchically syn- chronizes the models. Wang & Joshi (2019) and Balles et al. (2023) have respectively explored the optimal hyperparameter settings for Local SGD. SGD.|Shen al. (2021) advocated for gradually in- creasing synchronization intervals while decreasing learning rates to optimize model performance. Extensive theoretical analyses of Local SGD have also emerged. Yu et al. (019), Khaled et al. (2200), Spiridooof et al. (2020), and Deng et (2022) examined convergence rates under various conditions. Gu et al. (2022) found that Local SGD improves generalization with a small learning
rate and long training duration. Pan & Song (2023) demonstrated faster convergence by leveraging
second-order information.
Researchers have also explored the combination of Local SGD with asynchronous training paradigms that decouple computation and communication. Early works were predominantly based on the federated learning framework (Xie et al., (019). FedBuff (Nguyen et al.] 2022) updates the server model only after accumulating a certain amount of pseudo gradients. DN- -N-L (Liu et al, improves the buffer mechanism to employ delayed Nesterov update. TimelyFL (Zhang al,
2024
2023
dynamically adjusts the local training workload according to the real-time resource situation.
Subsequently, several works based on other architectures were also proposed. Gossip-PGA (Chen
iet al.
2021) incorporates periodic global averaging into the gossip SGD framework (Lian et al.j CO2 (Sun et al., 2023) utilizes Local SGD and asynchronous communication hide the overhead. A key challenge for asynchronous training is the staled model problem, resulting in infe- rior performance compared to synchronous training methods (Liu et al. 2024).
2017
Local SGD methods (Lin et al. 2019 Wang et al. 2019 Sun al., (023) hold complete model parameters on each GPU, making difficult to handle model sharding for LLM training. Although [Sun et al. (2023) claims that they can combine CO2 with ZeRO series optimizers (Rajbhandari et al., 2020). the additional communication introduced de- grades CO2 to synchronized mode, negating the performance gains from periodic synchronization and overlapped communication. Furthermore, the extra parameters and outer momentum further proposed
Notably, current
increase memory pressure, limiting their scalability to larger models. In contrast, EDiT and A-EDÍT methods effectively utilize the characteristics of model sharding, leveraging de- vice mesh, layer-wise parameter synchronization, prefetch strategy, and CPU offload to minimize communication and memory overhead, making it more suitable for LLM training.
3 METHOD
3.1 OVERVIEW
Our proposed EDiT method integrates model sharding with periodic synchronization to acceler- ate the training of LLMs. The detailed procedure of EDiT is illustrated in Figure [ and for- mally outlined in Algorithm in Appendix. To start with, EDiT builds an M x N device mesh across K workers : M model sync groups g" = [9i,... .hr with each comprising N workers G.- = (W(a)): -i..)..... W(.N)) 2 and N model shard groups 9d = { ....... Gy } with each comprising M workers 9 (Wa,s), W(2.s.... W,M where M x N K. This structured arrangement aims to tailor communication patterns to the diverse capabilities and network latencies inherent in the distributed system. For instance, a multi-node GPU cluster where intra-node com- munication is significantly faster than inter-node communication, all GPUs within the same node can be connected as a model shard group, while GPUs of the same rank across different nodes can be connected as model syne group. Model parameters are sharded uniformly in each model
TThe employment of double subscripts herein is merely notational convenience to denote the relationship hetween workers and groups. Similar considerations apply the cases discussed helow.
3

---

Published as a conference paper at ICLR 2025
shard group and each worker W; retains fraction of each parameter for the complete L modules: ((0) = (0(01)
(o ... ((.. In this way, workers as whole within model shard group 9 main- tain a complete replica of model parameters: O Concat(f@()) O E ); while workers within model sync group G maintain an identical shard of the parameters. The EDiT method centralizes communication-intensiiy operations within the model shard groups and utilizes peri- odic synchronization to mitigate the communication overhead in the model sync groups, thereby achieving training acceleration. To enhance the stability of the initial training process, EDiT utilizes a two-phase training strategy. This begins with a warmup phase using standard mini-batch SGD, followed by periodic synchronization phase utilizing Local SGD. More specifically,
During the forward pass of the I-th module, if the current updated step requires model synchronization, i.e, (1+ bearm and = 0 where is the outer step, p is the inner step, wwarm the number of warmup steps and r the synchronization interval, parameters are synced in model syne groups, outlined in lines 7 to 9 in Algorithm [ In practice, the communication overhead is minimal due to the large synchronization interval (r >> 1) and sharded parameters. Herein a novel pseudo gradient penalty strategy is introduced to enhance training stability that will be detailed in Section 3.2 After that, each worker gathers the full module parameters through its model shard group for forward computations and promptly frees excess parameters to conserve memory.
2. During the backward pass of the I-th module, workers again aggregate parameters via model shard groups for gradient calculations, followed by a reduce-scalter operation to average gradients across each model shard group. the current step within the warmup phase, i.e., S twarm an additional all-reduce operation will be performed within each model syne group to synchronize gradients across all workers; otherwise this operation will be skipped (lines 19 to 21). Afterwards, each worker frees the excess parameters.
3. Once modules have completed one forward-backwar the local parameters of each worker. Note that to distinguish from the outer optimizer (OuterOpt) used in parameter synchronization, we refer to the optimizer for local updates the inner optimizer (InnerOpt).
iteration, the optimizer updates
Different from other Local SGD methodologies that synchronize parameters before next step, EDiT performs layer-wise parameter synchronization during forward pass. In practice, we normally em- ploy a prefetch strategy that aggregates parameters for the upcoming module concurrently with ongoing computations, with which communications within model sync groups can be effectively overlapped with forward computations. In this way, EDiT further diminishes the additional commu- nication overhead introduced by parameter synchronization.
is also noteworthy that EDiT is compatible with most current large-scale distributed training frameworks. Although this manuscript mainly discusses its integration within ZeRO-3/FSDP frame- work (Rajbhandari et al. (2020), can be transposed with relative case to other frameworks such as 3D parallelism (Shoeybi et al. 2019).
3.2
PSEUDO GRADIENT PENALTY
Gradients Penalty
Worker A
Anomaly
Eliminate Anomalies
OuterOpt
Worker
157
OuterOpt
Clip
Sync
Worker C
OuterOppt
Worker D
Outertppt
Figure 2: Illustration of model synchronization and our proposed pscudo gradient penalty method, depicted with an example of four workers in a model sync group.
Despite diligent data cleaning efforts, there are still significant amount of low-quality data in the L.M pre-training corpora (Albalak al. 2024), resulting in training instability manifested as loss
4

---

Published as a conference paper at ICLR 2025
spikes. This issue can be addressed by large batch sizes typical of synchronous training regimes, but becomes salient in Local SGD regimes where each worker operates on relatively smaller batches.
To tackle this issue, introduce novel pseudo gradient penalty strategy the parameter syn- chronization stage, as depicted in Figure 2 and Algorithm 2 in Appendix. This strategy consists of anomaly elimination, weighted averaging. and gradient clipping. To illustrate the idea, use model syne group 9T = (W...... } as an example. We begin by computing the pseudo gradi- ents A (+.)) ( -0(-.) for each worker, where ( is the sharded parameters of module held
by worker W; at outer step t and inner step T, and o((.)) denotes the corresponding synchronized parameters at the beginning of outer step t.
Anomaly elimination. We first eliminate the significantly anomalous workers reduce their ad- verse impacts on the overall model performance. Since anomalies cause substantial parameter fluc- tuations and lead large aseudo-gradient norms, we use the aseudo-gradies norm the criterion. Here we utilize an Exponential Moving Average (EMA) z-test method for statistical analysis. Let G( A (t.0) 222 denotes the pseudo gradient norm for the worker W;, then the EMA z-score can 15""" (6.1) where Ui (i.d) and o; (i.d) are the EMA mean and standard devi-
be calculated by (e.I) G. ((.
(e.1) > is identified as an anomaly and its G( ation of G( respectively. worker W; with 21
will be set to infinity, where is threshold, typically set to 3 in practice. Both 14 and of (1.)) are updated at each step using an exponential moving average to capture the convergence trend of the gradient norm during the training process:
(3.1) Mi+1 = ac! + - (1-a) 6.0
2.-00 -
a(G) - m2t1)?
(1)
where a is a weighting coefficient, commonly assigned a value of 0.02 practical applications. The update of Equation | will be skipped if ((.)) is infinite. In the preliminary stage, warm-up period is set to establish stable values for μ4: (5.0) and of (8.1) during which no workers are flagged as anomalies. Notably, to maintain consistent updates within the same module, compute the pseudo gradient norm for the entire module, and subsequently introduced gradient norm corm-related operations follow the same procedure. Because this process only introduces model shard groups, the overhead is negligible. If all workers are identified anomalous, all the parameters will be effectively rollbacked to the last synchronized parameters 0f
scalar communication in the
Weighted averaging. Furthermore, considering that large pseudo gradients may still exert consid- erable impacts on the overall update direction, propose to weigh the pseudo gradients of each worker based on the norms, which was similarly demonstrated in Thakkar et al. (2023). The weight assigned the pseudo gradients corresponding W,
9% is calculated by
exp(-G,
Wtii
Σ, (xp(-GG-1)
In this way, larger pseudo gradient norm leads stronger suppression, thereby allowing all workers to contribute equally to the update direction and thus increasing the likelihood to find the correct direction. Following that, by performing a weighted summation of all pseudo gradients in G" obtain the synchronized pseudo gradients:
A (i.))
Σ
(1) V W2 Gr wi.
Gradient clip. We then adopt a gradient clip strategy to constrain the update step size. Let Glddd) = ا (100) denote the synchronized pseudo gradient norm and denote the threshold, the clip coefficient computed by
B, = min // (G ((t.)) +e), 1),
(4)
where e is a small positive constant to avoid division by zero. The pseudo gradients are clipped by
A (1.1) B (6.))
(5)
5

---

Published as a conference paper at ICLR 2025
Proceeding further, we update of by 0441 OuterOpt((( ( each worker.
In EDiT, the extra parameters and outer momentum on each worker are sharded in correspondence with the sharded parameters. Compared previous methods that maintain full extra parameters and outer momentum on each worker, EDiT reduces the additional memory usage. Additionally, based on layer-wise synchronization and prefetch strategy, EDiT can further offload the extra parameters and outer momentum to the CPU and only transfer the corresponding layer's data to GPU needed, thereby further minimizing memory overhead. Since the data for each layer is relatively small, the GPU-CPU data transfer can be effectively overlapped with GPU computations and GPU-GPU communications, ensuring fast parameter synchronization.
3.3
ASYNCHRONOUS EDIT
Faster Worker
Faster Worker
Update
Updats
|
Sync
Update
Local Update
Local Update
Sync
Slower Worker
Slower Worker
Local Update
Update
Sync
Local Update
Local Updaate
Step
Time
(a) The synchronization scheme of EDiT.
(b) The synchronization scheme of A-EDiT.
Figure 3: comparison of the synchronization scheme of EDiT and A-EDiT.
EDiT requires periodic synchronization at every т inner iterations. However, the fastest worker idles awaiting the peers finish iterations even if it completes its own iterations earlier. As consequence, the overall training efficiency is pegged to the slowest worker. This issue becomes more pronounced in heterogeneous clusters, where nodes are equipped with diverse devices.
Intuitively, it would be beneficial to allow different workers to train at their own pace and remove the constraint of fixed-step synchronization. Therefore, we propose an asynchronous variant of the EDiT method, named A-EDiT. The differences are depicted in Figure 3 Herein, we set a fixed time interval Ttime. and let each worker update locally until surpassing this specified time threshold. Then, parameter synchronization ensues. This modification enables faster workers to undertake more iterations each inner loop. Theoretically, no worker will wait longer than the single step time of the slowest worker at each parameter synchronization. We empirically verified that A-EDiT achieves faster training in all scenarios with comparable model performance.
EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Models We consider four different scales of Llama models (Touvron et al. 2023) in our experi- ments: 350M, IB, 3B, and 7B. Their specific configurations are detailed Table Appendix.
Datasets We use new large-scale open-source dataset, Fine -du (Lozhkov et al. 2024 in our experiments. Additionally, we also utilize an in-house private out of production pre-training dataset, which we will refer to as in-house dataset below.
Baselines We consider several state-of-the-art methods, including standard mini-batch (Baseline), Post Local SGD (Lin et al.] 2019). DiLoCo (Douillard et al. (003), and CO2/CO2* (Sun et al.) 2023 Here CO2#1 is the memory-
version of CO2 that shards extra parameters and outer momentum across workers (Sun et al. 2023).
Training Following DiLoCo (Douillard et al. 2023), we use AdamW (Loshchilov & Hutter 2019) as the inner optimizer and Nesterov momentum (Nestervv) 1983) the outer optimizer. The models are initialized with jP (Yang et al. 2021) for efficient hyperparameter search. Synchronization intervals and Trime set 128 and 600s, respectively. Experiments are conducted eight Nvidia A 100 GPU nodes with 64 GPUs and an 8 x 8 device mesh. Φ is 10 for pseudo gradient clip.
For more detailed setups, please refer the Appendix A.2

---

Published as a conference paper at ICLR 2025
4.2
CONVERGENCE AND GENERALIZATION
4.5
2.50
30B 250
155
1056
115%
150c
255
306
(a) FineWeb-Edu loss.
(b) Fine-eb-Edu PPL.
(c) In-house loss.
(d) In-house PPL.
Figure 4: The loss and PPL curves of different methods the (a) & (b) Fine Web-FEdu dataset and
(c) & (d) in-house dataset. The final values are marked, with the best ones in bold. Here we use the average of the last 10 values as results prevent randomness. PLS short for Post Local SGD.
'able 1: The evaluation results for different methods the public benchmarks (Fourrier et al. 2023 OpenCompaas Contributors 2023). with the best results in bold and second-hes, underlined. PLS is short for Post Local SGD.
FineWeb-Edu dataset
in-house dataset
Benchmark
Baseline
PLS
DiLoCo
CO2
EDIT
A-EDIT
Baseline
DiLoCo
EDIT
A-EDIT
MMLC (t)
32.28
30.86
32.55
31.33
32.29
31,96
24.12
24.63
24.47
24.56
ARC-E (1)
59.90
57.60
58.60
57.00
60.00
57.70
36.80
36.70
38,70
37.40
ARC-C (1)
30,20
28.60
31.00
30.50
32.40
30.20
22.50
22.80
23.00
22.40
HellaSwag (f)
50.99
48.03
51.64
48.66
51.75
$1.60
40.60
40.80
40.90
40.20
PIQA (t)
69.90
67.80
69.50
67.00
68.10
69.90
67.10
66.80
67.00
66.40
CommonSense-QK (1)
37.40
33.80
35.40
34.40
36.30
35.30
18.50
18.20
18.50
17.90
OpenBookQA (f)
25.40
22.80
25.20
24.40
26.00
24.00
18.00
17.80
18.00
18.20
WinoGrande (t)
50.70
49.20
47.80
50.70
51.70
50.50
49.10
49.20
49.10
48.80
Average (1)
44.60
42.34
43.96
43.00
44.82
43.90
34.59
34.62
34,96
34.49
We first applied different methods to train the Llama 1B model on the FineWeb-Edu dataset and in-house dataset separately. Here we only compared the best-performing methods, ie.. Baseline, DiLoCo, EDiT, and A-EDiT, on the in-house dataset. The training loss (L) and validation PPL (1) results are shown in Figure
As can be seen, our proposed EDiT and A-EDiT both achieve consistently good performance. Specifically, EDiT achieves the lowest training loss on both datasets and achieves the lowest val- idation PPL on the FibeWebEEdu dataset, even surpassing the Baseline. A-EDiT marginally lags behind the sync version due to the lagging workers, but it still performs better than other methods in most scenarios. Because the in-house dataset contains diverse data types and lower-quality corpora, DiLoCo (Douillard et al. 2023) experienced a noticeable decline in performance. In contrast, EDiT and A-EDiT filtered out low-quality data with the pseudo gradient penalty strategy, achieving results that were nearly comparable to the Baseline.
We evaluated the trained models on public benchmarks (Fourrier et al. 2023 OpenCompass Con- tributors 2023). Table D
presents the evaluation results. As be seen, the models trained with EDiT both achieve the best average performance, and A-EDiT also performs well on the eight eval- uation benchmarks. These results demonstrate that both EDIT and A-EDiT exhibit strong conver- gence and generalization capabilities.
Besides, we additionally trained the Llama 350M, 3B, and 7B models on the FineWebEEdu dataset using EDiT, the results in Figure 8 and Table 5 demonstrate that EDiT performs consistently well across different model scales.
4.3 ACCELERATION
We measured the speeds of different methods when training Llama models of four different scales on two A100 nodes. The synchronization interval was set to 5, and the results are the average
In this manuscript the means the bigger the hetter and the J. means the smaller the better.

---

Published as a conference paper at ICLR 2025
The speeds of different methods on training models of various scales. The values in the table correspond to throughput (tokens/sec) and TFLOPS, respectively.
Table 2:
Baseline
Post Local SGD
DiLoCo
CO2
CO2*
EDiT
A-EDiT
350M
4.52e5/107
4.67c5/111
4.56c5/108
4.84c5/116
4.66e5/110
4.81c5/114
4.82e5/115
1B
2.08c5/146
2.12c5/149
1.87e5/131
OOM
2.12e5/148
2.25e5/158
2.27c5/160
3B
1.05e5/177
OOM
OOM
OOM
OOM
1.11551187
1.12c5/189
7B
5.14c4/200
0OM
OOM
OOM
OOM
5.42c4/211
5.45c4/213
throughput (tokens/sec) and TFLOPS over 100 steps. As shown in Table 2 all Local SGD-based methods achieved higher throughput than the Baseline. It should be noted that when training the Llama 1B model with DiLoCo, extra parameters and outer momentum were placed on CPUs to prevent out of memory (OOM). resulting in non-overlapped extra GPU-CPU data transfer overhead. While CO2 achieved the highest throughput on the smallest model, holding extra parameters and outer momentum caused significant memory overhead preventing the method to be scaled beyond 350M model. CO2* alleviates memory pressure by sharding extra parameters and outer momentum, but introduces additional non-overlapping communication, causing throughput drop. Our pro- posed methods synchronize sharded parameters layer-by-layed during the forward pass, and utilize prefetch strategy to overlap computation with communication, achieving nearly the same through- put as CO2 (-0.5%). We also performed profiling analysis of the synchronization operations for different methods, and detailed results can be found in Appendix
A.3.2 and Figure
240
240
TFLOPS
200
69
200
TFLOPS
200
160
160
180
120
120
Buxelie
120
0
1.5
2.5
3.5
4.5
1.5
2.5
3.5
4.5
80
10
20
30
40
Lag Time (s)
Lag Time (s)
Repeat Times
(a) Random Straggler.
(b) Consistent Straggler.
(c) Limited Bandwidth.
Figure 5: The TFLOPS of different methods under different training scenarios.
We further evaluated the training speed of our proposed EDiT and A-EDiT methods against the Baseline method in various more challenging training scenarios. Here we manually introduced stragglers and communication delays. Specifically, simulated stragglers by pausing the training process of one selected node at each step, and simulated inter-node bandwidth constraints by artifi- cially repeating inter-node communications. Experiments were conducted the Llama 7B model. Detailed experimental results are presented in BigureBJand Table Appendix.
The results reveal consistent trend where A-EDiT and EDiT outperforms the Baseline method. As anticipated, the Baseline's training speed declines rapidly with increased lag time or inter-node congestion. In the random straggler scenario, EDiT and A-EDIT experience only slight speed ductions. This is attributed to the periodic synchronization that ensures relatively uniform training speeds across workers. In the consistent straggler scenario, since the cumulative delay at a single node cannot be eliminated by periodic synchronization, the performance of EDiT declines visibly. A-EDiT, leveraging its asynchronous nature, maintains high training speed. In the bandwidth- constrained scenario, both EDiT and A-EDíT are not affected. This is due to the large synchro- nization interval, which minimizes inter-node communication overhead. In summary, our proposed methods consistently demonstrate superior training speed compared to the Baseline method across various scenarios, and A-EDiT further effectively addresses the issue consistent stragglers.
4.4 SCALABILITY
Elastic training is the ability to dynamically adjust the resources in accordance with workload fluc- tuations. However, varying the resources alters the global batch size and requires additional learning rate tuning. Intuitively, the optimal learning rate of the Local SGD methods may be solely related to
8

---

Published as a conference paper at ICLR 2025
69
34
Validation Perplexity
60
Valldation Perpiceity
31
Basakined 2-4-0
EDIT11-2-488 17.59
Basufmu8--22-11 18.11118..
28
EDIT(6-4-2-1)
25
Valisttion
Warkee-
50
22
80
Warter
45
worker
her-d
warser 16
worker-
14.0-1335--33
135 -200-115 -1.30
-14.0 11.5--13--130---11
13.00-133-12-5 12,5 -12 -11.5-1111000 -11.0
Loarning Rate (Log2)
Loarnin Rata (Log2)
(a) PPL-LR curves of Baseline.
(b) PPL.-R curves of EDiT.
(c) PPL curves in elastic training.
Figure 6: (a) (b) The PPL results against learning rates (log 2 scale) under different number of
workers for the Baseline and EDiT methods. (c) The curves in the simulated training scenarios.
the per-worker batch size, which has not been extensively studied in prior research. To validate this hypothesis, we conducted experiments on the Llama 350M model to investigate the optimal learn- ing rate shift for the Baseline method and EDiT method under different worker numbers, fixing the batch size per worker at 128. The validation PPL results are shown in Figure 6a and Figure and the detailed training losses are shown Figure IO in Appendix. It can be seen that as the worker number increases, the optimal learning rate for Baseline gradually increases, whereas that for EDiT consistently remains 1.5e-4. These results validate our hypothesis. The scalability of EDiT makes it suitable for elastic training scenarios. Besides, this property enables us to economize resources by initially tuning the learning rate on single worker before scaling up hundreds of workers. We also note that the training loss for EDIT is more stable than that of the Baseline method across different worker numbers and learning rates, as shown in Figure [0] This not only demonstrates the robustness of EDiT but also highlights its potential to maintain consistent performance across diverse training configure gurations..
We further simulated realistic elastic training scenario. We conducted experiments on the Llama IB model, setting the batch size per worker to 128 and fixing the learning rate at 1.5e-4. We system- atically scaled the worker number upwards (1-2-4-8) and downwards (8-4-2-1), training for 25,000 steps at each worker number, and observed the validation PPL for both the Baseline and EDiT meth- ods. As illustrated in Figure [6c| although the Baseline method initially decreases faster than EDIT, EDiT maintains a significant decline rate the later stages and achieves the optimal PPL values in both scaling scenarios, yielding 4.5% and 2.6% improvement, respectively. These findings affirm the EDiT's viability and advantage in real-world, elastic training scenarios.
4.5 ABLATION STUDY
Sendin
6.0
5.5
Worke
Valtaaton Perpicxtty
12.85
5.5
UA 11
5.0
Training Lops
4.5-
4.5
4.0
4.0
Warker
18
3.0
12%
2.
25
Stepe
Validation PPL.
(b) Training loss of DiLoCo.
(c) Training loss of EDIT.
Figure 7: (a) The validation PPI. curves of different versions of EDIT with the final PPL. values marked. (b) & (c) The training loss curves for DiLoCo and EDiT. respectively.
We conducted ablation studies on the pseudo gradient penalty strategy to better understand its capa- bilities. In this experiment, we employ the in-house dataset as is of higher diversity and thus serves as ideal testbed. We individually removed anomaly elimination (w/o AE), weighted averaging (w/o WA), and gradient clip (w/o GC) from EDiT, as well as all three components simultaneously (w/o ALL). The validation PPL results shown in Figure 7a It can be observed that without the pseudo gradient penalty strategy (w/o ALL), the PPL curve exhibits noticeable spikes and deviates considerably from the Baseline. Individually removing anomaly elimination, weighted averaging.

---

Published as a conference paper at ICLR 2025
gradient clip each adversely affects stability and validation PPL, demonstrating that every compo- nent of the pseudo gradient penalty strategy is effective. We further investigated the training losses across eight different workers. As depicted in Figure 7b and Figure 7c all workers in DiLoCo si- multaneously encounter loss spikes and take a long time to recover. contrast, EDiT can swiftly rectify deviations in individual workers. Even when all workers experience abnormal losses, they can promptly revert to normal loss levels through the rollback strategy. These results demonstrate the effectiveness of the pseudo gradient penalty strategy.
THEORETICAL ANALYSIS
In this section, we choose SGD (Robbins & Monro 1951) the inner optimizer and the outer opti- mizer for simplicity. Under the framework developed in Wang et al. (201)), we have the following convergence theorem.
Theorem 1. Suppose that the following assumptions satisfied:
1. L is differential and lower bounded, i.e., L(0") > -oo where 0" is an optimal solution. L is also L-smooth, i.e., Vu, V E RR, , we have ((u) 1 [(v) + (VL(v), u - v) + 1 u v112.
2. At the outer step and inner step V W; E Gim. m E 1, 1, M, the algorithm can access bounded noisy gradient and the true gradient bounded, i.e., |1 |x Go., [|[gga))l S
Gx v [
{0, -
1}.
3. The noisy gradient is unbiased and the noise is independent, i.e., Pi, Elgi.p (3) + St.p E(GE
(
(8) 0 and (() is independent of o ift # t' p p'.
4. The learning rate of the inner optimizer is Htp 11/ VT + 1, and the learning rate of the outer optimizer v.
Then Algorithm yields
min 16/7-111,EE- B[1
2VFn(VT-1) 1) [(00.o) LoGGoo + In(TT))
(1In(())))))))
bonG" In In(TT)) (6)
26
where the meaning of n, and are listed in Table of Appendix A.4
The proof of Theorem iis presented in Appendix A.4 Therefore, the convergence (to the stationary point) rate of EDiT is O(log(T)/VT).
6 CONCLUSION
In this work, we investigate the challenge of training LLMs on large-scale clusters. We analyze the fundamental characteristics of large scale clusters and the limitations of the existing Local SGD-type methods. On this basis, we propose a novel Efficient Distributed Training method for LLMs called EDiT. This method effectively integrates model sharding strategies with tailored Local SGD mech- anisms. We propose layer-wise synchronization to achieve overlap of computation and communica- tion and reduce communication and memory overhead. We enhance the convergence and stability of EDiT by introducing pseudo gradient penalty strategy. We also present an asynchronous variant of EDiT (A-EDiT) to tackle the problem of consistent stragglers in heterogeneous clusters. Extensive experimental results demonstrate the superior capabilities of our proposed methods across multiple dimensions, and the convergence analysis provides a thcoretical foundation for our method.
Several potential avenues for future research are identified. First, for the A-EDiT, the stragglers negatively impact the overall performance. Mitigating the impact of these stragglers warrants further investigation. Second, simulation of elastic training currently entails halting and restarting the training process upon node addition or subtraction. We look forward to a truly clastic framework that can swiftly adjust training resources without disrupting the ongoing training process.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Ke Zhang for bringing the straggler and communication issues in distributed training to our attention and fully supporting our research on this topic. We thank Ji Zhang for providing us with insights into Local SGD. We thank Haitao Zhang for his technical advice and assistance. We thank Chunjie Shen for helping to apply our techniques to practical training scenarios.
REFERENCES
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang- A survey on data selection for language models. 6827
arXiv preprint arXiv:2402. /6827, 2024. htt.ps:/arrxv..org///eae///
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint (tt::2309.16609, 2023.
Lukas Balles, Cedric Archambeau, al. On the choice of learning rate for local sgd. Transactions on Machine Learning Research, 2023.
Timothy Castiglia, Anirban Das, and Stacy Patterson. Multi-level local sgd: Distributed sgd for heterogeneous hierarchical networks. International Conference on Learning Representations, 2020.
Yiming Chen, Kun Yuan, Yingya Zhang, Pan Pan, Yinghui Xu, and Wotao Yin. Accelerating gossip sgd with periodic global averaging. In International Conference on Machine Learning, pp. 1791 1802. PMLR, 2021.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc' aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. Large scale distributed deep networks. Advances in neural information processing systems, 25, 2012.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Local sgd optimizes overpa- rameterized neural networks in polynomial time. In International Conference on Artificial Intel- ligence and Statistics, pp. 6840 6861. PMLR, 2022.
Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc' Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low- communication training of language models. arXiv preprint arXiv:2311.08105, 2023.
Clémentine Fourrier, Nathan Habib, Thomas Wolf, and Lewis Tunstall. Lighteval: lightweight framework for Ilm evaluation, 2023. URL
lighteval
Xinran Gu, Kaifeng Lyu, Longbo Huang, and Sanjeev Arora. Why (and when) does local sgd gen- eralize better than sgd? In The Eleventh International Conference Learning Representations, 2022.
Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. Tighter theory for local sgd on identi- cal and heterogeneous data. In International Conference on Artificial Intelligence and Statistics, pp. 4519-4529. PMLR, 2020.
Jiamin Li, Hong Xu, Yibo Zhu, Zherui Liu, Chuanxiong Guo. and Cong Wang. Lyra: Elastic scheduling for deep learning clusters. Proceedings of the Fighteenth European Conference on Computer Systems, pp. 835-850, 2023.
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems, 30, 2017.
Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning, pp. 3043-3052. PMLR, 2018.
11

---

Published as a conference paper at ICLR 2025
Tao Lin, Sebastian U Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini-batcher use local sgd. In International Conference on Learning Representations, 2019.
Bo Liu, Rachita Chhaparia, Arthur Douillard, Satyen Kale, Andrei A Rusu, Jiajun Shen, Arthur Szlam, and Mare' Aurelio Ranzato. Asynchronous local-sgd training for language modeling. arXiv preprint ar arXiv:2401.09135, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL
?id=
Bkg6RicqY7
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu, May 2024.
URL https:://hhgiingggeeteeee cot.a/cceecccataatttteeeee http://///ggggggccc//////eeeeeee
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vi- jay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-Im. In Proceed- ings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-15, 2021.
Y. E. Nesterov. A method for solving the convex programming problem with convergence rate 0 ( Proceedings of the USSR Academy of Sciences, 269:543-547, 1983. URL https:
//ci.nii http..a.....pmaa////0029263322..
John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Mike Rabbat, Mani Malek, and
Dzmitry Huba. Federated learning with buffered asynchronous aggregation. In International Conference Artificial Intelligence and Statistics, pp. 3581-3601 PMLR, 2022.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. httpa//.ww....eeeeeo.ooo//a//paa
2023.
Linxuan Pan and Shenghui Song. Local sgd accelerates convergence by exploiting second order information of the loss function. arXiv preprint 1rXiv:2305.15013, 2023.
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024.
wtt.://///////////// 17557
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statistics, pp. 400-407, 1951.
Shuheng Shen, Yifei Cheng, Jingchang Liu, and Linli Xu. Stl-sgd: Speeding up local sgd with stage- wise communication period. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9576-995659 077666954,27 2021.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model par- allelism. arXiv preprint 1.Xiv:990900805. 2019.
Artin Spiridonoff, Alex Olshevsky, and loannis Ch Paschalidis. Local sgd with communication overhead depending only on the number of workers. arXiv preprint arXiv:2006.02582 2020.
Weigao Sun, Zhen Qin, Weixuan Sun, Shidi Li, Dong Li, Xuyang Shen, Yu Qiao, and Yiran Zhong. Efficient distributed training with full communication-come computtiioon overlap. In The Twelfth Inter- national Conference on Learning Representations, 2023.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In International conference on machine learning. pp- 1139-1147. PMLR, 2013.
12

---

Published as a conference paper at ICLR 2025
Megh Thakkar, Tolga Bolukbasi, Sriram Ganapathy, Shikhar Vashishth, Sarath Chandar, and Partha Talukdar. Self-influence guided data reweighting for language model pre-training. arXiv preprint arXiv:2311.00913. 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.099888 2023.
Jianyu Wang and Gauri Joshi. Adaptive communication strategies to achieve the best error-runtime trade-off in local-update sgd. Proceedings of Machine Learning and Systems, 1:212-229, 2019.
Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Slowmo: Improving communication-effiienl distributed sgd with slow momentum. In International Conference on Learning Representations, 2019.
Cong Xie, Sanmi Koyejo, and Indranil Gupta. Asynchronous federated optimization, arXiv preprint 1rXiv:1903.0395, 2019.
Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34:17084-17097, 2021.
Hao Yu, Sen Yang, and Shenghuo Zhu. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 5693-5700, 2019.
Jian Zhang, Christopher De Sa. loannis Mitliagkas, and Christopher Ré. Parallel sgd: When does averaging help? arXiv preprint arXiv: 1606 07366, 2016.
Tuo Zhang, Lei Gao, Sunwoo Lee, Mi Zhang, and Salman Avestimehr. Timelyfi: Heterogeneity- aware asynchronous federated learning with adaptive partial training. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5064-5073, 2023.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1
METHOD
Here we provide the formal descriptions EDiT method and model synchronization with pseudo gradient penalty strategy in Algorithm | and Algorithm respectively, to help readers better under- stand our work.
Algorithm EDiT Algorithm
Require: workers W = (W..... Wx} with each worker W; contains L sharded modules 00,00 0/x0) (0)
) K data shards D {Di, Dx> y; M x N device mesh with the columns forming model shard groups g { 99 } and rows forming model sync groups 9 [9 .... GM h outer training steps T; sync period т; warmup steps learm
1:
for 1 0 to 7 1 do
2:
for 0 to r - do for worker W; parallel do
3:
4:
Confirm W; is in the model syne group Ih and model shard group Ø
( (2.0) (8)
5:
Vi.p) Dr
6:
for = 1 do
Forward Pass
7:
if + > waarm and p == then
8:
Sync parameters in G..:: ((i.)) 0 (
Sync (6.)
( 9T)
9:
end if
Gather module parameters in 0 0 AlIGather(O (1.))
10:
(t.)) f( LtP ((.-1))
11:
Forward calculation: Fi.p
Free module parameters: (..)) = (h
12:
end for
14:
Calculate loss: L = E(x: for 1
{ -t.p)
(t.L) (8)
do Gather module parameters in ; 0 AllGather(O ;9:)
Backward Pass
16:
17:
Backward calculation: Di.r
(!)) V ui ((0))) t.p tip
(1) ((d-11)
18:
Sync grads in ;ü; 9t.
(t.)) = ReduceScatter (G.p:
(1) 9
19:
if (t * + p) S warr then
Syne grads in G (Warmup): gt.p end if
(4.1) AllReduce (1.1),
9m)
21:
Free module parameters: o/2 (hard(O!))
22:
23:
end for
end for
25:
for module (1.)) parallel do
D. Local Update
Update parameters: (f.))
Inner Ont (t.)) (1.))
26:
-t.
end for
28:
end for
end for
14

---

Published as a conference paper at ICLR 2025
Algorithm 2 Sync() in Algorithm
Require:
The m-th model sync group 9t = (W,,... WN J: the sharded parameters ( of the
1-th module in the i-th worker W; at outer step t and inner step т; the corresponding last synced
sharded parameters e((-)) at outer step
1: Calculate the pseudo gradient: A (a.d) = 0 (i.1)
0 (1)))
2: Calculate the pseudo gradient norm: G G( = || (1.0
1|
3: if IsAnomaly(G,. then
Eliminate Anomalies
4: G( =
5: end if
6: Sync the pseudo gradient norms: = Σn exp(-G( for Wj in 9%
7: if =- 0 then
Rollback parameters: ( 0(:10)
8:
9: else
10:
Calculate the weight: W1,; = exp( G. 3
D Weighted Average
Syne the pseudo gradients: A, (1.1) E; W A (1d) for W, in Si
Clip the pseudo gradient: A (i.l) Clip( (2.0)
Clip Grad, Eq 2
Update parameters: ( OuterOpt ( (1))
A (1)))
13:
!+1
Syne parameters: 0 1+10 0
15:
end if
return 0(54)
16:
A.2 EXPERIMENTAL SETUPS
Here we provide a more detailed description of the experimental setups facilitate readers re- producing the experimental results of this paper.
Models We consider four different scales of Llama models (Touvron al. 2023 our experi-
ments: 350M, IB, 3B, and 7B. Their specific configurations are detailed in Table 3 We configure
the models have the same number layers and head dimensions, which facilitates the utilization of μP Yang et al. 2021) for hyperparameter search.
Table Configurations for the four scales of Llama models.
Hyperparameter
350M
1B
3B
7B
Number of Layers
32
Hidden Size
768
1,536
2,560
4,096
Intermediate Size
2,048
4,096
6,912
11,008
Number of Heads
20
Number of K/V Heads
12
20
32
Vocab Size
79,800
Datasets Departing from small language datasets used in prior works (Douillard al., 2023), we employ a new large-scale open-source dataset, FineWeb-Edu (Lozhkov et al., 2024) in our exper- iments. This dataset comprises 1.3T tokens of premium educational web pages filtered from the extensive Fin repository (Penedo al.) (2024). Additionally, also utilize in-house private pre-training dataset, which consists of a diverse collection of corpus of varying quality.
Baselines We compare the proposed EDiT and A-EDIT method against several state-of-the- art methods, including standard mini-batch (Baseline). Post Local SGD (Lin et al.] 2019), DiLoCo (Douillard et al. 2023), and C02/CO2* (Sun al. 2023). Here CO2* is the memory- efficient version of CO2 that shards extra parameters and outer momentum across workers (Sun) et al. 2023). Since Parallel SGD (Zhang al. 2016) and SlowMo (Wang al. 2019) are equiva-
15

---

Published as a conference paper at ICLR 2025
lent Post Local SGD (Lin al. 2019) and DiLoCo (Douillard al. 2023). respectively, do not include them in comparisons.
Adam W (Loshchilov & Hutter 2019) as the inner optimizer and Nesterov momentum (Sutskever et al., 2013) as the outer optimizer. The models initialized with pP (Yang et al.. 2021). enabling the hyperparameters transfer from the smallest scale model (350M) to models of larger magnitude. To balance efficiency and performance, the synchronization interval т and Trime are set 128 and 600s, respectively. Across all experi- ments, a context length of 4,096 tokens and a cosine learning rate decay schedule are consistently applied. For the Fine FineWeb-EE Web-Edu dataset (Lozhkov al. 2024), the total batch size is set 1,024 and the training step is set to 100,000 (~420B tokens). The learning rate for Baseline, inner learning rate, outer learning rate, and outer momentum are set to 3e-4, 1 1.5e-4, 0.8, and 0.85, respectively. For the in-house dataset, the total batch size is set to 1,536 and the training step is set to 150,000 (~950B tokens). The learning rate for Baseline, inner learning rate, outer learning rate, and outer momentum are set to 6e-4, 6c-4, 1.0, and 0.8, respectively. We list the searched hyperparameters in detail in T The experimental infrastructure comprised eight Nvidia A10 GPU nodes with 64 GPUs and an x 8 device mesh. For the hyperparameters in the pseudo gradient penalty strategy, we set Φ 10.
Training Follow DiLoCo (Douillard al. 2023).
Table 4: The hyperparameters searched in the experiments.
Hyperparameter
Value
Inner Learning Rate
3e-5
6e-5
1.5e-4
3e-4
6e-4
Synchronization Interval
16
64
128
256
512
Outer Learning Rate
0.5
0.7
0.8
0.9
1.0
Outer Momentum
0.6
0.8
0.85
0.9
0.95
A.3 ADDITIONAL. EXPERIMENTAL RESULTS
A.3.1 CONVERGENCE AND GENERALIZATION
In the main text, present the performance of EDIT and other Local SGD methods in training the Llama IB model in Figure $ and Table To demonstrate that EDiT performs consistently well across different model scales, we additionally trained Llama 350M, 3B, and 7B models using EDiT on the FineWeb-Edu dataset, each with total of 420B tokens. The corresponding training loss, validation PPL, and evaluation results are shown Figure & and Table can be observed that EDiT is robust across various model scales. Besides, to our knowledge, this is the first time to train a 7B model on a large-scale dataset with a Local SGD-related method. Although CO2 (Sun et al.) 2023) also claimed that they trained 7B model, they only used about 50B tokens and provided only the final validation PPL results.
5.0
39
350M 24.97
4.5
38 2.629
1B 2,848
Validation Perplexity
34
Ahy Jww. "it
Training Loss
78 2,421
76 10.91
4.0
29
3.5
24
3.0
19
2.5
14
2.0 0
9
20k
40k
60k
80k
100k
20k
40k
60k
Bök
100k
Steps
Steps
(a) Training loss curves.
(b) Validation PPL. curves.
Figure 8: The training loss and validation PPL curves for the 350M, 1B, 3B, and 7B models trained with the EDiT method on the FineWeh-BB dataset. The final loss and PPL values are marked in the figures, which the average values of the last 10 values prevent randomness.
16

---

Published as a conference paper at ICLR 2025
Table 5: The evaluation results on the public benchmarks (Fourrier al. 2023
OpenCompass
Contributors 2023 for the 350M, IB, 3B, and 7B models trained with the EDiT method.
Benchmark
350M
IB
3B
7B
MMLU
28.96
32.29
34.70
36.20
ARC-E(1)
53.00
60.00
67.60
68.30
ARC-C
25.40
32.40
36.10
39.20
HellaSwag (t)
39.44
51.75
58.55
62.31
PIQA
65.00
68.10
72.40
73.70
CommonSense-On (1)
29.50
36.30
37.30
42.60
OpenBookQA (1)
21.40
26.00
27.40
28.60
WinoGrande (t)
50.50
51.70
52.00
52.40
Average (f)
39.15
44.82
A.3.2 ACCELERATION
In the main text, we analyzed the throughput and TFLOPS of different acceleration methods. Here, we further profile the synchronization operations of different methods when training the Llama 1B model. As shown in Figure 9 Post Local SGD introduces significant additional communication overhead of 160ms during model synchronization. Although CO2* successfully overlaps model synchronization communication with the forward computation ments of non-overlapping communication overhead to deal with the sharded extra parameters and outer momentum, causing a delay of approximately 300ms. This delay negates the acceleration ben- efits gained from overlapped parameter synchronization. However, without the memory-efficiee mode, the complete copies of model parameters and outer momentum in CO2 lead to severe mem- ory usage, resulting in OOM in this scenario. In contrast, EDiT synchronizes sharded parameters layer-by-layer during the forward pass, reducing communication volume and overlapping compu- tation with communication through prefetch strategy. It achieves the same performance as CO2 without introducing additional communication burdens or memory overhead. As result, EDiT only introduces 19ms delay this scenario.
the next step, it incurs two seg-
Besides, we provide the detailed TFLOPS corresponding to the Figure in Table 6
Table 6: The TFLOPS of different methods under different training scenarios.
Random Straggler
Consistent Straggler
Limited Bandwidth
Lag
Baseline
EDIT
A-EDIT
Lag
Baseline
EDiT
A-EDiT
Repeat
Baseline
EDIT
A-EDIT
0
225.75
236.50
237.45
225.75
236.50
237.45
0
225.75
236.50
237.45
1.5
175.21
228.06
230.05
1.5
175.12
181.20
230.12
10
205.71
234.74
237.85
2.5
150.26
219.72
224.38
150.03
154.12
227.58
20
136.64
236.20
238.04
3.5
130.94
214.36
219.49
130.80
134.00
225.08
30
105.06
236.46
237.73
4.5
115.29
209.44
214.53
115.94
118.47
223.07
40
85.18
236.39
238.03
A.3.3
SCALABILITY
Here we provide the detailed training loss curves the Baseline and EDiT methods under different numbers of workers and distinct learning rates in Figure which correspond to the Figure 6 in the main text.
17

---

Published as a conference paper at ICLR 2025
750309000
Frred Inythol
110
Baseline.
8833457855
Fread toyhorl
Computation
Modol Bynshronization
(b) Post Local SGD.
-5001343M
006873406
Computation
steee
Model Synchronization
steer
Model Synchronization Efficient
(C) CO2
Fread lncheo)
Computation
stear
Model Synchronization
stear
EDIT.
Figure 9: The profiling results of Baseline, Post Local SGD, CO2, and EDiT during synchronization while training the Llama IB model. The parts corresponding model synchronization and compu- tation are highlighted with red boxes.
18

---

Published as a conference paper at ICLR 2025
12
12
Training Loss
1ed
1.5e
Training Loss
10
10
2507
4750
5000
2nd
3644
4750
3000
30
26-4
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Steps
Steps
(a) Baseline worker 1.
(b) EDiT worker 1.
12
Training Loss
264
1.50.
Training Loss
10
1ed
1564
4750
5000
36-4
4p4
973 12055
3o-4
8-
5ad
6-
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Steps
Steps
(c) Baseline worker 2.
(d) EDiT worker 2.
36-5
Je-
4.0
Training Loss
Training Loss
10
4750
5000
3nd
4753
2n-
40.4
30
8
B6-4
0b
2000
3000
4000
5000
Steps
Steps
(e) Baseline worker 4.
(f) EDiT worker 4.
36-5
3e.5
32
3.9
Training Loss
10
Training Loss
10
$ 11000
4750
5000
564
4503
4750
5000
36-4
8-
Ro-4
8-
6e-4
8o44
3000
4000
5000
Steps
Steps
(g) Baseline worker 8.
(h) EDiT worker 8.
12
3a5
Bw-5
6e-5
1e
3.7
3.5
Training Loss
10
Training Loss
10
5005
4750
5000
Snd
200
4750
5000
8
664
1000
2000
3000
4000
5000
1000
2000
3000
Steps
Steps
(i Baseline worker 16.
EDIT worker 16.
Figure 10: The training loss curves of the Baseline and EDiT methods under different numbers of workers and distinct learning rates.
19

---

Published as a conference paper at ICLR 2025
A.4 PROOF OF THEOREM
Table 7: The hyperparameters of Theorem
Symbol
Meaning
The initial learning rate of the inner optimizer
The learning rate of the outer optimizer
The synchronization interval
The gradient norm clip threshold
The loss function L-smooth
The small positive constant avoid division by zero
The dimension of model parameters
Go The upper bound of the gradients
Proof. Since the inner and outer optimizer are both SGD, W; E G m (1,-.. MM, we have
(())
0 (
(7)
t,0
(1))
1pt
L.p
Th
(8) Pt.p
(8)
0.+1 = 0: v A
where nip and are the inner optimizer learning rate and the outer optimizer learning rate, respec- tively. Here, we omit the superscripts of variables in Equation 9 as they remain identical across
all workers in Gh The following proof will also adopt this simplified notation without risk of confusion. Hence, by the Equations
and we have
A 3, 31
Σ
W
Be
Wt, (
(.0)
(10)
= Be
Σ
We (nt.
(
+ +
-050 = Be Σ Wh.t
nt t.p
p=U
Let
ht.p = Bt Σ We.gr ()
(11)
then
E E B Σ
Wt (1)
(12)
Combining Equation
Equation 10 and Equation
into Equation
we have
O.+1,0 0c.0
(13)
For proving the convergence (0 J. we need to define the auxiliary sequence ( Denote
40,00 = 0o.0:
14++1.0 = vi.00 - v. Ex-d tenpte
1i,pt1 = Dhipp - Vm..hpp
is easy to prove 1/++1,0 the...
Then we have
p-1
p-1
p-1
E(pbt,p) E((O,p]
Elsbr,o
Σ
EE 0
Σ
m.ggu = v3.))
Σ
F(gr,t)
k=0
k=0
(14)
20

---

Published as a conference paper at ICLR 2025
By assumption 1, we have
L
((1b..+t)) S [((bt.p) + (VL(4)))), 21.. Vint) + 11
2 x,pt - 12
2
((bb vhap (V kx)p) + L
|11
LuP
=((b)p)) VTL (L((v)))) -VC(()))))V((((())))) ( map (VC(O())) 1p) +
11e.p1
2
LuP
S L(bbtp)) + Vmpp 1m l1
0 / - 1M1ep
(V((O()) (ip): m.p) + 2
|11.21??
(15)
Rearranging Equation
15 and taking expectation both sides, by assumption 2, assumption 3, Equa-
tion
12 and Equation 14
we get
VTh 3
Lu
m
<)( [(bbt )] + vrje, LEIl/ 0c.plll h
htp 11
2
Lu nppppppBCC SRnC2
<B([((bp) - ((br.p+1) + pr L3 5l - 0c,pll
2
p-1
LV ni Go
-)))) [(vb. )1 + VTt L, L.. vB+))
Σ
Th 7B
2
Lv hh p S2nG2
<B(((()p)) - ((br,p+1) + LB
2
(16)
Telescoping Equation 16 for 0 and t- to
1, we have
T-1
LvenGa T-17-1
<F([((oo.0) [( + 22
Σ
Boh +
ΣΣ
(17)
T-1
L22nG2
T-1
<C(00,0) + L 72
Σ
Suni +
ΣΣ
Tipp
2
1=0
1=0 p=0
Since from Equation 4 we have S B1 S $ Combining with Equation 17
get
[(00.0) + LnG22 72
T-1
LunG2 T-
nep (
m +
2
60
nhp (18)
2ea
0= pmo
0
1= pm0
Since
1 VTn
ntp
[=0 VT+I
1=0 p=0
t=0
T++
I,'
T+1
1
Vrn
ds + +
VT ds
2 Vin
v8 ds
VI
= (V >
WFn(VT - 1),
ΣΣ h S
T-1
Σ
11.0 11²
T-1
Σ
++I=1 =イ²
1
1 Jo
1 ds +
T-1
1
ds
t+1
T T 1 +
p=
t=0
(T-1
1
ds
= (1 + +In(T - + 1)) 17 +1+++01 +In(TT)),
J0
(19)
21

---

Published as a conference paper at ICLR 2025
substituting Equation 19
into Equation 17
we have
min 11
((0,0)
InG 72
T-1
LonG n2
T-1r-1
$ T-1 21=0 Ep=0 Mtp
Σ
11.0 +
ΣΣ
Th
V
2e
p=U
[(00,0)
LnG T
(1+1((1++
LunG
+ In(TT))
(V
26
(20)
This completes the proof.
22


Reviews:
[{"review_id": "PyoaZYyhgd", "reviewer": "Reviewer_8feR", "summary": "The paper presents EDiT (Efficient Distributed Training) and its asynchronous variant A-EDiT, which aim to improve the efficiency of distributed training for large language models (LLMs). EDiT solves issues in existing distributed training such as communication bottlenecks, straggler delays, and limited scalability in heterogeneous environments. A pseudo-gradient penalty strategy is introduced to enhance training stability. Experimental results suggest EDiT and A-EDiT addresses the straggler issue and is more stable compared to baseline like DiLoCo.", "questions": "Please see weakness section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper introduces an innovative approach that combines Local SGD with asynchrony and gradient penalty,, which addresses communication overhead and resource elasticity.\n* The paper rigorously evaluates EDiT and A-EDiT on multiple benchmarks, demonstrating improved performance in training speed, stability, and generalization compared to state-of-the-art methods.", "weaknesses": "* EDiT and A-EDiT rely on a range of hyperparameters, how should we think about and choose $\\alpha, \\phi, \\beta$ on a new training task?\n* The authors propose using gradient norm as a metric for anomaly elimination and gradient penalty. However, in llm training, gradients often have outliers on some of the examples. If we ignore examples with large gradient norms, will it create bias on training?\n* It would also help if the authors can provide convergence analysis for the proposed EDiT method. Do EDiT and A-EDiT have convergence guarantee?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NOSPsLQWYY", "reviewer": "Reviewer_zyrD", "summary": "This paper introduces EDiT, a distributed training method leveraging local SGD to reduce communication overhead when training LLMs. The method relies on a \"2D-grid\" graph topology with a total of $K= M \\times N$ data parallel models. On highly connected GPUs (e.g., on the same compute node in a cluster), $N$ models are sharded and communicate at each optimizer step. On the other hand, $M$ of these islands of highly connected machines are connected through possibly lower bandwidth communication links (e.g., on separate compute nodes), and leverage the lower communication frequency needs of local SGD to alleviate communication bottlenecks. To counter the destabilizing effect of using lower batch sizes in local SGD, a gradient penalty method is introduced. Finally, an asynchronous version of the method is also introduced to mitigate the effect of stragglers.", "questions": "* **Intra node for model shards:** on standard GPU clusters using nodes with 8 A100 or H100 GPUs with 80GB of memory, what size of model can be loaded & how many intra-node model replicas does it lead to?\n* **line 168:** a warm up phase is used as in post-local SGD. However, DiLoCo finds that it is not really helpful (cf their Fig.3), do you observe a different behavior here?\n* **Tab.1 & Fig.4**: Why not experiment with A-EDiT on the in-house dataset?\n* **Fig 5.a)**: can you explain the reason why A-EDiT and EDiT exhibit the same throughput in the random-straggler scenario?\n* **In ablation Fig 7.a):** the Gradient Clipping doesn’t seem to have much effect in the validation PPL curve, is it really necessary to keep it along with WA and anomaly detection (which both seem to smooth the spikes observed in the validation PPL curve during training)?\n\n=========REBUTTAL==========\n\nThe authors correctly addressed my concerns and provided further precision on their method, I raise my score.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* **The novel gradient penalty method seems effective**: In the experiment on the in-house dataset (Fig 4.d), the unstable behavior of standard local-SGD methods is highlighted, and the gradient penalty introduced seems to be an effective solution to alleviate it.\n* **Advantages of a proper distributed implementation of a local SGD algorithm are highlighted**: Section 4.3 displays the practical advantages (in terms of throughput) given by a proper implementation of local SGD algorithms using modern distributed methods (such as ZeRO-3/FSDP) when communication links between distributed workers have limited bandwidth.\n* **Asynchronous extension**: In the presence of stragglers, the advantage (in terms of throughput) of using an asynchronous extension of EDiT are highlighted in Sec.4.3.", "weaknesses": "* **Claiming novelty on the distributed implementation of a local SGD algorithm seems too strong:** As the sharding strategy itself cannot be claimed as novel (see the ZeRO series, FSDP), the “distributed implementation” cannot either. In fact, apart from the gradient penalty method, the Algorithm seems to be a straightforward and natural way of implementing local SGD for LLM training (using ZeRO-3) and not particularly novel (previous local SGD papers such as CO2 [[Sun et al., 2024]](https://arxiv.org/pdf/2401.16265) seem to already make use of ZeRO for their Autoregressive Language Modeling task, and SlowMo [[Wang et al., 2020]](https://arxiv.org/pdf/1910.00643 ) already consider all GPUs inside a cluster node to be a single local worker in their experiments so that each local worker reaches larger batch sizes through data-parallel). Thus, although having a *detailed* pseudo-code for this implementation is welcome, making it the main contribution of the paper and dedicating so much space for explaining it weakens the paper for me. For instance, as the stabilizing effect of the gradient penalty is empirically demonstrated, focusing on this novel observation and contribution seems more relevant to me. \n* **No comparison with CO2**: while experiments with SlowMo/DiLoCo are performed, no comparison with state-of-the art method CO2  [[Sun et al., 2024]](https://arxiv.org/pdf/2401.16265) is done.\n* **Hyper-parameters introduced:** EDiT introduces 3 novel hyper-parameters $\\alpha, \\delta, \\phi$, but the impact of their values on the method is not discussed.\n* **Lack of clarity and imprecision in the writing:**\n  * It was not clear at first that the $N$ models were *data parallel sharded* model replicas and **not** model-parallel splits. Maybe clarifying this distinction early in the paper could avoid potential confusion.\n  * **line 21:** *“ensure training stability and improve performance.”* this is a bit cryptic. Maybe detailing the reason for these instabilities (as done lines 210-213) early in the paper would help understand the challenges tackled here more clearly.\n  * **line 50:** *“Current Local SGD methods do not integrate well with modern distributed strategies”*. Why? Can you explain your point? For instance, the CO2 paper claims that their algorithm is compatible with ZeRO-series optimizers.\n  * **lines 73-75:** the problem described is exactly the one the CO2 paper aims at solving, so saying *“current local-SGD method”* seems to be an overstatement here.\n  * **line 116:** *“DiLoCo extends the slow momentum in SlowMo to the outer optimizer.”* this statement does not seem accurate: the momentum in SlowMo is already in the outer-loop, and seems to be roughly equivalent to DiLoCo (with nesterov SGD as the outer optimizer, which is also noted line 752). Can you clarify in which way DiLoCo extends SlowMo in this context?\n  * **line 170:** *“model synchronization”*. What does this mean? Can you provide a clear definition or explanation of this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MyUYO2Jat5", "reviewer": "Reviewer_eSB4", "summary": "This paper proposed a distributed training approach which arranges the participating workers into a two-dimensional device mesh, i.e., model replica group and the model shard group. It also introduces a pseudo gradient penalty mechanism that eliminates the significantly anomalous worker and averages and clips the gradients. It also introduces an asynchronous variant of the proposed scheme. Experiments have been conducted to evaluate the performance of the proposed scheme on the distributed training of LLMs.", "questions": "1. The proposed hierarchical distributed training method on a 2-D device mesh should be better explained, like how workers within a model shard group coordinate with each other. What are the messages transmit between them (should be intermediate results instead of parameters?)\n2. Is there any design in the proposed method addressing the concern that each worker within model shard group may need to wait for the backward gradients of each batch from the next worker in the same group? \n3. How this proposed hierarchical distributed training method accelerates the distributed training? It should be evaluated in the numerical results. \n4. In Section 4.5, it seems suggesting that DiLoCo scheme with the pseudo gradient penalty mechanism is the proposed scheme EDiT?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "It is interesting and novel idea of arranging the participating workers into a two-dimensional device mesh, i.e., model replica group and the model shard group. The experimental results also demonstrate the effectiveness of the proposed pseudo gradient penalty mechanism and asynchronous version of the proposed scheme. The presentation of this paper is fair and easy to follow.", "weaknesses": "1. The novelty of this paper is limited. The author claimed that this paper is the first to integrate Local SGD with modern distributed strategies. What does \"modern distributed strategies\" refers to here is not clear. Is't local SGD a modern distributed strategy. The way to eliminate the significantly anomalous worker and averages and clips the gradients is not a new idea. \n2. The  hierarchical distributed training method on a 2-D device mesh may be problematic. Since each worker within model shard group runs a subpart of the model, it may need to wait for the backward gradients of the same batch before processing the next batch, leading to significant idle time and slowing down the training. \n3. The improvement over the existing work since quite insignificant as shown in Figure 4. The performance of the proposed scheme is only compared to baseline, which is not sufficient to demonstrate its superiority. The author should compare it with a more advance asynchronous scheme, such as following work:\n[1] Nguyen J, Malik K, Zhan H, et al. Federated learning with buffered asynchronous aggregation[C]//International Conference on Artificial Intelligence and Statistics. PMLR, 2022: 3581-3607.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)