Pseudo-Asynchronous Local SGD: Robust and Efficient Data-Parallel Training
Authors: Hiroki Naganuma
Xinzhi Zhang
Man-Chung Yue
Ioannis Mitliagkas
Russell J. Hewett
Philipp Andre Witte
Yin Tat Lee
Venue: Accepted by TMLR ()

Abstract: Following AI scaling trends, frontier models continue to grow in size and continue to be trained on larger datasets.  Training these models requires huge investments in exascale computational resources, which has in turn driven developtment of distributed deep learning methods. Data parallelism is an essential approach to speed up training, but it requires frequent global communication between workers, which can bottleneck training at the largest scales. In this work, we propose a method called Pseudo-Asynchronous Local SGD (PALSGD) to improve the efficiency of data-parallel training. PALSGD is an  extension of Local SGD (Stich, 2018) and DiLoCo (Douillard et al., 2023), designed to further reduce communication frequency by introducing a pseudo-synchronization mechanism. PALSGD allows the use of longer synchronization intervals compared to standard Local SGD. Despite the reduced communication frequency, the pseudo-synchronization approach ensures that model consistency is maintained, leading to performance results comparable to those achieved with more frequent synchronization. Furthermore, we provide a theoretical analysis of PALSGD, establishing its convergence and deriving its convergence rate. This analysis offers insights into the algorithm's behavior and performance guarantees. We evaluated PALSGD on image classification and language modeling tasks. Our results show that PALSGD achieves better performance in less time compared to existing methods like Distributed Data Parallel (DDP), and DiLoCo. Notably, PALSGD trains 18.4% faster than DDP on ImageNet-1K with ResNet-50, 24.4% faster than DDP on TinyStories with GPT-Neo-125M, and 21.1% faster than DDP on TinyStories with GPT-Neo-8M.

Published in Transactions on Machine Learning Research (09/2025)
Pseudo-Asynchronous Local SGD: Robust and Efficient Data-Parallel Training
Hiroki Naganuma Xinzhi Zhang
naganama.hiin hwroki@mmila quebec, minzhi20@wwa.c edu
Mila, "Unieessiéé de Montréal, 3 University of Washington
Man-Chung Yue Ioannis Mitliagkast.25
meyue Carueehkk..an ioannis @mill.. quebec
The University of Hong Kong, Conada CIFAR Al Chair
Philipp A. W Russell J. Howett A Yin Tat L.c
moriite@@iivvoofftcco com, rheueed@@micrrooft.ccm yintat@eu..co edu Microsoott, 7 NVIDIA
Reviewed on OpenReview: https: openreview. net/ forum? id= dV-VTrrS55
Abstract
Following Al scaling trends, frontier models continue grow in size and continue be trained larger datasets. Training these models requires huge investments in exascale computational resources, which has in turn driven development of distributed deep learn- ing methods. Data parallelism is an essential approach to speed up training, but it re- quires frequent global communication between workers, which can bottleneck training at the largest scales. In this work, we propose a method called Pseudo- Asyndo-syyncroooos Lo- cal SGD (PALSGD) improve the efficiency of data-parallel training. PALSGD an extension of Local SGD (Stich, 2018) and DiLoCo (Douillard et al., 2023), designed to fur- ther reduce communication frequency by introducing pseudo-synnerroiization mechanism. PALSGD allows the use of longer synchronization intervals compared to standard Local SGD. Despite the reduced communication frequency, the pscudo-synchronizatinn approach ensures that model consistency is maintained, leading to performance results comparable to those achieved with more frequent synchronization. Furthermore, we provide a theo- retical analysis of PALSGD, establishing its convergence and deriving its convergence rate. This analysis offers insights into the algorithm's 's behavior and performance guarantees. We evaluated PALSGD image classification and language modeling tasks. Our results show that PALSGD achieves better performance in less time compared to existing methods like Distributed Data Parallel (DDP), and DiLoCo. Notably, PALSGD trains 18.4% faster than DDP on ImageNet-Il 1K with ResNet-50, 24.4% faster than DDP on TinyStories with GPT- Neo- 12-M, and 21.1% faster than DDP on TinyStories with GPT-Neo- -M..
1 Introduction
Training neural networks has become more computationally expensive, requiring distributed deep learning techniques to handle the growing data and model sizes. Distributed training usually uses some form of data parallelism such as DDP (Li et al., 2020a) ZeRO (Rasley et al., 2020), where batch of training samples is further split into multiple micro batches that assigned to different workers. These workers perform
Alphabetical order, -These authors contributed equally to this work. This work was partially done when Naganuma and X. Zhang were Microsoft Research interns
Alphabetical order Work done while at Microsoft

---

Published in Transactions on Machine Learning Research (09/2025)
forward and backward passes their local data shards and synchronize their model updates through op- erations like ALI-REDUCE. However, synchronization at every step introduces significant communication overhead, especially as the number of workers increases, because all model gradients have to be synchro- nized between workers (Lin et al., 2018). In addition, increasing the batch size to improve throughput can negatively impact model generalization, resulting suboptimal performance (Keskar al., 2016).
A common solution to reduce communication overhead is Local SGD (Stich, 2018), which allows workers to perform multiple local updates before synchronizing. While effective, Local SGD suffers from model di- vergence when synchronization intervals become too large, degrading its convergence properties (Lin et al., 2018). Recent methods such as DiLoCo (Douillard et al., 2023) and FedProx (Li et al., 2020b) introduce tech- niques to mitigate these issues, yet they still rely on periodic full synchronization, making them susceptible to performance bottlenecks in high-latency environments.
To address these issues, we propose deeudo-Asynchronuus Local SGD (PALSGD), a novel extension of the Local SGD (Stich, 2018) framework that. incorporates a pseudo-asynchronouu model update mechanism. Unlike standard synchronization methods that require workers to exchange gradients at fixed intervals, PALSGD allows workers to gradually synchronize with a local copy of the global model. Instead of waiting for an ALL-REDUCE operation, workers probabilistically mix their local paramcters with an outdated version of the global model, reducing the frequency of full synchronizations. By introducing probabilistic updates, workers operate more independently between syuchronization points, leading to better training efficiency. This geeudo-syccroorizatin siguificantly alleviates communication bottlenecks while maintaining model consistency, making PALSGD particularly suitable for large-scale distributed training.
Our contributions are as follows:
Pseudo Synchronization: We introduce a probabilistic pseude-synchronizatin mechanism to allow workers to loosely syuchronize with the global model, reducing the need for frequent full synchronization. This approach balances communication efficiency and model consistency (Section 5 and Algorithm 1).
Theoretical Analysis:
We provide a rigorous convergence analysis of PALSGD, demonstrating how the interplay between psendo-syncrronization probability and synchronization interval affects the overall convergence rate (Section 6).
Empirical Validation: We demonstrate the effectiveness of PALSGD through experiments on ImageNet-11K (Deng et al., 2009), TinyStories (Fldan & Ti,, 2023), and CTFAR1O datasets. We show that it achieves superior training efficiency compared compared to existing methods like Distributed Data Parallel (DDP) and DiLoCo (Douillard et al., 2023) (Section 7, Figures 2, 3, and 4).
Our work advances the field of efficient distributed deep learning by bridging the gap between synchronous and fully decentralized framework. By introducing a flexible and efficient synchronization strategy, PALSGD provides a scalable alternative for training large-scale neural networks in real-world distributed environments.
2 Background
Efficient communication in distributed training is a fundamental challenge as models and datasets continue to scale. Large-scale models, such as LLaMA-3 (Dubey et al., 2024), spend a significant portion of training timc-up to 20-30%-o ALL-REDUCE operations, which synchronize gradients and model states across workers. As models and datasets grow larger, this synchronization overhead becomes major bottleneck in distributed deep learning. Local SGD (Stich, 2018), which reduces communication frequency by allowing workers to perform local updates for several steps before synchronizing, offers a partial solution. However, it was observed that Local SGD struggles when the synchronization interval H exceeds about 8 steps, leading to degraded model performance and slower convergence (Lin et al., 2018). This limitation prevents Local SGD from effectively scaling to larger models, where minimizing communication overhead essential.
Moreover, as distributed training systems scale, frequent synchronization steps reduce the system's robustness to worker slowdowns or failures. In these scenarios, single delayed or failed worker can bottleneck the entire
2

---

Published in Transactions on Machine Learning Research (09/2025)
training process. These issues highlight the need for communication methods that not only lower the cost and frequency of synchronization but also ensure sulficient alignment between workers' models.
3 Preliminaries
We consider the following stochastic optimization problem:
min F(x), F(x) = (z, zER4
(1)
which aims to minimize the expected value of the objective function f(r,E), where training samples & are drawn from an underlying data distribution D.
In distributed training, we consider system of K workers running in parallel, each initialized with the parameter z!O)_ The training dataset is uniformly partitioned into K independent data shards, D1,..., DR. such that each worker performs local computations exclusively on its assigned subset of data. Each worker updates its local model using general optimization rule:
[+1 WORKEROPT( gi. 1t), sk f
(2)
where o the local model on worker k, g is the stochastic gradient computed using the local data sample S Da: and WORKEROPT(.) represents the local optimization algorithm (e.g., SGD or AdamW) with step size n)--
3.1 Synchronization in Distributed Training
To maintain consistency across workers, distributed training requires a synchronization mechanism to ag- gregate updates. In our setting, this achieved via ALL-REDUCE, a decentralized communication protocol where workers exchange and average an arbitrary state variable (which could represent model parameters, gradients, optimizer states, or other training-related variables). Specifically, after a synchronization step, all workers update their local state to the globally averaged state:
K
StD
$
(3)
K
k=1
Distributed Data Parallel (DDP) is a widely used distributed training algorithm. It performs an ALL- REDUCE operation on the gradient variable at each training step before applying the optimizer update. This ensures that all workers compute the same parameter updates based on the globally averaged gradi- ent, maintaining consistency across the training process. However, frequent synchronization at every step introduces substantial communication overhead, which can significantly limit scalability as the number of workers increases.
3.2
The Pseudo Seeudo-Asynchromouu Setting
To alleviate synchronization bottlenecks, distributed training can be performed in preado-osyncchrmous setting, where workers operate independently between synchronization points. In this approach, synchro- nization occurs only at pre-scheduled ALL-REDUCE operations, rather than every training step. During ALL-REDUCE, all workers must reach the same iteration count before synchronization occurs, meaning that faster workers must wait for slower ones.
Between two consecutive synchronization points, workers perform local updates without waiting for each other, allowing them to progress at their own pace. This reduces total idle time and lowers communica- tion overhead, improving computational efficiency. However, less frequent synchronization introduces the challenge of model divergence, necessitating strategics balance communccttio efficiency and model con- sistency.
3

---

Published in Transactions on Machine Learning Research (09/2025)
4 Related Work
4.1 Local SGD and Variants
Local SGD is a widely adopted technique for reducing communication overhead in distributed optimization. It allows workers to perform multiple local updates before synchronizing with a central server, minimizing the need for frequent gradient exchanges and improving communication efficiency. Introduced federated learning by McMahan et al. (2017a), Local SGD has been extensively studied for its convergence properties. Stich (2018) demonstrated that it converges at the same rate as mini-batch SGD, particularly in smooth and strongly convex settings. Subsequent works, such as Haddadpour et al. (2019) and Koloskova et al. (2020), extended these results by analyzing Local SGD under more generalized frameworks, including varying network topologies, different convexity settings, and data heterogencity. Similarly, Yu et al. (2019) provide an analysis of Local SGD, which they term Parallel Restarted SGD, further demystifying the effectiveness of model averaging in deep learning. The key insight of our algorithm is that it allows workers to mix with the "central model' more frequently without incurring extra communication overhead, leading to better alignment of local models compared to Local SGD.
Several variants of Local SGD have been proposed to further enhance scalability. Elastic Averaging SGD (EASGD) (Zhang et al., 2015) allows local models to diverge from the global model within bounded range, improving convergence in non-convex settings. While our proposed algorithm similarly introduces slackness through proximal terms in the loss function, it further reduces communication by employing stochastic syn- chronization rather than stepwise synchronization. Moreover, our algorithm reduces variance between local models more effectively by cnsuring that all workers start from the same global model every H local steps, leading to improved consistency across workers. Cooperative SGD (Wang & Joshi, 2021) expands on Local SGD by enabling direct communication between workers, reducing reliance on a central server and improv- ing robustness in decentralized systems. The Scaffnew algorithm (Mishchenko et al., 2022) extends Local SGD by introducing probabilistic communication updates and control variates, with commuicattin handled through a more complex proximal optimization process. However, its reliance on exact control-variate up- dates makes it difficult to extend to modern optimizers like AdamW without significant modifications to the algorithm and its memory footprint. In contrast, our approach uses a simpler control mechanism, making PALSGD substantially easier to integrate into existing deep learning pipelines.
Additionally, Post-Local SGD, combination of mini-batch SGD and Local SGD, was introducod by Lin et al. (2018) and shown to strike better balance between communication efficiency and generalization performance in deep learning tasks (Gu et al., 2023). In our proposed method, we adopt a similar strategy to Post-Local SGD by using mini-batch SGD in the warmup phase, followed by the application of our PALSGD algorithm in the second phase. This approach allows us to leverage the fast initial convergence of mini- batch SGD before transitioning to our more communication-effiinni method. SCAFFOLD (Karimireddy et al., 2020), addresses the issue of client drift by using control variates to correct the local update direction, which differs from our probabilistic mixing approach. In addition, SCAFFOLD is primarily designed to address non-IID scenarios and has been reported to perform worse than FedAvg (McMahan et al., 2017b) in IID settings. Another key related work, FedProx (Li et al., 2020b), explicitly penalizes local model divergence through a proximal term. Unlike the above-mentioned works, our method implicitly controls avoiding the need for additional hyperparameter model alignment via probabilistic calculation. Our mothod also contrast with synchronous approaches like tuning and extra forward-backward DropCompute (Giladi et al., 2023), which handles stragglers by dropping slow gradient computations rather than extending the local update period Furthermore, while our approach shares similarities with L,2GD Bergou et al. (2022), we incorporate warmup phase and an inner-outer loop optimization structure, which are particularly crucial when using AdamW for large-scale vision and language model training.
4.2 Negative Momentum
Mosentum-based optimization methods are widely employed to accelerate the convergence of gradient- based algorithms. Recent study (Gidel et al., 2019), particularly in the context of adversarial learning such Generative Adversarial Networks (GANs), have emphasized the role of negative ummentum in improving
4

---

Published in Transactions on Machine Learning Research (09/2025)
game dynamics. In their study, negative momentum was introduced as stabilizing mechanism to address oscillatory behavior in adversarial settings. Their results demonstrated that alternating gradient updates with a negative momentum term achieve more efficient convergence, both theoretically and empirically, especially in bilinear games and challenging scenarios like saturating GANs.
Our proposed method, PALSGD, shares conceptual similarities with negative momentum in the context of distributed learning, despite the focus being on single-objective function optimization rather than adversarial learning. The pseudo-synchronizaton introduced in PALSGD can be interpreted as a form of regularization, akin to how negative momentum explicitly modifies the update direction in adversarial games. While previous study (Gidel et al., 2019) applicd negative momentum to mitigate instability in adversarial games, our method regularizes model divergence in large-scale data-parallel SGD, reducing the instability often observed in such setups. Thus, negative momentum and our pseudo-asyuchronons approach provide complementary insights into enhancing the stability and efficiency of gradient-baser methods, albeit in distinct settings: adversarial games versus large-scale distributed learning.
4.3 Robust Aggregation through Decoupled Method
While Local SGD is theorctically fast-converging and communictioon-cffiiatt it faces empirical limitations in large-scale optimization lasks (Ortiz et al., 2021). One of the challenges is that simple averaging of local models, used in Local SGD, struggles in scenarios involving adaptive optimizers like SGD momentum and AdamW, which are common in large-scale training. Recent works such as Slomo (Wang et al., 2019) and FedOpt (Reddi et al., 2020) have focused on more robust aggregation techniques by decoupling the inner optimizer for local training and the outer optimizer for model aggregation. More recent approaches such as DiLoCo (Douillard et al., 2023) and Asynchronous Local SGD (Liu et al., 2024) have validated the effectiveness of using AdamW for local updates and Nesterov momentum for outer optimization in largo-scale language modeling tasks, offering improved performance and robustness. Our proposed algorithm intergrates the decoupled method from the DiLoCo framework with the pscudo-synchroonzzaioo process. We showed that our method siguificantly outperforms DiLoCo on image classification and language modeling tasks.
4.4 Asynchronous and Pseudd-AAyyncrrooous Methods
Asynchronous and pseudo-asynccrroouu methods have been developed to address inefficiencies in syn- chronous training, particularly the "straggler effect", where faster workers are forced to wait for slower ones. This issue has been widely observed in synchronous distributed settings (Koh et al., 2006; Lian et al., 2015; 2018; Dean et al., 2012). Dean et al. (2012) introduced one of the earlicst asynchronous frameworks, enabling each worker to update the global model independently, which significantly improved computationa utilization. However, this approach introduced the challenge of stale gradients, where outdated updates from slower workers are applied to newer models, hindering convergence. Methods like Asynchronous SGD with Delay Compensation (Zheng et al., 2017) addressed this issue by approximating fresher gradients. Other approaches such as Polyak Averaging (Xie et al., 2019) proposed downweighting stale updates to improve robustness. More recently, Asynchronous Local SGD (Liu et al., 2024) introduced a decoupled method, demonstrating that the strategic use of momentum can alleviate many of the challenges posed by staleness. In federated learning, methods like Moshpit SGD (Ryabinin et al., 2021) and TimelyFI. (Zhang et al., 2023) have explored asynchronous approaches to better manage unreliable or heterogencous devices in largo-scale distributed systems. Tyurin & Richtárik (2024) proposes a time complexity framework for parallel stochastic optimization under fixed computation model.
Recent works have also explored fully asynchronous and decentralized settings to further mitigate synchro- nization bottlenecks. Methods like Shadowheart SGD (Tyurin el al., 2024) and SWIFT (Bornstein el al., 2022) leverage parameter-sere architectures to enable wait-free communicttinn, while others have explored techniques like model fragmentation to boost asynchronous lcarning (Biswas et al., 2025). These approaches differ from our synchroossss framework, which maintains periodic global synchronization points avoid issues like unbounded gradient staleness.
5

---

Published in Transactions on Machine Learning Research (09/2025)
5 Proposed Method: PALSGD
Algorithm 1: Pssendo-Assycchrooooss Local SGD with Decoupled Optimizers
Data: 200) (initial model), K > 0 (number of workers), p (0, 1) (probability of mixing step), '/e > 0 (mixing rate), H > 0 (syne interval), optimizers INNEROPT and OUTEROPT, ar (learning rate for INNEROPT)
for worker k 1,.. K do (0)
Fk (0)
for t 0,... T 1 do
~ U|0, 1]:
if b < then
22
(t) (t) aalk (z x(0);;;
peauoossynchroizzttion step
else
Sample data & ~ Dx:
9 t Vs(E -E;;
F& ((+1) INNEROPT(2 99 at
gradient step
end
if (t+1) 1) mod H = 0 then
( t ALL-REDUCE( 1/ ( (1+1) OUTEROPT(Z(), AA
aggregate outer gradient
update global model
else
(++1) x(1)
end
end
end
The primary challenge in extending the synchronization interval H Local SGD is the problem of model divergence, where worker models drift from the global optimum and degrade convergence. To address this, we propose Pseudo-Asynchrooous Local SGD (PALSGD), novel extension of the Local SGD framework. PALSGD introduces a pseudo-synchrooous step that acts a lightweight, communication-free regularizer to explicitly counteract this divergence. As detailed in Algorithm 1, this step allows workers to probabilistically align with a locally stored global model copy, enabling the use of longer synchronization intervals without sacrificing model consistency.
5.1 Probablistic Regularization via Pseudo-Synchronizzaton
In our distributed setting, each worker maintains two key model states: a local model x which is updated using data from its assigned shard, and a global model copy 2ff), which stores the last synchronized state and remains unchanged between full synchronization steps. Training progresses over multiple itcrations, where each worker performs local updates independently for H steps before a full synchronization occurs.
At each local step, the worker follows one of two paths based on probability p. With probability - it performs a standard gradicnt- based update using its local data and the optimizer INNEROPT:
(I+1) INNEROPT(2)) 2
(4) at 9 11pp"
(4)
where df ff(((" {) the stochastic gradient computed from the worker's local data shard.
Alternatively, with probability p, the worker performs pseudoosynnchroizzaton update:
(1+1)
<- x
(1)
at7) (t)
(
(1))).
(5)
p

---

Published in Transactions on Machine Learning Research (09/2025)
This update is the core of PALSGD and serves principled mechanism to control model divergence. Its form can be justified in several ways:
As a Proximal Regularizer: The update in Equation (2) can be interpreted as a single gradient step on a quadratic penalty term, / -1 - 20 This term explicitly regularizes the local model by pulling it towards the last-known global state <d(), It penalizes deviation from this shared reference point, thereby constraining the local optimization paths and preventing workers from drifting too lar apart.
As an Exponential Moving Average (EMA): The update can be rewritten as a mixing step: 2k ((+1) + (1 3)x + B2((), where the mixing coefficient is B -ets, This shows that the worker's parameters are an exponential moving average of their own trajectory and the global trajectory. This form is well-known for stabilizing updates by dampening oscillations and reducing the variance of the paramcter updates.
p
By probabilistically applying this step; PALSGD ensures that local models remain sufficiently aligned, acting lightweight mechanism to reduce model divergence without incurring any communication cost. We refer to this as a peeudo-synchronization step to distinguish it from true synchronization event like ALL-REDUCE. In a true synchronization, workers exchange information and compute a global average, which requires all workers to pause and communicate. In contrast, a preudo-synchronizaati Each worker moves its local model towards its own stored copy of the global model, which may be stale by up to (H-1) steps. This step involves no communication with other workers, thus avoiding network latency and idle time for faster workers.
step is a purely local operation.
5.2 Reducing the Frequency of Full Synchronization
By employing the probabilistic regularization described above to maintain model consistency, PALSGD safely reduce the frequency of full, communcationn heavvy synchronizations. Unlike fully synchronous methods where workers communicate at every iteration, PALSGD leverages peeudo-synchronizztinn to partially align workers' models, reducing the need for frequent global updates. This approach mitigates the problem of idle time, where faster workers must wait for slower ones, and improves scalability, especially large distributed systems.
A full global synchronization occurs only every H iterations via ALLREDDCE operation, which aggregates the model differences across all K workers:
A (0) ALLREDUCE((( -x (2)).
(6)
The aggregated difference A (1) is then used to update the global model with an outer optimization step:
((+1) ".
By performing this costly operation less frequently, PALSGD significantly reduces communication overhead while the secudo-yyncroonizatin mechanism preserves model consistency between these updates.
5.3 Practical Enhancements and Implementation Details
To further improve empirical performance and ensure stability, PALSGD incorporates several practical tech- niques with minimal overhead. First, similar to Post-Local SGD (Lin et al., 2018), we initialize training with a DDP warm-up phase to address potential instability when gradients are large and noisy. Following this, we employ a decoupled optimizer strategy, as validated the Dil.oCo framework (Douillard et al., 2023). We use AdamW (Kingma, 2014) as the INNEROPT for rapid local progress and Nesterov momentum (Sutskever et al., 2013) as the OUTEROPT to robustly aggrogate updates.
From implementation standpoint, these enhancements are lightweight:

---

Published in Transactions on Machine Learning Research (09/2025)
Memory Requirements: Each worker stores the local model 2k the global model copy x0 and the outer optimizer state. This overhead, equivalent to one extra model and its optimizer state, is well within the capacity of modern GPUs.
(8)
The pseudo-synchroniiattin step is a purely local computation and adds no communication overhead. Communication occurs only during the ALL-REDUC oper- ation every H steps. The primary efficiency gain of PALSGD stems from its ability to use a larger H effectively, thereby reducing the total number of these expensive communication events.
Communication Requirements:
Together, these modifications ensure that PALSGD not only reduces communication costs but also achieves faster convergence and better model performance across various deep learning tasks.
6 Theoretical Results
Building on this framework, we showed the following theoretical convergence bound for simplified version of our algorithm, where the innor optimizer is standard SGD and the outer model updated by taking the average across inner models. We include the proof in Appendix A.
We make the following assumptions on the objective function F for our theoretical analysis:
Assumption 1 (L-Smoothness). There erists a constant L O such that for each in the support of D, and for each I, E Rd,
[))( ff - L|| yl1-
Assumption (u-Strongly Convex). There erists a constant μi > such that for each & in the support of D, f(x,E) f) is ju-strongly convex. Moreover, write x² = arg gin F(x) as the global minimal solution. Assumption 3 (Identical Data Distributions among Workers). Lel D1, D Dr be the dala distribulions for K workers. Assume that these distributions are identical and independent, denoted by DI = D2 DK D
Assumption 4 (Bounded Variance at Optimal). There exists 2 0 such that (
Theorem (Convergence of PALSGD, Informal). Let (0)) xx (T-1) be the sequence generoted by Algo- rithm 1 with INNEROPT as SGD and OUTEROPT as SGD with step size 1. Under Assumptions 1, 2, 3, and 4, let x == 4. For any 0 < p S 2, and for any T > 0, there erists sequence of inner step size (a a sequence of mixing rate (no and a weight sequence {wiYoo) such that for ir 1
Zr Et= wix!! where
Zr -E-- Wh, and ignorining the logrithmic and exponentially decaying terms, we have
kH HT2
[[ - F(x") < z
(8)
WKT
The convergence bound in Theorem contains two parts. The first term, pKT. reflects a linear speedup with respect to the number of workers K and is independent of the hyperparameters for the pseudo-sypccronizztionn steps. When is sufficiently large, this term dominates and matches the Cramer-Rao bound for estimating a single random variable. The second term arises from the stochastic optimization process. As K increases and communication becomes frequent (i.e., is small constant), we can simplify the error bound to xa? n which matches the lower bound for stochastic gradient descent in the strongly convex case (Koloskova et al., 2020).
7
Experiments
7.1
Experimental Setup
We performed experiments on five different workloads to assess the performance PALSGD in a distributed training setup. For image classification, used CIFAR-10 (Krizhevsky al., 2009) with small CNN and
8

---

Published in Transactions on Machine Learning Research (09/2025)
VGGI6(Simonyan & Zisserman, 2014), ImageNet-1K (Deng et al., 2009) with ResNet-50 (He et al., 2016). For language modeling, we employed TinyStories (Eldan & Li, 2023), using GPT-Neo" with 8 million and 125 million parameters.
To ensure a fair comparison, we evaluated PALSGD alongside three widely used distributed training baselines: Distributed Data Parallel (DDP), Local SGD, and DiL.Co (Douillard al., 2023). Tn the CTFAR-10 experiments, we varied the number of workers from to 64. For ImageNet-1K3 experiments, we trained the models with K 4 workers, while for the TinyStories experiments, trained the models using 4 to 8 workers.
Following the standard approach for post-Local SGD method (Lin et al., 2018), all three algorithms Local SGD, DiL.Co, and PALSGD used DDP as warmup during the initial training steps. This warmup phase stabilizes training and improves model initialization before transitioning to a more communication-effiienn distributed method. The training hyperparameters of all algorithms, including learning rates and 7) were tuned independently for each choice of K (number of workers) and H (synchronization interval). This tuning process ensures that each algorithm is evaluated under its best-performing configuration for given distributed training setting. Further details including the ablation studies are provided in Appendix B and E.
7.2 Preliminary Experiments: CIFAR10 Small CNN
We begin by presenting small-scale results, including comparisons with Local SGD (Ortiz et al., 2021). DiL.Co (Douillard et al., 2023) can be interpreted as equivalent to Local SGD, or to FedAvg (McMahan et al., 2017b) in the IID case, when the outer optimizer is set to SGD. As shown in Figure 6 of the Douillard et al. (2023), DiLoCo outperforms these methods. We conducted preliminary experiments to examine how accuracy changes as the number of workers increases for DDP, Local SGD, DiLoCo, and PALSGD. The results are shown Figure 1. The synchronization interval was fixed at 32.
The results indicate that LocalSGD degrades substantially the number of workers increases, compared to the other three methods. Our result consistents with observations in Lin et al. (2018). DiLoCo and PALSGD show similar performance when worker counts or synchronization intervals are small. In contrast, PALSGD maintains better performance when synchronization intervals become large. This occurs because DiLoCo permits extended intervals without ensuring model consistency. PALSGD addresses this issue by imposing seeudo-synchronization constraints.
Based on these findings, we do not include Local SGD in large-scale experiments. Instead, we use DDP the baseline and DiLoCo the main comparison method for evaluating PALSGD. Methods such et al., 2020) are excluded, since prior work has shown that they underperform LocalSGD in the IID setting with reasonable epochs budget (see Table 3 in Karimireddy et al. (2020)).
SCAFFOLD
The following subsections report results from more practical experiments on ImageNet and TinyStories.
7.3 Image Classification Tasks: ImageNet- 1K on ResNet 50
In the ImageNet-1K experiments using ResNet-50, we measured the time required for different training algo- rithmss to reach a specific validation accuracy. All three methods PALSGD, DiLoCo, and DDP achieved the same target top-1 accuracy of 75% (which is the standard thresholds for image classification tasks Lin et al. (2018); Mattson et al. (2020)), but PALSGD demonstrated an 18.4% speedup in terms of training steps compared to DDP and 6% speedup over DiLoCo (See Figure for details). This highlights PALSGD's ability to improve training efficiency while maintaining competitive accuracy.
Unlike previous findings Local SGD Ortiz et al. (2021), PALSGD's final validation accuracy not lower than DDP, as all methods converged to the same 75% target. This result contrasts with prior concerns that PALSGD might suffer from slight accuracy degradation in vision tasks. Given that ImageNet-1K was not originally evaluated in the DiLoCo paper (Douillard et al., 2023), our experiments provide a more comprehensive comparison across different domains. To further support our results on image classification
hhttp::/hhugginggacee
doc/gpt_nes

---

Published in Transactions on Machine Learning Research (09/2025)
Heatmap Accuracy Algorithm Number Workers
Test Accuracy Trend by Algorithm and Number of Workers
0.7
Aigenitmm
DDP
DOM
0.038
0.206
0.5/2
Local 5GB
DioCoo
0.8
0.713
0.500
0.271
Aigathin
0.o
Test Accu
0.5
DiuaCo
0.732
5.222
0.007
0.050
0.4
003
0.3
FALSGE
0.731
0.705
0.050
0.907
0.
Number Workers scale)
Figure 1: Comparison number of workers across algorithm simmulation experiments with CIFAR10 dataset
workloads, also evaluated the effectiveness of our method on workload involving CLEAR-10 training using VGG-16 (Simonyan & Zisserman, 2014). As a result, PALSGD demonstrated superior performance compared to DiLoCo and DDP. Details are provided in Appendix C.4.
Training Convergence / ResNet-50 (K=4)
Convergence Detail
75.2
80
24.62h 26.22h 19h
Validation Accuracy (%)
75.0
60
74.8
40
74.6
DDP (Baseline)
74.4
DiLoCo
20
PALSGD (Ours)
74.2
Target Accuracy:
75.00%
74.0
0
10
15
20
25
30
25
30
Wall Time (hours)
Figure 2: ImageNet-IK 1K on ResNet-50 Experiments (K= = 4/ H = 64): PALSGD demonstrates faster training compared to DDP and DiLoCo, while achioving the same target validation accuracy. The results regarding training accuracy presented in Figure C.1 in Appendix C.I.
7.4 Language Modeling Tasks: TinyStories on GPT-Neo
In the TinyStories experiments on GPT-Neo with K = 8, PALSGD demonstrated significant reductions in communcatton overhead by minimizing the frequency of synchronization steps. Specifically, PAL.SGD reduced the total number of synchronization steps by 93.75% compared to DDP". As a result, in the GPT- Neo-125M experiment, PALSGD was 24.4% faster than DDP while achieving the same target validation loss, as shown in Figure 3. Similarly, in the GPT-Neo-8M experiment, PALSGD reduced the total training time by 21.1% while still reaching the target validation loss, as depicted in Figure 4. In contrast, DiLoCo converged more slowly and failed to reach the larget loss within the same training timeframe. Comparable results were also observed for the case of K = 4, with details provided in Appendix C.2.1. These findings further
2This is the theoretical value when the communication frequency reduced one-sixteenth.
10

---

Published in Transactions on Machine Learning Research (09/2025)
Training Convergence GPT-NEO-125M (K=8)
Convergence Detail
DDP (Baseline)
DiLoCo
8.8 10
Validation Loss
PALSGD (Ours)
Target Loss: 0.84643
x.7
8.6 10
10
x 10
Not Reached
8.5 10
25.99h
34.40h
8 10
8.4 10
10 Wall Time (hours)
20
30
25 30
35
Figure 3: GPT-Neo- 125M Experiments (K=8 / H=1 Training time comparison across distributed algo- rithm to achieve target validation loss. While PALSGD achieves fastest and lowest loss, DDP is slowest and DiLoCo did not achieve larget loss. The results regarding training accuracy presented in Figure C.4 in Appendix C.3.
Training Convergence GPT-NEO-8M (K=8)
Convergence Detail
1.9 10
1.18 x 10
1.8 10
DDP (Baseline)
1.7 x 10
DiLoCo
x.17
Validation Loss
1.6 10
PALSGD (Ours)
1.16 x 10
x.5
Target Loss: 1.12799
1.15 x 10
1.4 10
1.14 x 10
1.3 x 10
1.13 x 10
1.2 x 10
Not Reached
1.12 10
x.1x
5.93h
7.52h
1.11 10
10
2
4
6
6
Wall Time (hours)
Figure 4: GPT-Neo-8M Experiments (K=8) / H=16): Training time comparison across distributed algorithm to achiove target validation loss. PALSGD achioves fastest and lowest loss, DDP is slowest and DiLoCo did not achieve larget loss. The results regarding training accuracy are presented in Figure C.3 Appendix C.2.2.
confirm PALSGD's effectiveness in reducing communication cost without compromising model quality or convergence speed.
Our experiments demonstrate that PALSGD effectively balances communication efficiency and model per- formance. The probabilistic pseudo-synchromizatio mechanism allows workers to update their local models independently, leading to faster convergence and reduced communication overhead. Compared to existing methods, PALSGD achieves significant improvements in both training speed and model loss, making it a compelling choice for scalable distributed training modern language model workloads.
In addition, we performed series of ablation studics on the hyperparameters of GPT-Neo-S0 experiment, and present the result. of synchronization interval H here. Ablation studies other hyperparame-
11

---

Published in Transactions on Machine Learning Research (09/2025)
Validation Loss by H Values
Training Time (hours) by H Values
1.155 x 10
6.05
1.154
6.050
1.15 10
Training Time (hours)
1.146
Validation Loss
6.03
10a
6.02
1.14 10
6.01
1.135 10P
6.00
5.989
5.99
5.986
1.128
1.126
5.98
5.976
1.125 x 10
32.0
64.0
128.0
256.0
32.0
64.0
128.0
256.0
H (Synchronization Interval)
H (Synchronization Interval)
Comparison of H, GPT-Nee-8MM Experiments (K 17 16). (Left) Validation Loss Sensitivity Analysis of H, (Right) H Effects for Training Time.
Figure 5:
including the optimizer, random seeds, 17, and p are provided in Appendix E. Overall, across all workloads we observed consistent trend: values of p 0.25 led to faster training but degraded perfor- mance, while in practical applications such ImageNet-1K and TinyStories, values around 0.05 yielded better results.
For synchronization interval, smaller H leads to longer training times due to more frequent communication during explicit synchronization. Conversely, a larger reduces the number of ALL-REDUCE operations, thereby shortening the time needed to complete an epoch. To assess the effect of H, we conducted an ablation study using GPT-Nee-8M with K workers. The results, shown in Figure 5, illustrate how different values of H impact validation loss (left) and training time (right).
Because the outer learning rate was tuned for H 64, we observed improved loss and accuracy at this value compared to smaller H. However, increasing H beyond 64 (e.g., H 2 128) led to degraded validation loss, likely due to excessive local updates causing model drift before synchronization ion. Meanwhile, training time consistently decreased with larger H, indicating that less frequent synchronization effectively reduces communication overhead. These results highlight a key tradc-off between training efficiency and model performance. They suggest. that moderate values of H can offer the best balance for large-scale distributed training.
8 Discussion and Conclusion
We introduced BReudo-Asynchrood in large-scale distributed learning through probabilistic pseudo-syncarooizatio. Extending Local SGD, PALSGD decreases the frequency of ALL-REDUCE operations, allowing for extended local updates. This method is particularly effective in high-latency environments, such intercontinental data centers, where it enables more efficient, scalable training.
Local SGD (PALSGD): which reduces communication overhead
Summary of Contributions
Our key contributions are as follows:
We introduced PALSGD, a novel extension of Local SGD that incorporates probabilistic pscudo- synchronization, significantly reducing the cost of synchronization without sacrificing model perfor- mance (Section 5 and Algorithm 1).
We provided theoretical convergence bounds for simplified version of PALSGD (Section 6).
12

---

Published in Transactions on Machine Learning Research (09/2025)
We empirically validated PALSGD image classification and language modeling tasks, demon- strating its effectiveness in reducing training time and improving model performance compared to baseline methods such as DDP, Local SGD and DiLoCo (Section 7, Figures 2, 4, and 3).
Limitations and Future Work
The limitations of our work include several factors.
First, our current approach assumes homogencous hardware and network configuration across workers. Future research could explore adaplive methods address heterogeneous environments, where worker speeds or network latencies vary.
Second, thcoretical analysis simplified, focusing PALSGD with SGD the inner and outer optimizer and assuming strongly convex functions, whereas training deep models is inherently non-convex. Extending this theoretical framework to more complex optimizers like Adam or other adaptive methods may offer deeper insights into the algorithm's performance.
Third, we do not include asynchronous baselines in this work. These methods have been studied extensively in Liu et al. (2024); Hadjis al. (2016), and including them here would not only be redundant but could also obscure the focus of our study on semi-syychronous approaches. For this reason, we explicitly state their exclusion limitation here.
Finally, while PALSGD enhances commuicattion efficiency, future studies could investigate further reducing synchronization costs, for example, by employing gradient compression techniques or decentralized commu- nication patterns.
Broader Impact Statement
Our work improves the efficiency of distributed deep learning by reducing communication overhead, mak- ing large-scale model training more accossible and cost-effective. These advancements benefit various applications, including healthcare, scientific research, and industry-scale Al deployments. As our method primarily focuses optimization and scalability, we do not foresee significant societal risks or ethical con- cerns.
Acknowledgement
Our deepest gratitude goes out the anonymous reviewers and the Action Editor, whose invaluable insights substantially enhanced the quality of this manuscript. We also acknowledge that this research was enabled in part by computing resources, software, and technical assistance provided by Mila and the Digital Research Alliance of Canada. Man-Chung Yue is supported in part by Hong Kong Research Grants Council under the GRF project 15304422.
References
El Houcine Bergou, Konstantin Burlachenko, Aritra Dutta, and Peter Richtárik. Personalized federated lcarning with communication compression. arXiv preprint arXiv:2200.0511 2022.
Sayan Biswas, Anne-Marie Kermarrec, Alexis Marouani, Rafael Pires, Rishi Sharma, and Martijn De Vos. Boosting asynchronous decentralized learning with model fragmentation. In Proceedings of the ACM Web Conference 2025, pp. 685 696, 2025.
Marco Bornstein, Tahseen Rabbani, Evan Wang, Amrit Singh Bedi, and Furong Huang. Swift: Rapid decentralized federated learning via wait-free model communication. arXiv preprint arXiv:2210. 14026, 2022.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc'aurclis Ranzato, Andrew Senior, Paul Tucker, Ke Yang, al. Large scale distributed deep networks. Advances in neural information processing systems, 25, 2012.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pollern recognilion, pp. 248-255. Teee, 2009.
Arthur Douillard, Qixuan Feng, Andrei Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc' Marc'Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low-communication training of language models. arXiv preprint 0rXiv:331.008105, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 2011:2447.2178.. 2024.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint Xiv:22900 07759,, 2023.
Gauthier Gidel, Reyhane Askari Hemmat, Mohammad Pezeshki, Rémi Le Priol, Gabriel Huang, Simon Lacoste-Julien, and Ioamnis Mitliagkas. Negative momentum for improved game dynamics. In The 22nd International Conference on Artificial Intelligence and Stalistics, pp. 1802 1811. PMLR, 2019.
Niv Giladi, Shahar Gottlieb, Asaf Karnieli, Ron Banner, Elad Hoffer, Kfrr Y Levy, Daniel Soudry, et al. Dropcompute: simple and more robust distributed synchronous training via compute variance reduction. Advances in Newral Information Processing Systems, 36:48403 48416, 2023.
Xinran Gu, Kaifeng Lyu, Longbo Huang, and Sanjeev Arora. Why (and when) does local sgd generalize better than sgd? arXiv preprint urXim:2303.012 012,, 2023.
Farzin Haddadpour, Mohammad Mahdi Kamani, Mehrdad Mahdavi, and Viveck Cadambe, Local sgd with periodic averaging: Tighter analysis and adaptive synchronization. Advances in Neural Information Processing Systems, 32, 2019.
Stefan Hadjis, Ce Zhang, Toannis Milliagkas, Dan Iter, and Christopher Ré. Omnivore: An optimizer for multi-device deep learning on cpus and gpus, arXiv preprint arXiv:1606.044 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Sai Pranceth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference machine learning, pp. 5132 5143. PMLR, 2020.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXio preprint arXiv:1609. 1609.04466 2016.
Diederik P Kingma. Adam: method for stochastic optimization. aixio preprint arXiv: 100:1412.680. 2014.
Byung-ll Il Koh, Alan George, Raphael T Haftka, and Benjamin J Fregly, Parallel asynchronous particle swarm optimization. International journal for numerical methods in engincering, 67(4):577-595) 2006.
Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning, pp. 5381-5393 PMLR, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Picter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training. Xin preprint 2rXiv:2006.157 15704, 2020a.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous works. Proceedings of Machine learning and systems, 2:429-444 2020b.
Xiangru Lian, Yijun Huang, Yuncheng Li, and Ji Liu. Asynchronous parallel stochastic gradient for non-
convex optimization. Advances in newral information processing systems, 28, 2015.
Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning, pp. 3043 3052. PMLR, 2018.
Tao Lin, Sebastian U Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini-batches, use local sgd. arViv preprint Xix: 10ff::800.0000 008.008177777 2018.
Bo Liu, Rachita Chhaparia, Arthur Douillard, Satyen Kale, Andrei A Rusu, Jiajun Shen, Arthur Szlam, and Mare' Aurelio Ranzato. Asynchronous local-sgd training for language modeling. arXiv preprint arXiv:2401.09 01,0135,, 2024.
Loshchilov. Decoupled weight decay regularization. arXic prepriut orv: 1att:111110000 2017.
Peter Mattson, Christine Cheng, Gregory Diamos, Cody Coleman, Paulius Micikevicius, David Patterson, Hanlin Tang, Gu- Yeon Wei, Peter Bailis, Victor Bittorf, et al. MIperf training benchmark. Proceedings of Machine Learning and Systems,
2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efffeeee learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273-1282 PMLR, 2017a.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Commmiicttion-effcieenn learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273-1282. PMLR, 2017b.
Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, and Peter Richtárik. Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally! In International Conference on Machine Learning, pp. 15750-15769. PMLR, 2022.
Jose Javier Gonzalez Ortiz, Jonathan Frankle, Mike Rabbat, Ari Morcos, and Nicolas Ballas. Tradc-offs of local sgd at scale: An empirical study. arXiv preprint arXin:21 00.0813,, 2021.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 3505-3506, 2020.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00200 2020.
Max Ryabinin, Eduard Gorbunov, Vsevolod Plokhotnyuk, and Gennady Pekhimenko. Moshpit sgd: Commmication-effiiett decentralized training heterogeneous unreliable devices. Advances in Neu- ral Information. Processing Systems, 34:18195-18211, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint Xrxi::1409. 1556, 2014.
Scbastian U Stich. Local sgd converges fast and communicates little. arXiv preprint arx 097677 2018.
Tlya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference machine learning, pp. 1139 1147. PMLR, 2013.
Alexander Tyurin and Peter Richtárik. Optimal time complexities of parallel stochastic optimization methods under a fixed computation model. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Alexander Tyurin, Marta Pozzi, Ivan Ilin, and Peter Richtárik. Shadowheart sgd: Distributed asynchronous sgd with optimal time complexity under arbitrary computation and communication heterogeneity. arXiv preprint 0440:22022044855, 2024.
Jianyu Wang and Gauri Joshi. Cooperative sgd: A unified framework for the design and analysis of local- update sgd algorithms. Journal of Machine Learning Research, 22(213):1-50, 2021.
Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Slowmo: Improving communication- efficient distributed sgd with slow momentum. arXiv preprint vrXim:111000064: 2019.
Cong Xie, Sanmi Koyejo, and Indranil Gupta. Asynchronous federated optimization. arXiv preprint arXiv:1903.0300 03933, 2019.
Hao Yu, Sen Yang, and Shenghuo Zhu. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAT conference artificial intelligence, volume 33, pp. 5693-5700, 2019.
Sixin Zhang, Anna Choromanska, and Yann LeCun. Deep learning with elastic averaging sgd. Advances in neural information processing systems, 28, 2015.
Tuo Zhang, Lei Gao, Sunwoo Lee, Mi Zhang, and Salman Avestimehr. Timelyfl: Heterogeneity-aware asyn- chronous federated learning with adaptive partial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5064-5077 2023.
Shuxin Zheng, Qi Meng, Taifeng Wang, Wei Chen, Nenghai Yu, Zhi-Ming Ma, and Tie Yan Liu. Asyn- chronous stochastic gradient descent with delay compensation. In International conference on machine learning, pp. 4120 4129. PMLR, 2017.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Appendix Table of Contents
A Missing Proofs
18
A.1 Main Result..
18
A.2 Basic Definitions
18
A.3 Main Technical Lemmas
19
A.4 Proof of Thcorem 2
A.5 Proof of Main Technical Lemmas
22
A.5.1 Proof of Lemma 2
22
A.5.2 Proof of Lemma 3
24
A.5.3 Proof of Lemma 4
24
B Experimental Details
26
B.1 Settings
26
C Full Results of Experiments
29
C.1 ImageNet-1K on ResNet50
................................
29
C.2 TinyStories GPT-Neo-8M
................................
30
C.2.1 GPT-Nco-8M Experiment with 4 GPUs (K=4)
................................
30
C.2.2 GPT-Neo-8M Experiment with & GPUs (K=8)
31
C.3 TinyStories on GPT-Neo-125M
32
C.4 CIFAR-10 on VGG-16
33
D Convergence Analysis by Global Steps
34
D.1 Image Classification Tasks: ImageNet-1K 1K on ResNet 50
D.2 Language Modeling Tasks: TinyStories on GPT-Neo
D.2.1 GPT-Neo-SM Experiments
35
D.2.2 GPT-Neo-125M Experiments
36
E Ablation Studies
37
E.1 Sensitivity of Seeds
37
E.2 Optimizer Selection for Loss
37
E.3 Optimizer Selection for Convergence
38
E.4 1) Selection
38
E.5 p Selection
39
F
Additional Study
40
F.1
CIFAR10 Experiments Scaling Performance
40
17

---

(1). Published in Transactions on Machine Learning Research (09/2025)
A
Missing Proofs
In this section, we prove the convergence theorem for PALSGD in the strongly convex case, where both the inner and outer optimizers are gradient descent.
A.1 Main Result
We restate the main thcorem as below:
Theorem 2 (Convergence of PALSGD). Let x0 (T-) be the sequence generated by Algorithom with
INNEROPT as SGD and OUTEROPT as SGD with step size 1. Under Assumptions 1, 2, 3, and 4, for any
0 << S I 2, lel Og = a,, 1/ = _Ho and Wi = (1 ((a-1(+t) where
In(p dop"0"
p
a
48LH
nT
For any T > 0, let BT = 2r 2i w,x(1) where ZT = Et=
T-
Wh: we have
LH
pf
k212022 1.22
[[F( F(x*) <
expl
P
K.H.
PKT
x2 and n L. # where Rg = 11x(0)
The roadmap for the remainder of this section is follows: Section A.2 introduces the basic definitions used in the proof. Section A.3 provides proof sketch and presents the main technical lemmas. Section A.4 contains the full proof of Theorem 2. Finally, Section A.5 proves the technical lemmas stated in Section A.3.
A.2 Basic Definitions
For any E [K],t= 0, .....TT-- 1, let (6 denote a Bernoulli random variable with parameter p (i.e., ((0)
with probability and (()) 0 with probability p)- Let
1- (1)
9k
I 1-p ( <l0) +
(x
pK
The for any t such that (t + 1) mod H + O, we can rewrite the inner step as
(t+1)
(8) k
(8)
I
ar 9k
Let
(0)
9k
2
(9)
K
KE(K]
Let z(1) = Enepnq (1) as the current mean of the client servers at step t. Then we have
((++)) (t) - -agg (t)
(t) Let &k denote the data sampled by the k-th server at step I. Let F. = (st Jo=0,-..-11.es U (b(h) ) )==0...--..EEE for t 2 1 and Fo = 0. Define g(t) as the expectation of g0t) over the randomness at step t, i.c.
g = ( Fd = 1 Σ (VF(=
+
(1) - =2(1))
(10)
K
KE[K]
For any 2 0, lct ビ- < be the largest integral multiples of that is most t, i.c. t- the last iteration at or before such that a!t) updated. Similarly, let ++ 2 be smallest integral multiples of H that is least I, i.e. the next round after such that (²) is updated.
18

---

Published in Transactions on Machine Learning Research (09/2025)
A.3 Main Technical Lemmas
We now sketch the proof of Proof of Theorem 2. Our analysis employs the framework of Local SGD (Stich, 2018). The main technical lemmnas are as follows:
Lemma 1. Under Assumption 1 with L and Assumption with μ 2 0, for any t 2 0 with Or S
1 zr. it holds
that
E | - x² x²1|] $ (1 - (1-pat) E || - -x'13] + + E [19()) - g0tu
at E [(( - -F(x') 20r I. 2
E [1=f -1a1)~z(01) 1
kE(K)
Lemma 1 can be proved almost verbatim to (Stich, 2018, Lemma 3.1).
Lemma 2. Under Assumption with L 2 0, Assumption 3 and Assumption 4. Suppose that S 2
1
12T.² S n /p, and Bg~DlIV +)]? S o². Let g(t) and g(t) be defined as equation 9 and equation 10 respectively. Then for any t 2 0, it holds that
E -1000)0000 -00
(1 p)
1
K 3
Bl
B
(2)
101
24L
Σ
K
B(F(
)[
12x²
p
K2
K² k=
K
The proof of Lemma. 2 can be found in Section A.5.1.
We then show the following lemma that upper- bounds 1 [KE(K] B11l2 - x110) [22] by a recursion:
Lemma 3. Suppose < p S 1 and the sequences (210 { 1120 satisfy (1) aell = सा and (2) ar S p
6L.
Suppose that Be-DllIVf(>> fx=", ( < o². Then for any 2 0, if (t+1) mod 0 we have E, 0, if (t+ mod + 0 we have
Σ
x 122
KE[K]
s 12LH
Σ
Σ - KE[K]
1²
Σ s=t
-
x 11.
2H
s=t
kE[K]
The proof of Lemma 3 can be found in Section A.5.2.
We further simplify the recurrence of 1 ΣeE(K El1a (t) 111 by taking weighted sum from f = 0 to T. K
Lemma Suppose the sequences (Err ferheo satisfy (1) for all (t+ 1) mod H E, 0, and (2)
for all (1 + 1) mod H + 0,
t-
E, < 2H
Σ
o
0 o 12LH
(11)
Suppose that for all 12 0, 0r
p and we wi++ (1 $ Ww. 6LH
Then for all T have
T
Σ W.E. S 2a² 22
Σ w2
p²
Σ
wgGg.
48L
e=0
==0
The proof of Lemma 4 can be found in Section A.5.3.
A.4 Proof of Theorem 2
We able to show Theorem using the previous lemmas.
19

---

Published in Transactions on Machine Learning Research (09/2025)
Proof. Combining with Lemma 1 and Lemma 2, we have
1| x01[p] S (1 - (aa) || x111]]
24La?
)) [[((z(0) - FF(x))
ar)
+
K²
2
20yL
+ (1 -p)
|1240 - - + 12az0²
+
E
K
pk²
K
kE(K]
< (1 -mo))) E
- x²1 -1
+
2ALai
ac
E [(() - -((x))
K2
2
121/aaf (1 -p))
| (1) 200 + 12a 0²
+
+
(12)
K
pk²
K
kE|K]
where the second inequality comes from
1 K KK
Σ B[| - 1000
Σ
B/21,1 (1) - 12 + +2(1=(1 - -0000197
K
kE[K]
and
E|| E| 20001/]= = K2
1
Ell Σ
( (1) 20))
S
1 K
Σ
Bll 11 - -0100)
AE[K]
kE[K]
For simplicity, write de = E - 11] [[]] Gt = [F(()) - F(x*)] Then we can rewrite equation 12 as
-
and
E K EkEiK] B(( 1110
24La K2
at
e + (SarL+ + 12n -p))
E, + 120/0²
d++1 S (1 - (1-par) de +
2
pk
K
I and rearranging, we have Multiplying both sides by at
(2- -1-2111 24Tor) --24Tar
1-100 d:
deti+ + 8L+ 12n - p) E +
120,0²
e S
K2
a2
at
pK
K
From 02 = a S
p S aI.
and A/- = 1) = 2Ha
p
we can further simplify the above inequality as
48HL
1
Et < pa de
1 do+1 + +LLE, 12ao2
4
K
Next, by taking the weighted average from = 0 to T - 1 with weight wi = (1 - pax) -(++1) and normalizing
factor Wr = 1-1 Wt, we have
T-1 waa)wad - wide+1)
Welt 5
((1
4 0
WT
1=0
9L
12a0²
T-1
+
Wr +
WT
K
WT t=0
(w-- wede+1)
WT
Wr + 12a02 T-1
76
+
Σ wi
(By (1-aa)w; = w;-1)
WT
K
Wr
<
Σ
(w--lda - widr+1)
WT
20

---

Published in Transactions on Machine Learning Research (09/2025)
+
76
90
2 H²
T-1
Σ
w2
+
p
T-1
W/Ct
+
12ao²
(By Lemma 4)
WT
2L
K
1=0
1=
1
(do
wrd(+1)
0 WT
9L
T-1
p
12
90
H2
Wio²
w/et
Wr
48I
K
t=
t=
(Taking telescoping sum on the first term)
Rearranging and using p 2, get that
1 1
1 1
12a
Σ
Wcet S
(do wrdh+1) 88LHHa2 +
0²
WT
Wr
K
1=0
do
(1 nam² + (81HH2a 12a K
o²
a
do
(81HH + 12a
K a²
a
Let
g(a) = du exp(-uTa) + 81La 220 + 12ad
a
K
We wish to minimize g(a) with respect to To do so, differentiate g(a) with respect to we get
1 x3 + uT ) + 162T.HPoaa+ 12o"
g'(a) = do expexxp(xuTe
(13)
K
and we want to find a² such that g'(a*) 0. Since (a) > such a" is the minimizer of g(a).
However, it is complicated to get the accurate value of a". But since we will ignore the poly-logarithmic terms in our final bound of g(a), it suffices to approximate a within poly-logrrithmic factor. This would give a proper approximation of the the second and the third terms of g(a*). For the first term, we can use Equation equation 13 to derive the value of exp(-ıTTo)) and simplify the term to a polynomial of a.
Suppose a" + In A. Our goal is to show that O(poly H, H.K.oo')) First, by plugging + In equation 13 we get
do
(InA+ + 1)22T2
162TH A +
12x²
g(aa) A
= 0.
(14)
ln² A
μT
K
Simplifying both sides, we have
do
(In A + 1)
162LH1o0 In
12o²
A
2 A
μT
K
and
In A +1
1 In A 0²
12
A A
27
Assuming T to be sufficiently large and
162xH² T
12
we can further simplify the above equality as K
In A 1
Co²
(15)
Al A
dourTTK
for some constant C 12. Notice that for any > 4,
A
A
< and
In² 2 VĀ
we have
A+1
A+
In +1
1
<
Al A
A
21

---

Published in Transactions on Machine Learning Research (09/2025)
for any A 2 4. Plugging back to equation 15, have that
Co? 1
3 douTTKK A2
Therefore A is polynomial in T, H, K and o-1
Now we upper-bound the minimal value of g(a). Since Theorem also requires that a < discuss the following two cases:
48L we need to
P
If a² 1SLH> then g(a) takes the minimum when a². Plugging in a² 1 In A we have
g(a) = g(a")
do exp(= + 111 +
12x"
a²
K
1662LH202
12
1
81LH 0² (a")2
1202a
AT
K
162LH20 A
12o²
8ILH In?
12o² In A pKT
11T22
KKT
T2
KHH22 o²
+
NT²
kKT
Here the step comes from equation 13, the third step come from a" = A is polynomial in T, H. K and a1
In A, and the last step comes from 1 nT
If 2 48LH then since g(a) is convex in a, it is monotonically non-increasing when S 48LH Thus g(a)
takes the minimal value when = 48LH Therefore
g(a) gl 48LH Ō doLi
uDT
RH?
G²
exp(
p
48T.
NT
μKT
Here 48TH a s Ō w the second and the third terms in the second step. Combining the two cases, we have
T-1 1 S
LgLH
RH?
exp(
+
5 WT
p
48T.H
1sT2
nKT
1=
The proof then follows from
B[F(+T)]
Σ
w. E(F(((())) F(x*)
1
Σ weer.
WT
WT
A.5 Proof of Main Technical Lemmas
A.5.1 Proof of Lemma
Proof. Noting that {g
(t)
(o + 200))(
(4)
(= are independent zero-mean random vectors, we
have that for any i 2 0,
1 K
27
E (1)x) - -00)p]]
(6)
(F( + max()) m2 (1)
E
9k
K
22

---

Published in Transactions on Machine Learning Research (09/2025)
K
(1) 9k
VF(2(() + na(!)) - m (1) 11)
(16)
K
1=1
where the last term comes from the independence of {.² Recall that
(t) 1 1- 6(4)
1obk (1) pk (a (0 x(1)).
2 +
gk -
where (()) is Bernoulli random variable that equals to 1 with probability p and 0 with probability 1
p. Also &x denotes the data sampled by the k-th server at step t. Let F, = 14 ---0....-1,,. U
(66²) >==0,...,- L-Lke] Conditional on any fixed Fe, we then rewrite each of the summands as
E
(8)
n (t)
+ m)x(1) - VF(2Z ||1
= Bg e(0)
9/ 11 + +,x(())= - vr(u!")
(t)
(1)
(1) - k
Tob (1)
3 200
V (1) _((1 +
pk (0 -
(1)
- -1( -m) + +mo(()) - VF(ak (0)
(2)
p
1
v (1) ) - vF(2 - = + mar 1
-11 - p) BB00
1 1-p p
(
1
1/1 (1 (1) - -00) - vF(af()) |
+ p.
p -1
E Ed
(0) (00) (= + (1 - p)
|:00 (1) (()
p v (z
1 p
1 I-p p
(1 - p)2
1-p VF(1(()) /|
(t) - m0 (0)
+
n
p
p v (x (0)) 2
| (0) (11)) ( vF(a ||1]
+ 1-p p.
[x(= (t)
x0)) 22))
E (
1-p
p
1-p p
Notice that for all &e E [K],
E 1
[ (8)
k , -110 vf(a ||]
f( (1) {())
$ E
(0) ((0)) Vf(2", <((1) + Vf(2" x()))
s E (
E m
(=
x
(1)
(/0) Vigo(( (f( Vf(("' ((0) + Vf(=", )1
s 3- E. B
(1) - Vfggol (x )11 + /V gg (z - Vf(=" -0 )1 + V /f(2",<0011
$ 3 - 201 + 6LE un)
(x)
(0) ) - f(x",110) +1))++30 302
31'E||>> - 1| (LE(F(2 F(x )1 302
(17)
and from Jensen's inequality,
[V < E000 [IV (1)
. [<()r]]] -3L' + - F(x*)) + +00,
we then have
v F( (t) 27
E (t) (
+
1-p
(6)
(110)2110 (()))
1- p
p
1 p
23

---

Published in Transactions on Machine Learning Research (09/2025)
[Vf 00 -50) - )1( 2n - p) |10 - =211)//2++ + 1-p p
2p |V |V )1
_ 1- "
E Beg
(1+ 2p)
(3L21
(t) 12 + 79 (F(x(0) - F(x*)) + +300) + 2n (1 p)
x (t)
- z(() 12
1-p
< 3mp (1 - p)
| - ( 12 + 24L (F(z!0) (Fc1))-F(28))) FF(x)) + 1202
(1)
Where the last step comes from p S 1 and 12L < zi Substituting the last inequality into equation 16 and taking expectation over Fe yields the desired inequality.
A.5.2 Proof of Lemma
(17) Here the last equality comes from that the client model synchronizes with the average model at step t- Therefore, we can write # ΣxE(K] B11l x: 111) as
Proof. Note that 211) 1/2 Exe(k] 2b (17) x(1") Ix
1
Σ
(t) =00
Σ kE[K]
Ella (t)
IE (t >11
(18)
K
kE|K]
Using
Σ
22112 < H. ΣE1
||>; 1|
we have
1
E
|
2k
1e
12]
K
kE[K]
H
[1o( - 21
E
K
kE[K]
H
02 p1-p)
ag??
Σ
-B BB [V -f 11
E1|=1 (3) xx00)
K
p
kE(K]
H
(6LE(F(((() ([=")+++0²)) + (a + 6a? Bllbob" - x(0 ((8l
20
K
p
KE[K]
H
6o -LE(F(((0)-
+ 2H²
p²
Bl1lz/
(0) - rle ((alc d
K
KE[K]
< 12LH
2 Σ
Σ kE[K]
a 0²
Σ K
Ellak (+) - (1) (19)
K
2H
kE[K]
where the third step comes from equation 17 and p S 2, and the fourth step comes from a²312
2²
and
a3 2 The lemma then follows.
A.5.3 Proof of Lemma 4
Proof.
Substituting E,
, on the right-hand-side of equation 11 by equation 11, we get
E <
Σ
E. + a
Σ 02 + 12LH Σ d'e.
2H
8-1
8-1
s-1
<
p
Σ
Σ E, + 6H
E0² 12LH
Σ
aze,
2H
2H
s=t
r=t
r=t
r=t
24

---

Published in Transactions on Machine Learning Research (09/2025)
t-1
H-
+
2
Σ
0² + 12LH
Σ
eees
s=t
p²²
Σ
+ +
Σ (3Hpa o + oH + Σ
(6LHpa + 12 + 12LHa e--1
4H1
Repeat the above step for t - times and using E
gets
p== p)(1 20-r-
1-1
p" p²-
E < Σ
(1
-0-
)02 + Σ
(1+
20-1- )es
==
8=t-
[-1
< 99
+ Σ
es
(20)
Taking average E, from = 0 T with weight wi (1 ca)', we get
T
T
1-1
Σ w - 9H2 a
w
1
W/
C2
t=0
1=0
$ *9H20
2
18LHa
C2
Ws
t=0
t=0
T
T
S *9H20 02
Wio
18LH
0
W/Ct
Σ (1+
P par
H
t=0
1=
s=t
S -9H20 2
W40
WiCt
16L
t=0
t=
where the third step comes from u1+1 S (1+ + 4 )W,, and the last step comes from 04 S 15TT.- The lemma then follows.
25

---

Published in Transactions on Machine Learning Research (09/2025)
B Experimental Details
Our code is available at http:://whh...ttttttttt..///tetp
We conducted our experiments on three datasels: CIFAR-10, ImageNet-1K, and TinyStories. The CIFAR- 10 and ImageNet-1K datasets were used for image classification tasks, while TinyStories was employed for language modeling experiments. We implemented the distributed algorithm (DDP, Local SGD, DiLoCo and PALSGD) on three different distributed systems, referred to as Cluster A, Cluster B, and Cluster which have varying GPU configurations and interconnects, detailed below.
B.1 Settings
Hardware Configurations:
We utilized three types of clusters for our experiments:
Cluster A
GPU: NVIDIA Tesla T4 (16GB)
GPU Bandwidth: 320.0 GB/s
GPU Interconnect: PCIe with NUMA Node Interconnecl (No NVLink)
Cluster B
GPU: NVIDIA Tesla V100 DGXS (32GB) 8
GPU Bandwidth: 897.0 GB/s
GPU Interconnect: NVLink, 150 GB/s per GPU
Cluster C
GPU: NVIDIA L40s (48GB) x
GPU Bandwidth: 864.0 GB/s
GPU Interconnect: PCle with NUMA Node Interconnect (No NVLink)
Software and Library Configurations:
All GPU clusters used the following software environment:
Python: 3.11.6
PyTorch: 2.3.1+cu122
CUDA: 12.1
CUDNN: 8902
Workloads:
Small CNN on CIFAR-10 (Preliminary Simulation Experiments): The CIFAR-10 datasct is widely used in machine learning research, particularly for image recognition tasks. It contains 60,000 color images, each measuring 32x32 pixels, evenly distributed across ten distinct classes including airplanes, automobiles, birds, cats, deer, dogs, frogs, horses, ships, and trucks. The dataset consists of 50,000 training images and 10,000 test images, with cach class represented by 6,000 images. We employed a small CNN model in Table B.1 and the CIFAR10 dataset. We investigate accuracy degradation across varying numbers of workers, including extremely large worker configurations. Due to limited computing resources typical of academic institutions, experiments were performed
hhttps://www.co teronto. edu/ -kriz/ciar..tt html
26

---

Published in Transactions on Machine Learning Research (09/2025)
on a simulated multi-worker environment using single GPU and relatively small-scale model. When controlling seeds and other conditions, such simulated experiments accurately represent actual distributed learning in terms of accuracy. However, execution time results are not comparable, as parallelization within a GPU differs fundamentally from distributed scttings. Therefore, only accuracy results are reported.
VGG-16 on CIFAR-10: We conducted experiments on CIFAR-10 using the VGG-16 architecture (Simonyan & Zisserman, 2014) to evaluate algorithm performance. Unlike prior experiments with smaller CNNs, this setup aims to leverage a deeper and more expressive model to assess the impact of algorithmic changes under realistic training conditions. The VGG-16 model we used follows the standard configuration defined the PyTorch torchvision models module, consisting 13 convolutional layers followed by 3 fully connected layers. The convolutional layers use 3x3 filters with padding to preserve spatial resolution and are interleaved with ReLU activations and max- pooling layers for downsampling. After the convolutional blocks, the resulting feature maps are passed through an adaptive average pooling layer and flattened. The classifier consists of two fully connected layers with 4,096 units each (activated by ReLU and followed by dropout), and a final linear layer that maps to 10 output classes corresponding to the CIFAR-10 categories.
ResNet50 ImageNett11K: The ImageNet-11K dataset is widely used benchmark for image classification lasks, providing comprehensive test for evaluating model performance on large-scale visual recognition. We employed the ResNet50 model 5 with 25.6 million parameters to test the effectiveness of the PALSGD algorithm in distributed training settings 6. The ResNet50 model used in this experiment features a total of 50 layers, which include a series of convolutional, batch normalization, and ReLU activation layers arranged residual blocks. The input size is set to 224x224 pixels, a standard for ImageNet classification. Each residual block has bottleneck structure that reduces the number of parameters while maintaining the accuracy of the model. The final fully connected layer produces 1,000 output logits corresponding to the 1,000 ImageNet classes.
GPT-Neo on TinyStories: The TinyStories dataset (Eldan & Li, 2023) designed for small- scale text. generation lasks, providing a benchmark for language modeling performance on short narrative texts. We evaluated the PALSGD algorithm under distributed training conditions using the GPT-Nco model?. Experiments were conducted using two different model configurations: one with 8 million parameters and another with 125 million parameters.
For the 8M parameter configuration, the GPT-Nco model features a maximum positional embedding size of 300, which allows the model to handle input sequences of up to 300 tokens. The hidden size is set to 128, defining the dimensionality of the model's internal representations. It employs 8 attention heads to capture diverse relationships among tokens through its self-attention mechanism, and it is built with 8 hidden layers to learn complex hicrarchical patterns in the data.
In contrast, the 125M parameter configuration is designed to handle longer sequences and capture more complex patterns. In this sctup, the model uses a maximum positional embodding size of 1024, a hidden size of 768, 12 attention heads, and 12 hidden layers. This higher-capacity configuration enhances the model's ability to process and understand more intricate language patterns compared to the 8M version.
Training Configuration:
All experimental results, unless otherwise noted, refer to the hyper parameter configuration with the best results for that metric. The results of the ablation study are shown in Section E. The target loss shown in the training curve plots is based on the loss achieved by DDP within the predefined epoch budget.
Small CNN on CIFAR-10 (Preliminary Simulation Experiments): The number of workers
ranged from 4 to 64, while synchronization intervals tested were 32. PALSGD was set with p
"nttps://wwv.immge-nr wttp:/eenntttto//ooooeeeeoo wrg//oonnoad.
"nttps://pytorch http://ppoorrcooorrriiioo//////e hatal////oollllggggerteeerrrrrre /gw.geaatte/ooorrriiiiiooo
html
WW have followed official pytorch implementation and have not done any special data augmentation: https://githu:
7 "httpe://hugginggace httpe///tt//ttaaaarrrreeeee//eee en/nddel doc/ doc/gpt_neo
27

---

Published in Transactions on Machine Learning Research (09/2025)
Table B.1: Architecture of the small CNN model used for simulation experiments
Layer
Type
Configuration
Convolution
channels: 3 + 32, kernel size: 5, padding: 2
2
Activation
ReLU
3
Max Pooling
kernel size: 2, stride: 2
Convolution
channels: 32 - 64, kernel size: 5, padding: 2
Activation
ReLU
Max Pooling
kernel size: 2, stride: 2 input: 64x8x8, output: 1024
Fully Connected
8
Activation
ReLU
Fully Connected 9
input:
1024, output: 10
0.25. The outer optimizer used was Nesterov momentum, and the inner optimizer was momentum SGD. Inner learning rates of 0.01, 0.001, and 0.0001 were explored. Additionally, was fixed at 1, and outer learning rate included 1, 0.1, and 0.01. No weight decay was applied, and training was conducted for 100 epochs.
CIFAR10 Training
VGG-16:
We trained the model for 200 epochs using 4 GPUs a single node on Cluster C. The local batch size was set to 128 per GPU, resulting in global batch size of 512. No gradient accumulation used. The model architecture was selected from the torchvision model, with VGG-16 as the default. We used Momentum SGD as the inner optimizer with learning rates selected from {0.025, 0.05, 0.075), and Nesterov Momentum SGD as the outer optimizer. The outer learning rate was chosen from {0.2, 0.4, 0.8, 1.2, 1.6]: with a warm-up period of 10 epochs followed by cosine annealing (CosineAneeallng(R)) for scheduling. Weight decay was fixed 1 x 10-4. The synchronization interval H was set to 128, and Local SGD variants started after 2049 iterations (approximately epoch 40). We used the PALSGD algorithm with decoupled updates enabled. For PALSGD--stll adaptive synchronization, we explored probabilistic synchronization with {0.02, 0.05, 0.1} and local step size E {0.1, 0.25, 0.5).
ImageNet-11K Training on ResNet-50:
We trained the model for 90 epochs with local batch size of 64 per GPU, using 4 GPUs on Cluster C. The global batch size was sot to 256. The inner optimizer's lcarning rate was fixed at 0.001 with the Momentum SGD. For outer optimizer, we used Nesterov Momentum SGD with outer learning rate fixed at 0.1 to 0.2 same GPT--eo experiments. The synchronization interval H was set to 64. The variants of the Local SGD algorithm started after 200K iterations (39 epoch). For PALSGD experiments, the probabilistic synchronization parameter = 0.05 and is 0.5 to 16.
TinyStories Training on GPT-Neo-8M and 125M:
Both the 8M and 125M models were trained following the training protocol described below. We trained the model for 15 epochs with a local batch size of 512 per GPU, and use Cluster A in the experiments with 4 GPUs, and Cluster B in the experiments with 8 GPUs. The global batch size was set to 2048. The inner optimizer learning rate was fixed at 0.001, and we employed AdamW (Toshchilov, 2017) for the inner optimizer with gradient clipping enabled. Regarding outer optimization for DiLoCo and PALSGD, we use Nesterov Momentum SGD with the outer learning rate fixed at 0.1 to 0.2. The synchronization interval probabilistic synchronization parameter p = 0.1, 7/ = 16. The variants of the Local SGD algorithm started after 1024 itcrations. For ablation study, the synchronization interval is sct 32 to 256 the probabilistic synchronization parameter p 0.025 to 0.5 and n) is 0.25 to 64.
was set to 16 (125M) or 64 (SM), the
28

---

Published in Transactions on Machine Learning Research (09/2025)
C Full Results of Experiments
C.1 ImageNet- 1K on ResNet50
Figure C.1 presents the training accuracy curves for ResNet-50 on ImageNet-IK with K = 4 workers and a synchronization interval of = 64, conducled on Cluster C. Consistent with our validation accuracy results, PALSGD is 15.4% faster than DDP and 4.3% faster than DiLoCo, while all methods converge to the same final accuracy of 74.0%.
Training Convergence ResNet-50 (K=4)
Convergence Detail
74.2
80
24.22h 25.46h 28.63h
Training Accuracy (%)
74.0
60
73.8
40
73.6
DDP (Baseline)
73.4
DiLoCo
20
PALSGD (Ours)
73.2
Target Accuracy:
74.00%
73.0
5
10
15
20
25
30
25.0
27.5
30.0
Wall Time (hours)
Figure C.1: Training Accuracy ImageNet-1KK -K on ResNet-50 Experiments (K = 4/ = =64): PALSGD demonstrates faster training compared DDP and DiLoCo.
29

---

Published in Transactions on Machine Learning Research (09/2025)
C.2 TinyStories GPT-Neo-BN
C.2.1 GPT-Neo-8MM 8M Experiment with 4 GPUs (K=4)
We conducted an additional experiment using GPT-Neo-8M with 4 workers on cluster A. Figure C.2 presents both the training and validation loss curves for this setup. Consistent with previous findings, PALSGD achieved the fastest convergence, reaching the target loss significantly earlier than both DiLoCo and DDP. Specifically, PALSGD reduced training time by 23.0% compared to DDP, while DiLoCo provided more modest 7.9% improvement over DDP. DDP exhibited the slowest convergence, taking the longest time to reach the target loss. While DiLoCo was faster than DDP, it failed to reach the target loss within the given training time.
Training Convergence / GPT-NEO-8M (K=4)
Convergence Detail
2.4 10
x.2x10
DDP (Baseline)
x.13x10
DiLoCo
1.12 10
2 10
PALSGD (Ours)
Training Loss
1.8x 10
1.11 10
Target Loss: 1.08057
1.6x
1.1 x 10
1.09 x 10
x04x100
1.08 x 10
1.2 10
Not Reached
1.07 10
24.93h
32.38h
10
1.06 x 10
10
20
30
30
35
Wall Time (hours)
Training Convergence / GPT-NEO-8M (K=4)
Convergence Detail
1.6 10
1.14 10
DDP (Baseline)
1.5 10
1.13 10
DiLoCo
Validation Loss
1.4x 10
PALSGD (Ours)
1.12 10
Target Loss: 1.08725
x.11x10"
1.3x 1
1.1
1.2x 10
1.09 10
Not Reached
1.08 x 10
1.1 10
25.82h
32.37h
1.07 x 10
10
20
30
25
30
35
Wall Time (hours)
Figure C.2: GPT-Neo Experiments (K=4 H=16): Training time comparison across distributed algorithm to achieve target training loss (top) and validation loss (bottom). PALSGD achieves fastest convergence and lowest loss, while DDP is slowest and DiLoCo did not achieve target loss.
30

---

Published in Transactions on Machine Learning Research (09/2025)
C.2.2
8M Experiment with 8 GPUs (K=8)
Figure C.3 presents the training loss curve for the GPT-Neo experiment with workers on Cluster B. Consistent with our validation loss results, PALSGD demonstrates the fastest convergence, reaching the target training loss 20.8% faster than DDP. Among the three methods, DDP converges the slowest, while DiLoCo does not achieve the target loss within the given training time.
We remark that while it might seem that the communication bottleneck would be larger due to the doubled number of GPUs compared to the 4-GPU cluster, this is not necessarily the case. Since the GPUs in Cluster B do not share the same bandwidth and FLOPS as those in Cluster A, furthermore, Cluster B differs in that the GPUs are connected via NVLink, which provides faster communication. This makes an ideal computing environment for distributed deep learning. As result, we only achieved a 20% improvement training speed. What we want emphasize here is that even an optimal communication environment like NVLink, which does not span across nodes, there is still room for 20% increase in training speed.
Training Convergence / GPT-NEO-8M (K=8)
Convergence Detail
3 x 10
DDP (Baseline)
1.17 x 10
DiLoCo
PALSGD (Ours)
1.16 x 10
Train Loss
x 10
Target Loss: 1.12360
1.15 x 10
1.14 10
1.13 x 10
1.12 10
Not Reached
1.11 10
5.95h
7.52h
10
2
4
8
Wall Time (hours)
Figure C.3: GPT-Neo Experiments (K=8) H-64) on DGX-1 (8 V100 GPUs Connected by NVLINK):
Training time comparison across distributed algorithm achieve target loss.
31

---

Published in Transactions on Machine Learning Research (09/2025)
C.3 TinyStories OPT-Neo-125M
Training Convergence / GPT-NEO-125M (K=8)
Convergence Detail
2 10
8.5 10
DDP (Baseline)
DiLoCo
8,4
PALSGD (Ours)
Training Loss
Target Loss: 0.80473
8.3 x 10
8.2
10
Not Reached
8.1 x0
25.31h
34.40h
8 x 10
10
20
30
25
30
Wall Time (hours)
Figure C.4: GPT-Neo- 125M Experiments (K=8 / H-16) Training time comparison across distributed algo- rithm to achieve target training loss.
Figure C.4 compares the training time required by PALSGD, DiLoCo, and DDP to reach common target training. PALSGD and DDP eventually achieve the target loss, whereas DiLoCo fails to reach this target. Furthermore, PALSGD converges significantly faster than DDP. Specifically, PALSGD achieves the target training loss 26.4% faster than DDP. These results highlight PALSGD's superior efficiency in accelerating training while maintaining model quality.
32

---

Published in Transactions on Machine Learning Research (09/2025)
C.4 CIFAR-10 10 VGG-16
We conducted experiments on the CIFAR-10 datasel using the VGG16 archilecture Simonyan & Zisserman (2014) on cluster A with k = 4 workers. Figure C.5 presents the training loss and validation accuracy of the DDP, DiLoCo, and PALSGD algorithms for varying synchronization intervals (H). As expected, the results show an inverse relationship between and regularization strength for DiLoCo and PALSGD-iA. i.c. training loss tends be lower when using DDP or smaller values of H. The relatively poor performance of DDP is likely due its use of large batch sizes. In contrast, DiLoCo and PALSGD operate with batch sizes reduced by factor of four, which may result in stronger implicit regularization effect. Compared to DiL,CC, PALSGD consistently achieves lower training loss and higher validation accuracy, suggesting better generalization performance. Morcover, PALSGD is less sensitive to the choice of H, showing only a 0.058% drop in accuracy when increasing H from 32 to 1024. These results demonstrate the effectiveness of PALSGD in reducing communication overhead without compromising model performance.
Train Loss (arch=vgg16)
DDP 1-1.1.111----- 101-8--031118e 18e-08883331 1.8--00 1. 0.11e-0
Train Loss (Log Scale)
Algorithm
DiLoCo 4.57e-041.07e-000
98.-033..44-03
10-3
PALSGD 2 5.45-005..00 5.60e-04 1. 48e-03 3. 9.300-03 11.669--03 -.3
32
64
128
256
512
1024
Sync Interval
Best Val Accuracy (arch=vgg16)
DDP
89.80
89.80
89.80
89.80
89.80
89.80
91.5
Best Val Accuracy
Algorithm
91.0
DiLoCo
91.34
91.20
90.95
90.77
91.07
90.99
90.5
PALSGD
91.77
91.69
91.15
91.13
91.14
91.24
90.0
32
64
128
256
512
1024
Sync Interval
Figure C.5: CIFAR-10 with VGG16 Experiments: (Top) Comparison of training loss with respect H (Syne Interval); (Bottom) Comparison of validation accuracy with respect to H. PALSGD achieves lowest loss and highest accuracy consistently.
33

---

Published in Transactions on Machine Learning Research (09/2025)
D
Convergence Analysis by Global Steps
D.1
Image Classification Tasks: ImageNet-1K ResNet 50
Training Convergence / ResNet-50 (K=4)
Convergence Detail
413512 steps
74.2
80
425949 steps
Training Accuracy (%)
74.0
424078 steps
60
73.8
40
73.6
73.4
DDP (Baseline)
20
DiLoCo
73.2
PALSGD (Ours)
73.0
100000
200000
300000
400000
4.00
4.25
Global Step
1e5
Training Convergence ResNet-50 (K=4)
Convergence Detail
436059 steps
80
75.2
Validation Accuracy (%)
434138 steps
435402 steps
75.0
60
74.8
40
74.6
74.4
DDP (Baseline)
20
DiLoCo
74.2
PALSGD (Ours)
74.0
100000
200000
300000
400000
4.25
4.50
Global Step
1e5
Figure D.1: Training (Top) and Validation (Bottom) Accuracy of ImageNet-IK on ResNet-50 Experiments (K 4 H 64) with using Cluster C.
Our main results Figure present wall-clock time comparisons. To isolate the source of performance gains, this section provides complementary analysis. We compare methods based the number of global steps required to reach target metric. This approach distinguishes algorithmic convergence speed from system- level speedups due to reduced communication. Image Classification ImageNet-1K For the ResNet-50 experiment, all three wethods-PALSCHE DiLoCo, and rop-roached a given validation accuracy in a nearly identical nunber of global steps, as shown in Figure D.1 (bottom). DDP demonstrated a marginal advantage the convergence of training accuracy. These results suggest the convergence properties of the algorithms are comparable for this task. The practical wall-clock advantage of PALSGD shown in Figure therefore stems primarily from its reduction in commuication overhead, from superior step-wise convergence.
34

---

Published in Transactions on Machine Learning Research (09/2025)
D.2 Language Modeling Tasks: TinyStories GPT-Neo
D.2.1 GPT-Neo-8M Experiments
The GPT-Neo-8M experiments show that PALSGD and DDP again achieve similar convergence for both training and validation loss. In contrast, DiLoCo failed converge, with its loss increasing during training. PALSGD's advantage DDP this model its communication efficiency. Its advantage over DiLoCo is its stable convergence.
Training Convergence GPT-NEO-8M (K=8)
Convergence Detail
x 10
DDP (Baseline)
1.17 x 10
DiLoCo
PALSGD (Ours)
1.16 10
Train Loss
x 10
Target Loss: 1. 13660
1.15 x 10
x.14 10
1.13 x 10
Not Reached
x.12
6940 steps
1.11 10
6986 steps
10
2000
4000
6000
6.2e+03 7.7e+03
Global Step
Training Convergence / GPT-NEO-8M (K=8)
Convergence Detail
2 10
1.18 10
DDP (Baseline)
1.8 x08
DiLoCo
x.17
Validation Loss
PALSGD (Ours)
1.16 10
1.6 x 10
Target Loss: 1.12799
x.15x10'
x.4x100
1.14 x 10
1.13 10
Not Reached
1.2 10
X691 steps 6986 steps
1.12 10
1.11 x 10
0
2000
4000
6000
6.2e+03 7.7e+03
Global Step
Figure D.2: GPT-Neo 8M Experiments (K 8
64) DGX-1 (8 V100 GPUs Connected by
NVLINK): Training time comparison across distributed algorithm to achieve target loss.
35

---

Published in Transactions on Machine Learning Research (09/2025)
D.2.2 GPT-Neo- 125M Experiments
Experiments with the larger GPT-Neo-125M model reveal a different dynamic. Here, PALSGD achieves the larget loss in fewer sleps than both DDP and DiLoCo. PALSGD required 15.8% fewer steps than DDP reach the target validation loss. DiLoCo again failed converge. This algorithmic improvement contributes significantly to the overall performance gain. The 24.4% wall-clock time reduction reported in Figure 3 is product of both laster algorithmic convergence and reduced communication costs.
Training Convergence GPT-NEO-125M (K=8)
Convergence Detail
2 x 10
DDP (Baseline)
8.4 10
DiLoCo
PALSGD (Ours)
8.3x 10
Training Loss
Target Loss: 0.80473
8.
8.1x
10
8 10
Not Reached
11426 steps
13973 steps
7.9 x 10
2500
5000
7500
10000
12500
1.25 1.50 1e4
Global Step
Training Convergence GPT-NEO-125M (K=8)
Convergence Detail
DDP (Baseline)
8.8 10 -1
DiLoCo
Validation Loss
PALSGD (Ours)
8.7 10
Target Loss: 0.84643
8.6 10 -1
10
8.5 10
9 x 10
Not Reached
8.4 x 10
11752 steps 13973 steps
8 x 10
8.3 x 10
2500
5000
7500
10000
12500
1.25 1.50
Global Step
1e4
Experiments (K = / 16) on DGX-1 (8 V100 GPUs Connected by NVLINK): Training time comparison across distributed algorithm to achieve target loss.
Figure D.3: GPT-Nco 125M
36

---

Published in Transactions on Machine Learning Research (09/2025)
E
Ablation Studies
Here, we present the results of an ablation study on hyperparameters using GPT-Nco-8M to train on the TinyStories dataset with K=8. We conducted ablation studies on the following hyperparameters: seeds, optimizer selection, 7 P. and H. Default hyperparameter setting is described in Appendix B.
The soods represent the error bars in our experiments. While optimizer solection and 1) influence the final loss and accuracy, they have minimal impact on training time, so focused on evaluating these parameters using accuracy and loss metrics. On the other hand, and H affect training time, we report the time taken in addition accuracy and loss. The result of the ablation study for H is already included in Section 7.4.
Unless otherwise specified, the experiments follow scheme in which only one hyperparameter is altered, with the rest set to their optimal values. Detailed descriptions of the hyperparameters are provided in the previous section.
E.1 Sensitivity of Seeds
We evaluated different algorithms using three random seeds (2022, 2023, 2024). The results showed that the variance was minimal, confirming that it had no significant impact on the performance of the training process within the scope of our workload.
Algorithm Mean Validation Loss DDP
Variance
1.1247
1.0047e-05
PALSGD
1.1273
5. 7.76680-07
DiLoCo
1.1646
4.6850e-05
Table E.1: TinyStories on GPT-Nee-8M/ Mean and Variance of Final Validation Loss for PALSGD with
Different Optimizer Combination
E.2 Optimizer Selection for Loss
In line with the findings suggested by Douillard et al. (2023) for language modeling workloads, the combina- tion of using AdamW as the inner optimizer and either Nesterov momentum or SGD with momentum as the outer optimizer not only achieved the best loss, but also revealed that using AdamW as the outer optimizer significantly worsened the loss.
Validation Loss by Optimizer and Outer Optimizer Comt 1175.660
Validation Loss (log scale
135.257
10a
10
4.100 2.9552.955
1.305 126
10
nesterov + adamw
nesterov + sgd
nesterov + sgd m
nesterov + nesterov
adamw + adamw
adamw + sgd
adamw + sgd m
adamw + nesterov
Inner Optimizer Outer Optimizer
Figure E.1: Comparison of inner and outer optimizer with GPT-Neo Experiments on Cluster B (K=8,
H=64). 'sgd m² stands for SGD with Momentum
37

---

Published in Transactions on Machine Learning Research (09/2025)
E.3 Optimizer Selection for Convergence
Figure E.2 compares the time required for different outer oplimizers to reach the larget validation loss. The results show that the Nesterov optimizer achieves the shortest training time. In contrast, using SGD as the outer optimizer is significantly slower than using either Nesterov or SGD Momentum. Notably, when the outer optimizer SGD, our algorithm reduces to a randomized variant Local SGD.
sgd
sgd momentum
adamw
Opt: nesterov
Target Validation Loss: 1.31
Target Validation Loss: 4.11
Time Target (seconds)
21590.88 8s
21517.46
20000
15000
10000
5969
5977
5930.6s
5931.7s
5000
5000
mestrro
sge
rgd. momentumm
mestrre
mommentumm
Outer Optimizer
Outer Optimizer
Figure E.2: Comparison of inner and outer optimizer with GPT-Nco Experiments on Cluster B (K=8, H-64).
E.4 1) Selection
7) is a parameter that controls the update range in peaudo-syachronizatio,, and larger values can be inter- preted as stronger regularization. The experimental results showed that eta increases, the validation loss also deteriorates significantly, likely due excessive regularization, Additionally, it found that when eta is below 4, no substantial changes occur.
Validation Loss by n Values
1.133
Validation Loss
1.129
1.126
1.125
1.124 1.124 1.124 1.124 1.124
0.250
0.500
8
16
32
64
n (Coefficient of Pseudo Synchronization)
Figure E.3: Comparison of on GPT-Neo Experiments on Cluster B (K=8, H-64)
38

---

Published in Transactions on Machine Learning Research (09/2025)
E.5 p Selection
The parameter p controls the frequency of pseudo synchronization. When is too large, updates without gradients become more frequent, leading to insufficient progress in learning. However, since this skips forward and backward processes, the same number of stops is reached in a shorter amount of time. This intuitive trade-off aligns well with our experimental results.
Validation Loss by p Values
Training Time (hours) Values
1.16 10
1.16$
x 10
6,138
6.099
5.996
Training Time (hours)
5.793
Validation Loss
5.591
x015
5.362
5 10
1.139
1.14 x
1.133
x 1
1.126
1.127
1.124
1.125
10
3.869
0.1
p (Probability of Pseudo Synchronization)
(Probability of Pseudo Synchronization)
Figure E.4: Comparison of pfp/ GPT-Neo Experiments on Cluster (K-8, H-64)
39

---

Published in Transactions on Machine Learning Research (09/2025)
F Additional Study
F.1 CIFAR10 Experiments on Scaling Performance
In this section, we conduct distributed training experiments using multiple nodes on Cluster C, rather than the simulated environment described in the previous section. PALSGD, DiLoCo, and DDP were evaluated using the CIFAR-10 dataset on the VGG-16 architecture. Experiments were executed on Cluster with configurations of 2, 4 and 8 nodes corresponding to 4, and 32 GPUs, respectively.
For optimization, local models utilized SGD with momentum. The outer optimizer was Nesterov momentum. Workers synchronized parameters every 128 steps. The local SGD phase began after 2048 global iterations, except for the K-16, 32 setting, where it started after 1024, 512 global iterations.
A grid search determined key hyperparameters. Inner learning rates were selected from {0.025, 0.05, 0.075). Outer learning rates ranged across {0.2, 0.4, 0.8, 1.2, 1.6]. Algorithm-specific parameters included a gradient- drop probability selected from 0.02, 0.05, 0.1 and an eta coefficient chosen from {0.1, 0.25, 0.53.
Several parameters remained constant. The batch size per GPU was set at 256, with no learning rate decay applied throughout training. To ensure high computational efficiency, used the largest possible per-GPU batch size. The total number of training epochs fixed for all experiments. This selup means that as the number of GPUs increases, the global batch size grows proportionally, while the total number of training steps decreases.
Architecture: VGG16, Sync interval: 128
Architocture: VGG16, Sync Interval: 128
30P
DDP Biato
3Lodo
4x10--
1AL56
1M,300
1x-1--
Acouraty 1351
Total Runtiee (hours)
02.5
00.0
27.5
10
73.2
22.5
Nomber of GPUS
Number GPUS
Figure F.1: Comparison of K CIFAR10 Experiments on Cluster C (K=4, 8, 16, 32, H=128)
Our baseline, DDP, showed 4% drop in validation accuracy when scaled from 4 to 16 GPUs. The degra- dation was more severe at 32 GPUs, where accuracy fell by over 15%. This sharp decline on 32 GPUs is likely attributable to the large-batch problem on CIFAR-10. The global batch size becomes 8,192, which means an entire epoch over the 50,000-image dataset completes in just over six steps. Training time for all methods decreased linearly on log scale, indicating good system scalability. PALSGD and DiLoCo have nearly identical runtimes. Their advantage in training time becomes more significant when scaling from 16 to 32 GPUs. This result suggests that both methods effectively mitigate the communication bottleneck that emerges at larger scales. However, they also suffer from accuracy degradation at 32 GPUs. Despite this, both methods consistently achieve higher validation accuracy than DDP. More importantly, PALSGD demonstrates superior accuracy DiLoCo across all scales.
40


Reviews:
[{"review_id": "vhrWMCZxXg", "reviewer": "Reviewer_fWvt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. Reducing the synchronization bottleneck in distributed training for large models is an important problem.\n2. The approach is simple and straightforward to implement.\n3. The evaluation is performed on both image classification and language modeling tasks.\n4. Limitations of the method have been acknowledged.\n\n## Weaknesses\n1. The parts of PALSGD algorithm have not been justified well in the paper. In particular:\n    - For instance, why should there be a pseudo-synchronization step and not just constrain the optimization in INNEROPT to penalize distance from the global average?\n    - Why does the pseudo-synchronization step need to be probabilistic and not every X rounds?\n    - There's no experimental justification for this particular choice of these components. For instance, what happens if we do not perform synchronous ALL-Reduce ever, and only perform Asynchronous updates? Similarly, for the choice of probabilistic pseudo-synchronization step over a regularization term.\n2. The related work lacks comparison to recent works in distributed training [1,2,3,4]. The authors should defend how their work differs from these works or compare PALSGD against them experimentally.\n3. The experimental section is weak.\n    - The paper only compares against DDP and DiLoCo experimentally. There has been no experimental comparison against Local-SGD with comparable `H`, Asynchronous Local SGD [5], and DropCompute [1] at least?\n    - Alongside wall-clock time, it would have been nice to see other metrics like the number of gradient computation steps or communication rounds on the x-axis.\n    - The choice of `H` in DiLoCo being the same as `H` in PALSGD for the same experiment is not explained well. Is there a better H for DiLoCo?\n    - The algorithm introduces a new hyperparameter `p`. However, the choice of `p` in Section 7 is not clear. Section D.4 just shows that lower `p` leads to better validation loss but higher training time. So, the authors should clarify how `p` should be chosen/tuned?\n\n## References\n[1] Giladi, Niv, et al. \"DropCompute: simple and more robust distributed synchronous training via compute variance reduction.\" Advances in Neural Information Processing Systems 36 (2023): 48403-48416.\n\n[2] Tyurin, Alexander, et al. \"Shadowheart SGD: Distributed Asynchronous SGD with Optimal Time Complexity Under Arbitrary Computation and Communication Heterogeneity.\" The Thirty-eighth Annual Conference on Neural Information Processing Systems.\n\n[3] Bornstein, Marco, et al. \"SWIFT: Rapid Decentralized Federated Learning via Wait-Free Model Communication.\" The Eleventh International Conference on Learning Representations.\n\n[4] Biswas, Sayan, et al. \"Boosting asynchronous decentralized learning with model fragmentation.\" Proceedings of the ACM on Web Conference 2025. 2025.\n\n[5] Liu, Bo, et al. \"Asynchronous local-sgd training for language modeling.\" arXiv preprint arXiv:2401.09135 (2024).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents PALSGD, a Local-SGD variant that aims to address the communication bottleneck of Local SGD by reducing the number of all-reduce operations against the Distributed Data Parallel approach. This is achieved by replacing the usually synchronous all-reduce operation with a probabilistic synchronization with a global model. This probabilistic aspect makes the all-reduce operation pseudo-asynchronous. The system is studied both theoretically and empirically across various datasets and models, and is able to reach a target accuracy with less wall-clock time than the chosen baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Address the weaknesses mentioned above. I shall be happy to reconsider weaknesses given proper justification.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "75KJ6pfy4v", "reviewer": "Reviewer_JWgr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: A pseudo-asynchronous method is proposed. It is investigated from both perspectives of theory and experiments. \n\nWeakness: \n1. It is unclear how the proposed method can reduce communication frequency intuitively.  \n\n2. Why the global model is not immediately used as local model after synchronization? \n\n3. The theoretical analysis, although looks correct, has several concerns. i) The theoretical analysis has chosen H to be constant under discussion of Theorem 1, which is not considered communication-efficient. However, looks the analysis could allow $H = O(\\sqrt{\\mu T/K})$. ii) The analysis has transferred the dependence of H to p. Can the authors explain the motivation for this? iii) In  page 17, it is assumed that $p \\leq \\frac{1}{\\kappa H}$, which is a very small value considering both $\\kappa$ and $H$ are large. It in not consistent with Theorem 1 where $0< p \\leq 1/2$.\n\n4. Some important literature has been missed. For example, [1] analyzed asynchronous federated learning, and [2] has used variance reduction technique to reduce communication frequency in heterogeneous setting. Both should be discussed and compared to the proposed algorithm in experiments.\n\n[1] Yu, Hao, Sen Yang, and Shenghuo Zhu. \"Parallel restarted SGD with faster convergence and less communication: Demystifying why model averaging works for deep learning.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019.\n\n[2] Karimireddy, Sai Praneeth, et al. \"Scaffold: Stochastic controlled averaging for federated learning.\" International conference on machine learning. PMLR, 2020.\n\n5. The number of clients is small in experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission proposes a pseudo-asynchronous local SGD algorithm, in order to reduce the communication frequency. The global model after all-reduce is gradually incorporated into local model according to some probability at each iteration. Theoretical analysis of convergence is provided and experiments have been shown.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YOgvYlNheE", "reviewer": "Reviewer_oaZf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper address an important problem of communication bottleneck with large-scale training.\n2. The paper processes an extension of local-SGD to improve the total training time required to reach a given performance.\n3. The papers has exhaustive related work session.\n4. The experimental results show that the proposed methods has  6% speedup over DiLoCo (ImageNet use-case).\n5. Paper is well written and easy to follow.\n\nWeaknesses/questions:\n1. The paper motivates communication bottleneck for large-scale training but conducts experiments on 4-8 workers. \n2. Theoretical analysis assumes strongly convex functions.\n3. Section D.4 is not clear to me.  Can you please elaborate the setup used for both the figures?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. The paper introduces PALSGD, an extension of Local SGD that incorporates probabilistic pseudo-synchronization.\n2. The paper provided theoretical convergence bounds for a simplified version of PALSGD ie., µ-Strongly Convex case, the inner optimizer is standard SGD and the outer model is updated by taking the average across inner models.\n3. Experiments were conducted on two different datasets: 1) ImageNet-1K with ResNet-50, and 2) TinyStories using GPT-Neo1 with 8 million and 125 million parameters. ImageNet was trained with 4 workers where as TinyStories was trained with 4 to 8 workers. PALSGD is compared against DDP and DiLoCo in terms of wall clock time to reach the same performance (Accuracy or loss). The experiments show that PALSGD reach the same performance faster as compared to DDP and DiLoCo.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. How does PALSGD compare to Scaffnew[1]? Since Scaffnew claims better performance than local-SGD, it might be insightful to compare PALSGD with Scaffnew.\n2. The abstract highlights speedup with respect to DDP. However, the most competitive baseline in the current experiments is DiLoCo and therefore the speedup should be reported wrt DiLoCo (second best method). Is there a reason to pick DDP over DiLoCo for speedup computation?\n\nReference:\n1. Mishchenko, Konstantin, et al. \"Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally!.\" International Conference on Machine Learning. PMLR, 2022.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)