HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training
Authors: Geon-Woo Kim
Junbo Li
Shashidhar Gandham
Omar Baldonado
Adithya Gangidi
Pavan Balaji
Zhangyang Wang
Aditya Akella
Venue: ICML 2025 poster (2025)
Keywords: Hierarchical Geo-Distributed LLM Training
Asynchronous Optimization
Local SGD with Momentum
Primary area: optimization->large_scale_parallel_and_distributed
TL;DR: We propose a hierarchical asynchronous optimization method that accelerates geo-distributed LLM training by combining local and global model updates with theoretical guarantees, achieving faster convergence without sacrificing accuracy.

Abstract: Training large language models (LLMs) increasingly relies on geographically distributed accelerators, causing prohibitive communication costs across regions and uneven utilization of heterogeneous hardware. We propose HALoS, a hierarchical asynchronous optimization framework that tackles these issues by introducing local parameter servers (LPSs) within each region and a global parameter server (GPS) that merges updates across regions. This hierarchical design minimizes expensive inter-region communication, reduces straggler effects, and leverages fast intra-region links. We provide a rigorous convergence analysis for HALoS under non-convex objectives, including theoretical guarantees on the role of hierarchical momentum in asynchronous training. Empirically, HALoS attains up to 7.5× faster convergence than synchronous baselines in geo-distributed LLM training and improves upon existing asynchronous methods by up to 2.1×. Crucially, HALoS preserves the model quality of fully synchronous SGD—matching or exceeding accuracy on standard language modeling and downstream benchmarks—while substantially lowering total training time. These results demonstrate that hierarchical, server-side update accumulation and global model merging are powerful tools for scalable, efficient training of new-era LLMs in heterogeneous, geo-distributed environments.

HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training
Geon-Woo Kim
Junbo Li
Shashidhar Gandham 2
Omar Baldonado Adithya Gangidi 2
Pavan Balaji²
Zhangyang Wang'
Aditya Akella
Abstract
challenges. Today's de facto standard distributed opti- mization approach to pretrain LLMs is fully synchronous Stochastic Gradient Descent (SGD), where the massive amount of model states are synchronized across multiple accelerators (e.g.. GPUs and TPUs) after every training step. To overcome this communication overhead, practition- ers attempt to deploy homogeneous accelcrators in close proximity and connect them with highly optimized network infrastructure (Dubey et al.. 2024; Jiang et al., 2024). For example, to pretrain Llama-3 405B, 16K H100 GPUs are deployed within a single datacenter, each equipped with 400 Gbps network bandwidth and network connectivity with latency in the tens of microseconds (Dubey al., 2024).
Training large language models (LLMs) increas- ingly relies on geographically distributed accel- erators, causing prohibitive communication costs across regions and uneven utilization of hetero- geneous hardware. We propose HALoS, hier- archical asynchronous optimization framework that tackles these issues by introducing local pa- rameter servers (LPSs) within each region and global parameter server (GPS) that merges up- dates across regions. This hierarchical design minimizes expensive inter-region communication, reduces straggler effects, and leverages fast intra- region links. We provide rigorous convergence analysis for HALoS under non-convex objectives, including theoretical guarantees on the role of hierarchical momentum in asynchronous train- ing. Empirically, HALoS attains up 7.5x faster convergence than synchronous baselines in geo- distributed LLM training and improves upon ex- isting asynchronous methods by up to 2.1 x. Cru- cially, HALoS preserves the model quality of fully synchronous SGD matching or exceeding accu- racy on standard language modeling and down- stream Benchmarks-whie substantially lower- ing total training time. These results demonstrate that hierarchical, server-side update accumulation and global model merging are powerful tools for scalable, efficient training of new-era LLMs in heterogeneous, geo-distributed environments.
While highly optimized homogeneous single-datacenter setup is desirable, it faces significant scalability challenges due to operational constraints, such as energy consumption and power management, when deploying large numbers of accelerators one region (Gherghescu al., 2024). Simi- larly, companies and researchers relying on cloud providers often struggle allocate sufficient GPUs in single region due limited availability (Strati et al., 2024).
Consequently, there is growing need to utilize accelerators that are geo-distributed across multiple regions in training LLMs (Yuan et al., 2022; Tang et al., 2024; Gandhi et al., 2024; Jaghouar et al., 2024; Strati et al., 2024). This setup presents new challenges: (1) inter-region communication orders of magnitude slower, with link bandwidths typically ranging from 1 to 10 Gbps and network latency in milliscc- onds (Jaghouar et al., 2024; Gandhi et al., 2024; Yuan et al., 2022), and (2) accelerators are highly heterogeneous, as maintaining homogeneous hardware across multiple regions is impractical (Li et al., 2022; Mei et al., 2024).
1. Introduction
Local SGD, a widely studied communication-efiin efffccccnntii exten- sion of synchronous SGD, allows workers' to perform mul- tiple local model updates before global synchronization, bal- ancing computation and communication overhead (Zhang et al., 2015; Stich, 2018; Lin et al., 2018). Recent efforts have adapted this approach to address the challenges in
Large Language Models (LLMs) have rapidly revolution- ized diverse domains (Dubey al., 2024; Achiam et al., 2023), but their training presents significant computational
'The University of Texas at Austin Meta. Correspondence to: Aditya Akella <akella@ es. edu..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'In this paper, a worker represents a set of accelcrators func- tioning as data-parallel group, maintaining model replica.

---

Hierarchical Asynchronous Local SGD
geo-distributed LLM training. DiLoCo (Douillard et al., 2023) demonstrates empirically that carefully tuned mo- mentum acceleration (Sutskever et al., 2013) enables local SGD to amortize slow inter-region communication costs effectively. However, under heterogeneous accelerators, DiLoCo's strictly synchronous design often results in signif- icant resource underutilization due to the straggler problem.
Slow Inter-Region Communication
Fast Intra-Region Communication
Global Parameter Server
G
1
2
Updates Accumulation
2. Global Model Merging
Bg
Bo
BD
03
84
15
BB
87
8B
Region
Local Parameter Server
Asyne-Local-SGD (Liu et al., 2024) addresses this by incor- porating asynchrony, allowing faster workers to proceed in- dependently. It employs global server that asynchronously collects and applies gradients to update the global model, while workers periodically pull the latest parameters, com- pute gradients, and push them back (Langford et al., 2009; Dean et al., 2012; Li et al., 2014). Async- -oGl-SSDD en- hances convergence through momentum correction mech- anisms that mitigate the effects of stale gradients and the variability inherent in asynchronous training.
Workers
$
$
R2
Local Parameter Server
Figure I. Overview of HALoS. Local parameter servers leverage fast intra-region networks (red solid lines) to update their models with asynchronously computed gradients from workers. Accumu- lated updates are sent to the global server via slow inter-region networks (blue dotted lines), and the latest global model is merged back into the local models.
However, Asyne-Local-SGD encounters two major chal- lenges. First, slow communication between the global server and workers hinders convergence. Although asynchrony al- lows workers to operate independently and mitigates the straggler problem, regular communication is still required for pulling the latest models and pushing computed gradi- ents. The slow inter-region communication can significantly extend these times. While increasing the number of local up- dates helps amortize communication overhead, it increases the bias and variance of gradients in local SGD (Ortiz et al., 2021; Balles et al., 2024), degrading convergence efficiency and requiring more tokens to achieve the same model per- formance, offsetting the benefits of reduced communication overhead. Second, theoretical analysis and convergence guarantees for asynchronous training methods under practi- cal geo-distributed conditions remain underexplored. While Soyne-Local-SGD empirically demonstrates faster conver- gence compared to synchronous methods, the lack of a theoretical basis limits broader adoption in practice. Key aspects, such as the impacts of communication delays and staleness in gradients, data heterogeneity, and the role of momentum, remain poorly understood.
communication through asynchronous updates. In particu- lar, local servers continue updating their local models during communications with the global server, and merge the up- dated local model with the model pulled from the global server (we refer to this as "global model merging").
This novel hierarchical design of HALoS offers several benefits, ensuring efficiency and scalability, and achieving good model performance in geo-distributed training. First, the introduction of an additional level of hierarchy signifi- cantly lowers the communication stress on high-delay and bow-dandwidth network links, and mitigates their impact training performance. Second, local servers accumu- late updates and communicate with the global server optimized intervals, significantly reducing the overhead on the global server and ensuring its scalability. Third, by combining asynchronous updates and global model merg- ing, HALoS minimizes computation idle time (compared to Asyne-Local-SCii and prevents valuable updates from being discarded in the presence of the remnant high inter- region communication delays.
Our Approach. We present a new technique for geo- distributed training of LLMs that overcomes the above chal- lenges. It effectively controls the impact of slow inter-region communication central to the geo-distributed setup.
We provide formal algorithm description and rigorously prove its convergence for general non-convex loss functions, with a focus on the role of momentum at each hierarchy level and the impact of delays in hierarchical asynchronous designs. Our analysis offers insights into balancing local and global updates enhance convergence efficiency in challenging geo- reo-distributed settings.
We develop HALoS. hierarchical distributed optimization framework. As shown in Figure 1, HALoS deploys local parameter servers within each region that asynchronously update their local models using gradients computed by work- ers in the same region, leveraging fast intra-region commu- nication. Here, each worker performs multiple local updates before communicating with their local parameter server. At the higher tier, global parameter server coordinates train- ing progress across local servers, hiding slow inter-region
In sum, our contributions are as follows:
We propose HALOS, a novel hierarchical asyn- chronous framework that achieves scalable, efficient geo-distributed LLM training, enabling fully asyn- chronous communication at each level with server-side updates accumulation and global model merging.
2

---

Hierarchical Asynchronous Local SGD
We present tight convergence proof for HALoS with general non-convex loss functions, offering insights and theoretical guarantees on the complex dynamics of asynchronous training. To the best of our knowledge, this is the first convergence proof for hierarchical asyn- chronous distributed optimization with momentum.
settings, their performance under slow inter-region com- munications can be limited due the trade-off between training and communication efficiency (Balles et al., 2024; Ortiz et al., 2021), as we show in Section 3 (Figure 2).
Asynchronous Federated Learning. Federated Learning (FL) focuses on training a global model across resource- constrained mobile devices while preserving data pri- vacy (McMahan et al., 2017; Li et al., 2020). In FL, several works also utilize asynchronous training to achieve better time-to-loss performance by addressing device heterogene- ity and reducing synchronization delays (Xie et al., 2019; Chen et al., 2020; Nguyen et al., 2022). To enhance scala- bility, asynchronous hierarchical structures have been pro- posed, where multiple servers are coordinated by a global server (Wang & Wang. 2022; Mitra & Ulukus, 2023; Xie et al., 2024) or operate in decentralized manner (Sun et al., 2023; Zuo et al., 2024; Liang et al., 2024). However, these methods require workers to wait for global model broadcasts or for synchronizations of neighboring servers, resulting in significant computational inefficiencies with slow inter-region communication in geo-distributed envi- ronments. Moreover, the hierarchical asynchronous FL ap- proaches are primarily designed for now-performance de- vices, and their performance in large-scale language model training remains underexplored, particularly in integrating momentum- a critical component for effective LLM train- ing and providing its theoretical convergence analysis.
We demonstrate that HALoS achieves up to 7.5x x faster convergence than DiLoCo and up to 2.1x faster con- vergence than Asyne-Local-SGii in geo-distributed en- vironments. Compared to fully synchronous SGD, HA- LoS achieves 68.6x faster convergence and matches model performance in standard benchmarks. To fos- ter reproducibilityy we release our implementation at https://gitubb.coo/uunssaa
2. Related Work
Geo-distributed LLM Training. Recent research has fo- cused enabling LLM training in the geo-distributed en- vironment, where computational resources such as GPUs and TPUs are distributed across different geographical re- gions (Yuan et al., 2022; Strati et al., 2024; Gandhi et al., 2024; Tang et al., 2024). These approaches leverage various training parallelism strategies, including data parallelism, tensor parallelism, and pipeline parallelism (Narayanan et al., 2021), formulate training costs and minimize them by finding the most communication-efficiee and computa- tionally balanced partitioning schemes. For example, they observe that mapping pipeline parallelism, where relatively small amount of activations are transferred, to slow inter- region communication is beneficial. However, they either do not account for heterogeneous accelerator speeds in their optimization search space (Yuan et al., 2022; Strati al., 2024; Gandhi al., 2024), potentially leading to suboptimal resource allocation, or they rely on gradient and activation compression techniques (Tang et al., 2024) without provid- ing rigorous theoretical proofs or detailed analyses of their impact on training dynamics and convergence.
3. Motivation
We aim minimize a loss function F(.) for model O collaboratively trained by N workers distributed across mul- tiple geographical regions with heterogeneous operating speeds. The optimization objective is formulated as:
N
+, = arggmng N
E((O)
(1)
where F; represents the local loss function computed by worker using its assigned training data. The training data for each worker is assumed to be independently and identi- cally distributed (i.i.d.)². Figure 2 (a) illustrates an example of the ceo-distributed training environment with 4 regions and 16 heterogeneous workers.
Local SGD. Local SGD enables efficient distributed op- environments,
timization in communication-constaaaine where local models are synchronized infrequently after mul- tiple updates (Zhang et al., 2015; Stich, 2018; Lin et al., 2018; Yu et al., 2019; Wang et al.. 2019; Castiglia et al., 2021; Sun et al., 2024). These methods significantly re- duce communication overhead while achieving linear con- vergence speed-ups with more workers, matching the the- oretical rates of synchronous SGD for both convex and non-convex objectives (Koloskova et al., 2020; Khaled et al., 2020; Wang & Joshi, 2021). DiLoCo (Douillard et al., 2023) and Asyne-Local--oel (Liu et al., 2024) adapt local SGD for geo-distributed LLM training and show improved training efficiency over synchronous SGD with carefully selected and designed momentums. However, in geo-distributed
In this geo-distributed setup, DiLoCo (Section 2) faces sig- nificant challenges due to high synchronization costs due to slow inter-region communication bandwidth, high delays and the straggler problem. where the slowest worker bot- tlenecks the overall training progress. On the other hand, Asyne-Local-SGD, while designed to mitigate some of these
"We follow standard practice in assuming i.i.d. data for analyti- cal clarity, but HALoS does not rely on this assumption in practice. See Section 5.4 for further discussion and empirical results.
3

---

Hierarchical Asynchronous Local SGD
Region
Region
Training n Time (se0)
Syme SGD
Compute
Stal
Communkcatio
x Workers
0.5 Glps
x Workers
10a
DILoCo
100%
Asymo-Loca--GD
Gans
60%
0.2 Cbps
0.1 Glppa
Hedd
He44
Gope
10*
4 x Workers
4 Workers
Hit8
He22
Hed4
20%
0.1 Gops
10
12
14
16
18
20
Region
Region 4
Trained Tokens (B)
Sync SGD
DiLoCo Async-Local-SGD
(b)
(c)
Figure 2. (a) A geo-distributed training environment with 4 regions with 16 heterogenous workers where we adapt recent measurement of region communication bandwidths from (Jaghouar et al., 2024). (b) Comparison of training time (in log scale) and token consumption for Pythia-70N model (Biderman et al., 2023) in (a) to reach the same validation loss under various training methods ("H²" denotes the number of local updates in workers.) (c) Runtime breakdown analysis of workers for synchronous SGD, DiLoCo (H=32), and Asyme-Local-SGD (H=32) in (b). We detail the experimental setup in Appendix A.
issues, encounters prolonged wait times when workers com- municate with a global parameter server, which is likely to be deployed in a different geographical region.
the frequency of communication with the GPS, lowering inter-region communication costs while leveraging fast intra- region links for local updates. By accumulating updates at the LPS level, HALoS effectively relieves the scalability bottleneck at the GPS, allowing for more efficient handling of large-scale distributed training.
Although both methods reduce communication overhead by increasing the number of local updates (H) performed by workers, this incurs trade-off: a higher H increases vari- ance in training, requiring more tokens to achieve the same convergence. As shown in Figure 2 (b), while infrequent communication (via larger H) accelerates training initially, it ultimately demands more tokens for the same validation loss and makes the convergence slower with too large H. Both methods achieve their fastest convergence H 32.
4.1. Algorithm
Specifically, HALoS operates on three levels (Algorithm I):
1. Global Parameter Server: The GPS receives accumu- lated updates A from an LPS (line 2), applies it to the global model O using a momentum- based update rule (line 3), then sends the model back to the LPS (line 4).
Figure 2 (c) illustrates the overheads associated with geo- distributed training methods that achieve the fastest times to convergence. For synchronous SGD, communication overhead dominates, consuming 93.7% of worker runtime due to frequent synchronization of large model states across regions. DiLoCo alleviates some cost by enabling multiple local updates, but its strictly synchronous design results in the slowest worker stalling the progress of others, leading to 47.5% of worker runtime being stalled. Lastly, Asyne- Local-SGD mitigates long stall times but workers still spend 29.2% of their runtime waiting for communication with the global parameter server, either to pull the latest global model or to push the computed local gradients.
2. Local Parameter Servers: Each LPS maintains a local model 9,, receives gradients 5 from workers (line 9), and applies them using a momentum-based update rule (line 11). After K updates (line 13), it sends the accu- mulated updates A to the GPS (line 14) and continues updating its local model with the workers. Upon re- ceiving an updated global model O (line 17), the LPS merges it with its local model using a weighted average controlled by a (line 19). Both worker gradients (6) and the latest global model (θ;) are placed in the same queue, and updates and merging occur in that order.
3. Workers: Each worker performs H/ local gradient de- scent steps on its assigned data (line 22) before sending the resulting gradient ő to its LPS (line 25).
4. Hierarchical Asynchronous Local SGD
To overcome the fundamental impact of slow communi- cation in geo-distributed training, we propose HALOS, a novel hierarchical distributed optimization framework that fully asynchronous at each level of its hierarchy. HAL.S employs local parameter servers (LPS) within each geo- graphical region, as illustrated in Figure 1. These servers aggregate updates from local workers and accumulate specified number of updates before communicating with the global parameter server (GPS). This approach reduces
The hyperparameters K and a are central to HALoS's abil- ity to balance communication efficiency and convergence quality. First, К regulates the frequency of LPS-to-GPS communication, effectively reducing the computational load on the GPS while controlling the variance of gradients from geographically diverse LPSs. Second, determines the ex- tent to which the global model influences the local model during the merging process. This parameter is crucial for accounting for regional learning progress in asynchronous

---

Hierarchical Asynchronous Local SGD
Algorithm 1 HALOS Update Rules.
4.2. Convergence Analysis
Require: Initial model Op. total training iterations T', learning rates fg. Mj. Ye. momentum coefficients Sg. 8t. model merging weight o, number of updates in local server K, and number of local steps H.
Next, we present the theoretical guarantee for Algorithm 1, accompanied by detailed proof and formal notation description in Appendix E. Our analysis begins by intro- ducing several standard assumptions commonly used in asynchronous distributed optimization. The first two as- sumptions correspond to typical SGD settings. The third is standard in distributed optimization, and the last two are frequently adopted in asynchronous optimization, e.g., (Nguyen et al., 2022).
Global Parameter Server (GPS):
1: for i 4- 1 to T do
2: Receive A from an LPS.
3: θ; t Modeeeeppaaleee--...ee A, 1/g. Bg)
4: Send θ to the LPS.
5: end for
Local Parameter Server (LPS):
Assumption 4.1 (L-smoothness)) Local loss functions are L-smooth, i.e., there exists L > 0, such that for any worker i, [[V V Fv(y)) S L||x y/1-
6: Initialize 0o <- Oo, & 4- 0, Mast <- O.
7: Schedule all workers lo.
8: repeat
9: if receive from a worker then
Assumption 4.2 (Unbiased gradient and bounded variance).
10:
+ +++111111
11:
Oe < ModelUppatt((0--1/ mi ())
The local stochastic gradients are unbiased estimator of full-batch gradients with bounded variance.
12:
Schedule the worker tc-
13:
if tlast = K then
Assumption 4.3 (Bounded heterogeneity). At every itera- tion, the variance of the aggregated gradients sent from an LPS to the GPS is bounded. Formally, we have o², where Vi(@) represents the aggregated gradients of the model & on LPS L.
14:
Send A < 0.- Oraa to GPS.
15: end if
16:
end if
17: if receive from GPS then
E V (+) [F)))]|
18:
tlast +
19:
O: + (1 a) + a
20:
end if
Assumption 4,4 (Bounded gradients). For any worker i, the local gradients satisfy || [(())] S G.
21:
until training completed
Worker 0,,0 0: scheduled:
Assumption 4.5 (Bounded staleness). The norm of the model difference between local training and global updates, caused by asynchronous training, is bounded by D, for GPS and Dj for LPS, respectively.
22: for 1 to H do
23: Pe, +- 0r,3-1 - nww To VP
24: end for
25: Send 6 <- 01.H 0c,0 to the LPS
Theorem 4.6. Denote Bg and 33 to be the global and local momentum hyperparameters and the learning rate satisfies
communication, maintaining an optimal interplay between local and global training dynamics.
T/o 2 Ttt 2 Mmin for all L. Under certain assumptions, the following holds:
min E VFF(0)) 4(F(On)) ( -((.)) (1+ 1 B,
Comparison with Existing Hierarchical Design. Ex- isting asynchronous hierarchical structures for federated learning often struggle with efficient, scalable optimiza- tion in geo-distributed settings. For instance, methods like FedAH (Wang& & Wang, 2022) and Timely-AHFL (Mitra & Ulukus, 2023) impose semi-synehronous constraints, re- quiring regional local servers to wait for fully transmit- ted global models, causing high delays. In contrast, our method enables fully asynchronous communication, allow- ing LPSs and colocated workers to update local models continuously, improving efficiency. We empirically show that HALoS's design outperforms existing methods in geo- distributed LLM training (Section 5.2).
ISST
mm
+ 70
6T.1o0
12L10
7m 33
(1- (2-
Go²
+ 122 + 1.2D9
31)(1 (g)
F(Oo) - =())))
mm - -By)T
Go²
+ +70
+y 0 ( DJ)
(2)
3:)(1 - -B,)
Asychronous spdatt delay
Hierorchicul structure
On the other hand, HGA-FL (Xie et al., 2024) also hides LPS-to-GPS communication and aims to utilize worker re- sources efficiently but requires the GPS to maintain multiple model replicas, increasing memory and computational de- mands. Our fully asynchronous design, leveraging side update accumulation and global model merging, eliminates these inefficiencies.
where 7 = O
L Piom 3 92)
As with previous distributed optimization algorithms (Xie et al., 2019; Wang et al., 2020; Liet al., 2023), our results in- clude a diminishing term (1//T) associated with the initial bias and a constant term specific to our hierarchical asyn- chronous optimization setting. The first component of the
5

---

Hierarchical Asynchronous Local SGD
constant term arises from the variance of global and local updates in the hierarchical structure, while the second is due to the delays inherent in the asynchronous setting. Our set- ting is the first attempt to tackle a hierarchical asynchronous setting with momentum updates, so it is not directly com- parable with previous works. By setting D2 = Dj = 0, we bound from (Liu et al., 2020) in terms of Br. with an additional G term arising from asyn- chronous optimization, Moreover, removing the momentum updates restores the result in (Nguyen et al., 2022). Overall, our derived bound is tight considering both perspectives.
5. Experiments
In this section, we evaluate the performance of HALoS in training LLMs within diostiiiteteed environments through custom execution trace-driven simulations. Our evaluation covers four aspects: (I) end-to-end pretraining performance compared to baseline methods (Section 5.1); (2) the im- pact of individual techniques and hyperparameters of HA- LoS (Section 5.2); (3) generalization across diverse cluster configurations and model families (Section 5.3); and (4) robustness to data heterogeneity (Section 5.4).
recover the momentum-
Evaluation Task. We measure wall-clock times and num- ber of tokens required to train LLMs to achieve specific model performance (i.e., test accuracy or validation loss) in geo-distributed setup. To ensure reproducible evaluations, we train the Pythia models (Biderman et al., 2023) using the deduplicated Pile dataset (Gao et al., 2020). Pythia pro- vides the order of trained tokens and fine-grained snaphots of models during full training across various LLM sizes. For all evaluations, we the same learning rates and batch sizes used in training the models and adhere to stan- dard optimization practices in LLM training: learning rates follow a linear warmup phase and decay to 10% of their maximum value using a cosine schedule, automatic mixed precision training with float16 (Micikevicius et al., 2017; PyTorch. 2025), AdamW optimizer with 0.1 weight de- cay (Loshchilov, 2017), and gradient clipping set to 1.0.
Influence of momentums Bg and St. To achieve effective convergence in training, HALoS incorporates momentum- based updates for both GPS and LPS, with coefficients 3g and Bt. Our theoretical bounds suggest that these two momentum terms have different magnitudes of influ- ence. Specifically, for local momentum 3r, the bound fol- lows ( (-m) in line with prior momentum- based studies, e.g., (Liu et al., 2020), which allows us to use typical values such 0.9 (Douillard et al., 2023; Liu et al., 2024).
In contrast, for global momentum Bg. the bound is more complex as ( (3(011111) Two insights emerge:
1. The factor (1 10--)))) B,y) appears with higher powers than usual momentum bound, preventing 3g from being set too high (e.g., 0.9).
Experimental Setup. We simulate geo-distributed envi- ronment consisting of four geographical regions, each host- ing four workers, as illustrated in Figure 2 (a). To model realistic wall-clock training times, we utilize recent measure- ments of inter-region network bandwidths (Jaghouar et al., 2024). The workers are assigned operating speeds uniformly at random within range of 1 to 10, reflecting the recent rapid advancements in accelerator computation capabilities, where higher values indicate faster workers. We profile the execution of training computations on H100 GPUs and use the profiled time to represent the computation time of the fastest worker. For other workers, their computation times are scaled based on their relative speeds. Our simulator esti- mates computation time based on the profiled runtime and communication time using widely adopted analytical mod- els (Valiant, 1990; Thakur et al., 2005). Then, it determines the order of updates for each local and global model and executes model updates based on this order. We describe further details about the experimental setup in Appendix A.
2. The additional B3 factor rules out very small values.
Consequently, By should be chosen as a trade-off value, such as 0.5, which approximately minimizes ...- Remark- ably, experiments confirm that these choices of (3g, 8i) are indeed optimal (Section 5.2).
Influence of heterogeneity 6² and staleness Dg, Di. Tak- ing both heterogeneity and staleness into account provides additional insight into the choice of momentum. Delays and staleness introduce bias in gradient estimation, and the the- ory indicates that stale information can inflate effective noise and reduce the benefit of higher-level momentum. When workers in different LPSs operate on highly heterogeneous data, the aggregated global gradient (e) may exhibit conflicting directions. In such cases, GPS-level momentum risks "averaging" non-stationary and contradictory signals, potentially exacerbating the mismatch and slowing conver- gence. Consequently, it is preferable not to use or use a smaller 3, By contrast, momentum at the LPS level works on more homogeneous data and tends to be more stable. We provide empirical evidence of this analysis Section 5.4.
Baselines. We compare HALoS with two baseline methods: DiLoCo (Douillard et al., 2023) and Asyne-Local-SGii (Liu et al., 2024). Asyne-Local-SGD adjusts the number of lo- cal steps based on the inverse of worker speeds, reducing gradient staleness by allowing workers to complete com- putations at similar times. To evaluate the impact of this dynamic adjustment on synchronous methods, we introduce
6

---

Hierarchical Asynchronous Local SGD
Time-to-loss
Tokens-to-boe
Mueto Ciads Cymtidd
Table 1. Training performance of Pythia-70M, Pythia- Pythia-160M, and Pythia-410M under various training methods. The table shows normalized simulated wall-clock times (relative to HALoS) and the number of tokens used to reach the same validation loss. The corresponding validation loss curves are detailed in Appendix B.
blaco Dynulp
922
Loyne d00
Asyle
Validaton
Validation
3.0
2.
28
00
2.5
5.0
75
10.6
16.0
Simuiation Wall-Clock (sec) x10
# Trained Tokens
PYTHIA-70M TOKENS
PYTHIA-16NM
PYTHLA-41OM
TIME
TIME
TOKENS
TIME
TOKENS
model for different local SGD-based methods to reach the same validation loss: (left) loss vs. simulation wall-clock time and (right) loss vs. number of trained tokens.
Figure 3. Validation loss curves of the Pythia-1600
DILoCo
7.2
11.88
7.1
12.9B
12.3B
DILoCo+ DYNUPD
4.0
10.28
3.8
11.58
3.9
10.8B
ASYNC LOCAL-
1.8
14.08
1.9
16
2.1
16.
HALOSS (OURS)
1.0
12.3B
1.0
13.2B
1.0
11.8B
MMLU (5 shots)
Hellaswag (10 shots)
"DiLoCoo + DynUpd," which integrates dynamic local up- dates into DiLoCo. Additionally, ensure a fair comparison with Asyne-Local-SCCe HALoS incorporates both dynamic local updates and the delayed Nesterov optimizer used in
fccurary.. (ha)
Aocuraoy (%%)
Syme StD
Syne SGD
Asyne-Local-SGii
HALLS (Our Method)
HALoB (Our Methhl
100
150
500
150
20
Hyperparameters. We conduct a hyperparameter sweep for HALoS and each baseline method using the smallest model, Pythia-70M, and use the pest-performin hyperparameters for all subsequent evaluations. Details of the candidate hyperparameters and the selected values for each method are provided in Appendix D.
Trained Takens (B)
8 Trained Tokens (B)
LAMBADA
WinoGrande
Recurae mary (16)
Acourssc (%)
& & *6 '6
Buno
Syne 30D HALo8 (Our Menoa
(Our
150
250
Trained Takens (B)
Trained Tokens (B)
5.1. End-to-end Training Performance
Figure 4. Test accuracies of the Pythia- model on the MMLU, Hellaswag, LAMBADA, and WinoGrande benchmarks, plotted against the number of trained tokens. The results compare syn- chronous SGD and HALoS. The shaded regions represent the standard errors in benchmark evaluations. We use 5-shot evalua- tions for MMLU and 10-shot evaluations for Hellaswag. Results for four additional benchmarks (SciQ. PIQA, ARC-Challenge, and ARC-Easy) are provided in Appendix B.
In this section, we evaluate the training efficiency of HA- LoS compared to local SGD-based methods and assess the generalization capabilities of LLMs trained with HALoS.
Compared to Local SGDebased Methods. We evaluate the end-to-end training performance of HALoS against existing local SGD-based methods. Specifically, we train the Pythia- 160M model from scratch on the first 12.9B tokens from the Pile dataset (with 0.6B tokens used for the initial warmup), measure the final validation loss, and then train the same initial model using different methods until each reaches the same validation loss. Following the original Pythia model configuration, we use a global batch size of 1,024 where each of the 16 workers uses a mini batch size of 64 with a sequence length of 2,048 per local step.
Compared to Asyne-Local-SGD HALoS overcomes limita- tions related to communication delays. Async-Local-SoC workers are periodically idle due to the need to fetch updated global models and upload gradients, which increases the number of required local steps and adversely affects conver- gence. In contrast, HALoS delivers 1.9x x faster convergence and reduces token consumption by 18.0%.
Figure 3 illustrates the validation loss curves as a function of training time (left) and the number of trained tokens (right) for each method. HALoS achieves 7.1x x faster training speed compared to DiLoCo, which suffers from straggler issues and high synchronization delays. When dynamic local steps are applied to DiLoCo (DiLoCo + DynUpd), performance improves by mitigating the straggler problem through adaptive step adjustments based on worker speeds. This modification also reduces token requirements by 10.8% due to increased synchronization frequency. Nevertheless, HALoS outperforms DiLoCo + DynUpd by 3.8x, benefiting from its hierarchical asynchronous design that effectively conceals synchronization delays.
We extend the analysis across models of varying sizes, as detailed in Table 1. The advantages of HALoS persist across scales. For example, when training the Pythia-410M model, HALoS achieves a 7.5 x speedup over DiLoCo, 3.9x over DiLoCo + DynUpd, and 2.1 x over Async-Local-SGI.
Benchmarks Performance. We further evaluate the test accuracies of HALoS on four representative benchmarks: MMLU (Hendrycks et al., 2020), Hellaswag (Zellers et al., 2019), LAMBADA (Paperno et al., 2016). and Wino- Grande (Sakaguchi et al., 2021). We fully train the Pythia- 160M model with HALoS until it matches or exceeds the performance of the publicly available model checkpoints
7

---

Hierarchical Asynchronous Local SGD
Updates Accumulation
Global Model Merging
Valication Loss
3.00
Validatiion Loss
3.6
Table 2. Training performance of HALoS with ! Global and Lo-
cal Momentums (8) 0.9, 8 0.5, K 1, a 1.0), Global Model Merging (a = 0.25), and ① Local Server Updates Accumulation (K 32), compared to FedAH (Wang & Wang, 2022). All methods are trained until they reach the same validation loss. The training times are normalized to HALoS
3.00
3,00
300
40
A Acoumulations
Merging Weight
Global Server Momentum
Local Server Momentum
Valddsion Log5
3.05
Validatian Loss
3.25
METHOD
TIME-TO-LOSS
TOKENS-TO-LOC
3.00
3.20
FEDAH
3.3
38,1B
3.90
19.15
HALOS
1.6
19.38
3.00
8:10
HALOS
1.3
15.6B
3.0
300
HALOS
1.0
12.3B
3.0
Momentum Coafficient &
Momentum Costlicient Bi
and local models during the merging process. As shown in Figure 5 (top right), the validation loss is minimized when a 0.25. When a is too low (approaching 0), the local models become overly isolated, failing to incorporate global learning progress effectively. This isolation leads to diver- gence across regions and hinders convergence, resulting in suboptimal performance. Conversely, when a is too high (close to 1), updates made during communication between LPSs and GPS are not effectively reflected after the merging process, leading to degraded convergence behavior.
Figure 5. Hyperparameter sensitivity of HALoS: validation losses of Pythia-70M after training 12.9B tokens, varying (top left) up- dates accumulation K, (top right) merging weight a, (bottom left) global server momentum Sy. and (bottom right) local server mo- mentum Bt- The best loss is achieved with K = 32, a = 0.25,
3, 0.5, and Bi 0.9.
trained by fully synchronous SGD on one epoch of the Pile dataset (207B tokens). In practice, this requires training on 223B tokens ensure broad coverage and stable con- vergence. As shown in Figure 4, the HALoS- based model consistently meets or surpasses the baseline's accuracy. HA- LoS achieves the performance with a 68.6x speedup in training time by effectively addressing the high communica- tion delays and straggler problem of synchronous SGD in geo-distributed environments.
Impact of Bg and 31. The training efficiency of HALoS is significantly influenced by the global momentum 3, and the local momentum 33. As shown in Figure 5 (bottom left and right), the optimal values are 3 = 0.5 and 3 = 0.9, which align with our theoretical analysis in Section 4.2. The lower optimal value for Bg supports the theoretical insight that global momentum needs to be more conservative to mitigate the accumulation of confticting signals from diverse regions. In contrast, the higher optimal value for St leverages the relative homogeneity of updates within local servers, allowing for more aggressive use of momentum.
5.2. Ablation Studies
We evaluate the impact of different values of hyperparam- eters used in HALoS: updates accumulation (K), global model merging weight (a), and server momentums (3) and $g). We train Pythia-70M model on 12.9B tokens and report the final validation losses using different values of hyperpa- rameters.
Contributions of Key Techniques in HALoS. Table 2 illus- trates the impact of HALoS's key techniques training ef- ficiency compared to FedAH (Wang & Wang, 2022), which applies staleness scaling with a polynomial form (3 0.5) following prior work (Xie et al., 2019; Nguyen et al., 2022). First, integrating local and global momentums (0) accel- erates convergence, achieving the same validation loss as FedAH with 2.0x fewer tokens. Second, applying global model merging prevents wasted local computations and better balances local and global training progress, further reducing training time and token requirements by 1.2x. Fi- nally, accumulating updates at local servers improves training efficiency by lowering the variance of accumulated updates across regions, achieving an overall 3.3x faster convergence than FedAH.
Impact of K. As we describe in Section 4.1, K plays a pivotal role in achieving scalable structure by managing the communication frequency between LPSs and GPS. It also reduces the overhead associated with global model updates in GPS. As shown in Figure 5 (top left), HALoS demonstrates robust performance across various K values, with the best performance observed at K = 32, where each LPS has 4 workers. Notably, we observe that high K not only reduces the overhead but also mitigates the variance of gradients from different LPSs This occurs because the accumulation of a minimum number of updates ensures stability, irrespective of the differences in LPS-to- GPS communication bandwidths.
5.3. Generalization Studies
Impact of a. The global model merging weight a is a critical parameter that balances the contributions of global
In this section, we assess HALoS 's generalization by mea- suring its performance across heterogeneous worker distri-
8

---

Hierarchical Asynchronous Local SGD
Table 3. Relative training time on heterogencous clusters (2, 4, 4, 6 workers per region) when training Pythia-70M HALoS (Naive Grouping) assigns one LPS to each region, whereas HALoS (Con- sistent Grouping) deploys 1, 2, 2, and 3 LPSs so that each LPS serves exactly two workers, Using the same hyperparameters tuned for the homogencous (4, 4, 4, 4 workers) setting, HALoS (Naive Grouping) diverges.
Table 4. Training performance of HAL.8 and Asyne-L Loca--Lo on the non-ii.d. Shakespeare dataset (Caldas et al.. 2018) when training the 6-layer Llama model. The table reports character-Ievel test accuracy and normalized training time.
METHOD
ACCURACY
TIMF-TO-CCCURACY
ASYNG-LOCAL-SGE
49.2%
1.6
HALOS (w/ GLOBAL MOMENTUM)
46.9%
1.0
HIAOOS (W/O GLOBAL MOMENTUM)
50.0%
1.0
METHOO
TIME-TO-LOSS
LSYNC--LCAL-SGD HALOS (NAIVE GROUPING)
1.7
model replica. In Appendix C.3, we further compare HA-
HALOS (CONSISTENT GROUPING)
LoS on training Pythia- 70M against three fully synchronous MP baselines: data parallelism (DP), DP + pipeline par- allelism (PP), and a heterogeneity-awee aware DP + PP variant. HALoS converges 8.26x faster than the strongest baseline (heterogeneity- aware DP + PP), demonstrating the effective- ness of its hierarchical asynchronous design in avoiding the high synchronization costs (e.g.. pipeline stalls) that hamper synchronous MP under heterogeneous accelerator speeds.
butions, different inter- and intra-region bandwidths, and multiple LLM families under the same hyperparameter set- ting. We also compare HALoS with fully synchronous model-parallel baselines in geo-distributed environments.
Heterogeneous Clusters. We evaluate HALoS under het- erogencous worker distributions (2,4,4.6 workers per re- gion), comparing against our strongest baseline, Asyne- Local-SGD. As shown in Table 3, when workers were naively grouped into one LPS per region, resulting in signif- icantly varying workers per LPS (2 to 6), training diverges due to large discrepancies in update progresses across LPSs. To address this, we employ consistent grouping strategy ensuring each LPS manages exactly two workers, resulting in 1,2,2,3 LPSs per region. Using this strategy, HALoS effi- ciently coordinates learning across heterogeneouss clusters, achieving 1.7x faster convergence than Asyne-Local-SGD. Here, we trained the Pythia-70M, model as described in Sec- tion 5.1. For consistent grouping, hyperparameters remained unchanged, except for adjustments to local updates accumu- lation (K) to and local momentum update delay (d)) to 4, reflecting the smaller number of workers per LPS.
5.4. Robustness to Data Heterogeneity
We evaluate HALoS on the standard non-i.i.d split of the Shakespeare dataset, where each character's lines are as- signed to a distinct worker (Caldas et al., 2018; McMahan et al., 2017: Li et al., 2023). As shown in Table 4, HA- LoS without global momentum reaches 50.0% test accuracy 1.6x faster than Asyne-Local-See
With global momen- tum, HALoS maintains the speed-up but reduces accuracy by 3.1 percentage points, which confirms our analysis in Sec- tion 4.2 that averaging conflicting directions into momentum can undermine convergence. These results highlight HA- Lo$ 's robustness to data heterogeneity and its suitability for geo-distributed LLM training under non-i.i.d. workloads.
Experiment Details: We train 6-layer Llama-style model (hidden size 128) for next-character prediction (among 79 unique characters) given the previous 80. The training set contains 3.15 M samples; the same 16 workers in Ap- pendix A process 64-sample batches each. Evaluation uses 0.52 M held-out samples. Training runs for one epoch with a peak local learning rate of 0.01. HALoS employs its default hyperparameters, while Esyne-Local-See adjusts local steps (H) from 32 to 16 to avoid divergence.
Different Network Bandwidths. We evaluate the impact of network bandwidths on HALoS by comparing its perfor- mance against local SGD methods in Appendix C. Even with 2x faster inter- and intra-region network bandwidths, HALOS consistently outperforms the baselines, achieving 5.7x and 1.5x x faster convergence than DiLoCo and Asyne- Local-SGI, respectively.
Different Model Families. We evaluate the generality of HALoS across three popular LLM families, Llama (Dubey et al., 2024), Qwen (Yang et al., 2025), and Pythia (Bider- man et al., 2023), using exactly the same hyper- parameters tuned on Pythia (Appendix C.2). HALoS consistently out- performs the strongest baseline, Asyne-Local-SGD, achiev- ing 2.1 x faster convergence on Llama-70M and 2.3x Qwen-70M without any additional tuning, underscoring its generality to architectural choices of LLMSs.
6. Conclusion
We present HALoS, a novel hierarchical asynchronous opti- mization framework for geo-distributed LLM training. HA- LoS integrates server-side update accumulation and global model merging to mitigate communication delays and re- source heterogeneity, supported by tight convergence proof and insights into the effects of momentum, delays, and staleness in the hierarchical asynchronous structure. Empir- ically, we demonstrate that HALoS achieves significantly faster convergence than the baselines methods.
Comparison with Model Parallelism Techniques. HALoS integrates model parallelism (MP) seamlessly by treating each worker as a set of accelerators that jointly train a single

---

Hierarchical Asynchronous Local SGD
Acknowledgements
Chen, Y., Ning, Y., Slawski, M., and Rangwala, H. Asyn- chronous online federated learning for edge devices with non-iid data. In 2020 IEEE International Conference on Big Data (Big Data), PP. 15-24, IEEE, 2020.
Kim and Akella are supported by NSF grants CNS-2105890 and CNS-2232135. and by gifts from Meta and Cisco Re- search. Z. Wang part supported by NSF Award 2145346 (CAREER) and 2212176. This research has been supported by computing support the Vista GPU Cluster through the Center for Generative AI (CGAI) and the Texas Advanced Computing Center (TACC) at the University of Texas at Austin.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 00.80050077, 2018
Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., et al. Large scale distributed deep networks. Advances in neural information processing systems, 25, 2012.
Impact Statement
This paper presents a new distributed optimization method to improve the efficiency of LLM training using geographi- cally distributed resources. The proposed hierarchical asyn- chronous framework, HALoS, addresses slow inter-region communication delays and hardware heterogeneity to im- prove training efficiency. The main social and ethical im- plications of this work relate enabling more resource- efficient and accessible LLM training scale. While these advances have broad applicability, we do not foresee any direct societal harm arising from this study.
Douillard, A., Feng, Q. Rusu, A. A., Chhaparia, R.
Donchev, Y., Kuncoro, A., Ranzato, M., Szlam, A., and Shen, J. Diloco: Distributed cowmmummuination training of language models. arXiv preprint arXiv:231 1.00201 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Daale, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178, 2024.
References
Gandhi, R., Tandon, K., Bhattacherjee, D., Padmanabhan, V. N.. et al. Improving training time and gpu utilization in geo-distributed language model training. arXiv preprint arXhv:2411.14456 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08772 087744 2023.
Gao, L.. Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 2020.
Balles, L., Archambeau, C., et al. On the choice of learning rate for local sgd. Transactions on Machine Learning Research, 2024.
Gherghescu, A. M., Bădoiu, V.-A., Agache, A., Dumitru, M.-V., Vasilescu, I., Mantu, R., and Raiciu, C. I've got 99 problems but flops ain't 't one. Proceedings of the 23rd ACM Workshop on Hot Topics Networks, pp. 195-204, 2024.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O' Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for ana- lyzing large language models across training and scaling. In International Conference on Machine Learning, pp. PMLR, 2023.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint WXXi:20099033000 2020.
Bisk, Y., Zellers, R., Bras, R. I.,, Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. In Fourth-Fourth AAAI Conference on Artificial Intelligence, 2020.
Jaghouar, S., Ong. J. M., and Hagemann, J. Opendiloco: An open-source framework for globally distributed Iow-communication training. arXiv:2407.07852 2024.
Caldas, S., Duddu, S. M. K., Wu, P., Li, T., Konečný, J., McMahan, H. B., Smith, V., and Talwalkar, A. Leaf: A benchmark for federated settings. arXiv preprint arXiv: 18/2.01097 2018
arXiv preprint
Jiang, Z., Lin, H., Zhong, Y., Huang, Q., Chen, Y., Zhang, Z., Peng. Y., Li, X., Xie, C., Nong, S., et al. MegaS- cale: Scaling large language model training to more than 10,000 gpus. In 2 USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pp. 745- 760, 2024.
Castiglia, T., Das. A., and Patterson, S. Multi-level local sgd: Distributed sgd for heterogeneous hierarchical networks. In International Conference on Learning Representations, 2021.
10

---

Hierarchical Asynchronous Local SGD
Khaled, A., Mishchenko, K., and Richtárik, P. Tighter theory for local sgd identical and heterogeneous data. In International Conference Artificial Intelligence and Statistics, pp. 4519 4519-4529, PMLR, 2020.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and
y Arcas, B. A. Communication-effiiee learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Koloskova, A., Loizou. N., Boreiri, S., Jaggi, M., and Stich, S. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning. pp. 5381-5393 PMLR, 2020.
Mei, Y., Zhuang, Y., Miao, X., Yang. J., Jia, Z., and Vinayak, R. Helix: Distributed serving of large language models via max-flow on heterogeneous gpus. arXiv preprint arXiv:2406 24X:::2400.01666 2024.
Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al. Mixed precision training. arXiv preprint 1710003774 2017.
Langford, J., Smola, A., and Zinkevich, M. Slow learners are fast. arXiv preprint arXiv:0911.049/ 2009.
Li, D., Wang, H., Xing, E., and Zhang. H. Amp: Automati- cally finding model parallel strategies with heterogeneity awareness. Advances Neural Information Processing Systems, 35:6630-666 6639, 2022.
Mitra, P. and Ulukus, S. Timely asynchronous hierarchical federated learning: Age of convergence. In 2023 I International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt), pp. 509-515. IEEE, 2023.
Li, J., Li, A., Tian, C., Ho, Q.. Xing, E., and Wang, H. Fed- nar: Federated optimization with normalized annealing regularization. Advances in Nearal Information Process- ing Systems, 36:74753-7447 2023.
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Pat- wary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. Efficient large-scale language model training on gpu clusters using megatron- Im. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-15, 2021.
Li, M., Andersen, D. G., Park, W., Smola, A. J., Ahmed, A., Josifovski, V., Long, J., Shekita, E. J., and Su, B.-Y. Scaling distributed machine learning with the parameter server. In 11th USENIX Symposium on operating sys- tems design and implementation (OSDI 14), pp. 583-598, 2014.
Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M., Malek, M., and Huba, D. Federated learning with buffered asynchronous aggregation. In International Con- ference on Artificial Intelligence and Statistics, pp. 3581- 3607. PMLR, 2022.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020.
Ortiz, J. J. G., Frankle, J., Rabbat, M., Morcos, A., and Ballas, N. Trade-offs of local sgd at scale: An empirical study. arXiv preprint arXiv:2110.0813: 2021.
Liang, X., Tang, J., and Quek, T. Q. Large-scale decentral- ized asynchronous federated edge learning with device heterogeneity. In ICC 2024-IFFFF International Confer- ence on Communications, pp. 4566-4571 IEEE, 2024.
Paperno, D.,, Kruszewski, G., Lazaridou, A., Pham, N. Q..
Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Anmual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 1525- 1534, 2016
Lin, T., Stich, S. U., Patel, K. K., and Jaggi, M. Don't use local sgd. arXiv preprint
use large arXiv: 1808.888877 072 17, 2018
Liu, B., Chhaparia, R., Douillard, A., Kale, S., Rusu, A. A., Shen, J., Szlam, A., and Ranzato, M. Asynchronous local-sgd training for language modeling. arXiv preprint 24KV:22001.01133 2024.
Py PyTohh. Automatic mixed precision package torch.amp. https: https:s/pptorrhhhh org/ docs clocs/staalle.. sttss/aable/aaaaeea amp. html, 2025.
Liu, Y., Gao, Y., and Yin, W. An improved analysis of stochastic gradient descent with momentum. Advances in Neural Information Processing Systems, 33:18261- 18271, 2020.
Sakaguchi, K., Bras, R. L.,, Bhagavatula, C., and Choi, Y.
Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106 2021.
Loshchilov, L. Decoupled weight decay regularization. arXiv preprint Xrviv 100 2017.
Stich, S. U. Local sgd converges fast and communicates little. arXiv preprint 1 1805.09767, 2018

---

Hierarchical Asynchronous Local SGD
Strati, F., Elvinger, P., Kerimoglu, T., and Klimovic, A. MI training with cloud gpu shortages: Is cross-region the answer? In Proceedings of the 4th Workshop Machine Learning and Systems, pp. 107-116, 2024.
W17-4413. URL httpss//aclanntoolooy.ooom W17-4413/.
Xie, C., Koyejo, S., and Gupta, I. Asynchronous federated optimization. arXiv preprint urXiv:/903.033 2019.
Sun, W., Qin, Z., Sun, W., Li. S., Li, D., Shen, X., Qiao, Y., and Zhong, Y. Co2: Efficient distributed training with full communication-coomuttaai arXiv:2401.16263 2024.
Xie, W., Xiong, R., and Luo, J. Hierarchical global asyn- chronous federated learning across multi-center. In The 16th Asian Conference on Machine Learning (Conference Track), 2024.
overlap. arXiv preprint
Sun, Y., Shao, J., Mao, Y., Wang. J. H., and Zhang, J. Semi- decentralized federated edge learning with data and de- vice heterogeneity. IEEE Transactions on Network and 2023.
Yang, A., Li, A., Yang, B., Zhang. B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, Lv, C.,, et al. Qwen3 technical report. arXiv preprint S.lv:25055093800 2025.
Service Management,
Yu, H., Yang. S., and Zhu, S. Parallel restarted sgd with faster convergence and less communication: Demystify- ing why model averaging works for deep learning. In Proceedings of the AAAI conference on artificial intelli- gence, volume 33. pp. 5693-5700, 2019.
Sutskever, I,, Martens, J., Dahl, G., and Hinton, On the importance of initialization and momentum in deep learn- ing. In International conference machine learning, pp- 1149. PMLR, 2013.
Tang, 7., Kang, X., Yin, Y., Pan, X., Wang. Y., He, X., Wang, Q., Zeng, R., Zhao, K., Shi, S., al. FusionIlm: A decen- tralized Ilm training system on geo-distributed gpus with adaptive compression. arXiv preprini 127077 2024.
Yuan, B., He, Y., Davis, J., Zhang, T., Dao, T., Chen, B., Liang. P. S., Re, C., and Zhang, C. Decentralized training foundation models in heterogeneous environments. Advances in Neural Information Processing Systems, 35: 25464-2544 2022.
Thakur, R., Rabenseifner, R., and Gropp. W. Optimization of collective communication operations in mpich. The International Journal of High Performance Computing Applications, 19(1):99-66, 2005.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sen- tence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Valiant, L. G. A bridging model for parallel computation. Communications of the ACM, 33(8):103-111, 1990.
Zhang, S., Choromanska, A. E., and LeCun, Y. Deep learn- ing with elastic averaging sgd. Advances in neural infor- mation processing systems, 28, 2015.
Wang, J. and Joshi, G. Cooperative sgd: A unified frame- work for the design and analysis local-update sgd algo- rithms. Journal of Machine Learning Research, 22(213):
Zuo, Y., Cox, B., Chen, L. Y., and Decouchant, J. Asynchronous multi-server federated learning for geo- distributed clients. arXiv preprint arXiv:2406.01439, 2024.
1-50, 2021.
Wang, J., Tantia, V., Ballas, N., and Rabbat, M. Slowmo: Improving communicaiion-cffiicii distributed sgd with slow momentum. arXiv preprint arXiv:1910.000 00643,, 2019.
Wang, J., Liu, Q. Liang, H., Joshi, G., and Poor, H. V. Tack- ling the objective inconsistency problem in heterogeneous federated optimization. Advances neural information processing systems, 13:7611-7622 2020.
Wang, X. and Wang, Y. Asynchronous hierarchical feder- ated learning. arXiv preprint wrxiv:2206.00054, 2022.
Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. In Derczynski, I..,
Xu, W., Ritter, A., and Baldwin, T. (eds.), Proceedings of the 3rd Workshop on Noisy User-generated Text, pp- 94-106, Copenhagen, Denmark, September 2017. Asso- ciation for Computational Linguistics. doi: 10. 18653/v1/
12

---

Hierarchical Asynchronous Local SGD
A. Details on Experimental Setup
Region
Resource Specification
GPS Gg
Real GPUs
GPS
>100 Gbps
100 Gbps
LPS-2
LPP-1
Worker-3 (Speed=10)
LPS-1 19
Worker-1 (Speed=8)
Worker-4 (Speed=5)
Worker-2 (Spead=2)
W-1
Pull
Comp
|Push Compusation
Region 1
0.5 Gbps
Region 2
W-2
Push
Region 2
LPS-2
Profiled Mini-batch Computation Time
Model Specification
Comm
W-3
Push
Simulation Inputs
Trace Generation
Trace Execution
Figure 6. Simulation approach for evaluating geo-distributed training methods. The simulator accepts: (i) resource specifications (c.g.. network bandwidths and the placements the Global Parameter Server (GPS), Local Parameter Servers (LPSs), and workers along with their relative operating speeds), (ii) model specifications, and (iii) profiled per-step computation time on real GPUs. Using these inputs, it determines the computation and communication events under each training method, calculates the duration of each event, then generates a trace capturing the sequence of model updates (and their dependencies) in GPS and LPSs. Finally, the simulator executes the trace on actual GPUs, respecting the recorded update ordering and dependencies. We release our code publicly at https://gthhh..com//// wwwp:tsslll//aaaaaaa.ooaoo
R-1
R-2
R-3
R-4
REGIONS
WORKER SPEEDS
MODEL
TIME (MS)
R-1
100.0
0.537
0.935
0.202
R-2
0.537
100.0
0.386
0.117
R-1
10.0
9.1
3.8
2.6
PYTHIA-70M
238.4 .4
R-3
0.935
0.386
100.0
0.127
R-2
9.4
8.0
6.3
5.8
PYTHIA-160M
623.0 3
R-4
0.202
0.117
0.127
100.0
R-3
9.9
5.7
2.1
1.5
PYTHIA-410M
1589.7
R-4
9.1
8.7
5.8
1.2
Table 7. Profiled computation times of different models. Each entry is the du- ration for one training step (forward, back-
Table 5. Inter- and intra-region communication bandwidths (Gbps). Four regions used in the Table 6. Relative speeds of the 16 work- evaluation, following measurements in (Jaghouar ers. Speeds are drawn uniformly in the ward, and parameter update) al., 2024) for inter-region bandwidths (0.117 range [1.0. 10.0]. reflecting heterogeneous H100 GPU with a mini- batch size of 64, 0.935 Gbps). The intra-region bandwidth is set to operating speeds of workers. 100.0 Gbps.
single
measured on AWS P6 EC2 instances.
This section explains the methodology used to evaluate the performance of various training approaches under geo-distributed environments. custom trace-driven simulator is developed to capture the effects of network latency, bandwidth constraints, and heterogeneous accelerator speeds. Figure illustrates the overall simulation pipeline.
The simulator takes as input the resource specifications for each region, including both inter- and intra-region bandwidths, well the placement of the Global Parameter Server (GPS), Local Parameter Servers (LPSs), and workers, whose speeds vary from 1.0 to 10.0. By default, we place the GPS in the first region (R-1) and use the network bandwidths in Table 5. Table 6 summarizes the relative speeds of 16 workers. Additionally, the simulator takes as input a model configuration along with a profiled computation time measured on real GPUs for one training step. then uses this profiled time as the simulated computation time for the fastest worker. Table shows these measured times.
With the given inputs and a specified training method, the simulator generates computation and communication events over time, along with the dependencies between model versions, and calculates the duration of each event. For point-to- point communications, such as those used in pulling local models by workers, the simulator uses the well-known a-3 model (Valiant, 1990), given by
T T22--comm = a + 3C,
where a denotes the propagation delay, 3 is the inverse of network bandwidth, and C' is the data size. For collective communications, such as the all-reduce operation in DiLoCo, the simulator assumes a widely used fin-basedd implementation. Specifically, it searches for the ring that maximizes the communication bandwidth of the slowest link and calculates the
13

---

Hierarchical Asynchronous Local SGD
communication time using the standard analytical model (Thakur al., 2005):
Tall reduce = (N - C)C
N
where N is the number of participating workers, B is the slowest bandwidth, and C' is the data size.
To account for heterogeneous worker speeds, the simulator calculates per-worker computation time by scaling the profiled computation time. Given T prodited as the time measured for the fastest worker with speed Stustest the time to perform H local steps on a worker with speed S is computed as:
S
T compute H T profiled
Slower workers thus incur proportionally longer computation times, while faster ones complete local steps more quickly.
Once all communication and computation times have been determined, the simulator generates final trace of model update events at each LPS and the GPS. The order of updates is strictly enforced so that the correct model version is referenced at every update. By replaying this trace on real GPUs, the simulator provides an accurate depiction of how different distributed optimization algorithms perform when subject to realistic network latencies and resource heterogeneity.
B. Extended Experimental Results
Time-to-loss
Time-to-lose
Time-to-lass
Validtion Loes
34
MluCC Dyetlp
Validation Loes 12
Diado Dynl.pd
Di
Validation Loss
BluCe
DyeUpp
0.00
0.25
0.0
0.75
1.00
1.25
1:50
1.75
0.0
022
0'4
64
0.8
1.0
Simulation Wall- Clack (sec)
Simulation Wall-Clock (sec)
x10
Simulation Wall-Clock (sec)
Tokens-to-loss
Takens-to-loss
Tokens-to-loss
3.4
Validatoo Loss
bia tyellpd
Diacs Dynk.pd
Validation Loss
bioco Dyellpd
200
Locdl trob
500
32-
10.55 /Our Matho
Validation
Fil.S 10ur Meilodd
3..
1AA (Our Netho
2.a
8 Trained Tokens (3)
Trained Tokens
Trained Tokens
(a) Pythia-70M
(b) Pythia- 160M
Pythia-160M
(c) Pythia-410M
Figure 7. Validation loss curves for Pythia-70M, Pythia- 160M, and Pythia-410M models trained using different methods from Table 1.
MMLU (5 shots)
Hellaswag (10 shots)
LAMBADA
WinoGrande
20
Acciracy (96)
Rocuraoy (%)
Accuracy (%)
Accuracy (%)
do
Re
so
Sync SGD
Syne SGD
10
SyD
Syn SG3
HALOS (Dur Mechol
HALOS (Dur Mothoed
HALCS you Mathed
KALLS yOu Method)
150
90
200
190
a00
160 Trained Tokens (B)
Trainad Tokens (B)
# Trained Tolkens (B)
Trained Tokens (B)
SciQ
PIQA
Arc Challenge
Arc- Easy
24
5
Accuracy 1%6)
Accuracy (%)]
Accuracy (%))
Accuracy (%)
t
67.6
20
25.
HALIB (Dur Mcchoo
Sync 5GD
625
HALoS (Our Method)
Sync SGD
Byne SGD HALE your Mathedi
Syn 5G3 HALLS yOu Mathoal
150 Trained Tokens (B)
250
193 Trained Tokens (B)
Trained Tokens
Trained Tokens (B)
Figure& 8, Extended benchmark performance results corresponding Figure 4 for Pythia 160M trained with fully synchronous SGD and HALOS. At the bottom, we additionally includes results for the SciQ, PIQA, Are-Challenge, and Arc-Easy benchmarks.
14

---

Hierarchical Asynchronous Local SGD
We present the validation loss curves for all evaluated models from Table 1 in Figure 7. Additionally, extended benchmark performance results, including four additional benchmarks (SciQ (Welbl et al., 2017), PIQA (Bisk et al., 2020), Are- Challenge, and Arc-Easy (Clark et al., 2018)), are shown in Figure 8.
C. Additional Experimental Results
C.1. Impact of Network Bandwidths
1x NETWORK BANDWIDTHS
2x NETWORK BANDWIDTHS
TIME
TOKENS
TIME
TOKENS
DILOCo
7.2
11.8B
5.7
11.8B
DILOCo + DYNUPD
10.2B
2.3
10.2B
ASYNC LOCAL-SGD
1.8
14.0B
1.5
14.8B
HALOS (OURS)
1.0
12.3B
1.0
13.2B
Table 8. Comparison of training performance for the Pythia-70M model using different methods under the default (1 x) and network bandwidths.
Time-to-loss
Time-to-lass
Validation Losa
3.4
Dl:Co
DioCo + Dyrlpd
Validatioo Loss
DlaCs Dyniio
BlaCs
Looa 800
3.2
HALLS (Our Melhon)
3.2
HAL08 (Our Mehhod
0.03
0.55
0:75
0.0
0.2
0.4
0.6
0.6
12
144
Simulation Wall-Clock (sec)
K10
Simulation Wall-Clock (sec)
x101
Tokens-to-loss
Tokens-to-loss
Validation Loss
Di
Validdetion Loss
800
3.0
Trained Tokens (B)
Trained Tokens (B)
(a) 1x Network Bandwidths
(b) 2x Network Bandwidths
Figure 9. Validation loss curves for training the Pythia-70M model with different methods
under (a) the default (1x) network bandwidths and (b) network bandwidths. These results
correspond to those presented in Table 8.
We evaluate the impact of varying network bandwidths on the training performance of HALoS and three baseline methods: Specifically, we train the Pythia- 70M model using the same config-
DiLoCo, DiLoCo + DynUpd, and Asyne-Local-Soe uration detailed in Section 5. comparing the default (1x) network bandwidths with 2x faster inter- and intra-machine bandwidths. As shown in Table 8, even when the network bandwidth is doubled, HALoS continues outperform the baselines, providing up 5.7x, 2.3x, and 1.5x faster convergence compared to DiLoCo, DiLoCo DynUpd, and Async-Local-SGD, respectively. Figure 9 illustrates the validation loss curves under both bandwidth settings.
C.2. Generalization to Different LLM Families
MODEL
TRAINING SPEEDUP OF HALOS
LLAMA-70M QWEN-70M
2.1x
2.3x
PYTHIA-70M
1.8x
Table Training speedups of HALoS compared Asyne-Local-SGD for different LLM familics.
15

---

Hierarchical Asynchronous Local SGD
To demonstrate the robustness and practical applicability of HALoS across diverse model architectures, we extended our evaluation beyond the Pythia models to include two additional widely-used LLM families-LLlama (Dubey et al., 2024) and Qwen (Yang et al., 2025).
As shown in Table 9, HALoS consistently outperforms the strongest baseline (Asyne-Local -SDD) across all tested LLM ar- chitectures. Importantly, the hyperparameters initially optimized for Mithia-70M-withoee any additional tuning-exhibited strong generalization, achieving even greater relative improvements (up to 2.3x with the Qwen-70M model).
We assessed whether the optimal hyperparameters. identified from our theoretical analysis and validated on the Pythia-70M model, could effectively generalize to these models. To ensure fair comparison, selected Llama and Qwen models closely matching Pythia-70M in size, using the same number of layers and hidden dimensions, and conducted the same experiments described in Section 5.1.
C.3. Comparison with Synchronous Model Parallelism Techniques
METHOD
RELATIVE TIME-TO-LOSS
DP
85. 12
DP+PP
8.34
HETERO- AWARE DP+PP
8.26
HALOS
1.00
Table 10. Relative training time to reach the same vali-
dation loss for various synchronous model parallelism techniques and HALoS.
In HALoS, each worker is Sata-parallel group of accelerators, each maintaining a full model replica, enabling seamless in- tegration of Model Parallelism (MP) techniques, including Tensor Parallelism (TP) and Pipeline Parallelism (PP) (Narayanan et al., 2021). Below, we further compare HALoS with strictly synchronous MP methods.
We evaluate relative convergence times across three methods: synchronous DP (DP), synchronous DP with PP (DP+PP), and heterogeneity-awann version of DP+PP (Hetero-Aware DP+PP). As shown in Table 10, HALoS achieves 8.26x faster convergence compared the strongest baseline, Hetero-Aware DP+PP. PP allows relatively small activations transferred cross-region and improves convergence speed compared to pure DP (85.12x - 8.34x). However, PP inherently suffers from computational inefficiency from imbalanced pipeline stages (i.e., pipeline bubbles) due to heterogeneous accelerator speeds-4e even with heterogeneity-aware workload partitioning. In contrast, HALoS effectively mitigates slow inter-region communications and heterogeneous accelerator speeds, achieving superior performance.
We trained Pythia-70M as described in Section 5.1. DP+PP method used a DP degree of 4 and a PP degree of 4, placing pipeline stages across distinct regions. Hetero-Aware DP+PP employed heterogeneity-awan partitioning and a simulated annealing- based heuristic for placement from (Li al., 2022).
D. Hyperparameters
In this section, we detail the procedure for deciding the best performing hyperparameters for HALoS and the baseline methods. As described in Section 5, we conduct hyperparameter sweeps using the smallest model, Pythia-70M. Specifically, for each method, the model is pretrained for x 1024 steps, using a batch size of 1,024 and a sequence length of 2,048. This corresponds to processing 12.9B tokens, including an initial 300-step warmup phase. In HALoS and Async-Local-SGii each of the 16 workers uses 1022/11 16 64 sequences for each local step. The hyperparameter set yielding the lowest validation loss is selected as the optimal configuration.
Table 11 presents the candidate hyperparameters and highlights the best-performing ones. For HALoS, we employ the delayed Nesterov momentum (Liu et al., 2024) to ensure a fair comparison with Asyne-Local-SGD. The hyperparameter search for delayed Nesterov updates is conducted using identical candidate sets. For DiLoCo, we observe that the optimal hyperparameters align with those reported in the original paper (Douillard et al., 2023), specifically a learning rate of 0.7 and momentum coefficient of 0.9.
16

---

Hierarchical Asynchronous Local SGD
Table 11. The best performing hyperparameters for different training methods. 'Delayed Nesterov momentum optimization (Liu al., 2024) effectively divides the learning rate by the delay interval. We search for learning rales on scaled ones, as this approach shows more stable comparisons across different momentum update delays.
HALOS
ASYNC-LOCAL-SGD
DILOCo
Global Learning Rate" (gg/
0.03, 0.05, 0.1, 0.15, 0.2. 0.25
0.03, 0.05, 0.1, 0.15, 0.2, 0.25
0.1, 0.3, 0.5, 0.7, 0.9
Global Nesterov Momentum (9g)
0.1, 0.3, 0.5, 0.7, 0.9
0.1, 0.3, 0.5, 0.7, 0.2
0.1, 0.3, 0.7. 0.9
Global Momentum Update Delay (dg)
2. 8, 16, 32, 64
2, 4, 8, 16, 32, 64
Local Learning Rate (n//d)
0.03, 0.05, 0.1, 0.15, 0.2. 0.25
Local Nesterov Momentum (3:)
0.1, 0.3, 0.3, 0.5, 0.7, 0.9
Local Momentum Update Delay (d))
4, 8, 16, 32, 64
Local Updates Accumulation (K)
4, 8, 16, 32, 64
Global Model Merging Weight (a)
0.0, 0.25, 0.5, 0.75, 1.0
Number of Local Updates (H)
8. 16, 32, 64
8. 16, 32, 64
8, 16, 32, 64
E. Proof of Theorem 4.6
For clarity, we restate the problem and assumptions. We solve the following distributed optimization problem:
N
O. arg minl θ
R((O),
using Algorithm 1. We have the following assumptions:
Assumption E.1 (L-smoothnesss) Local loss functions are L-smooth, i.e., there exists L > 0, such that for any worker i, [[V[(()) (x(il S L|| yyl-
Assumption E.2 (Unbiased gradient and bounded variance). The local stochastic gradients are unbiased estimator of full-batch gradients with bounded variance.
Assumption E.3 (Bounded heterogeneity). At every iteration, the variance of the aggregated gradients sent from LPS the GPS is bounded. Formally, we have ()() - VF())] s o², where Vi(@) represents the aggregated gradients of the model θ on LPS I.
Assumption E.4 (Bounded gradients). For any worker i, the local gradients satisfy ||V < G.
Assumption E.5 (Bounded staleness). The norm of the model difference between local training and global updates, caused by asynchronous training, is bounded by Dg for GPS and D, for LPS, respectively.
In the above assumptions, consider both bounded gradients and bounded staleness. These conditions can be ensured in practice by employing standard stable training techniques, such as gradient clipping. We first clarify the momentum-based update rule of both LPS and GPS. Denote (0,,,) to be the aggregated gradient in LPS I at time t, and Vy(O,) to be the aggregated gradient in the GPS at time f. We denote two operators [g(l, i) and gl) for timestamp conversion between GPS and LPS. For LPS, we have the following update rule:
m1,+1 = 3.mi, + (i(O,.).
01.+11 = (1-a)0t, + a Th - (11--8)vii (O,,) + Bemre+1)
And for GPS, we have:
(O.) 1
Σ (11 Bi) (1)0)))) ((0))
M 1=1
Mg.1+1 = .y.... 1 Vg(O.),
O = +, - th [11
mg
17

---

Hierarchical Asynchronous Local SGD
By the L-smoothness assumption, have
Luñ B[ - F(())) S -V + (3 + 2
E ( + B.
(3)
We bound the two terms in the right side.
To bound the second term, bound the variance of aggregated gradients on GPS, and the variance of global momentum respectively. For gradients, we have
V 9 (+t) - VF(O:))1
M
E
-1- V + Simi.gut - VF(O.)]
M
2(1 - Bi)
2
<
E V VF(O:)|1 +
E ml,g((()) (F
M
M
- M
(0² + + +12D))
+
E
3 V (8 (010)) -FF())))))
M
1=0
0²
1
< (0² + D2 1.2 122
G.
1 31
1 B1
And for the momentum, we have
(0,) 2
E
Mg.
Bg
E
V
(
-
Σ
8 V (0,)
1't+1
2G
G2G
3G
<
+
<
1
82
B1
1
Bi
8
Bi)
(1 (1) B1)
where the last inequality is because for any and l2.
2
(V, V,
(0,))1
1
Σ [a1 Bi)
(i)(V((O))))))) - v.(Oro(((z))) + 9e (mz,g((1)) m..
(4)
M
2G
$ 2G
31 Σ
B 1-
(5)
1=0
Now we can bound the second term (3). Denote
1 Bg (O,)) + Rg
mg.t
We have
32
(,,))) + 8.mg, (1+ 1- B
+ + -V(())))V())) + +,,
and
32
E (0,)) + +: m. 12 <3
3
3
1
B2
+ 3.9
(1 11- 31)(1 - 52)
2
+ 122 + 1.2D3)
8g
18

---

Hierarchical Asynchronous Local SGD
3² 1-
Denote 7/ = 1 +
For the first term in (3), we have
VF(O.)" + ()
[ + (ToAe + (8g
2 2 (7o +3)IF 1
2 (1gl14111 + R
2 2 (19- +)II= ( + (/)) (a-
Go²
(1- (1) (1- -83)
+ D2 + D2)
-83)
Substituting the two terms back in (3) gives:
G
-(F((++))-F(()))) F(0.))
tt 2
(7g -) + (1, +B9)
+ D3 + L3D3)
2
(1 1-) (g4
L
GØ²
D2 9 + L2D ]
+
2 n
3+2E|VF(0)))++
+ +389)
(1 Br)(1 - -8g
3
Linng
ig
2
Th (7g
(1g - 89) B|
2
2
L
Go²
+
(7g
+ (1) +
3 (3 +38)))
+ 1.2 + 1.22))
2
2
(1 Bi)(1 - (89)
7o 22 6/
Setting the range of mn satisfies 1m S 7t S 7/ S
the coefficient satisfies:
3 2
2- Tt (7g
7t (7g - (9) S 4
Tm (Tg- k2)
2
4
Rearranging the above inequality, and taking summation for 0 s t $ T 1 we have:
T
Tm 4 (7o-
13
|FOO) - 7 ((((O))FF(0)) F(("))
-
T
+ (7g + (8) + L
T0
nh + +395)) ) (a-
Go²
+ 1L2 + +12D}
2
2
B:) 339)
With
- By +
33
7g -
0g
82
- Bg
g
3g + 32
2 +
9
1 3g' g
2
2
+ 9
2 2 +
(1 - "g)
and moving the left coefficients to the right side, we simplify the second term:
110
(7g
(22)
+
L 2 " (31 3,5%))
7g
2
9
3
B2
1
2 +
+ 6L
2+
83
1 1 3
(1 - }2
6Lmu
<
3+ + 12LTTo +
33
(1-3,)
This finalize the proof.
19


Reviews:
[{"review_id": "GY20gyteNh", "reviewer": "Reviewer_eTdm", "summary": "The paper presents HALoS, an optimization framework designed to enhance cross-region training of large language models (LLMs). To address the challenges of communication costs and imbalanced hardware utilization, HALoS employs a hierarchical architecture with local parameter servers within each region and a global parameter server that aggregates updates across regions. By prioritizing fast intra-region communication and minimizing inter-region synchronization, the framework reduces latency, mitigates straggler effects, and optimizes resource efficiency. HALoS enables asynchronous updates at both local and global scales, facilitating overlap of computation and communication. The authors provide theoretical convergence guarantees for non-convex objectives, analyzing how hierarchical momentum influences training stability in asynchronous settings. Empirical evaluations demonstrate HALoS’s superiority.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# [W1] HALoS without i.i.d. assumption\nWe appreciate the reviewer’s insightful question regarding the i.i.d. assumption. While our theoretical analysis in Section 4.2 follows standard practice by assuming i.i.d. data for analytical clarity, **HALoS does not rely on this assumption in practice.** To demonstrate this, we evaluate HALoS under both i.i.d. and non-i.i.d. data distributions.\n\nFor the non-i.i.d. setting, we use the Shakespeare dataset [2], assigning each worker distinct speaking roles (e.g., Juliet, Romeo)—a widely adopted setup in federated learning (e.g., [1], [2], [3]). The i.i.d. setting is created by randomly shuffling data across workers. After training on the same amount of data, we report accuracy and relative training time below (see “Experiment Details” at the end for additional information):\n\n|Method|Test Accuracy|Relative Training Time|\n|-|-|-|\n|Async-Local-SGD|49.2%|1.6|\n|HALoS (w/ Global Momentum)|46.9%|1.0|\n|HALoS (w/o Global Momentum)|**50.0%**|**1.0**|\n\nEven under non-i.i.d. partitions, **HALoS (w/o Global Momentum) achieves 0.8% higher accuracy and 1.6× faster convergence** than our strongest baseline, Async-Local-SGD. As discussed in Section 4.2, global momentum may slow convergence when worker updates conflict due to data heterogeneity, while local momentum remains effective. This allows HALoS to mitigate inter-region communication delays and converge efficiently.\n\nIn the i.i.d. setting, HALoS achieves the same accuracy (50.4%) as Async-Local-SGD but with 1.6× faster training. These results highlight **HALoS’s robustness across both i.i.d. and non-i.i.d. settings,** demonstrating practical effectiveness beyond theoretical assumptions.\n\nThe slight accuracy drop under non-i.i.d. data (50.0% vs. 50.4%) is consistent with prior observations in many hierarchical or asynchronous methods, which are *partially robust* to data heterogeneity. To further improve non-i.i.d. performance, we plan to explore FL techniques such as gradient clipping and adaptive regularization ([3]).\n\nTheoretically, **HALoS can be readily extended to non-i.i.d. settings**. Existing literature on local or federated SGD often provides additional bounds involving the divergence across local distributions. In HALoS, we can introduce an extra term in the bound related to the data heterogeneity assumption when bounding the variance of local gradients, similar to prior work ([4], [5]). This would allow our analysis to formally reflect the effect of data heterogeneity.\n\nWe thank the reviewer again for prompting this valuable discussion. We will include both the empirical results and an extension of our theoretical analysis to the non-i.i.d. case in the final version.\n\n**Experiment Details:** We train a 6-layer Llama-style model (128 hidden dim) to predict the next character (among 79 unique characters) given the previous 80. We train on 3,145,728 samples with a batch size of 64 per worker using 16 workers in Appendix A. Evaluation uses 524,288 separate test samples. We use a max local learning rate of 0.01 and train for one epoch. For HALoS, we use the same hyperparameters in our paper. For Async-Local-SGD, we found that 32 local steps ($H$) caused divergence and adjusted it to $H=16$.\n\n[1] Brendan McMahan et al. Communication-Efficient Learning of Deep Networks from Decentralized Data. AISTATS 2017.\n\n[2] Sebastian Caldas et al. LEAF: A Benchmark for Federated Settings. arXiv preprint arXiv:1812.01097, 2018.\n\n[3] Junbo Li et al., FedNar: Federated Optimization with Normalized Annealing Regularization, NeurIPS 2023.\n\n[4] Xiang Li et al., On the Convergence of FedAvg on Non-IID Data, ICLR 2020.\n\n[5] Tian Li et al., Federated Optimization in Heterogeneous Networks, MLSys 2020.\n\n---\n# [W2] Computation in Global Parameter Server (GPS)\nWe acknowledge that minimizing computation in the Global Parameter Server (GPS) is crucial for scalable geo-distributed training, and this goal fundamentally guided our design of HALoS. Our method significantly reduces GPS computation by employing local server-side update accumulation: each local parameter server (LPS) aggregates multiple local updates before synchronizing with the GPS. This design reduces both the frequency of inter-region communication and the number of updates processed by the GPS—preventing it from becoming a bottleneck.\n\n|Method|# Global Updates|Time-to-Loss|Tokens-to-Loss|\n|-|-|-|-|\n|FedAH|5857|3.3|38.1B|\n|HALoS (Ours)|478|1.0|12.3B|\n\nAs shown in the above table (extending Table 2 in the paper), when training Pythia-70M, **HALoS reduces GPS updates by 12.3x** compared to FedAH, our baseline hierarchical asynchronous training algorithm. This reduction comes from two key factors: (i) 32 local update accumulations per LPS before GPS communication, and (ii) 3.1× fewer total tokens needed to reach the same loss due to HALoS’s improved efficiency. Together, these design choices substantially reduce GPS computation and ensure HALoS achieves scalable, efficient coordination.", "claims_and_evidence": "HALoS could greatly improve training speed under cross-region scenarios, and it ensures convergence. Both experiment results and proofs in the appendix have verified the claim.", "methods_and_evaluation_criteria": "The methodology is okay with a good motivation and experiment-verified results.\nThe work uses common LLM benchmarks. There is no problems in terms of benchmarks.", "theoretical_claims": "The convergence analysis is supported by a proof om the supplementary materials.", "experimental_designs_or_analyses": "Experiment designs are generally okay under the simulation regime. Randomness is introduced in terms of worker speed, which reflect real scenarios. But I think real experiment demos rather than simulations could be carried out to directly demonstrate the efficacy of the method.", "relation_to_broader_scientific_literature": "The work is contributive to the broader literature in that it has a hierarchical architecture: global & local. It also demonstrates good efficiency.", "essential_references_not_discussed": "Not at all.", "other_strengths_and_weaknesses": "Strengths:\n1. Practicalness of the proposed method; good simulations that reflect reality.\n2. Theoretical proofs on convergence.\n3. Scalability across various model sizes and network bandwidth conditions.\n\nWeaknesses: \n1. The proposed method assumes i.i.d. data distribution across all workers. \n2. Global parameter server may undergo huge computation burden, impacting efficiency.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "How would the method perform under non iid? Would the method work without the iid assumption?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kAbF122Jtf", "reviewer": "Reviewer_Lzk8", "summary": "This paper presents HALoS, a hierarchical asynchronous optimization framework for training large language models (LLMs) across geographically distributed hardware. HALoS addresses communication bottlenecks by using local parameter servers (LPSs) within each region and a global parameter server (GPS) that merges updates across regions. This design minimizes expensive inter-region communication while leveraging fast intra-region links. The framework allows asynchronous updates at each level, enabling efficient communication and computation overlap. The paper provides theoretical convergence guarantees for HALoS under non-convex objectives, demonstrating how hierarchical momentum affects asynchronous training. Empirical results show HALoS achieves up to 7.5× faster convergence than synchronous methods and faster than existing asynchronous methods in geo-distributed LLM training, while preserving model quality.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# [W1] Refining Introduction and Related Work\nWe appreciate the suggestion to enhance the clarity and readability of our paper. We designed the introduction to provide sufficient motivation and context for HALoS by outlining both the challenges of geo-distributed LLM training and relevant prior work to help readers understand our design choices early. That said, we greatly value readability for the community and will carefully revise the introduction and related work sections in the final version to further improve their structure and ease of understanding.\n\n---\n# [W2] Hyperparameter Search Strategy\nWe recognize that effective hyperparameter tuning is crucial for the practical deployment of new optimization frameworks across diverse models and environments. To address this challenge, we combined rigorous theoretical analysis with empirical validation, enabling systematic and efficient hyperparameter tuning across different LLMs and geo-distributed setups.\n\nOur paper provides **a thorough theoretical analysis and valuable insights** that guide the identification of optimal hyperparameters. For example, our theoretical results (Section 4.2) demonstrate that local momentum ($\\beta_l$) significantly stabilizes training due to relatively homogeneous updates within each region, while global momentum ($\\beta_g$) requires moderation to effectively handle stale updates at the global level. Specifically, our theory suggested optimal momentum parameters ($\\beta_l$=0.9, $\\beta_g$=0.5), which we empirically confirmed through ablation studies (Section 5.2).\n\nTo further reduce the hyperparameter search cost, we adopted **a scale-up strategy**. Initially, we conducted comprehensive hyperparameter sweeps on the smallest model (Pythia-70M) and subsequently transferred the best-performing settings to larger models (Pythia-160M and Pythia-410M). This approach consistently improved convergence efficiency, significantly lowering tuning overhead. Similar approaches have been recently validated by other studies as well ([1], [2]).\n\nOur results further indicate **strong generalization of these hyperparameters** across varying infrastructure conditions (e.g., inter- and intra-region bandwidths; see Section 5.3 of the paper) and different LLM families (e.g., Llama and Qwen; for related empirical results, please see the section \"Empirical Validation on Different LLM Families\" below), demonstrating their robustness.\n\nAs a next step, we plan to integrate automated hyperparameter tuning techniques—such as Bayesian optimization or adaptive online adjustment—by combining our theoretical convergence bounds with runtime monitoring. We believe this will facilitate easier adoption of HALoS in diverse scenarios without excessive tuning overhead.\n\n[1] Arthur Douillard *et al*. DiLoCo: Distributed Low-Communication Training of Language Models. WANT@ICML 2024.\n\n[2] Weigao Sun *et al*. CO₂: Efficient Distributed Training with Full Communication-Computation Overlap. ICLR 2024.\n\n---\n# Empirical Validation on Different LLM Families\nTo demonstrate the robustness and practical applicability of HALoS across diverse model architectures, we extended our evaluation beyond the Pythia models to include two additional widely-used LLM families—Llama and Qwen.\n\n|Model|Training Speedup of HALoS (vs. Async-Local-SGD)|\n|-|-|\n|Llama-70M|2.1x|\n|Qwen-70M|2.3x|\n|Pythia-70M|1.9x|\n\nAs demonstrated in the table above, **HALoS consistently outperforms the strongest baseline (Async-Local-SGD) across all tested LLM architectures**. Importantly, the hyperparameters initially optimized for Pythia-70M—**without any additional tuning**—exhibited strong generalization, achieving even greater relative improvements (**up to 2.3x** with the Qwen-70M model).\n\n**Experiment Details:** We assessed whether the optimal hyperparameters, identified from our theoretical analysis and validated on the Pythia-70M model, could effectively generalize to these models. To ensure a fair comparison, we selected Llama and Qwen models closely matching Pythia-70M in size, using the same number of layers and hidden dimensions, and conducted the same experiments described in Section 5.1 of the paper.", "claims_and_evidence": "Claim1: HALoS enables efficient geo-distributed LLM training.\nEvidence1: experiments show the reduced amount of communication and enhanced speed.\nClaim2: HAloS has tight convergence.\nEvidence2: The proof in Appendix E.", "methods_and_evaluation_criteria": "I think the methods make sense. HALoS could reduce inter-region communication.\nBenchmarks (MMLU, Hellaswag, LAMBADA, and WinoGrande) are commonly-used ones in LLM evaluation.", "theoretical_claims": "Convergence analysis includes an order approximation of gradient expectation. I briefly go through Appendix E and found no significant problem.", "experimental_designs_or_analyses": "Experiments are carried out in a simulation manner. The designs are okay with four regions, 16 workers for LPS. Worker speeds are not constants; rather, they are randomly set via uniform distribution.", "relation_to_broader_scientific_literature": "On top of sync optimizers, the proposed HALoS targets cross region training scenarios via async training, and proposes an algorithm that reduces cross-region communication costs, yielding higher training speed. It is also claimed by the authors to be the first giving theoretical proofs on convergence.", "essential_references_not_discussed": "No (from my knowledge).", "other_strengths_and_weaknesses": "Strengths:\n1. I think the research field is of practical usages. During pretraining of LLMs, cross-region training is common.\n2. The experiments are sufficient.\n\nWeaknesses: \n1. The introduction needs to be polished. It is much longer than I expected, and I think some chunk of paragraphs could be re-allocated to \"related work\".\n2. Too much hyperparameters are introduced.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Since many hyperparameters are introduced in this work, how to effectively adjust new hyperparameters under a different scenario (different LLM models, different # of regions, etc.)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GRbMYtEP8E", "reviewer": "Reviewer_T99B", "summary": "This paper presents a framework for geo-distributed LLM training named HALoS. To reduce the staleness effect, HALoS introduces local parameter server (LPS) for workers within each region, and periodically syncs LPS with global parameter server (GPS). This paper also introduces a convergence analysis for this approach under certain assumptions. The results show that HALoS significantly reduces the amount of sync bottlenecks while maintaining competitive model performance.\n\n## update after rebuttal\n\nI appreciate the reply from the authors during the rebuttal process and have carefully read them. I will upgrade my score because most of the concerns are addressed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Clarification on Questions\n**(Q1)** We assign the same mini-batch size per worker. For heterogeneous clusters, we avoid ineffective convergence from imbalanced numbers of workers per LPS by adopting **a consistent grouping strategy** that assigns an equal number of workers to each LPS. For example, if four regions originally have (2,4,4,6) workers, we deploy (1,2,2,3) LPSs, each covering two workers. Empirical results and further discussion are provided in the last section ([W3]) of this response.\n\n**(Q2)** We apply momentum at both local and global levels to mitigate gradient/model staleness effectively (Figure 5). We observe that this approach surpasses prior methods (e.g., gradient penalty), which aligns with previous findings (see Section 2). Our theoretical analysis (Section 4.2) and empirical results (Section 5.2) demonstrate **optimal momentum configurations that minimize staleness effects while maintaining strong performance.** Exploring more advanced, staleness-aware momentum update techniques is left as our future work.\n\n**(Q3,Q4)** We clarify that each LPS accumulates updates during LPS-GPS communication and then $K$ additional updates (Algorithm 1). When $K$ is too small (e.g., $K=1$), regions with higher LPS-to-GPS bandwidth (up to 0.9 Gbps in Figure 1(a)) can push updates more frequently than those with slower links (as low as 0.1 Gbps), causing high variance. By using a modest $K$, each LPS accumulates enough updates before syncing with GPS, ensuring more balanced contributions and reducing variance (Section 5.2).\n\n**(Q5)** We clarify that HALoS consistently achieves **the fastest wall-clock time** to reach the same target loss (top row in Fig. 7 and 9). For example, when training Pythia-410M (Table 1, Fig. 7), **HALoS achieves 3.9x faster convergence** than DiLoCo+DynUpd, with less than 10% additional tokens.\n\n---\n# [W1] Different Models\nWe evaluate HALoS using different LLM families (Llama and Qwen) and observe that HALoS consistently outperforms the baseline. Please see the 'Empirical Validation on Different LLM Families' section in our response to Reviewer Lzk8.\n\n---\n# [W2] Model Parallelism\nWe thank the reviewer for highlighting model parallelism (MP). In HALoS, each worker is a data-parallel group of accelerators, each maintaining a full model replica, enabling seamless integration of MP techniques (e.g., tensor and pipeline parallelism). Below, we further compare HALoS with strictly synchronous MP methods.\n\n**Comparison with Strictly Synchronous MP:** We evaluate relative convergence times across three methods: synchronous DP (DP), synchronous DP with PP (DP+PP), and a heterogeneity-aware version of DP+PP (Hetero-Aware DP+PP).\n\n|Method|Time-to-Loss (Relative to HALoS)|\n|-|-|\n|DP|85.12|\n|DP+PP|8.34|\n|Hetero-Aware DP+PP|8.26|\n|HALoS (Ours)|1.00|\n\nAs shown in the above table, **HALoS achieves 8.26x faster convergence** compared to the strongest baseline, Hetero-Aware DP+PP. PP allows relatively small activations transferred cross-region and improves convergence speed compared to pure DP (85.12 → 8.34). However, PP inherently suffers from computational inefficiency from imbalanced pipeline stages (i.e., pipeline bubbles) due to heterogeneous accelerator speeds—even with heterogeneity-aware workload partitioning. In contrast, HALoS effectively mitigates slow inter-region communications and heterogeneous accelerator speeds, achieving superior performance.\n\n**Experiment Details:** We trained Pythia-70M as in Section 5.1. DP+PP method used a DP degree of 4 and a PP degree of 4, placing pipeline stages across distinct regions. Hetero-Aware DP+PP employed heterogeneity-aware partitioning and a simulated annealing-based heuristic for placement from [1].\n\n[1] Dacheng Li et al. AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness. NeurIPS 2022.\n\n---\n# [W3] Heterogeneous Cluster Setup\nWe evaluate HALoS under heterogeneous worker distributions (2,4,4,6 workers per region), comparing against our strongest baseline, Async-Local-SGD.\n\n|Method|Relative Time-to-Loss|\n|-|-|\n|Async-Local-SGD|1.7|\n|HALoS (Naive grouping: 4 LPSs with 2, 4, 4, 6 workers)|-|\n|HALoS (Consistent grouping: 8 LPSs with 2 workers each)|1.0|\n\nWhen workers were naively grouped into one LPS per region, resulting in significantly varying workers per LPS (2 to 6), convergence slowed due to large discrepancies in update progresses across LPSs. To address this, we employ a consistent grouping strategy ensuring each LPS manages exactly two workers, resulting in 1,2,2,3 LPSs per region. Using this strategy, HALoS efficiently coordinates learning across heterogeneous clusters, **achieving 1.7x faster convergence** than Async-Local-SGD.\n\n**Experiment Details:** We trained Pythia-70M as in Section 5.1. For consistent grouping, hyperparameters remained unchanged, except for adjustments to local updates accumulation ($K=8$) and local momentum update delay ($d_l=4$), reflecting the smaller number of workers per LPS.", "claims_and_evidence": "The claims are generally well-supported by the evidences.", "methods_and_evaluation_criteria": "The methods are mostly sound. But I have a few questions:\n\nQ1: In the geo-distributed training setting, we may not have the same number of workers for each region. The evaluation were conducted on 4x4 setting. How would you split the batch size among regions if the number of workers are difference from region to region?\n\nQ2: when pushing from LPS to GPS, are there mechanism to prevent a staled model weight from \"poluting\" global model weight, or they just merge with GPS model weight?", "theoretical_claims": "I checked the correctness of the Theorems in Section 4.2, but I am not entirely sure if the assumptions made are valid or realistic.", "experimental_designs_or_analyses": "Q3. For the impact of K (Fig.5), why is the validation loss the highest for smaller K? I assume that a smaller K conrresponds to more frequent sync between LPS and GPS, and a smaller K should be closer to Sync SGD baseline in term of training performance.\n\nQ4. Can you elaborate a bit more on the claim _\"the accumulation of a minimum number of updates ensures stability\"_ in Section 5.2?\n\nQ5: Fig.7 and Fig. 9shows that DiLoCo+DynUpd has the best convergence speed. Is this because Sync SGD still outperforms Async variants?", "relation_to_broader_scientific_literature": "Outside the geo-distributed training area, this paper is also related to model parallel training (under strictly synchronous setting).", "essential_references_not_discussed": "Not applicable.", "other_strengths_and_weaknesses": "Strengths:\n* The paper presents an important problem -- straggler mitigation for async sgd, and introduces a hierachical architecture that effectively hide the communication for syncing with computation. \n* The speedup is significant compared to baselines. \n* This is well-written and easy to follow.\n\nWeakness:\n* More models (eg. Llama, QWen) and more model parallelism techniques beyond DP should be explored\n* Heterogenous cluster setup (each region having difference number of workers) should also be explored, especially for geo-distributed setting.", "other_comments_or_suggestions": "Please refer to the previous comments.", "questions_for_authors": "Please refer to the previous comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission presents HALoS, a hierarchical asynchronous optimization framework for geo-distributed LLM training, addressing communication bottlenecks and heterogeneous hardware challenges. The reviewers initially raised concerns about non-i.i.d. data robustness, GPS computational overhead, hyperparameter tuning, and generalizability across heterogeneous clusters and model architectures. The authors provided thorough rebuttals, including empirical validation of HALoS’s performance under non-i.i.d. settings, demonstrating reduced global update frequency, and extending evaluations to other models. Theoretical arguments were also extended to account for data heterogeneity. Reviewers acknowledged these clarifications, with the unanimous positive reviews. All reviewers ultimately support acceptance.
Decision: Accept (poster)