Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUs
Authors: Guoliang HE
YOUHE JIANG
Wencong Xiao
Jiang Kaihua
Shuguang Wang
Jun Wang
Du Zixian
Zhuo Jiang
Xinlei Zhang
Binhang Yuan
Eiko Yoneki
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models Training
Data Center Topology
Resource Scheduling
Primary area: infrastructure

Abstract: The scaling law for large language models (LLMs) depicts that the path towards machine intelligence necessitates training at large scale. Thus, companies continuously build large-scale GPU clusters, and launch training jobs that span over thousands of computing nodes. However, LLM pre-training presents unique challenges due to its complex communication patterns, where GPUs exchange data in sparse yet high-volume bursts within specific groups. Inefficient resource scheduling exacerbates bandwidth contention, leading to suboptimal training performance. This paper presents Arnold, a scheduling system summarizing our experience to effectively align LLM communication patterns to data center topology at scale. In-depth characteristic study is performed to identify the impact of physical network topology to LLM pre-training jobs. Based on the insights, we develop a scheduling algorithm to effectively align communication patterns to physical network topology in data centers. Through simulation experiments, we show the effectiveness of our algorithm in reducing the maximum spread of communication groups by up to $1.67$x. In production training, our scheduling system improves the end-to-end performance by $10.6\%$ when training with more than $9600$ Hopper GPUs, a significant improvement for our training pipeline.

Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUs
Guoliang He Youhe Jiang" Wencong Xiao*, Kaihua Jiang', Shuguang Wang", Jun Wang', Zixian Du², Zhuo Jiang', Xinlei Zhang', Binhang Yuan', Eiko Yoneki!
Universitt of Cambridge, 'ByteDance Seed, 'HKUST
{gh512, yj367) ac. uk, biyuan@ust. hk, eiko. yonekknneiicll... cam. ac.uk, Thanli.hl, jiangkaihua, wangshuguang, wangjun 289)abytedance com, {duzixian, jiangzhuo.co es, zhangxinlei. tby)etyeeeeeeeeeeee.......oe com
Abstract
The scaling law for large language models (LLMs) depicts that the path towards machine intelligence necessitates training large scale. Thus, companies con- tinuously build large-scale GPU clusters, and launch training johs that span over thousands of computing nodes. However, LLM pre-training presents unique chal- lenges due to its complex communication patterns, where GPUs exchange data sparse yet high-volume bursts within specific groups. Inefficient resource schedul- ing exacerbates bandwidth contention, leading to suboptimal training performance. This paper presents Arnold, scheduling system summarizing our experience effectively align LLM communication patterns with data center topology at scale. An in-depth characteristic study is performed to identify the impact of physical network topology to LLM pre-training jobs. Based on the insights, we develop a scheduling algorithm to effectively align communication patterns with the physical network topology in modern data centers. Through simulation experiments, we show the effectiveness of our algorithm in reducing the maximum spread of com- munication groups by up to 1.67x. In production training, our scheduling system improves the end-to-end performance by 10.6% when training with more than 9600 GPUs, significant improvement for our training pipeline.
1 Introduction
Pre-training large language models (LL.Ms) at scale is highly resource-intensive process that requires vast computational infrastructure. The performance L.M training is fundamentally dependent on three factors: dataset size, computational power, and model parameters
[19] To meet these demands, companies continually enhance their computing infrastructure by incorporating cutting-edge GPUs and redesigning network architectures [44
32 40) However, LLM pre-training presents unique challenges that distinguish from conventional deep learning tasks - this paper, we explore how to develop an efficient resource scheduling mechanism to support the LLM training workflow to accommodate the resource- inteesiie and complex communication patterns in modern data centers.
LLM pre- tre-training is an exceptionally resource- resource-intensive process. Given the pressing need to com- mercialize L.LMs swiftly, accelerating the training process is paramount. However, training these models often spans weeks, requiring the deployment of thousands of GPU nodes per run. The ability to efficiently schedule and allocate resources critical for both performance optimization and cost
"Equal contribution.
Work done during internship ByteDance Seed. Cooresponndecee to: Youhe Jiang <yj367 @ @cam.ac.uk>
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

management, Furthermore, the unique data transmission patterns in LLM training wherein GPUs communicate sparsely but a high volume within specific groups pose an additional challenge in leveraging modern multi-tier network topologies effectively.
fail integrate network topology-aware scheduling specific to LLM workloads. The primary limitation is their lack of awareness of the high-volume, yet sparsely active distributed communication patterns inherent in LLM training. For example. Figure I indicates that 30% 50% of the time is spent on communication during production LLM training, but studies [40] show that more than 99% of the GPU pairs do exhibit direct traffic, with data exchange occurring exclusively within specific communication groups, as shown in Figure [b] Meanwhile, modern GPU clusters use multi-tier, fat-tree network topologies [[] (Figure[2)) and inefficient job placement leads to significant bandwidth loss and communication overhead. Current schedulers are not designed optimize network-aware placement the scale required by L.LM pre-training jobs (LPJs).
Existing cluster schedulers (e.g.,
To enable effective scheduling of LPJs data centers, we identify two key challenges that limit the effectiveness of existing cluster schedulers.
Corm Vitume
Parcentgg (%)
1220
tooes
256
Misalignment of communication patterns with data center topology. Schedulers opti- mize GPU locality through bin-packing but lack awareness of LPJ communication struc- tures. Consequently, jobs may experience in- efficient cross-switch communication despite being tightly packed.
44
512
Mall
Detse
(a) Time break-down.
(b) Traffic heatmap.
Figure 1: Communication characteristics of LLMs
training in production.
Unaddressed trade-offs across communica-
tion dimensions. There is fundamental trade-off between aligning data parallel (DP) and pipeline parallel (PP) communication patterns. Since GPUs participate in both groups, perfect alignment for both is unachievable, and schedulers must carefully balance the two during placement.
To address the challenges, present Amool,, system that co-designs training frameworks and cluster scheduling, effectively aligning LPJs with modern data center network topology. To optimize training performance, we performed an in-depth characterization study to investigate the impact of physical network topology on LLM training. Based on the observation, we devise a scheduling algorithm reduce the maximum weighted spread of communication groups for LPJs. We also develop a resource management policy that manages job queues to reserve nodes for imminent LPJs..
Through mase- experiments, we show the effectiveness of our scheduling algorithm by bench- marking against other SOTA algorithms. We also perform production run with 9600+ GPUs and show our proposed system improves the end-to-end training performance by 10.6%. In summary, contributions include the following:
Contribution 1. We identify the challenge of aligning LLM communication patterns with modern data center topology for large-scale pre-training. We characterize the impact of physical network topology on individual communication operations and end-to-end training performance in modern data centers.
We design a scheduling algorithm to effectively align communication patterns to the topology of the data center for LPJs, and a resource management policy reserve nodes for the placement.
Contribution 2.
Contribution 3. Through comprehensive simulation experiments, we evaluate the effectiveness of the scheduling algorithm in reducing the maximum spread of communication groups by up to 1.67x. In production run, verify the proposed scheduler improve 9600+ GPUs LPJ by 10.6%.
Background
Distributed training. L.LMs are billion-paramnn parameter transformer-based models that must be trained with multi-GPU sy9tems[39] 35). Common training frameworks [6 34 employ hybrid parallelization strategies to parallelize and accelerate the training process, including:

---

Data parallelism (DP). The Zero Redundancy Optimizer (ZeRO) 33 shards model weights and gradients across data parallel processes and performs synchronization the end of a training step by all-gather and reduce-scatter communication operations.
Pipeline parallelism (PP). The layers of models are divided into several stages (PP size), and each stage interleave communication to adjacent stages as well as the computation within the stages. Inter-stage communication is performed by P2P communication operation like send-recv.
Tensor parallelism (TP). Model weights within an PP stage further sharded across multi-GPUs to alleviate the memory pressure. All-gather and reduce-scatter are necessary to synchronize the intermediate activation during forward pass and backward pass.
The combination of parallelism, i.e. hybrid parallelism, forms diverse communication patterns for GPUs, and training frameworks use communication group to manage the complexity. Each GPU is assigned to a DP, TP, and PP communication group at initialization. The illustrations of different parallelisms and communication groups are demonstrated in Figure 2a] Other parallelism strategies, such as sequence parallelism and expert parallelism, are excluded from discussion as they are for specific scenarios (long context and MoE models respectively).
Previous works 24 36 have identified that TP communication groups should be prioritized to GPUs located within the same node utilize the nigh-bandwidth NVLink interconnection due to stringent data dependencies. Thus, the inter-node communication only takes place within the DP and PP groups As only inter-node communication sensitive to physical network topology, the communication patterns of DP group and PP groups are the focus of this paper.
Model Parameters
Intermediate Activations
Paralleliom
Parallelism
Pipeline Parallelism
Core Layer
GPU
GPU
G
GPU
GPU
GPU
Minipod
L-SW
LSW
L-SW
LSW
Model
Stage
Stage
Hybrid Paralletimm (DP=2, TP=2,
NICT
MC2
MCC
NICT
NYUst
(b) Data center topology. S-SW: spine switches. L-SW: leaf switches. H: computing nodes.
(a) Hybrid parallelism.
Figure 2: LLMs parallelism and data center topology.
Data center topology. Figure 2b gives an overview of our HPC cluster, which is similar to other modern data centers. More than 2000 nodes are interconnected by three layers of switches, forming CLOS-like topology [8] The leaf switch (L-SW) is denoted as sO, which interconnects nodes within the same rack. Then, several s0 switches link spine switch (s1), forming minipod of nodes. Finally, sl switches link to core switches, enabling communication between minipods. The switches in each layer have 32 ports both for uplinks and downlinks. The greatest number hops occurs when the nodes of two different minipods communicate with each other. The compute nodes are equipped with H800 GPUs, each of which is connected an InfiniBand [26] NIC. GPUs within a node are connected by gigh-banddidth links such as NVLink (29 with a bandwidth of 400GB/s, while moter-ooe communication is achieved via the InfiniBand network (400GB/s).
3 Observation and Challenges
Given a user-specified number of GPUs and degree of hybrid parallelism of an LPJ, job scheduling systems enqueue the job and perform resource scheduling to find the best placement in GPU clusters. However, we observe existing scheduling systems fail to align L.M communication patterns with data center topology in practice.
3

---

S-SW
S-SW
S-SW
L-SW
L-SW
L-SW
L-SW
L-SW
L-SW
Group
H
H
Grofp
(a) Fully misaligned.
(b) PP-aligned.
(c) DP-aligned.
Figure 3: Alignment of communication pattern.. One DP group and PP group is highlighted.
Observation 1: Misalignment job placement results in increased cross-switch communication.
SOTA cluster schedulers 42 2 44 apply bin-packing strategy to enhance GPU locality of LPJs. However, shown in Figure Sa even if the scheduler packs an 4-node (32 GPUs) LPJ inside minipod, the communication groups may still not be aligned, because both DP and PP groups engage cross-spine-swited communication that has a longer distance. This misalignment stems from the scheduler's 's lack of awareness of the LPJ's communication structure at scheduling time, limiting its ability to allocate GPU resources according to the job's s communication patterns.
Observation 2: Unresolved trade-off's between DP and PP communication priorities. Figure 3b and [ show two potential alignments of the LPJ, with one prioritizing DP communication and the other prioritizing PP. This presents a fundamental trade-off between the two, because DP and PP are orthogonal parallelism strategies widely used in LLM training. Each GPU participates both DP and an PP group, making it impossible to perfectly align both communication patterns simultaneously. well-designed scheduler must consider this trade-off and balance the alignment needs of both group types during job placement.
Communication and topology-aligned scheduling for LPJs. To effectively schedule LPJs, the scheduler must be aware of the diverse communication patterns and minimize their spread in data centers. Furthermore, effective balance between the spread of DP and PP groups is critical, which requires in-depth characterization communication patterns in modern data centers.
Challenge:
4 Characterization of Communication Patterns for LPJs
Although prior studies 44 have explored locality and topology, their scope is constrained by (1) focus on data-parade parallel (all-reduce) workloads and (2) limited consideration of inter-node topology. To address these gaps, we conduct NCCL tests, a benchmarking suite designed to measure the latency and bandwidth of communication operations used by NCCL [27] 28). to study the impact of ioder-node topology. We focus on inter-node topology across minipods, as the scale of LPJs typically necessitates allocating computing nodes across multiple minipods, where the slowest communication path often dictates the overall communication overhead.
Communication operation performance. Figure studies the performance of communication operations. We use the bus bandwidth (BusBw) as performance measurement, which reflects the peak hardware bandwidth by accounting for the number of ranks for collective communication.
NMMB
6MM
BusBw (GBs)
175
175
140
170
105
165
160
35
155
150
0
2
6
10
12
14
Message Size (2"
MB)
Number of Minipods
Number of Minipods
(a) BusBw over message sizes.
(b) BusBw of collective operations.
(c) BusBw of send-reev.
Figure 4: Performance of communication operation. (AR: all-reduce, AG: all-gather, RS: reduce-
scatter, SR: send-recv)
Figure 4a measures the inter-minipod communication bandwidth and illustrates the performance of different communication patterns over message sizes. For collective communication, the message size must be larger than 2 ( 256) megabytes to fully utilize the bandwidth, while for P2P communication like send-recv, small message size (Re 2 megabytes) can saturate the bandwidth. Using a widely

---

adopted analytical model detailed in Appendix B and substituting the parameters with 7B GPT- based model, we can obtain that the data volumes of the DP and PP groups are GB and 30 MB respectively, indicating that the bandwidth is fully utilized.
The degradation in BusBw by expanding communication groups across minipods is illustrated in F 4c] Performance decreases by to 17% for collective operations and 70% for the P2P operation as communication extends over additional minipods, highlighting the critical importance of GPU locality and alignment. Additionally, our findings suggest that co-located jobs may experience reduced bandwidth contention in multi- teltttteennt cluster environments, as evidenced by the inter-job interference patterns documented in Appendix C
End-to-end training performance. Based the characterization of individual communication operations, we proportionally down-scaled a production model and ran the workload with 96 GPUs, spanning 2 minipods, to further understand the impact of network topology LLM training.
unaligned
pp
Dense
MOB
Peuaflops
3.1
1.80
2,9
3.2
2
3.1
1.70
11
1.1
3.0
1000
1000
3,5x
Dense training steps
Moli trinning steps
Model Size
(a) Comparison of three different placement strategies two types of LLMs.
(b) Performance improvement by scaling model sizes under the optimized alignment.
Figure 5: End-to-end training performance.
Figure 5a shows the throughput of the LPJ under the three different placement strategies. The pp-aligned placement is illustrated in Figure 12 The throughput becomes stable after 200 steps except for a slight fluctuation around the 550th step due to garbage collection. PP-aligned placement consistently outperforms the other two, demonstrating that prioritizing PP group communication leads to improved performance. The average improvement for the dense model and the MoE model is 2.3% and 1.8% respectively. For the dense model, also observe that the PP communication dominates the communication overhead, since prioritizing the placement of DP groups leads to no speedup. For the MoE model, reducing the spread of both the DP and PP groups contributes to performance gains, with the optimizations of the PP group providing more improvements.
Figure shows that if we scale the model size by adding more layers, the performance improvement continues to increase. We attribute this to communication being the primary performance bottleneck, with larger models further amplifying communication volume. Thus, more pronounced benefits obtained as the model size increases.
We further examine the sensitivity of training performance to intra-minipod network topology by varying node placement within a single minipod. For a dense 24B parameter model, the maximum observed performance variation is 0.3%, and the impact is negligible for other models. Since the communication overhead group is typically dominated by the slowest link, and LPJ communi- cation groups frequently span multiple minipods, we conclude that training performance is largely insensitive to intra- mira-od topology.
We repeated the characterization experiment in another GPU cluster detailed in Appendix D and found that the best placement can be subjected model sizes and GPU types. Since LPJs are typically scheduled in advance and deployed for long duration, it is essential to perform a characterization beforehand identify communication bottlenecks within the group. This enables the optimization of placement strategies accordingly and balances the trade-off, as detailed in $5)
5 Scheduling Algorithm
The core component of Arnold is its scheduling module, which is designed to effectively allocate GPUs to LPJs based on the user-specified number of GPUs and the degree hybrid parallelism. In this section, formally define the scheduling problem and present proposed solution.
5

---

5.1 Workload Representation
Arnold represents an LPJ by a communication matrix, where row represents a PP group and a column represents a DP group. Formally, given job specification including the total number of GPUs, the degree of PP, TP, and then Arnold computes the size of the communication matrix using Equation
DP = /GPPs
#row = DP/(8/TP) #col PP
(I)
An example of 96 GPUs and DP - PP 2 is shown in Figure 12 communication matrix, is attached with vector [Fee, Ua, 2pl. representing the size of weight, the DP and PP communication volume per GPU respectively. Those values are computed using the analytical model in Appendix and is used balance the trade-off between DP and PP groups.
For node Mij in the
5.2 Objectives
The scheduling objective function aims to minimize the maximum spread for both DP and PP groups. For node Uij in the communication matrix, the number of possible assignments minipod is k. Therefore, it has a one-hot decision vector Tay of length k, representing the decision to place to one of the minipods. Then the objective function can be written Equation
MIN (ar max(a(aa (12, ...n))) + 3 (x( T2j Tmi))) ))
Where, the distance D between the vectors is defined as follows:
D(e ......, Un) Ki: 3j, E (1, n..j & U,yl + wi
(3)
Intuitively, the distance measures the spread of communication group, i.e. if any two vectors differ in the äth position, the ith position adds one to the distance. The objective function aims to minimize the weighted of the maximum spread of DP and PP groups, and the maximum is taken because the slowest communication group is the straggler to slow down the whole training process. The weight parameters and /3 represent the affinity that controls the trade-off between DP and PP groups, and a + 3 = 1. Equation 2 cannot be solved by off-the-shelf solvers efficiently for online scheduling due to the discrete distance calculation, and seek for simplifications.
Domain-specific simplification. We identify that communication groups are homogeneous and synchronous for LPJs because nodes are seng-scheduled and must synchronize their gra- dients at the end of a training step. As a result, each PP group always starts approximately at the same time and performs the same amount of computation and communication. Similarly, DP groups perform gradient synchronization at the same time. The characteristies allow us to simplify the scheduling objective function by coarsening scheduling unit communica- tion group. We therefore transform Equation 2 into a bin- ton-packkng-lin formulation:
MIN
Jax
(N;)
ST]
(4)
S.L.
Vi: Σ st ST
(Max Spread)
(5)
Vj Σ Pij S csti
(Capacity Const.)
(6)
Wi Σ Pij =1
(Allocation Const.)
(7)
Vi,j P. Ssy (Minipod Selection)
(8)
Vj 3) E [0,1}
(9)
Vi,j :8;5 E (0, 1). E [0
(10)
Where y indicates whether the j-th minipod is used and cy is the normalized capacity of the minipod, updated dynamically based on the number of available nodes. Sij denotes whether the i-th communication group is allocated to minipod j. Piy denotes the percentage of the i-th communication group allocated to minipod j. T is introduced auxiliary variable that allows to minimize the maximum spread of communication groups. a and 33 are the affinity parameters as before. Overall, minimizing T' effectively consolidates communication groups into the smallest possible number of minipods, while the objective term Σ, (y;) controls the spread of the other communication group.
simple example. We consider the PP group as a scheduling unit. By setting a = O. the scheduler can place each PP group into minipod, causing more cross-switch communication for DP groups,

---

while by setting 3 = O, the scheduler minimizes the overall usage of minipods, although the placement may cause cross-switch communication for PP groups. Together, Equation (4) is the balanced optimization objective function. Equation (5) restricts the maximum spread of PP groups to be less than or equal T. Equation (6) restricts that the allocation cannot exceed the capacity of minipod. Equation (7) suggests that the sum of the allocation percentage must be equal to 1 for every PP group. Equation (8) suggests that if a minipod has some percentage of a PP group, then it is considered used. Finally, Equation (9) and (10) define the range of the binary variable y/, 8ij and the continuous variable Pij. As a concrete example, the values for those variables in simulated experiment are listed in Appendix H
In this formulation, all variables are either an integer or a fraction. Therefore, the objective function can be solved using off-the-shelf muxed-intege programming (MIP) solvers efficiently for online scheduling [4] After solving the MIP, we assign continuous rank indices to nodes belonging to the same minipod to reduce cross-switch communication within each communication group.
Balancing the trade-ofT. The affinity parameters in Equation 4 require balance of the trade-off between the DP and PP groups, which depends on the model configurations and GPU types (44) To perform online scheduling, store the characterization results in $ database, and we search for the best match to determine the values of the affinity parameters. The communication matrix computes the per-GPU parameters (Vn)) and communication volumes (vd. Up). We then compute the average ratio of pnppuuaaioont--commniiiii
and DP-to-PP volume as r1 1b6xe and r2 Ea Up where mb is the size of the microbatch. These ratios are used to find the best matching job in the database by comparing the Fuclidean distance, i.e. MIN rury V r2)² + (r2 )² because GPUs exhibit comparable performance characteristics if they have similar computational load and communication volume.
va+tp
LPJs are associated with metadata (GPU Jop) where Idp- corresponds to the improvement of alaaligned, and Pl-aligned placement strategies. The affinity parameters a and |3 are then derived idr
and B jon based on the relative performance improvement of jap and jpp. i... a Jep
Due to the importance of LLM training and their unified architectures, LPJs are scheduled in advanced and pre-characterized, so the profiling data in the database can be looked up in online scheduling. For example, for a 24b dense model in the H800 GPU cluster, the scheduling unit is set to the PP group and a is set zero as PP groups clearly dominate the communication overhead. For the 24b MoE model, a 0.3 and B = 0.7.
5.3 Resource Management
The scheduling algorithm computes globally optimal placement for LPJs in the GPU cluster, which inevitably confficts with other jobs. To address this, we develop a queuing policy to manage the job queue and reserve resources for the imminent LPJ.
Algorithm Tillustrates our scheduling policy. Once the LPJ is planned, the scheduler solves the MIP equation and reserves the resources. Since then, incoming jobs are preferentially allocated outside the reserved zone. Otherwise, to improve resource utilization, if the predicted job completion time (JCT) of an incoming job precedes the arrival time of the LPJ., it may still be scheduled within the reserved zone. If neither of the conditions can be satisfied, the job delayed also employ an ML-driven JCT predictor balance the trade-off of queuing delay and resource utilization. The setup and evaluation are detailed in Appendix and G respectivvely
the scheduling interval. We
6 System Implementation
We have implemented a prototype of Arnold with more than 3k lines of Python codes. The prototype consists of the scheduling module and a trace-driven simulator that can replay production traces. We also have version of the deployment integrated with Kubernetes (21). For training frameworks, we build on top of Megatron [24] and modify it to ensure that communication groups follow the placement provided by Arnold. Figure gives overview of Arnold.
7

---

Uses
Jobs
Scheduler
Prediction
Matudats
Param
Training Framework
Predictor
Monitor
Testure
Group
lisk
Database
Figure 6: Architecture overview of Amold.
Evaluation
We evaluate Arnold using both simulation and real-cluster experiments. To benchmark scheduling algorithms cost-effectively, we develop a simulator, as direct evaluation on production clusters is prohibitively expensive. After identifying the highest-performing scheduling algorithm through simulation, we deploy it on our production cluster to validate its effectiveness under real workloads.
7.1 Simulation Experiments
Baselines.
We compare the scheduling algorithm with the following baselines.
1. Best-fit 31 assigns the nodes the minipod with the least remaining resources.
Random-hh fit assigns nodes minipods randomly such that the assignment is balanced and fair.
3. GPU-packing 22 is effective placement strategy applied by state-of-the-art GPU cluster schedulers that pack multiple jobs to the same GPU. We modify the algorithm to pack multi-GPU jobs to minipod to satisfy the network topology semantics.
GPU topology-aware placement algorithm. It represents the workload job graph (similar to our communication matrix) and the topology as physical graph. Then recursively bi-partitions the physical graph and maps the job graph the sub-graphs (Hierarchical Static Mapping Dual Recursive Bi-partitioning [10). The graph bi-partitioning is implemented by the Fiduccia Mattheyses algorithm
Topo-aware
Metrics. The weighted sum of the maximum spread as in Equation 2 used to evaluate scheduling algorithms. To evaluate scalability, we measure the scheduling latency.
Table I: Benchmark setting. Network topology {x), {v} represent [x] minipod and (y) nodes in total, and the numbers in job configurations are the degree of DP, TP, PP. The scheduling unit the PP group.
Settings
Network Topology
Job Configs
3. 18
12, 4, 2
(ii)
5, 438
24, 4, 8
(111)
11. 1019
46, 8, 8
Setups. We use 3 settings in the benchmark as listed in Table I where the network topology is taken from subset of GPU cluster, and the job configurations are representative for small, medium, large jobs respectively. We also vary the value of a to investigate different degree of affinity.
Figure compares the performance of different algorithms. Our algorithm consistently outperforms other baselines and up to 1.67x compared to the best baseline. On average, leads to 1 1.2x reduction of the weighted sum of the maximum spread for communication groups. In the simple topology (setting (i)), our algorithm achieves the same score as best-fit, gpu-pack and topo-aware, because the network topology and job configurations are relatively simple, there no room improve the placement. For medium and large jobs, our algorithm is better than the other baselines due to the large search space of possible placement.

---

Random-Fii
GPU-Pack
We also observe that a increases, our scheduling algorithm is closer to other baselines. This is because controls the affinity of the DP group, and if is 1, the objective function reduces to a bin-packing formulation and therefore has difference from other bin-packing algorithms. In prac- tice, would not set greater than 0.5 as observed from our characteri- zation results ($ As result, our algorithm usually scores higher than other baselines.
Weightedd Spread
0 0.2 0.5aa 0.8
am 0.2 a
a.5
122200. (i
(
(60
Figure 7: Weighted spread communication groups under differ- ent scheduling algorithms.
Scalability. To evaluate the scalabil- ity of our algorithm, we implement an enumeration approach and com- pare the scheduling latency. Figure investigates the scalability. The enu- meration approach is guaranteed to obtain the optimal placement strategy. However, it is not scalable, as its com- putational complexity is O(k")) for scheduling m nodes to l minipods. Empirically, it cause scheduling latency of 30s in a simple topology (setting (i)) when scheduling 14 nodes. In a medium topology (setting (ii)) it takes 100s+ schedule job with 10 nodes. In contrast, our algorithm has low latency even if it is required to schedule a 512 node job in a cluster with 1000+ nodes.
Ours
Enumeration
100
30
Seconds
20
50
5-
10
5.0
7.5
200
Requested Nodes
Requested Nodes
Requested Nodes
()
(iii)
Figure 8: Scheduling latency under different configurations.
7.2 Cluster Experiment
To evaluate the effectiveness of Arnold in real-world environments, we run experiments in our GPU cluster. The specific information such as the number of GPUs and the model size, hidden due to business concerns. One of our LLMs is a MoE variant and was trained previously with more than 9600 GPUs (1200+ nodes). We first run the experiment by scheduling validate the speedup achieved by Arnold. We then run the pre-training at full scale. We compare Arnold with an SOTA production system for L.MMs, MegaScale [8] which takes a full-stack solution to optimize LLMs training and scale to O(10, 000) GPUs.
job with 208 GPUs, and
1.8
240
250
Petsllops
1.6
200
Amaad
220
MayiScalic
1.2
150
Terch-Profie
400
500
MigaSccole
Ansodd
200
MegiScale
Arsold
Steps
(a) 208 GPUs
(b) 9600+ GPUs.
(c) Throughput over steps.
Figure 9: Cluster experiments.
End-to-end experiments. Figure 9a and 9b illustrate the average throughput of the two systems. Amold achieves average speedup of 5.7% and 10.6% respectively. We observe that Arnold reduces the maximum spread for the DP group and the PP group by 3x and 2x in the medium- scale experiment, while for the full-scale experiment, the reduction is 1.5x and 1.3x. This is because it is more likely to spread nodes across minipods in the cluster for medium-scan However, for the full-scale experiment, the requested GPUs take up more than 50% of the total number GPUs in the cluster, and therefore the space of scheduling is shrunk.
experiment not planned carefully.
Nevertheless, we observe that as the model size scales and more nodes are added, the speedup achieved by Arnold also increases. The finding is consistent with Figure [5b] in $4 in which the effectiveness of optimized placement more prominent as the model size scales up. Our production

---

deployment encompasses significantly larger models exceeding 400 billion parameters distributed across a substantially higher number of nodes, resulting in intensive network communication demands.
Figure 9c plots the performance of the full-scale experiment over the training steps. Despite the performance fluctuation at the 160-th step due to torch profiling, we observe that the Arnold out- performs MegaScale consistently. The LPJ runs for more than one month, and the improvement is significant for downstream tasks as well as cost savings (GPU hours and human resources), since cloud providers price one Hopper GPU at $2.99/h. Moreover, the proposed optimization is orthogonal to those reviewed in $8 allowing it to be applied in conjunction with existing methods. Importantly. the optimization is fully transparent to end users.
We also evaluate the scheduling algorithms on an open- source model (Llama3 8B), replacing the InfiniBand network with a RoCE network and using 12 nodes equipped with 1.20 GPUs, and the results shown in Appendix [ It indicates our scheduling algorithm generalizes well across models and hardware configurations.
Breakdown analysis. Figure 10 shows kernel-level ex- amination of both systems using the Torch profiler. It re- veals that communication and topology-aligne placement strategies yield a nuanced impact: while they enhance the performance of communication kernel as expected, they simultaneously introduce performance degradation in other kernels, including computation kernel. Through systematic ablation studies in Appendix[] [ we identify re- source contention between GPU streams the fundamental mechanism underlying this phenomenon, which presents when communication and computation kernels execute concurrently across multi- ple streams. These observations broaden the scope of topology-aware scheduling by showing that its impact extends beyond communication efficiency, influencing the execution characteristics of computation kernels as well.
DiE (mn)
250
96
Broadoas
RS-FP32
GEMM
Figure 10: Breakdown analysis.
Related Works
LLMs training. LLMs have become significant workload in the field of machine learning (30
37
49), and the computing infrastructure continues to evolve to adapt to the challenging workload.
For example, efficient parallelization strategies are searched by model parallelizers [ST] 22
38).
training frameworks specialized for training scalability are built to orchestrate large-scale worker and perforrrrfrmece operators are developed to maximize the utilization of
nodes 18 36 24
hardware accelerators 9 6 12]. However, those works are orthogonal to the optimization proposed in this paper, as the physical network topology is only visible at the cluster scheduling layer. Therefore, our work is transparent to the underlying infrastructure codes and the speedup is achieved on top of existing effort to accelerate the training performance.
Deep learning job schedulers. Job scheduling systems for deep learning tasks have been widely deployed by companies
25 However, none of deep learning jobs have come even close to the scale and importance of LLM pre- pre-training jobs. Amool addresses this gap by providing a solution tailored to scheduling LLM pre-training jobs. complementing existing schedulers. Recent studies have begun to explore the characteristics of LLM workloads in GPU clusters [3]. In contrast, our work specifically targets the optimization of L.LM pre-training performance.
44
43
Emerging network architectures. Emerging network architectures 40 32 have significant impacts on future design of scheduling algorithms, and we believe that the presented algorithm is capable of handling other network topologies, for production GPU clusters follow a multi-tier design. The collective communication is bottlenecked by the highest hierarchy, i.e. the inter-minipod communica- tion in our case. In our case, we observe little impact of intra-minipod topologies; however, we can also solve the MIP in intra- mira-dd level if we can observe any performance degradation.
Conclusion
In this work, we present Arnold, a scheduling system that summarizes our experience in effectively scheduling LPIss at scale. In-depth characterization is performed and a scheduling algorithm is developed to align LPJs with the topology of modern data centers. Through experiments, we show the effectiveness both in simulation-based and real-world GPU clusters.
10

---

References
[1] Mohammad Al-Fares, Alexander Loukissas, and Amin Vahdat. A scalable, commodity data center network architecture. In Proceedings of the ACM SIGCOMM 2008 Conference Data Communication, SIGCOMM '08, page 63-74, New York, NY, USA, 2008, Association for Computing Machinery.
[2] Marcelo Amaral, Jordà Polo, David Carrera, Seetharami Seelam, and Malgorzata Steinder. Topology-aware gpu scheduling for learning workloads in cloud environments. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 17, New York, NY, USA, 2017. Association for Computing Machinery.
[3] George Amvrosiadis, Jun Woo Park, Gregory R. Ganger, Garth A. Gibson, Elisabeth Baseman, and Nathan DeBardeleben. On the diversity of cluster workloads and its impact research results. In 2018 USENIX Annual Technical Conference (USENIX ATC 18), pages 533-546, Boston, MA, July 2018. USENIX Association.
[4] Suresh Bolusani, Mathieu Besançon, Ksenia Bestuzheva, Antonia Chmiela, João Dionísio, Tim Donkiewicz, Jasper van Doormaalen, Leon Eifler, Mohammed Ghannam, Ambros Gleixner, Christoph Graczyk, Katrin Halbig, Ivo Hedtke, Alexander Hoen, Christopher Hojny, Rolf van der Hulst, Dominik Kamp. Thorsten Koch, Kevin Kofler, Jurgen Lentz, Julian Manns, Gioni Mexi, Erik Mühmer, Mare E. Pfetsch, Franziska Schlösser, Felipe Serrano, Yuji Shinano, Mark Turner, Stefan Vigerske, Dieter Weninger, and Liding Xu. The seip optimization suite 9.0, 2024.
[5] Jiamin Cao, Yu Guan, Kun Qian, Jiaqi Gao. Wencong Xiao, Jianbo Dong, Binzhang Fu, Dennis Cai, and Ennan Zhai. Crux: Gpu-efficient communication scheduling for deep learning training. In Proceedings of the ACM SIGCOMM 2024 Conference, ACM SIGCOMM '24, page 1-15, New York, NY, USA, 2024. Association for Computing Machinery.
[6] Li-Wen Chang, Wenlei Bao, Qi Hou, Chengquan Jiang, Ningxin Zheng, Yinmin Zhong, Xuanrun Zhang, Zuquan Song. Chengji Yao, Ziheng Jiang, Haibin Lin, Xin Jin, and Xin Liu. Flux: Fast software-basse communication overlap gpus through kernel fusion. 2024.
[7] Arnab Choudhury, Yang Wang, Tuomas Pelkonen, Kutta Srinivasan, Abha Jain, Shenghao Lin, Delia David, Siavash Soleimanifard, Michael Chen, Abhishek Yadav, Ritesh Tijoriwala, Denis Samoylov, and Chunqiang Tang. MAST: Global scheduling of ML training across Geo-Distributed datacenters at hyperscale. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 563-580, Santa Clara, CA, July 2024. USENIX Association.
[8] Charles Clos. A study of non-blockeee blocking switching networks. The Bell System Technical Journal, 32(2):406 4244, 1953.
[9] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory memory-efficient efficiient exact attention with io-awareness, 2022.
[10] Ercal, Ramanujam, and P. Sadayappan. Task allocation onto a hypercube by recursive mincut bipartitioning. In Proceedings of the Third Conference on Hypercube Concurrent Computers and Applications: Architecture, Software, Computer Systems, and General Issues Volume C3P, page 210-221, New York, NY, USA, 1988. Association for Computing Machinery.
[11] C.M. Fiduccia and R.M. Mattheyses. A linear-time heuristic for improving network partitions. In 19th Design Automation Conference, pages 175-181, 1982.
[12] Guoliang He and Eiko Yoneki. Cuasmrl: Optimizing gpu sass schedules via deep reinforcement learning, 2025.
[13] Qinghao Hu, Zhisheng Ye, Zerui Wang. Guoteng Wang, Meng Zhang, Qiaoling Chen, Peng Sun, Dahua Lin, Xiaolin Wang, Yingwei Luo, Yonggang Wen, and Tianwei Zhang. Characterization of large language model development in the datacenter. In NSDI, pages 709-729, 2024.
[14] imbue team. From bare metal to 70b model: infrastructure set-up and scripts. https Accessed: 2024-12-01.

---

[15] Youhe Jiang, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, and Bin Cui. Osdp: Optimal sharded data parallel for distributed deep learning. Proceedings of the Thirty-Secont International Joint Conference Artificial Intelligence, pages 2142-2150, 2023.
[16] Youhe Jiang, Huaxi Gu, Yunfeng Lu, and Xiaoshan Yu. 2d-hra: ivo-dimensional hierarchical ring-based all-reduce algorithm in large-scale distributed machine learning. IEEE Access, 111888818844999 2020.
[17] Zhifeng Jiang, Zhihua Jin, and Guoliang He. Safeguarding system prompts for llms, arXiv preprint MrXiv:2412.13426. 2024.
[18] Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong. Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng. Zhuo Jiang. Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie. Leqi Zou, Sida Zhao, Liang Xiang, Zherui Liu, Zhe Li, Xiaoying Jia, Jianxi Ye, Xin Jin, and Xin Liu. MegaScale: Scaling large language model training to more than 10,000 GPUs. In 21st USENIX Symposium Networked Systems Design and Implementation (NSDI 24), pages 745-760, Santa Clara, CA, April 2024. USENIX Association.
[19] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[20] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[21] Kubernetes Authors. Kubernetes: Production-grade container orchestration. kubernetes.. 1o 2023. Accessed: 2024-12-01.
https://
[22] Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. Galvatron: Efficient transformer training over multiple gpus using automatic parallelism. Proc. VLDB Endow, 16(3):470-47, November 2022.
[23] Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. Galvatron: Efficient transformer training over multiple gpus using automatic parallelism. Proceedings of the VLDB Endowment, 16(3):470-49, 2022.
[24] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron- In SC21: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-14, 2021.
[25] Andrew Newell, Dimitrios Skarlatos, Jingyuan Fan, Pavan Kumar, Maxim Khutomenko,, Mayank Pundir, Yirui Zhang, Mingjun Zhang, Yuanlai Liu, Linh Le, Brendon Daugherty, Apurva Samudra, Prashasti Baid, James Kneeland, Igor Kabiljo, Dmitry Shchukin, Andre Rodrigues, Scott Michelson, Ben Christensen, Kaushik Veeraraghavan, and Chunqiang Tang. Ras: Continuously optimized rede datacenter resource allocation. In Proceedings of the ACM SIGOPS 28th Symposiun on Operating Systems Principles, SOSP 221, page 505-520, New York, NY, USA, 2021. Association for Computing Machinery.
[26] NVIDIA. Infiniband networking solutions.
http::/wwwwwniiiic com/en-us Accessed: 2024-12-01.
[27] NVIDIA. Nccl. httpes//eeraapper.vvvdddac.occc Accessed: 2024-12-01.
[28] NVIDIA. Necl-test, http:://ggtabb.../////////cceeee Accessed:
Accessed 2024-12-01.
[29] NVIDIA. Nvlink. http::/wwwwaiiaaacccco...aaaaaas Accessed:
2024-12-01.
[30] OpenAl. Chatgpt. httper//peeaai.ccm//neee///at/g Accessed: 2024-12-01.
12

---

[31] Rina Panigrahy, Vijayan Prabhakaran, Kunal Talwar, Udi Wieder, and Rama Ramasubramanian. Validating heuristics for virtual machines consolidation. Technical Report MSR-TR-2011-4 January 2011.
[32] Kun Qian, Yongqing Xi, Jiamin Cao, Jiaqi Gao, Yichi Xu, Yu Guan. Binzhang Fu, Xuemei Shi, Fangbo Zhu. Rui Miao, Chao Wang, Peng Wang, Pengcheng Zhang, Xianlong Zeng, Eddie Ruan, Zhiping Yao, Ennan Zhai, and Dennis Cai. Alibaba hpn: A data center network for large language model training. Proceedings of the ACM SIGCOMM 2024 Conference. ACM SIGCOMM '24, page 691-706, New York, NY, USA, 2024. Association for Computing Machinery.
[33] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models, 2020.
[34] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 200 page 3505-3506, New York, NY, USA, 2020. Association for Computing Machinery.
[35] Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely aparsely-gated gated mixture-of- experts layer. In International Conference on Learning Representations, 2017.
[36] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-In Training multi-billion parameter language models using model parallelism, 2020.
[37] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[38]
Colin Unger, Zhihao Jia, Wei Wu, Sina Lin, Mandeep Baines, Carlos Efrain Quintero Narvaez, Vinay Ramakrishnaiah. Nirmal Prajapati, Pat McCormick, Jamaludin Mohd- Yusof, Xi Luo, Dheevatsa Mudigere, Jongsoo Park, Misha Smelyanskiy, and Alex Aiken. Unity: Accelerating DNN training through joint optimization of algebraic transformations and parallelization. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 267-284, Carlsbad, CA, July 2022. USENIX Association.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[40] Weiyang Wang, Manya Ghobadi, Kayvon Shakeri, Ying Zhang, and Naader Hasani. Rail-only:
A low-cost high-performance network for training Iims with trillion parameters, 2024.
[41] Yujie Wang, Youhe Jiang, Xupeng Miao, Fangcheng Fu, Shenhan Zhu, Xiaonan Nie, Yaofeng Tu, and Bin Cui. Improving automatic parallel training via balanced memory workload optimization. IEEE Transactions on Knowledge and Data Engineering, 3906633900 2024.
13

---

[42] Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, and Yu Ding. MLaaS in the wild: Workload analysis and scheduling in Large-Scale heterogeneous GPU clusters. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), pages 945-960, Renton, WA, April 2022. USENIX Association.
[43] Qizhen Weng, Lingyun Yang. Yinghao Yu, Wei Wang, Xiaochuan Tang. Guodong Yang, and Liping Zhang. Beware of fragmentation: Scheduling GPU-Sharing workloads with fragmenta- tion gradient descent. In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pages 995-1008, Boston, MA, July 2023. USENIX Association.
[44] Wencong Xiao, Romil Bhardwaj. Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, and Lidong Zhou. Gandiva: Introspective cluster scheduling for deep learning, In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 595-610, Carlsbad, CA. October 2018. USENIX Association.
[45] Wencong Xiao, Shiru Ren, Yong Li, Yang Zhang, Pengyang Hou, Zhi Li, Yihui Feng, Wei Lin, and Yangqing Jia. AntMan: Dynamic scaling on GPU clusters for deep learning. In 14th USENIX Symposium Operating Systems Design and Implementation (OSDI 20), pages 533-548, USENIX Association, November 2020.
[46] Dian Xiong, Li Chen, Youhe Jiang, Dan Li, Shuai Wang, and Songtao Wang. Revisiting the time cost model of allreduce. arXiv preprint arXiv:2409.04202. 2024.
[47] Ran Yan, Youhe Jiang, Xiaonan Nie, Fangcheng Fu, Bin Cui, and Binhang Yuan. Hexiscale: Accommodating large language model training over heterogeneous environment. arXiv preprini arXiv:2409.01143 2024.
[48] Ran Yan, Youhe Jiang. and Binhang Yuan. Flash sparse attention: An alternative efficient implementation of native sparse attention kernel. arXiv preprint arXiv:2508. 18224, 2025.
[49] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong. Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang. Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang. Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin. Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024.
[50] Li Zhang, Youhe Jiang, Guoliang He, Xin Chen, Han Lv, Qian Yao, Fangcheng Fu, and Kai Chen. Efficient mixed -reccisin large language model inference with turbomind. arXiv preprint arXiv:2508. 15601, 2025.
[51] Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. Alpa: Automating inter- and Intra-Operator parallelism for distributed deep learning. 16th USENIX Symposium Operating Systems Design and Implementation (OSDI 22), pages 559-578,6 Carlsbad, CA, July 2022. USENIX Association.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
In the abstract and introduction, we state our contributions and the paper scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Appendix A we explicitly discuss the limitation of the algorithm and future
work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

This paper presents scheduling system for LLM training. The results are empirically evaluated both on simulation and real experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [No]
The simulation results are fully disclosed and reproducible. The cluster experiment contains sensitive information regarding the size of the LLM, the number of the GPUs our cluster, and we are unable to provide that information.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will open source the simulator and the scheduling algorithms.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praips.cc/publicc/uiieeee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all details regarding the scheduling system. Other information such data splits is irrelevant.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The simulation experiment is deterministic and can be fully reproduced. The cluster experiment is shown with several steps of stable performance measurement.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes].
We specify both the GPU types and network topology to reproduce the experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We conform to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: The paper improves the LLM training in modern data centers, and has no negative societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper describes LLM training from a system perspective, and has no risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators have been credited and licensed.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
We have not introduced new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
We do not have any human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
We do have any human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We did not use LLMs to develop our core method.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
21

---

A Limitation and Future Works
Failure recovery. On hardware failure, the optimal placement of LPJs will inevitably change, but is too expensive to solve the MIP and then migrate to the new placement. A potential approach is to increase the number of GPUs of communication groups in the initial scheduling as backups, which only rum preemptive jobs and can be replaced with failure nodes when needed.
Other network topology. The characterization results, while derived from our in-house data center environments, exhibit broad applicability to CLOS-based network topologies, which represent the predominant network architecture in modern data center deployments. By varying the affinity parameters, one can effectively trade-off the balance of DP and PP groups in their own data centers. Therefore, the characterization methodology and the scheduling algorithm are generalizable to other data centers for large-scale LPJs.
B Analytical Estimation for Communication Volume
To estimate the communication volume of pre-training jobs, we adopt an analytical model for GPT- based variants. We use the same notation from previous work [24] by denoting the vocabulary size V. global batch size gb, micro-batch size mb, sequence length s, hidden dimension the number of layers 4, the DP size dp, the PP size pp, the number of VPP size up, number of microbatches m. We have:
gb
m
(11)
mb dp
DP groups. GPUs within the same group replicate the model weights and exchange parame- ters as well as gradients, so the communication volume can be computed using Equation
DP - volume h (V 8) + 1/pp (4h² + 2h + 81: + 7h)
(12)
dense layer
For MoE models, we can replace the number of parameters with MoE layers accordingly.
PP groups. GPUs exchange intermediate activation to adjacent PP stages, and thus we apply Equation 13 to estimate the data volumes.
PP volume = 2 x mb
(13)
C
Sensitivity to Shared Load
23.5
BusBw (GB/s)
23.0
job
22.5
job
22.0
job
21.5
01:20
01:30
01:40
01:50
02:00
02:10
02:20
Time
Figure 11: Bandwidth interference.
Since GPU clusters are usually multi-tenant to improve resource utilization, we also study the interference between inter-node communication quantitatively. Before we bring our cluster online, we perform large-scale stress test on our cluster by running NCCL tests. We record the time series The stress test consists of 3 jobs, requesting thousands
of bus bandwidths and show in Figure II
22

---

of GPUs each, and spanning across multiple minipods in the cluster. Each job performs all-to-all communication with message size of 2GB constantly, which simulates jobs performing extensive inter-node communication busy cluster because all-to-all generates large amount of flows in the network.
We can observe performance fluctuation for all three jobs. For example, after job is launched at 01 22, job 3 has slight performance degradation of 0.5GB/s (3%). The maximum performance degradation is to 5% for job 3 during the stress test period. This suggests that jobs spanning larger number of minipods only suffer from increased bandwidth loss but are also more exposed to interference from other workloads in the cluster.
D Ada Lovelace GPUs
We repeat the characterization experiment in another GPU cluster, where each node is equipped with L,20 GPUs, to ensure finding is not limited to H800 GPUs, and briefly summarize the results in 2
Best placement Model size
speedup
DP-aligged dense 7b
1.4%
PP-aligned dense 14b
0.5%
Table 2: Results on L20 GPUs cluster.
We observe that Dl-aligned yield greater speedups for certain model configurations. This is likely because during training, L20 GPU uses 8-bit data format, which halves the communication volumes between PP stages. However, DP groups still use 32-bit for parameter and gradient synchronization, so the communication volumes remain unchanged. As a result, DP group communication can become the dominant overhead, making placement strategy that prioritizes DP groups more beneficial. However, as the size of the model grows, the bottleneck shifts back to the communication of the PP group, so the PP-aligned is preferential.
E Communication Matrix
PP1
PP2
PP3
PP4
PP5
PP6
2
3
5
DP1
6
7
8
9
10
11
DP2
Figure 12: Example placement (96 GPUs and 12 nodes) I.P..
The example placement in Figure 12 has DP group size of 6, and PP group size of 2. Nodes in different colors represent they are placed in different minipods. The number the rank of the node. In this example, DP group is aligned and the communication of PP group must cross spine switches.
Queue Management
Algorithm T iillustrates queuing policy. It manages the job queue and reserve resources for the imminent LPJ. It also employs an ML-driven job completion time (JCT) predictor to balance the trade-off of queuing delay and resource utilization
JCT predictor. The JCT prediction enables opportunistically scheduling short-lived jobs to reserved resources long as they can finish before the arrival of LPJ. This helps improve resource utilization and decrease queuing delay. The prediction is based on metadata associated with jobs, such the number of requested CPUs and GPUs, the requested amount of drives, the department of
23

---

Algorithm Job scheduling policy
1: J: I/job's configurations and metadata
2: V: //physical view of the cluster
3: Q: //job queue, sorted by priority and arrival time
4: O: //delay list
5: function SCHEDULER(J, V. Q)
6: while True do
7:
O < 0 while Q + do J < Q.pop()
8:
9:
10:
preemplable()) then
11:
sched(J, V)
else if J.request < V.Jree_resoure then
13:
sched(J, V)
14:
else pred JJT(J) < arrival time then
15:
sched(J,V)
16:
else
17:
end (.ad((J)
18:
19:
end while
20:
Q ++0 O
21:
sleep(interval)
22:
end while
end function
task owners, etc. Although estimating the exact JCT is inherently difficult, we adopt coarse-grained forecasting strategy, which classifies the JCT into different time intervals.
To train the JCT predictor, we retrieve historical trace data from the database. Then, pre-process the data, such as removing jobs that are early killed by users, and divide the JCT into 10-minute intervals. We then train models to predict the interval into which incoming jobs may fall by the metadata associated with the jobs. We tried both deep neural network (DNN) and a gradient boosting predictor (GBM) and found that GBM achieves higher performance, likely due to its ability to handle categorical variables.
Uncertainty
Predicted
Actual
Predicted
Actual
15
15
10
10
5
5
0
0
12-12
12-13
12-14
12-15
12-16
12-17
12-18
12-19
12-20
12-12
12-13
12-14
12-15
12-16
12-17
12-18
12-19
12-20
(a) GBM: RMSE 1.61.
(b) DNN: RMSE: 2.12.
Figure 13: JCT prediction.
To demonstrate effectiveness, we extract 4 4-month trace data and divide them into a training set (90%) and test set (10%). Figure s3 an example of prediction in the test set. We apply randomized grid search to optimize hyper-parameters and also use bagging to determine uncertainty estimation. We observe that the RMSR is 1.61 the test set, and recent studies suggest that such prediction could help scheduling decision
42 3].
24

---

No-Reserve
No-JCT
Arnold
Retention (%) Allocation (%)
100
80
60
20
100
40
20
18:00
18:30
19:00
19:30
20:00
20:30
21:00
21:30
22:00
Figure 14: Allocation and retention rate over time since a LLM pre-training job is planned.
G Evaluation of Queue Management
We collect job traces and replay the trace-driven simulator. Figure 14 shows the allocation and retention rate time. The allocation rate is determined by dividing the total number of nodes by the nodes with allocated jobs. The retention rate measures how many planned nodes for the LLMs job occupied by other jobs, which inevitably requires manual preemption when the LLMs job arrives. At 18: 00, Arnold is told the arrival time of the LLMs job at 22 00, and so it triggers the code path plan and reserve resources accordingly. We use a default bin-pack algorithm schedule other jobs.
The allocation rate is 0.9 initially and then gradually decreases below 0.5 due to the need to reserve 1200+ nodes in the cluster. At the beginning of the imminent period, the retention rate is relatively high and matches the allocation rate because the previous allocated jobs the incoming LLMs pre-training job. Then the retention rate decreases faster than the allocation rate since nodes have been reserved, and is close of the scheduling policy.
aware of
the end of the period, showing the effectiveness
For the reserving-and- packing policy 42]. it does not offer strong semantics for reservation (i.e. best effort). Thus, the scheduler will not able to generate a feasible solution as the LLM job arrives, not to mention optimal placement (orange line). The JCT prediction navigates the rrfde-off space between resource utilization and guarantees by scheduling opportunisticall short-lived jobs to the reserved zone. In its absence, both queuing delays and resource idle times increase, indicated by the green line.
Example MIP solution
25

---

Table 3: The MIP solution values for setting (1)
771
Index
2
0
0.0
0.0
1.0
$_
0.0
0.0
1.0
$_2_
0.0
0.0
1.0
8_3
0.0
1.0
0.0
$ 4.4
0.0
1.0
0.0
$_5_
0.0
0.0
Index
2
P_0
0.0
0.0
1.0
P_
0.0
0.0
1.0
p_2_* *
0.0
0.0
1.0
P_3
0.0
1.0
0.0
P_4_* *
0.0
1.0
0.0
P_5
0.0
1.0
0.0
Index
Value
y[O]
0.0
y[1]
1.0
y[2]
1.0
I Evaluation on open-source models
Table 4: Average throughput (PetaFlops) comparison on Llama3 8B.
Scheduling algorithm
llama3 8B
Bin Bin-pack (used by MegaScale) GPU-pack
3.80
3.82
Ours
3.86
26

---

Break-down analysis
Dift. (ms)
250
404
115
95
16
14
RS-FP32
RS-BF16
All-Gather
cuBLAS
GEMM-I
GEMM-2
Figure 15: Breakdown analysis of the ablation experiment.
We compare aggregated kernel-level metrics by summing the duration for each type of kernel. Figure 10 [Oillustrates the speedup achieved by Amold (in green) and the slowdown (in red) of the full-scale experiment. We only report kernels whose difference is significant. The most significant speedup the broadeast kernel (10%), which the optimized P2P implementation of our communication library. However, the speedup is slightly offset by the slowdown of a reduce-scatter kernel and even a computational kernel. The slowdown contradicts our expectations, Arnold's scheduling also reduces the spread of DP groups. Moreover, we have not changed other configurations, so the slowdown of the GEMM kernel is unexpected.
After thorough investigation, we suspect the slowdown is due to the interference between GPUs' streams. Due to hybrid parallelism, GPUs maintain multiple streams that issue operations concurrently during training. Although overlapping computation with communication indicates good performance optimization, also causes resource contention and interference.
Network topology affects computation kernels. To investigate the counter-intuitive results, we isolate the impact of streams by modifying NCCL. For example, we add additional environmental variables such as NCCL DP MIN NCHANNELS to have fine-grained controls on the DP stream. We disable channel auto-tuning and rerun jobs with and without setting the NCCL variable. Figure [5] shows the breakdown analysis. Communication kernels have speedups by setting the NCCL variable, whereas computation kernels have slowdown. Since the only change is the NCCL variable, it indicates we allocate more GPU SMs to communication, computation kernels suffer from performance loss for less available SMs.
In production training, the NCCL variables are dynamically auto-tuned, so given that network topology-optimizn scheduling influences the communication of DP and PP groups, ultimately it causes variations the computation kernels.
27


Reviews:
[{"review_id": "tBR6nx2U1K", "reviewer": "Reviewer_dkH5", "summary": "This paper introduces a new scheduler, dubbed Arnold, for distributed training of LLMs. Arnold is responsible for determining how to assign accelerators to perform data parallelism and pipelining provided information about hardware topology and communication bandwidth. The scheduling problem is formalized as a mixed-integer program. The highlight of the empirical result shows an improvement of about 10% in throughput when training an unspecified LLM on 9600 GPUs compared to megascale, which is the current standard.", "questions": "- how much better is the enumeration approach in terms of throughput across various cluster configurations?", "limitations": "The major limitation of this paper is the absence of connection to learning itself. This is a contribution better suited for a system conference.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n+ topic is very timely\n+ paper is overall clearly written\n\nWeaknesses:\n- This is a system paper, the authors made not much effort to connect this to learning. I think a system venue would be more suitable than NeurIPS.\n- The empirical validation is lacking. I think throughput should have been used as a metric throughout (and not just in the last section). The time to schedule is absolutely negligible relative to the total training time. \n- The reported gain when using 9600 GPUs is a bit marginal. If large runs take a month, let's say, Arnold let us save only 3 days. It is unclear to me whether this is because the scheduling is sub-optimal or mega-scale is close to optimal already.\n- The paper lacks insights a bit. It would have been nice to show how different hardware topologies / cluster configurations yield / benefit from different scheduling.\n- When scaling to very large models, a node may not be sufficient to host the entire model and tensor parallelism might be needed across nodes, but the paper considers only DP and PP.", "was_revised": true, "final_justification": "After reading the rebuttal and the other reviews, I have decided to keep a borderline rating but leaning towards acceptance, as there is nothing flawed in the paper as far as I can tell. \nI think that if the authors include the clarifications from the reviews (e.g., why gains are significant, articulate better how Arnold adapts to various topologies), the paper might get better received.", "rebuttal": "> **w1: This is a system paper, the authors made not much effort to connect this to learning. I think a system venue would be more suitable than NeurIPS.**\n\n**Scope of our paper.** The topic of \"**Infrastructure** (e.g., libraries, improved implementation and scalability, distributed solutions)\" is explicitly included in the call for paper statement in NeurIPS 2025. The scope of our paper falls in the **infrastructure track**, which includes improved implementation, scalability and distributed solutions. Other innovative solutions such as flash-attention, also have wide impacts on the NeurIPS community.\n\n**Connection to learning.** LLMs' learning is benefited by large-scale training, which necessitates innovative infrastructure support. In section 3 and 4, we share our insights and lessons learned from large-scale training on real-world clusters. In section 5, 6 and 7, we present a scheduling algorithm capable of handling very large-scale training jobs. We hope our work shed light on scalable training solutions and benefit the community.\n\n---\n\n> **w2: The empirical validation is lacking. I think throughput should have been used as a metric throughout (and not just in the last section). The time to schedule is absolutely negligible relative to the total training time.**\n\n**Throughput as the metric throughout the paper.** In the simulator described in section 7.1, we use Equation (2) as the score to benchmark scheduling algorithms. This is based on the observation from Figure 4 and 5. Alternatively, we can replace the metric in the simulator by throughput via using the characterization results from section 4 (line 141-155). For example, as communication groups span over more minipods, the bandwidth decreases from the maximum bandwidth, and we use the percentage of the decrease to estimate the decrease of throughput from maximum throughput.\n\n\nsetting (i) throughput (PetaFlops)\n\n| Method       | α = 0.2  | α = 0.5  | α = 0.8  |\n|--------------|--------- |--------- |--------- |\n| Best-Fit     | 1.68     | 1.63     | 1.59     |\n| Random-Fit   | 1.55     | 1.52     | 1.49     |\n| GPU-Pack     | 1.68     | 1.63     | 1.59     |\n| Topo-Aware   | 1.68     | 1.63     | 1.59     |\n| Ours         | 1.68     | 1.63     | 1.59     |\n\n\nsetting (ii) throughput (PetaFlops)\n\n| Method       | α = 0.2  | α = 0.5  | α = 0.8  |\n|--------------|--------- |--------- |----------|\n| Best-Fit     | 12.57     | 12.57     | 12.57  |\n| Random-Fit   | 12.22     | 11.70     | 11.18  |\n| GPU-Pack     | 12.57     | 12.57     | 12.57  |\n| Topo-Aware   | 12.57     | 12.57     | 12.57  |\n| Ours         | 13.44     | 13.11     | 12.78  |\n\nsetting (iii) throughput (PetaFlops)\n\n| Method       | α = 0.2 | α = 0.5 | α = 0.8 |\n|--------------|---------|---------|---------|\n| Best-Fit     | 46.85     | 44.85     | 42.86     |\n| Random-Fit   | 46.19     | 43.19     | 40.20     |\n| GPU-Pack     | 47.16     | 45.62     | 44.08     |\n| Topo-Aware   | 47.16     | 45.62     | 44.08     |\n| Ours         | 50.51     | 47.71     | 44.92     |\n\n\n\n**The time to schedule is not negligible.** We wish to gently point out that the time to schedule is not negligible as shown in Figure 8. The enumeration method has time complexity $O(k^m)$, i.e., for each of the $m$ nodes, we can place it in one of the $k$ minipods. This is not feasible for online scheduling. Moreover, cloud providers price Hopper GPUs at \\$2.99/h. For our largest job, this is \\\\$28,704/hour or \\\\$478/min. GPUs can be idle during scheduling time, which directly leads to significant cost. Figure 8 shows that our method scales well for large job configurations and data center topology.\n\n---\n\n> **w3: The reported gain when using 9600 GPUs is a bit marginal. If large runs take a month, let's say, Arnold let us save only 3 days. It is unclear to me whether this is because the scheduling is sub-optimal or mega-scale is close to optimal already.**\n\n**Arnold's impact on cost savings.** We wish to highlight that a 10.6% speedup has a huge impact on cost savings. If 9,600 GPUs save 3 days, this saves us \\\\$2,066,688, because cloud providers price Hopper GPUs at \\$2.99/h. This improvement has a significant impact on cost-saving, swift productionization, and fast iteration.\n\n**MegaScale's scheduling is suboptimal.** MegaScale, which employs the bin-pack algorithm, ignores the inherent sparse communication patterns in LLM pre-training (shown in Figure 1 and 3), while Arnold's MIP algorithm specifically considers these patterns. In section 7, we demonstrate how the improved scheduling algorithm works well in both simulation and real-world environments.\n\n---\n\n> **w4: The paper lacks insights a bit. It would have been nice to show how different hardware topologies / cluster configurations yield / benefit from different scheduling.**\n\n**Benchmarks in different setups.** We have shown benchmarks of different hardware topologies (TOR switch, spine switch), cluster configurations (small, medium, large job sizes), and different scheduling algorithms (a list of SoTA schedulers from literacture) in section 7.1 (line 290-306).\n\n**Scheduling insights.** We have shown that when scheduling small jobs in a simple topology, scheduling algorithms achieve similar scores. As the job size and scale become larger, Arnold's algorithm shows more advantageous, and it is able to scale to very large sizes, as detailed in line 321-325, section 7.1:\n>> We also observe that as $\\alpha$ increases, our scheduling algorithm is closer to other baselines. This is because $\\alpha$ controls the affinity of the DP group, and if $\\alpha$ is $1$, the objective function reduces to a bin-packing formulation and therefore has no difference from other bin-packing algorithms. In practice, we would not set $\\alpha$ greater than $0.5$ as observed from our characterization results. As a result, our algorithm usually scores higher than other baselines.\n\n\n---\n\n> **w5: When scaling to very large models, a node may not be sufficient to host the entire model and tensor parallelism might be needed across nodes, but the paper considers only DP and PP.**\n\n**Clarification.** Our model is not hosted by one node. As discussed in section 7.2 (line 354-358), our training systems have trained LLMs with more than 400 billion parameters, and it is sharded by TP and PP across multiple nodes.\n\n**Why only PP and DP are considered.** TP must stay within high-bandwidth low-latency network domain (e.g. intra-node NVLink), as Megatron-LM [1] has in-depth experiments to show that. PP can scale out to inter-node connections. Therefore, to scale even larger than our models, more PP stages can be added to accommodate the memory pressure.\n\n[1] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron-lm. In SC21: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–14, 2021.\n\n---\n\n> **q1: how much better is the enumeration approach in terms of throughput across various cluster configurations?**\n\n**Compare with enumeration approach on setting (i).** We add additional experiments on setting (i) (line 302) comparing the throughput of the enumeration approach with our scheduling algorithm:\n\nsetting (i) throughput (PetaFlops)\n\n| Method       | α = 0.2  | α = 0.5  | α = 0.8  |\n|--------------|--------- |--------- |--------- |\n| Best-Fit     | 1.68     | 1.63     | 1.59     |\n| Random-Fit   | 1.55     | 1.52     | 1.49     |\n| GPU-Pack     | 1.68     | 1.63     | 1.59     |\n| Topo-Aware   | 1.68     | 1.63     | 1.59     |\n| Ours         | 1.68     | 1.63     | 1.59     |\n| Enumeration  | 1.68     | 1.63     | 1.59     |\n\nAs shown in the table, our scheduling algorithm can match the performance of the enumeration method with much shorter scheduling time.\n\n**Compare on other settings.** For setting (ii) and (iii), the algorithm has complexities of $O(5^{96})$ and $O(11^{368})$. They are computationally intractable."}, {"review_id": "IG3tab5HKR", "reviewer": "Reviewer_awuk", "summary": "This paper introduces a system (Arnold) that improves LLM pre-training through a scheduling algorithm that takes into account data center topology to be more communication-aware. Claimed contributions are investigations into existing data center training, introduction of the design of Arnold, as well a resource scheduling policy, and evaluations of this system. The paper goes through various motivating experiments before introducing the scheduling algorithm.", "questions": "### Major\n\nWhy isn’t equation (4) symmetric? $\\alpha$ and $\\beta$ are symmetric, but T is the max spread across PP communicator groups as I understand, but for DP communicator groups all the minipods are summed. Wouldn’t it make more sense to have a max spread measure for this as well?\n\nWhy isn't $T$ penalized non-linearly?\n\nAre there any experiments against an external baseline (MegaScale) with experimental settings that can be disclosed?\n\n### Minor\n\nIs there any explanation for why all gather and send/recv bandwidths degrade over multiple minipods? Is this due to NCCL using a different algorithm? Perhaps the algorithm isn’t being selected intelligently going from 4 to 5 minipods.\n\nLines 175-177 discuss how training is insensitive to intra-minipod topology. But Figure 3 shows an intra-minipod example. Why is this?", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "I would first like to thank the authors for submitting their work to NeurIPS 2025.\n\n### Pros\n\nThis paper is interesting as it identifies two key problems with current cluster schedulers, namely the misalignment of schedulers with data center topology, and PP/DP placement trade off.\n\nThe background is overall clear, and has a good exposition for DP/PP/TP. It also includes easy to understand figures for Hybrid parallelism and an example of a data center topology.\n\nSection 3 and 4 also add additional exposition for the reader, and set up the problem in a very nice way. I like the background given on both the data center topology as well as the motivating experiments. Particularly, the communication operation performance and E2E training performance motivating examples are useful for understanding. Additionally, the E2E training performance motivating example is taken over a few different settings including MoE and Dense models.\n\nThe system design/architecture makes sense, and seems like it would work in practice. \n\n### Cons\n\nIn line 82 (DP), it can be made more clear that this is different from pure DP and is in fact FSDP. It would also be good to at least spend a couple of lines on some form of context/sequence parallelism to show that that’s been acknowledged even though that will not make a difference from the DP/PP perspective.\n\nIn figure 4(a), it is not clear where the bandwidth is measured between, i.e. is this within a minipod.\n\nI think the biggest issue with sections 3 and 4 is that it would be good to show what the alignment pattern is with SoTA communication frameworks - i.e. demonstrate that this is not actually what occurs in practice today, as it seems it would be fairly simple to align by PP or DP.\n\nI may have misunderstood, but it seems that section 4 implies that $T$ should be penalized in a non-linear manner as collective performance varies with the number of minipods in a nonlinear manner. This is not reflected in the MIP.\n\nAlthough the distance to nearest database entry in section 5.2 makes sense, this seems like it will be inaccurate for models with a large euclidean distance with the current entries. In other words, this won’t generalize easily to other large training jobs (for example, different size, parameter shape, GPU) without running a characterization experiment.\n\nAnother issue here is the lack of discussion regarding related work. I see this is in Appendix A, but should be part of the main paper, especially since the experimental results are only against 1 external baseline. In addition, it is hard to make sense of the results when the specific number of GPUs and model size is hidden. In my humble opinion, a much shorter job, but with an open source model and arbitrary number of GPUs would make much more sense as an experiment from a reproducibility and use point of view.\n\n---\n\nOverall, this paper is a welcome study in pre-training efficiency, and I am looking forward to the open sourced code. Clarity is good, with sufficient background introduced. I am concerned about whether the theoretical formulation of the MIP makes sense (and hope this will be clarified in the rebuttal) but more importantly, whether there is enough significance/novelty as a research work suitable for NeurIPS - the main research contributions are elucidating the DP/PP misalignment and introducing the MIP although there is clearly a significant amount of engineering that went into the work.\n\n#### Other Comments\n\nNits:\n\nLine 109 - include Infiniband bandwidth if possible.\n\nFigure 3(a) - Suggest to call this “Fully misaligned”\n\nSection 3 - Add a forward pointer to section 4 E2E performance.\n\nFigure 2(b) can also be made clearer - what is S-SW, L-SW, H, etc? Readers familiar with data center topology will know about spine-leaf topology. Should be stated in the caption or text. Also, In lines 100-109 the notation s0 is used for leaf, and s1 for spine, but this is nowhere in the figures.\n\nLine 199 - assignment -> assignment(s)\n\nEquation (3). Notation is a little confusing.\n\nLine 265 - JCT should be full-form the first time. JCT (Job completion time).", "was_revised": true, "final_justification": "Overall, this paper is strong. I am still on the fence about it.\n\nI reiterate my statement from the Strengths and Weaknesses section:\n\n> Overall, this paper is a welcome study in pre-training efficiency, and I am looking forward to the open sourced code. Clarity is good, with sufficient background introduced. I am concerned about whether the theoretical formulation of the MIP makes sense (and hope this will be clarified in the rebuttal) but more importantly, whether there is enough significance/novelty as a research work suitable for NeurIPS - the main research contributions are elucidating the DP/PP misalignment and introducing the MIP although there is clearly a significant amount of engineering that went into the work.\n\nThe authors have satisfied me with their explanation of the MIP.\n\nStrengths:\n\n- Great speedup\n- Well written\n- Good motivational examples\n- Theoretical formulation makes sense\n- Timely topic\n- Great fit for NeurIPS\n\nCons:\n\n- Not sure there is enough novel material and insight.\n- Still not very satisfied with the E2E examples even with Llama 8B. Would prefer more examples.", "rebuttal": "> **q1: Why isn’t equation (4) symmetric? $\\alpha$ and $\\beta$ are symmetric, but $T$ is the max spread across PP communicator groups as I understand, but for DP communicator groups all the minipods are summed. Wouldn’t it make more sense to have a max spread measure for this as well?**\n\n**Asymmetry of Equation (4).** Equation (2) is symmetric because each node is a scheduling unit. Equation (4) is asymmetric because we coarsen the communication group (e.g. PP) as a scheduling unit. After the coarsening, we lose the granularity to express the spread of DP groups directly, but we can still control that by minimizing the usage of minipods.\n\n**A simple example.** Suppose we have $m$ DP groups and $n$ PP groups, after the coarsening, the scheduling problem becomes “how to put the $n$ unit into $k$ minipods?” - we are allowed to spread one unit across multiple minipods, but we want to minimize the spread, hence introducing the $T$ auxiliary variable. On the other hand, DP groups’ spread is “orthogonal” to PP group – GPUs in the same stage of different PP groups belong to the same DP group, so if each unit (PP) is packed into a different minipod (the optimal $T$), the spread of DP groups is maximized. Thus, we also minimize the overall usage of minipod, which controls the spread of DP groups. Equation (4) combines the two objectives. \n\n--- \n\n> **q2: Why isn't $T$ penalized non-linearly?**\n\nWe acknowledge $T$ could be penalized non-linearly because in Figure 4, it has non-linear bandwidth degradation. We have also thought about constraint $T<=4$. However, it is not feasible in production scenario.\n\nWe choose the current solution due to multi-tenancy. A running cluster has dynamic job submission, hardware failure and bursty network flows. A hard constraint $T<=4$ may fail to solve at real time and it is unacceptable. A soft constraint requires quantification of bandwidth degradation, which is often interfered by other jobs, as shown in Figure 11. Besides, soft constraints are hard to express for MIP solvers. \n\nAnother issue is the resource contention in Figure 10 – computation and communication kernels are overlapped for maximized efficiency during training, but they also interfere with each other, which exacerbates the difficulty of quantifying the bandwidth decay.\n\n---\n\n> **q3: Are there any experiments against an external baseline (MegaScale) with experimental settings that can be disclosed?**\n\n\n\nYes. We disclose the following information: \n\n| Model Size | Heads | Hidden Size | Layers | Experts  | \n|------------|-------|-------------|--------|----------|\n| 30B(medium)| 32   | 4096        | 8     | 128      | \n| 440B(large)| 32   | 6144        | 80     | 128      |\n\nThe experiment settings at framework level are the same, i.e.,\n\n- overlapping DP, TP and PP.\n- developping high-performance operators.\n- applying asynchronous data pipeline.\n- tuning the network performance. \n\nThe difference is the scheduling algorithm, and it is the goal of this paper to present a scheduling solution for LLM pre-training. Framework-level optimization is orthogonal. \n\n---\n\n> **q4: Is there any explanation for why all gather and send/recv bandwidths degrade over multiple minipods? Is this due to NCCL using a different algorithm? Perhaps the algorithm isn’t being selected intelligently going from 4 to 5 minipods.**\n\n**Degradation reasons.** The degradation is due to the increased routing distance and the switch is overwhelmed. The bandwidth degradation is expected as communication spans more minipods due to the increased physical routing distance. The specific 4-5 minipod degradation may be due to the switch is overwhelmed by the hundreds of bursty flows generated by the communication group over a short period of time, so the ECMP load balancing algorithm is not effective. The flows collide to certain links and cause link saturation, and the saturated link becomes the straggler to slow down the entire communication process. This means the scheduling algorithm needs careful design, not just relying on the network tuning.\n\n**NCCL algorithms and network tuning.** NCCL uses ring algorithm for the highest bandwdith. The algorithm is tuned by NCCL's cost model. Our network software stack has been tuned meticulously, as observed from the S-curves over the message sizes in Figure 4 (a), which is detailed in NVIDIA’s NCCL performance tuning technical blog.\n\n---\n\n> **q5: Lines 175-177 discuss how training is insensitive to intra-minipod topology. But Figure 3 shows an intra-minipod example. Why is this?**\n\nFigure 3 is an illustrative example of the alignment of communication and topology. The network architecture is hierarchical, so the illustrative example can be naturally extended to higher layer in the hierarchy. We will refine the figure to be clear about the network topology.\n\n---\n\n> **con1: In line 82 (DP), it can be made more clear that this is different from pure DP and is in fact FSDP**\n\nIt is DP, because FSDP is zero $3$ strategy that shards weights, gradients and optimizer states across all processes in DP groups. It introduces large communication overhead during forward pass. Our DP implementation is zero $1$.\n\n---\n\n> **con2: In figure 4(a), it is not clear where the bandwidth is measured between, i.e. is this within a minipod.**\n\nThis is measured between minipods. We will clarify this in the text. \n\n---\n\n> **con3: I think the biggest issue with sections 3 and 4 is that it would be good to show what the alignment pattern is with SoTA communication frameworks**\n\nExisting SoTA frameworks use the bin-pack strategy, and it has poor alignment as shown in Figure 3. The misalignment issue of a small training job is less significant due to the small data volume. In Figure $5 (b)$, we empirically observe smaller jobs suffer less from misaligned performance degradation.\n\nHowever, due to the scale of LLM training, the communication volume increases and the communication patterns become very sparse, as discussed in Figure 1. Thus, the misalignment problem is much more severe, and LLM pre-training needs a more specific scheduling algorithm.\n\n---\n\n> **con4: this won’t generalize easily to other large training jobs (for example, different size, parameter shape, GPU) without running a characterization experiment.**\n\nOur scheduling algorithm is LLM-specific and LLMs have unified model architecture (decoder-only architecture), increased model sizes by stacking more layers (7B, 33B, 70B), and hybrid parallelism. This makes it easy to pre-characterized automatically and generate database entries, i.e. launch cron-jobs and record the throughput.\n\n---\n\n> **con5: Another issue here is the lack of discussion regarding related work. I see this is in Appendix A, but should be part of the main paper, especially since the experimental results are only against 1 external baseline. In addition, it is hard to make sense of the results when the specific number of GPUs and model size is hidden.**\n\nWe will move the discussion of related work to main body in the final version with an extra page. We will add more discussion of exising DL scheduling, and why they are not sufficient for LLM pre-training jobs - mainly due to ignoring the sparse communication patterns as shown in Figure 1.\n\nAdditionally to the response to **q3**, where we release details model configurations and framework-level experiment settings. The largest job uses $1216$ nodes and each node has $8$ GPUs. \n\n--- \n\n> **con6: I am concerned about whether the theoretical formulation of the MIP makes sense (and hope this will be clarified in the rebuttal) but more importantly, whether there is enough significance/novelty as a research work suitable for NeurIPS**\n\nThe theoretical formualtion is clarified in the response to **q1**. \n\n**Suitability.** The topic of \"Infrastructure (e.g., libraries, improved implementation and scalability, distributed solutions)\" is explicitly included in the call for paper statement in NeurIPS 2025. The scope of our paper falls in the infrastructure track, which includes improved implementation, scalability and distributed solutions.\n\n**Significance/novelty.** LLMs' learning is benefited by large-scale training, which necessitates innovative infrastructure support. In section 3 and 4, we share our insights and lessons learned from large-scale training on real-world clusters. In section 5, 6 and 7, we present a scheduling algorithm capable of handling very large-scale training jobs. We hope our work shed light on scalable training solutions and benefit the community.\n\n---\n\n> **Other comment:**\n\nWe thank the reviewer for pointing out the grammar issues and lack of explanation in the caption. We will fix the grammar and add more text in the captions to make the terminology clear."}, {"review_id": "PPDMGVOKjz", "reviewer": "Reviewer_e9bD", "summary": "LLM training spends a significant amount of time on collective communication. However, existing infrastructures are not optimized for this type of communication pattern, where GPUs mostly communicate within groups. The key challenge is that data parallelism (DP) and pipeline parallelism (PP) are orthogonal, making it hard to arrange workloads such that all traffic avoids the spine network. This paper proposes **Arnold**, a workload-aware training framework that uses Integer Linear Programming (ILP) to search for optimal job placement that minimizes inter-minipod communication, achieving real-world speedups.", "questions": "Thanks for submitting to NeurIPS. The paper motivates the problem of misaligned GPU communication requirements with the underlying hardware, which is both educational and practically important. I enjoyed reading the paper. Overall, I think it addresses a timely and significant challenge. I have a few clarifications I’d like to ask:\n\n1. **Figure 4 Analysis**  \n   Figure 4 shows the profiling results across different numbers of minipods. After 4 minipods, the performance degrades drastically. Why is this the case? Is it due to the physical connectivity between minipods?\n\n2. **Bandwidth vs. Latency Bottlenecks**  \n   Is the performance degradation primarily caused by limited bandwidth between minipods, or by increased latency due to additional switches in the communication path?\n\n3. **Scaling Model Size**  \n   The paper shows that adding more layers to increasing model size makes Arnold more beneficial. Could you clarify why this is the case? Although the network becomes more important, compute cost also increases with model size.\n\n4. **ILP Approximation Details**  \n   The ILP formulation in equations (2) and (3) is clear. However, I am struggling to understand the approximation version in equations (4)–(10). Could you elaborate more on the rationale and implementation behind these approximations?\n\n5. **Network Topology Co-design**  \n   The paper focuses on scheduling-side solutions. If you had the ability to redesign the data center network topology, what strategies would you suggest to better accommodate LLM training workloads?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The problem is well-motivated and critical for large-scale LLM training.\n- Evaluation is based on real-world training runs, which is a strong validation of the system.\n- The proposed method effectively balances tensor parallelism (TP) and data parallelism (DP).\n- Arnold is shown to scale to clusters with thousands of GPUs.\n\n## Weaknesses\n- The description of the ILP formulation is unclear in parts.\n- No example of the search output is presented, which would help readers better understand the solution.", "was_revised": true, "final_justification": "The idea of the paper is novel and important. While the clarity can be improved, the rebuttal response is satisfactory. I would like to keep the score.", "rebuttal": "> **w1: The description of the ILP formulation is unclear in parts.**\n\nWe apologize for the presentation and will add more clarification in the text.\n\nIn Equation (2) each node is a scheduling unit, while in Equation (4) the communication group (e.g. PP) becomes a scheduling unit. \n\n**A simple example.** Suppose we have $m$ DP group and $n$ PP group, after the coarsening, the scheduling problem becomes “how to put the $n$ unit into $k$ minipods?” - we are allowed to spread one unit across multiple minipods, but we want to minimize the spread, hence introducing the $T$ auxiliary variable. On the other hand, DP groups’ spread is “orthogonal” to PP group – GPUs in the same stage of different PP groups belong to the same DP group, so if each unit (PP) is packed into a different minipod (the optimal $T$), the spread of DP groups is maximized. Thus, we also minimize the overall usage of minipod, which controls the spread of DP groups. \n\nCombining the above, Equation (4) is our optimizing objective function.\n\nEquation (5) constraints the maximum spread of PP groups less or equal to $T$, and $T$ is being minimized. \n\nEquation (6) says that the allocation cannot exceed the capacity of a minipod, e.g. if a minipod has $10$ nodes free, we cannot allocate more than that. The capacity is normalized by the number of node in a PP group.\n\nEquation (7) says that while we can spread a PP group across minipods, the summation of the allocation percentage must be equal to $1$.\n\nEquation (8) says that if we allocate to a minipod, that it is considered used in the solution, i.e. $y_j = 1$. \n\nFinally, Equation (9) and (10) define the range of the binary variable $y_j$ and $s_{ij}$, and continuous variable $p_{ij}$.\n\n---\n\n> **w2: No example of the search output is presented, which would help readers better understand the solution.**\n\n\nWith the explanation in **w1**, the search output is the specific values of each variable, i.e. $y_j$, $s_{ij}$ and $p_{ij}$. For example, $y_j$ indicates whether the $j$-th minipod is used. $p_{0,0}$ tells us how many nodes of the $0$-th PP group should be allocated to the $0$-th minipod. Thus, we use the value of $p_{i,j}$ to know which PP group goes to which minipods. \n\n\n--- \n\n> **q1: Figure 4 Analysis**\n\nThe degradation is due to the increased routing distance and the switch is overwhelmed.\n\nThe bandwidth degradation is expected as communication spans more minipods due to the increased physical routing distance. The specific 4-5 minipod degradation may be due to the switch is overwhelmed by the hundreds of bursty flows generated by the communication group over a short period of time, so the ECMP load balancing algorithm is not effective. The flows collide to certain links and cause link saturation, and the saturated link becomes the straggler to slow down the entire communication process.\n\n---\n\n> **q2: Bandwidth vs. Latency Bottlenecks**\n\nAs discussed in the response to **q1**, the increased physical routing distance and the collision of network flows lead to performance degradation. This is reflected by the measured bandwidth. The theoretical bandwidth is the same for inter-minipods connection. \n\n\n---\n\n\n> **q3: Scaling Model Size**\n\nThe training is bottlenecked by communication, and the optimized topology benefits GPU utilizes resources for communication, which is more pronounced when both computation and communication sizes increase.\n\nComputation is much faster than communication for GPUs (TeraFlops vs GB/s). Thus, computation kernels are blocked by the dependent communication kernels. In Figure 10, we observe that an optimized topology benefits communication, which effectively optimizes the bottleneck of training. \n\n\n---\n\n> **q4: ILP Approximation Details**\n\nAs explained in response to **w1**, we have clarified the details of each equations:\n\nEquation (4) is the main objective function that minimizes the spread of PP groups and DP groups.\n\nEquation (5) constraints the maximum spread of PP groups less or equal to $T$, and $T$ is being minimized. \n\nEquation (6) says that the allocation cannot exceed the capacity of a minipod, e.g. if a minipod has $10$ nodes free, we cannot allocate more than that. The capacity is normalized by the number of node in a PP group.\n\nEquation (7) says that while we can spread a PP group across minipods, the summation of the allocation percentage must be equal to $1$.\n\nEquation (8) says that if we allocate to a minipod, that it is considered used in the solution, i.e. $y_j = 1$. \n\nFinally, Equation (9) and (10) define the range of the binary variable $y_j$ and $s_{ij}$, and continuous variable $p_{ij}$.\n\n---\n\n> **q5: Network Topology Co-design**\n\nWe think the presented algorithm is in principle able to handle other netowrk topology. \n\nWe observe that production GPU clusters follow the multi-tier design (Alibaba’s HPN, NVIDIA’s DGX). The collective communication is bottlenecked by the highest hierarchy - the inter-minipod communication in our case. In our case, we observe little impact of intra-minipod topology, however, we can also solve the MIP at the intra-minipod level if we can observe any performance degradation. In short, at every level of the network hierarchy, we can solve the MIP to find the optimal placement at that level."}, {"review_id": "6cILNYrMaZ", "reviewer": "Reviewer_ShkR", "summary": "This paper introduces Arnold, a topology-aware scheduling system designed to optimize the placement of large language model (LLM) pre-training jobs (LPJs) in modern GPU clusters. The authors argue that existing schedulers fail to consider the complex, high-volume but sparse communication patterns typical in LLM training, leading to inefficient resource usage and prolonged training times.", "questions": "Here are additional comments for weakness 1:\n\n1. After reading twice, I can basically understand that the DP and PP make topology-related problems in this paper different from earlier works like https://www.comp.nus.edu.sg/~hebs/course/cs6284/papers/zhang-sigmod19.pdf ,BriskStream: Scaling Data Stream Processing\non Shared-Memory Multicore Architectures. SIGMOD 2019\n\n2. But the understanding is fully formula-based and fragmented to me; it could be better if the authors use a concrete paragraph to explain intuitions and summarise the differences.", "limitations": "N.A.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. Demonstrates clear empirical benefits with both simulation and production-scale experiments.\n\n2. Introduces a principled scheduling model that adapts to diverse LLM configurations and network conditions.\n\n3. Well-integrated system with resource management and job queue handling, including ML-based job completion time prediction.\n\nWeakness:\n\n1. Presentation issues: the solution is novel after a careful examination of the formulation, but novelty is not discussed well at a glance.\n\n2. Lack of discussion about data compression and emerging networking architectures, which are also impactful in distributed training.", "was_revised": true, "final_justification": "I am satisfied with response and will keep my score", "rebuttal": "> **w1 + q1 + q2: Presentation issues. But the understanding is fully formula-based and fragmented to me; it could be better if the authors use a concrete paragraph to explain intuitions and summarise the differences.**\n\nWe apologize for the presentation issues, and will clarify in the text. \n\nMulti-core CPU parallelism (as in BriskStream) and inter-GPU parallelism (as in LLM training) both aim to use multiple processing units to speed up computation. However, they are fundamentally different due to their underlying memory architecture, communication mechanisms, and architectural granularity.\n\n**Memory architecture:** Multi-core Parallelism is primarily concerned with efficiently managing a shared memory space among cores, with the main challenge being the performance penalty of Non-Uniform Memory Access (NUMA). \n\nInter-GPU Parallelism is primarily concerned with orchestrating computation across multiple devices that have their own separate and distributed memory, with the main challenge being the high cost of explicitly transferring data between them.\n\n**Communication mechanisms:** Multi-core Parallelism happens implicitly through memory. When a producer operator writes a tuple to a queue and a consumer operator reads it, they are accessing a shared location in RAM. \n\nInter-GPU Parallelism is explicitly managed via libraries like NCCL (NVIDIA Collective Communications Library).\n\n**Architectural granularity:** Multi-core Parallelism is designed for task parallelism on a single tensor computation level. For example, matrix multiplication for tensors operations.\n\nInter-GPU Parallelism is designed for massive data parallelism on homogeneous, throughput-oriented, and highly regular workloads. For example, a group of communication processes.\n\n> **w2: Lack of discussion about data compression and emerging networking architectures, which are also impactful in distributed training.**\n\nWe will add more discussion on data compression pipeline and networking architectures. Reviewer is right about these factors can impact distributed training. \n\n**Data compression and processing.** Data processing typically takes place on CPU and causes GPU idle. We adopt MegaScale's optimization, by overlapping data processing with gradient synchronization. They can be fully overlapped because they take place in different devices and the data processing overhead can be hidden. \n\n**Emerging networking architectures.** Emerging network architectures have significant impacts on the future design of scheduling algorithms. We think the presented algorithm is in principle able to handle other netowrk topology. \n\nWe observe that production GPU clusters follow the multi-tier design (Alibaba’s HPN, NVIDIA’s DGX). The collective communication is bottlenecked by the highest hierarchy - the inter-minipod communication in our case. In our case, we observe little impact of intra-minipod topology, however, we can also solve the MIP in intra-minipod level if we can observe any performance degradation. \n\nThere are more dedicated network architectures exist for LLM training, e.g. rail-only. However, the inability of any-to-any connectivity make it unlikely to be adopted in production training clusters in the near future."}]

Author rebuttal:

Meta-review: This paper presents a new scheduling system for LLM pre-training. It identifies the mismatch between physical network topology and LLM pre-training jobs and effectively align the communication patterns with the network topology to optimize communication efficiency. Large-scale simulations and end-to-end physical experiments demonstrate the effectiveness of the proposed system. 

The reviewers agree that this MLSys work is well written, addressing a timely topic, and the evaluation validates the superiority in terms of training efficiency. The majority of reviewer’ comments have been addressed in the authors’ responses. Based on these, this paper is recommended for acceptance. The authors are suggested to include the important experiment results and discussion in the responses to the paper. Besides, the authors should add more clarifications about the reviewers’ remaining points, such as highlighting the technical novelty and insights, and adding more clarifications about the system design and empirical results.
Decision: Accept (poster)