Title: Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models Training

Data Center Topology

Resource Scheduling
Abstract: The scaling law for large language models (LLMs) depicts that the path towards machine intelligence necessitates training at large scale. Thus, companies continuously build large-scale GPU clusters, and launch training jobs that span over thousands of computing nodes. However, LLM pre-training presents unique challenges due to its complex communication patterns, where GPUs exchange data in sparse yet high-volume bursts within specific groups. Inefficient resource scheduling exacerbates bandwidth contention, leading to suboptimal training performance. This paper presents Arnold, a scheduling system summarizing our experience to effectively align LLM communication patterns to data center topology at scale. In-depth characteristic study is performed to identify the impact of physical network topology to LLM pre-training jobs. Based on the insights, we develop a scheduling algorithm to effectively align communication patterns to physical network topology in data centers. Through simulation experiments, we show the effectiveness of our algorithm in reducing the maximum spread of communication groups by up to $1.67$x. In production training, our scheduling system improves the end-to-end performance by $10.6\%$ when training with more than $9600$ Hopper GPUs, a significant improvement for our training pipeline.


Peer reviews:
comments: SUMMARY:
This paper introduces Arnold, a topology-aware scheduling system designed to optimize the placement of large language model (LLM) pre-training jobs (LPJs) in modern GPU clusters. The authors argue that existing schedulers fail to consider the complex, high-volume but sparse communication patterns typical in LLM training, leading to inefficient resource usage and prolonged training times.

STRENGTHS AND WEAKNESSES:
Strengths:

1. Demonstrates clear empirical benefits with both simulation and production-scale experiments.

2. Introduces a principled scheduling model that adapts to diverse LLM configurations and network conditions.

3. Well-integrated system with resource management and job queue handling, including ML-based job completion time prediction.

Weakness:

1. Presentation issues: the solution is novel after a careful examination of the formulation, but novelty is not discussed well at a glance.

2. Lack of discussion about data compression and emerging networking architectures, which are also impactful in distributed training.

QUESTIONS:
Here are additional comments for weakness 1:

1. After reading twice, I can basically understand that the DP and PP make topology-related problems in this paper different from earlier works like https://www.comp.nus.edu.sg/~hebs/course/cs6284/papers/zhang-sigmod19.pdf ,BriskStream: Scaling Data Stream Processing
on Shared-Memory Multicore Architectures. SIGMOD 2019

2. But the understanding is fully formula-based and fragmented to me; it could be better if the authors use a concrete paragraph to explain intuitions and summarise the differences.

LIMITATIONS:
N.A.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLLM training spends a significant amount of time on collective communication. However, existing infrastructures are not optimized for this type of communication pattern, where GPUs mostly communicate within groups. The key challenge is that data parallelism (DP) and pipeline parallelism (PP) are orthogonal, making it hard to arrange workloads such that all traffic avoids the spine network. This paper proposes **Arnold**, a workload-aware training framework that uses Integer Linear Programming (ILP) to search for optimal job placement that minimizes inter-minipod communication, achieving real-world speedups.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The problem is well-motivated and critical for large-scale LLM training.\n- Evaluation is based on real-world training runs, which is a strong validation of the system.\n- The proposed method effectively balances tensor parallelism (TP) and data parallelism (DP).\n- Arnold is shown to scale to clusters with thousands of GPUs.\n\n## Weaknesses\n- The description of the ILP formulation is unclear in parts.\n- No example of the search output is presented, which would help readers better understand the solution.\n\nQUESTIONS:\nThanks for submitting to NeurIPS. The paper motivates the problem of misaligned GPU communication requirements with the underlying hardware, which is both educational and practically important. I enjoyed reading the paper. Overall, I think it addresses a timely and significant challenge. I have a few clarifications I’d like to ask:\n\n1. **Figure 4 Analysis**  \n   Figure 4 shows the profiling results across different numbers of minipods. After 4 minipods, the performance degrades drastically. Why is this the case? Is it due to the physical connectivity between minipods?\n\n2. **Bandwidth vs. Latency Bottlenecks**  \n   Is the performance degradation primarily caused by limited bandwidth between minipods, or by increased latency due to additional switches in the communication path?\n\n3. **Scaling Model Size**  \n   The paper shows that adding more layers to increasing model size makes Arnold more beneficial. Could you clarify why this is the case? Although the network becomes more important, compute cost also increases with model size.\n\n4. **ILP Approximation Details**  \n   The ILP formulation in equations (2) and (3) is clear. However, I am struggling to understand the approximation version in equations (4)–(10). Could you elaborate more on the rationale and implementation behind these approximations?\n\n5. **Network Topology Co-design**  \n   The paper focuses on scheduling-side solutions. If you had the ability to redesign the data center network topology, what strategies would you suggest to better accommodate LLM training workloads?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a system (Arnold) that improves LLM pre-training through a scheduling algorithm that takes into account data center topology to be more communication-aware. Claimed contributions are investigations into existing data center training, introduction of the design of Arnold, as well a resource scheduling policy, and evaluations of this system. The paper goes through various motivating experiments before introducing the scheduling algorithm.\n\nSTRENGTHS AND WEAKNESSES:\nI would first like to thank the authors for submitting their work to NeurIPS 2025.\n\n### Pros\n\nThis paper is interesting as it identifies two key problems with current cluster schedulers, namely the misalignment of schedulers with data center topology, and PP/DP placement trade off.\n\nThe background is overall clear, and has a good exposition for DP/PP/TP. It also includes easy to understand figures for Hybrid parallelism and an example of a data center topology.\n\nSection 3 and 4 also add additional exposition for the reader, and set up the problem in a very nice way. I like the background given on both the data center topology as well as the motivating experiments. Particularly, the communication operation performance and E2E training performance motivating examples are useful for understanding. Additionally, the E2E training performance motivating example is taken over a few different settings including MoE and Dense models.\n\nThe system design/architecture makes sense, and seems like it would work in practice. \n\n### Cons\n\nIn line 82 (DP), it can be made more clear that this is different from pure DP and is in fact FSDP. It would also be good to at least spend a couple of lines on some form of context/sequence parallelism to show that that’s been acknowledged even though that will not make a difference from the DP/PP perspective.\n\nIn figure 4(a), it is not clear where the bandwidth is measured between, i.e. is this within a minipod.\n\nI think the biggest issue with sections 3 and 4 is that it would be good to show what the alignment pattern is with SoTA communication frameworks - i.e. demonstrate that this is not actually what occurs in practice today, as it seems it would be fairly simple to align by PP or DP.\n\nI may have misunderstood, but it seems that section 4 implies that $T$ should be penalized in a non-linear manner as collective performance varies with the number of minipods in a nonlinear manner. This is not reflected in the MIP.\n\nAlthough the distance to nearest database entry in section 5.2 makes sense, this seems like it will be inaccurate for models with a large euclidean distance with the current entries. In other words, this won’t generalize easily to other large training jobs (for example, different size, parameter shape, GPU) without running a characterization experiment.\n\nAnother issue here is the lack of discussion regarding related work. I see this is in Appendix A, but should be part of the main paper, especially since the experimental results are only against 1 external baseline. In addition, it is hard to make sense of the results when the specific number of GPUs and model size is hidden. In my humble opinion, a much shorter job, but with an open source model and arbitrary number of GPUs would make much more sense as an experiment from a reproducibility and use point of view.\n\n---\n\nOverall, this paper is a welcome study in pre-training efficiency, and I am looking forward to the open sourced code. Clarity is good, with sufficient background introduced. I am concerned about whether the theoretical formulation of the MIP makes sense (and hope this will be clarified in the rebuttal) but more importantly, whether there is enough significance/novelty as a research work suitable for NeurIPS - the main research contributions are elucidating the DP/PP misalignment and introducing the MIP although there is clearly a significant amount of engineering that went into the work.\n\n#### Other Comments\n\nNits:\n\nLine 109 - include Infiniband bandwidth if possible.\n\nFigure 3(a) - Suggest to call this “Fully misaligned”\n\nSection 3 - Add a forward pointer to section 4 E2E performance.\n\nFigure 2(b) can also be made clearer - what is S-SW, L-SW, H, etc? Readers familiar with data center topology will know about spine-leaf topology. Should be stated in the caption or text. Also, In lines 100-109 the notation s0 is used for leaf, and s1 for spine, but this is nowhere in the figures.\n\nLine 199 - assignment -> assignment(s)\n\nEquation (3). Notation is a little confusing.\n\nLine 265 - JCT should be full-form the first time. JCT (Job completion time).\n\nQUESTIONS:\n### Major\n\nWhy isn’t equation (4) symmetric? $\\alpha$ and $\\beta$ are symmetric, but T is the max spread across PP communicator groups as I understand, but for DP communicator groups all the minipods are summed. Wouldn’t it make more sense to have a max spread measure for this as well?\n\nWhy isn't $T$ penalized non-linearly?\n\nAre there any experiments against an external baseline (MegaScale) with experimental settings that can be disclosed?\n\n### Minor\n\nIs there any explanation for why all gather and send/recv bandwidths degrade over multiple minipods? Is this due to NCCL using a different algorithm? Perhaps the algorithm isn’t being selected intelligently going from 4 to 5 minipods.\n\nLines 175-177 discuss how training is insensitive to intra-minipod topology. But Figure 3 shows an intra-minipod example. Why is this?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new scheduler, dubbed Arnold, for distributed training of LLMs. Arnold is responsible for determining how to assign accelerators to perform data parallelism and pipelining provided information about hardware topology and communication bandwidth. The scheduling problem is formalized as a mixed-integer program. The highlight of the empirical result shows an improvement of about 10% in throughput when training an unspecified LLM on 9600 GPUs compared to megascale, which is the current standard.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ topic is very timely\n+ paper is overall clearly written\n\nWeaknesses:\n- This is a system paper, the authors made not much effort to connect this to learning. I think a system venue would be more suitable than NeurIPS.\n- The empirical validation is lacking. I think throughput should have been used as a metric throughout (and not just in the last section). The time to schedule is absolutely negligible relative to the total training time. \n- The reported gain when using 9600 GPUs is a bit marginal. If large runs take a month, let's say, Arnold let us save only 3 days. It is unclear to me whether this is because the scheduling is sub-optimal or mega-scale is close to optimal already.\n- The paper lacks insights a bit. It would have been nice to show how different hardware topologies / cluster configurations yield / benefit from different scheduling.\n- When scaling to very large models, a node may not be sufficient to host the entire model and tensor parallelism might be needed across nodes, but the paper considers only DP and PP.\n\nQUESTIONS:\n- how much better is the enumeration approach in terms of throughput across various cluster configurations?\n\nLIMITATIONS:\nThe major limitation of this paper is the absence of connection to learning itself. This is a contribution better suited for a system conference.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}