Title: StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training
Venue: NeurIPS 2025
Accepted: yes
Keywords: Sequence Parallelism

Machine Learning System

Distributed Training
Abstract: Training Transformer models on long sequences in a distributed setting poses significant challenges in terms of efficiency and scalability. Current methods are either constrained by the number of attention heads or excessive communication overheads. To address this problem, we propose StarTrail, a multi-dimensional concentric distributed training system for long sequences, fostering an efficient communication paradigm and providing additional tuning flexibility for communication arrangements. Specifically, StarTrail introduces an extra parallel dimension and divides the peer-to-peer communication into sub-rings to substantially reduce communication volume and avoid bandwidth bottlenecks. Through comprehensive experiments across diverse hardware environments and on both Natural Language Processing (NLP) and Computer Vision (CV) tasks, we demonstrate that our approach significantly surpasses state-of-the-art methods that support Long sequence lengths, achieving performance improvements of up to 77.12% on GPT-style models and up to 114.33% on DiT (Diffusion Transformer) models without affecting the computations results.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes StarTrail, a new method to further reduce the communication overhead of training LLM with long sequences when the bandwidth between nodes and within nodes in a GPU cluster is different. Unlike Ring Attention, where each device processes only one input sequence block, StarTrail first groups multiple devices into a team to collaboratively process multiple sequence blocks, and then the intermediate results are aggregated to obtain the final result. The paper theoretically and empirically shows that StarTrail outperforms Ring Attention.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper studies an important problem.\n* The paper proposes a sequence parallelism method that does not rely on Attention-Head-Sharding.\n\nWeaknesses:\n* The paper does not discuss some important related work. For example, LoongTrain [1] proposes Double-Ring-Attention, which partitions the GPUs into multiple inner rings, to fully utilize available NICs for inter-node communication.\n* The theoretical analysis should not only compare StarTrail with Ring Attention but also include LoongTrain and Unified SP [2] (i.e., the combination of Ring Attention + DeepSpeed-Ulysses).\n* The experiments should not only compare StarTrail with Ring Attention but also include LoongTrain and Unified SP (i.e., the combination of Ring Attention + DeepSpeed-Ulysses).\n* Presentation could be improved.\n  * The order of Figure 2 and Figure 3 is reversed.\n  * There is no explanation for Figure 3 in the paper.\n  * The notations used in Algorithm 1 are different from those listed in Figure 5, which makes it difficult for readers to understand.\n\n[1] LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism  \n[2] USP: A Unified Sequence Parallelism Approach for Long Context Generative AI\n\nQUESTIONS:\nThe paper shows that StarTrail outperforms Ring Attention when there is a disparity between inter-node and intra-node bandwidths\nin GPU clusters. However, how does StarTrail compare with SoTA methods like LoongTrain and USP which are also robust to network topologies?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces StarTrail, a new system for training Transformer models on very long sequences. The method improves upon existing ring-based sequence parallelism techniques. StarTrail organizes GPUs into teams and creates multiple smaller communication rings, called sub-rings. This design reduces the total peer-to-peer communication volume, especially across slower network links between compute nodes. The authors demonstrate that StarTrail achieves significant throughput gains over the standard Ring Attention method on both language and vision models. This performance improvement comes at the cost of a modest increase in GPU memory usage.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The core idea of creating hierarchical communication rings is a very effective solution to a clear problem. The paper correctly identifies that Ring Attention's single large ring is often bottlenecked by slow inter-node communication. By creating sub-rings that can operate within high-speed intra-node networks, the method directly addresses this critical performance issue.\n\n2. The experimental evaluation is extensive and convincing. The authors test their method across multiple hardware types (A100, H100), different cluster topologies, and on two distinct Transformer architectures (GPT and DiT). This thorough testing on various setups provides strong evidence for the general applicability and robustness of their proposed system.\n\n3. The paper presents a clear analysis of the performance trade-offs. The theoretical breakdown of communication and memory costs helps readers understand the fundamental principles behind StarTrail. It clearly explains how the team size parameter C allows a user to balance peer-to-peer communication, collective communication, and memory overhead to optimize for a specific hardware environment.\n\nWeaknesses:\n\n1. The comparison to prior work seems incomplete. The paper does not experimentally compare against other recent and relevant methods for long context training, such as DistFlashAttention.\n\n2. The paper does not fully explore the scalability limitations of its own design. The All-Gather operation within each team could become a new bottleneck at a very large scale. The analysis does not discuss how the cost of this collective communication scales with the team size C or the total number of GPUs, nor does it address the C <= sqrt(P) constraint in depth.\n\n3. The method's performance appears to be highly dependent on the network topology, but this dependency is not explored. The benefits of StarTrail rely on mapping teams and sub-rings to the physical hardware layout, for instance, keeping a sub-ring within a single node. The authors do not discuss how the system would perform on clusters with less favorable or more complex topologies where this mapping is not possible.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, a novel sequence/context parallelism strategy called StarTrail is proposed. By introducing a new parallelism dimension, ring communication can be split into subrings, thereby reducing communication volume. This is particularly beneficial when inter-node communication is a bottleneck due to limited bandwidth. Evaluation results show that StarTrail significantly outperforms the original Ring Attention, achieving higher throughput across various settings.\n\nSTRENGTHS AND WEAKNESSES:\n1. The motivation and problem setting are reasonable: inter-node bandwidth is clearly a bottleneck for Ring Attention.\n\n2. The analysis and evaluation look promising.\n\n3. Inter-node and intra-node bandwidths differ significantly and can have distinct impacts. The analysis assumes a single bandwidth value W, which may undermine the correctness of the conclusions.\n\n4. The terminology is inconsistent: Team group, ring communication group, and subring appear to refer to the same concept but are used interchangeably.\n\n5. Only values of C = 2 and C = 4 are discussed. With only two data points, it's difficult to assess the full impact of C. Including results for C = 8 (especially for 64 or more GPUs) would be helpful. Additionally, further discussion on why C = 2 or C = 4 performs better across different settings would improve understanding.\n\n6. There is no discussion on convergence. Including a training curve comparison would provide the most straightforward evidence of the correctness of the proposed algorithm.\n\n7. The topology hierarchy is unclear. Based on my understanding, the hierarchy and corresponding communication are as follows:\n\n    Team - C GPUs - all-gather\n\n    Team group - P / C² teams - subring communication\n\n    What is the communication pattern between the C team groups?\n\nQUESTIONS:\n1. How does inter-node bandwidth affect performance? Intuitively, with higher bandwidth, the improvement might be limited since communication is no longer the bottleneck - especially if communication can be overlapped with computation. Some discussion or insight into the relationship between bandwidth and exposed communication would be helpful.\n\n2. Does the evaluation assume pure context parallelism (i.e., CP degree equal to the number of GPUs)? If so, this may not be a practical setup and could result in a weaker baseline, as hybrid parallelism generally yields better throughput or MFU. For example, the open-source training framework NeMo provides a training recipe for LLaMA3-128k [1], where the CP degree is only 8. A total CP degree much higher than 8 may not be realistic.\n\n3. In Figure 8, why is the rightmost subfigure normalized to 0.5 instead of 1?\n\n[1] http://github.com/NVIDIA/NeMo/blob/main/nemo/collections/llm/recipes/llama3_8b_128k.py\n\nLIMITATIONS:\nSee Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a training system named StarTrail, which uses a concentric ring parallelism scheme to train Transformer models on near-infinite-context sequences. Specifically, the method introduces an additional parallel dimension by grouping GPUs into teams and divides the communication workload into multiple sub-rings. This approach reduces the total peer-to-peer communication volume and avoids bottlenecks caused by slower inter-node connections. The authors conduct experiments on both Natural Language Processing (GPT) and Computer Vision (DiT) models using various hardware clusters equipped with Nvidia A100 and H100 GPUs. The experiments show that StarTrail achieves performance improvements over existing methods without affecting the computational results.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\nThe proposed method improves training throughput for long-sequence Transformer models. By dividing communication into concentric sub-rings, it reduces P2P communication volume and achieves performance speedups.\n\n**Weaknesses**\n\n1. The primary weakness in terms of significance is that StarTrail is an incremental improvement upon a specific existing method, Ring Attention.  It is a systems-level optimization that re-architects the communication pattern rather than a fundamental breakthrough in attention algorithms.Its scope is confined to this specific paradigm.\n\n2. The core idea involves dividing GPUs into teams and forming sub-rings. While Figures 4 and 6 are helpful, the description of the initialization process and the mapping of ranks to sub-rings and teams can be difficult to follow on a first read. The logic is spread between the main text, figures, and algorithms in the appendix, which could be integrated more smoothly to improve comprehension.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}