StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training
Authors: Ziming Liu
Shaoyu Wang
Shenggan Cheng
Zhongkai Zhao
Kai Wang
Xuanlei Zhao
James Demmel
Yang You
Venue: NeurIPS 2025 poster ()
Keywords: Sequence Parallelism
Machine Learning System
Distributed Training
Primary area: infrastructure
TL;DR: Concentric Ring Parallelism that reduces P2P communication volume and avoid inter-node communication.

Abstract: Training Transformer models on long sequences in a distributed setting poses significant challenges in terms of efficiency and scalability. Current methods are either constrained by the number of attention heads or excessive communication overheads. To address this problem, we propose StarTrail, a multi-dimensional concentric distributed training system for long sequences, fostering an efficient communication paradigm and providing additional tuning flexibility for communication arrangements. Specifically, StarTrail introduces an extra parallel dimension and divides the peer-to-peer communication into sub-rings to substantially reduce communication volume and avoid bandwidth bottlenecks. Through comprehensive experiments across diverse hardware environments and on both Natural Language Processing (NLP) and Computer Vision (CV) tasks, we demonstrate that our approach significantly surpasses state-of-the-art methods that support Long sequence lengths, achieving performance improvements of up to 77.12% on GPT-style models and up to 114.33% on DiT (Diffusion Transformer) models without affecting the computations results.

StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-InfiniteConneex Transformer Model Training
Ziming Liu" National University of Singapore liuziming@comp mus edw.eduu.s.
Shaoyu Wang" University of Southern California wangshao@usc edu
Shenggan Cheng
Zhongkai Zhao National University of Singapore zhongkai zhao@u. nus edu
National University of Singapore shenggan@comp nus edu. sg
Kai Wang
Xuanlei Zhao National University of Singapore xuanlei@comp mus. nus edu. sg
National University of Singapore kai wang@womnom.... nus edu.
James Demmel
Yang You National University of Singapore youydcomp. nus edu. sg
University of California, Berkeley demmel @berkelerreeeee edu
Abstract
Training Transformer models on long sequences in distributed setting poses significant challenges in terms of efficiency and scalability. Current methods are either constrained by the number of attention heads or excessive communication overheads. To address this problem, we propose StarTrail, a multi-dimeensinnl concentric distributed training system for long sequences, fostering efficient communication paradigm and providing additional tuning flexibility for communi- cation arrangements. Specifically, StarTrrail introduces an extra parallel dimension and divides the peer-to-peer communication into sub-rings to substantially reduce communication volume and avoid bandwidth bottlenecks. Through comprehensive experiments across diverse hardware environments and on both Natural Language Processing (NLP) and Computer Vision (CV) tasks, demonstrate that our ap- proach significantly surpasses state-of-the-ee methods that support Long sequence lengths, achieving performance improvements of up 77.12% GPT-style mod- els and up to 14.33% on DiT (Diffusion Transformer) models without affecting the computation results.
Introduction
Over the past decade, Transformer 338 models have made remarkable strides in diverse fields, including computer vision (CV) and natural language processing (NLP). As the technology has evolved, the ability to efficiently process long sequences with Transformer has emerged as a pivotal challenge. For instance, in text summarization, the ability to handle extensive sequences is vital, as the content to be summarized can range from lengthy chapters to entire books []] Similarly, applications, such ChatGPT [[] require the capacity to process extensive dialogue
Equaa Contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

histories to ensure conversational consistency. There are also applications in other fields like video generation [5] 30 and protein structure prediction [5 7
The long context in the above scenarios has introduced several challenges for model training and inference: 1) Efficiency and Adaptability. The challenge of efficiency predominantly lies in handling long sequences that require quadratic computations during attention, and in addressing the large amount of communication during distributed processing. 2) Memory. Besides the major obstacle of storing the model weight and optimizer states, the activation has also exceeded the capacity of single GPU and risen new memory challenge due to the extreme sequence length. 3) Scalability. Current Transformer models usually require thousands of GPUs for pre-training, even with datasets of regular lengths. For longer sequences, ensuring an acceptable scaling speedup rate with both the sequence length and the number of GPUs increasing is even more critical to reducing time and economic costs.
Traditional parallelisms such as Data Parallelism 12 37 [22 41). Tensor Parallelism 37
39
40], and
P2P Tatal Communication Volume
distribute the model, input batch, and the optimizer states, but not directly address the large memory requirement of extremely long sequences as the sequence length dimension remains unchanged. To break through this obstacle, Sequence Parallelism has been intro- duced, splitting the input on the sequence length di- mension. Mainstream Sequence Parallelism schemes can generally be classified into two categories, and are usually combined [II] to complement each other's s drawbacks. Methods like DeepSpeed Ulysses [14),
Pipeline Parallelism[]] [33 I0 23 25
Volume
60
StarTrail-2
StarTrail-1
80
64K
128K
256k
512K
IM
Training Sequence Length (Tiken)
DIT on Nvidia A100
GPT Nvidla H100
which are based on all-to-all communication, offer ef- ficiency but require the splitting of attention heads.
Hine
Star Irail-2
King
Star
Consequently, these methods are limited in scala- bility and can not be scaled to the number of attention heads. On the other hand, peer-to-peer communication methods [2 20), such as Ring Attention (24,, do allow for mear-infinite con- text lengths; however, they require the transmission of complete keys and values across all GPUs, leading significantly high communication loads. In sum- mary. there remains a deficiency in communication-effiee
Figure 1: StarTrail-2 and StarTrail-4 theo- retically save around 50% and 75% total P2P communication volume for various se- quence length and achieves up to 2x speedup in end-to-end training
devices than
methods that are capable of supporting near-infinite context lengths. In this paper, we focus on solving the communication inefficiency of ring-style sequence parallelism.
To solve these challenges, introduce StarTrail, novel wear-infinite-conee Transformer training system with concentric multi-ring sequence parallelism that incorporates an additional parallel dimension into the existing ring-style communication. Specifically, instead of including all GPUs in a single parallel group as done Ring Attention (24). StarTrail groups the GPUs into teams and divides the peer-to-peer communication within these teams. This approach fosters an efficient communication paradigm and provides extra tuning flexibility for communication arrangements. With very little additional memory cost, StarTrail parallelism significantly reduces the peer-to- peer communication volume, as shown in Figure T Compared to previous works, StarTrrall is not limited in supported sequence length by attention heads like DeepSpeed Ulysses[ [4] and Megatron Sequence Parallelism[18] and also shows better communication efficiency and scalability than Ring Attention 24). We perform experiments on mainstream Transformer models, including GPT-stylc and DiT-style B1], conducting performance and scaling tests across various computing clusters. Experiment results indicate that our StarTrail system outperforms Ring Attention by up to 77.12% on the GPT model and up to 11.33% on the DiT model, showcasing its efficiency and scalability.

---

Background and Related Works
2.1 Long Sequence Training and Sequence Parallelism
The key mechanism behind these Transformer-bast models is attention which captures the text feature by calculating the attention score between every two single tokens. However, the sequence length can reach hundreds of thousands, when dealing with multi-round chatting, or high-resolution long video generation. then becomes necessary to distribute the sequence across multiple GPUs. This distribution helps to reduce both the memory and computation demands on any single device. This strategy is also known as sequence parallelism. Presently. Sequence parallelism can divided The
into two main categories: attention-head teeaddsaardnng-baaae and pen--tooperr-ommmniiaaiiinn former involves distributing the attention heads multi-head attention across multiple GPUs, whereas the latter resembles a distributed version of FlashAntention, relying on peer-to-peer communication to transfer keys, values, and intermediate statistics.
2.1.1
The primary method in perrrooopprr-ommmuniitiiioo based strategies is Ring Attention (24], which also the main baseline of this work. Introduced in 2023, Ring Attention [24] innovatively partitions the se- quence dimension and utilizes ring-style peer-to-peer (P2P) communication pattern to transfer Keys and Val- across all GPUs. Each GPU receives the key and value matrices from the preceding rank, updates the lo- cal attention score, and then forwards them the next rank, as is shown in Figure 2 This method employs an online-softmax and updates attention scores incre- mentally, allowing the computation of attention scores without retaining the full sequence length. Thus, po- tentially supports infinite context, provided sufficient computing resources are available. However, the requirement for the same number of rounds of P2P communication as the number of GPUs renders this approach less efficient in environments with high-latency communication.
KVo
KWlg
KV
GPUB
GPUB
GPU4
GPU?
fKVs
K.y
aPu1s
KVis
K.V12
GPU12
KiV11
GPUT1
K.V10
K.Va
High Bandwicth Intro node Network
Low Banddw Intor mode Network
Figure 2: An example of Ring Attention Computation on 16 GPUs in two nodes. The Communication is largely limited by the inter-node bottleneck.
DPU 15
Tram
Btuthar Rashhrr
Boggui ice Parallvlon
Whole
Fing Attention
Neraliaaa
Camp she Madel
Team
KNNT
Thort E
Computing Meods
Chookpart
Caums
Teams G
Kibe
KStaza
Tamn
Temn
StarTrall Atlantion
Starhrall-2
Iwetiioss
aaved
Tundtoral Units
deng Heng 2- Hing
Trail-2 60% P2P volume on 16 GPUs compared with ring atrentionn
(b) collective communication.
ring
sub-rings, every
Figure 3: An overview of the StarTrail Training System
2.1.2
There are two representative methods, DeepSpeed- Ulysses and Megatron Sequence Parallelism in this category. SeepSpeed-Ulyseee
transitions from sequence parallelism to a method akin to tensor parallelism with two all-to-all communication. It divides the query, key, and value matrices across the attention heads, thereby preserving the original attention computation structure. Megatron Sequence Parallelism [8] focuses on minimizing memory usage and reducing the necessity for

---

activation recomputation rather than efficiency. The two methods both rely on the number of attention heads to split the sequence. thus limited in scalability. especially when employing techniques like grouped-query attention (GQA) (2) or multi-query attention (MQA) [36]. As these two sequence parallelism methods are orthogonal to the ring-based method, they are usually combined with ring attention to enable longer sequences.
In this paper, we focus on optimizing ring-style sequence parallelism, noting that integrating it with DeepSpeed-UTg Parallelism does not interfere with the underlying ring process.
StarTrail Training System
"aan
Team
Conmancziin Group
Ring Poor Poor
Fing hour Pour
X
C.K.V
Flank
QbiS
Hank
Park
Teamn
Hank
Hauure
KV1s-1
Norm
Team
Team
Team
Teem Rank
Over-
0-3
Pank
39
Iqueed
K.K.V
Parnk
Rog Peer Powrr
Team
Rayt
Ring Peerto Peer
Team
Storber
Cgpy
Pank
Op-3
Rank
Calher
KVguar
Ki
K.Ya
Team
Team
0-3
PPM
PoPi1
Block
Figure 4: An example of StarTraii Attention on 64 GPUs. Each team member forms sub-ring with members with the same local rank from other teams in the same ring communication group, reducing the communication volume of each team member by 75%.
3.1 Motivation
Through observation, we identify two main drawbacks of Ring Attention. First, the communication overhead exceedingly high because every GPU in the system must send and receive keys and values for nearly the entire sequence length before completing the attention computation. Second, variations in bandwidth between and within computing nodes can cause communication bottlenecks. As illustrated in Figure the bandwidths between GPUs 3 and 4 and between GPUs 11 and 12 are lower than those between other GPUs in the ring. Despite this, the system is forced to operate in complete circle, which can result in unnecessary idle times for other GPUs. To solve these drawbacks, we develop the StarTrail training system, which we will detail in the following section.
3.2 Stailrail Attention
As discussed in the previous section, a major limitation of Ring Attention is the extensive amount of peer-to-peer (P2P) communication required, which becomes problematic in envi- ronments with weak connections between com- puting nodes. We enhance the ring sequence parallelism by introducing an additional dimen-
Figure 5: Meanings of the symbols that are used this paper
P
The number of GPUs
C
The parallel size StarTrail size)
H
The hidden dimension size of the Transformer blocks
sion. The fundamental idea of StarTrail is akin
The total number of tokens within the whole sequence
to divide aad-conque strategy. During atten-
B
The training batch size
tion, each token must compute attention score with every other token in the sequence. While Ring Attention passes keys and values along ring of P GPUs over P iterations, our ap- proach introduces the concept of a team. this setting, each team member interacts only with designated portion of the overall sequence, and the results are later aggregated using collective
The communication bandwidth between GPUs
L
The communication latency between GPUs

---

communication. Thus, StarTraii can be devided into three phases: preprocessing. ring-phase, and postprocessing.
For preprocessing, we duplicate the queries within a team using an all-gather operation. This ensures that when a GPU receives keys and values, it compute the attention scores for the entire team's queries. Similarly, gathering the keys and values allows us to reduce the number of P2P communication iterations by transmitting longer sequences per iteration. Following the preprocessing. we enter the ring-style communication phase. With the number of GPUs in one team being C, CN/P tokens are exchanged in each iteration, and each GPU is responsible for computing N/C tokens. This leads to a number of iterations of N/C P. within a smaller ring, which we refer CN/P e
to as subring. For convenience, we group &or adjacent teams into team group for subring communication, where GPUs sharing the same local team rank form the subring. An initial P2P communication step is executed to ensure that each team group has access to the complete set of keys and values for the sequence (details are provided in the Appendix). After completing the subring iterations, each GPU holds 1/C" of the overall computation result for its team. With the help of online softmax, then apply a simple reduce- scatter to combine these results while eliminating the duplicate tokens, which we refer to as the postprocessing. Throughout the attention process, asynchronous communication is employed alongside the early launch of communication kernels to maximize the overlap of computation and communication tasks. Now we will delve into more details in the StarTraail training process.
3.2.1 Configurations of Arail Parallelism
In the StarTrail system, GPUs are grouped into Teams to coordinate computation and communication tasks more efficiently. StarTrrail introduces an additional parameter, C, which determines the replica- tion factor of the imput and, consequently, the number of GPUs within each team. The range of C from to vP. When equals one, the algorithm falls back to Ring Attention. When C equals vP, the algorithm becomes a completely collective-communicttoon baaed one with no rings. When 1 <<C < VP, it becomes a structure with multiple rings looping concurrently.
Algorithm StarTrail Attention Block (Forward)
Require:
Input sequence x, Linear Function query, key, and value, attention parallelism size e, global rank global size gs, team process group pg. init send/recv target rsend and Frecv
1: compute the gathered Ateam kream: Yream AllGather QKVmatmul( query, key, value, x, pg)
2: launch the asynchronous send and receive request
reqsend and req recv. sending keeam, Vteam to rsedd and receiving knext, Vnext from Frecy
3: get the ring P2P target nnext and Faat with P_PPPP2P_____________ gs,
4: initialize attention score O, extra statistics Ise to zero. // Ise stands for log-sum-exp
5: for S wworld_sizef do
6: wait for reqgend and reqrecv
7: Keurrent Knext, Veurrent nnxx
8: launch reqsond to send keurrent and Veurrent to Tnext, launch regrecv to receive Knext and Vnext
from Flast
9: calculate Ise, O
drwardd_terationnls,, O, 4ream: keurrent, Veurrent.)
10: end for
compute Otnal RReduceeSatterrccombiie O, pg)
12: return O Oinal
Forward Propagation. In Figure 4 we have example of one team of four GPUs out of all the 64 GPUs performing StarTrail-style attention. Each training iteration begins with the dataloader splitting the entire input sequence of length N into N/P sub-sequences, which are then loaded onto each GPU. As previously mentioned, the next step involves computing the queries, keys, and values. These computed separately via matrix multiplication, followed immediately by the launch of the all- -athea kernel, which gathers the above QKVs within the team, allowing for the overlap of up to two-thirds of the communication with computation.
Once this phase is complete, each GPU within the team possesses the same Q K, and V, each of length of CN p. To distribute the communication and computation tasks among the team members,
5

---

we divide the original workload based on four specific ranks assigned to each GPU. These ranks determine each GPU's partners and position within the P2P ring, as is shown in Figure
Following the setup, the Keys and Values are dis- patched to their designated locations within the cluster to establish the initial sub-ring, setting the stage for the multi-ring iteration phase of StarTrail attention. Given that each sub-sequence is CN long and each GPU is "p" tasked with computing the attention score for of the N/C
Team
Kigal
Kitul
Kiter Kil
whole sequence, results in CN/P - P/C²-1 rounds of communication. This implies that there are P/C² GPUs in one ring.
Figure 6: An example ring initialization process of 8GPUs and 4 sub-rings in Star- Trail.
The iteration process involves storing the log-sum-exp (Ise) and intermediate output O, which are updated step by step. Queries are retained locally, while Keys and Values circulate through the ring via P2P communication. After completing the iterations, each team member accumulates the attention scores for the entire team's 's sub-sequence of Queries with 1/C of the Keys and Values from the full sequence.
simple redute-scatter operation is then employed to amalgamate the intermediate results and distribute them among the team members. Each GPU ultimately contains the final attention score for its portion of the sequence over the entire sequence.
Backward Propagation. The major distinction between backward and forward propagation is the inability to calculate queries independently during the backward phase. Unlike forward propagation, the backward phase requires the complete set of keys and values to calculate the gradient for queries, and vice versa. manage this, we have structured the gradient calculation into two loops: the key & value outer loop and the query inner loop. In the outer loop, gradients for keys and values are tracked and maintained fixed on the corresponding GPUs within the sub-rings; these gradients do not transfer between GPUs. The inner loop, however, handles the gradients for queries, which start initialized as zero and are circulated along the sub-rings together with the Queries themselves. During each iteration, the approach mirrors the backward computation method used in FlashAttention [8], where the updated gradient of the current query shard is passed to the next GPU in the ring, while the gradients for keys and Values are retained for subsequent query shards.
3.2.2 Theoretical Analysis
During the analysis, we will employ a case study using the StarTrail system with an attention parallel size of C' = 4 on Ilama-30B model, which consists of 64 layers. For this model, referred to as model M, the batch size = set to 1, the sequence length to 65536, the hidden dimension H to 6656, and the number of GPUs P to 64. Additionally, the computation will utilize bíloatl6 precision.
Communication Analysis. Let's analyze the communication overhead within one forward Trans- former block on a single GPU. For Ring Attention, the communication primarily due to the ring P2P loop. As the total number of iterations done is P 1, the total communication overhead can be calculated as:
(P- 1){ 1)-1)(220) 2 + L) 2B -
2BNH PW
-P
WP and this overhead can be partially overlapped with the attention computation.
For StarTrail, the communication overhead comes from both collective and P2P. The collective overhead for all- all-gather and reduce-scatter is:
HB 1)
(2)
PW
while the P2P communication can be similarly computed as:
P
1)(
1)(
2CBNH
PW + +L)
-P- (P-C9)2BH
P C
1)L
(3)
C
CPW

---

The advantages of StarTrail over Ring Attention during the ring-P2P phase are evident in three main aspects: 1) Reduced Communication and Latency: Ring Attention requires C times more communication than StarTrail, significantly increasing the bandwidth requirement across the entire cluster. For the llama 30B model M, the total communication volume of ring P2P communication and collective communication volume for Ring Attention and StarTrrail can be computed as 1.625 GB and 0. 152 GB(collective) + 0.406GB (P2P) = 0,558GB. Furthermore, while Ring Attention necessitates P 1 iterations per attention block, StarTrail only requires , 1, reducing the latency overhead by around C?2 2) Localized Communication: In scenarios like those depicted in Figure 3] StarTrail's s ring P2P communication can be confined within the same computing node, where bandwidth is typically much higher than between computing nodes. Conversely, Ring Attention demands inter-node communication during every iteration, which can be less efficient. 3) Enhanced Overlap of Communication and Computation: During each iteration, the communication volume of StarTrail C times higher than that of Ring Attention, while the computational volume during ratio makes it easier for StarTraiil to overlap P2P communication with computation, enhancing overall efficiency.
attention approximately C2 times greater. This higher compucation---commmnieee
Memory Analysis. In this section, we estimate the theoretical peak memory requirements necessary to store the model weights, activations, and optimizer states. Our implementation utilizes the Adam Optimizer [16], bíloat16 precision, and Zero-2 optimization [34.. We name the memory cost for the model and optimizer as Mm+o- As for the activation, we refer to the size of one single activation of a sub-sequence one GPU
A N
P
As use the checkpointing scheme from 20 model of Y layers needs save Y activations as checkpoints. Now we calculate the approximate peak memory after Q K, and V are already calculated and before the attention computation the last layer of the whole model. For Ring Attention and StarTrail, the peak memories are:
PM Ring Mm+o + (Y + +4)
(5)
PMster = Mm+o + (Y + 3C + 1)A
(6)
where C is the StarTrrail attention dimension. And for the example model M, the peak memory would be Mm++ + 68A4 and M + 77A, and the extra memory cost compared with Ring Attention a lot less than 13.2%, while the P2P communication volume is reduced by about 75%. In a word, the extra memory cost is acceptable tradeoff for the communication reduction.
4 Evaluation
Table 1: Cluster and Model Configurations. All GPUs are connected with NVLink with computing
nodes.
GPU
dev. x node
Mem. (GB)
inter-node bandwidth
Model
#Heads
#Layers
Dim.
H100
8x8
80
8*400Gbps InfiniBand
GPT 3B
12
16
4096
A100
16 2
40
100Gbps Ethernet
GPT 7B
32
4096
A100
8 x
40
100Gbps Ethernet
DiT IB
24
24
1536
A100
4
100Gbps Ethernet
The computational resources we use in the experiments include local Nvidia H100 cluster with eight nodes and three Nvidia 1100 clusters, as listed in table We utilize two model types of total three settings, as listed in table [ For the DiT(Diffusion Transformer) model, we use similar configurations as those in Stable Diffusson 3 We utilize the backbone Diffusion Transformer only, without other components like the text and image encoders. During training. both models use bíloat precision and batch size of 1 to accommodate longer input sequences.
In the evaluation section, we aim to answer three major questions: 1) How much improvement in throughput can StarTraii bring? Additionally, how adaptable is StarTrail to clusters with both good and poor inter-node connections? 2) Is the additional memory cost incurred by StarTrail acceptable considering the throughput improvement it offers? 3) How does Star perform in scenarios of weak and strong scaling? Specifically, does outperform Ring Attention when sealed to handle longer inputs?
7

---

DTT 4255 1:88. 2280
OPT A230 4032 15 12, 6411
GIT A200 6003 3:35. 641
CPT N200 50G2 3.35. GR11
Throughput Tolens
5
AIT 45GO E 188 S12KI
SPT X100 A0GO 36.00 1284
GPT ASSS ASGB B 32841
GPT 1005 83GB, 8 178, 220K)
17.5
12.5
10.0
2.5
0.0
Figure 7: Throughput evaluation Ring Attention and StarTrail on GPUs from three different clusters. We place the performance of StarTrail with both C=2 and C=4 in the figure. The configura- tions are marked in the titles the sub- sub-figures. For instance, A100 40GB 8(1B, 512K) represents that the experiment has one billion parameters, and the sequence length 512k.
machines with 8 Nvidia A100 40GB GPUs in each node, the model used
4.1 Throughput and Adaptability
Our first experiment aims to assess the performance of StarTrail and Ring Attention across different clusters with varying environments, testing the adaptability of both methods. There are several factors influencing the efficiency of ring-style attention computation:
Volume Ratio: Primarily determined by the sequence length used during training. Attention computation exhibits a computational complexity of ONN2. whereas P2P communication complexity ONN H). Thus, the model configuration does not impact this ratio; only the sequence length does. A larger N increases the ceapuutaiio-commuiiiltii ratio, facilitating easier overlap of communication with computation.
Theoretical CeapuutationCCommniictt
The computing overhead, given specific volume, affects the computation-commuiiciii overhead ratio. Higher compute capabilities make overlapping more challenging. We utilize two sets of GPUs in this evaluation: Nvidia A 100 40GB and Nvidia H100 80GB, with the latter offering significantly higher theoretical tflops on 6 computations. Connectivity is considered in two parts: intra-node and inter-node. Our clusters are equipped with NVLink, ensuring robust intra-node communication. For inter-node communication. our H100 nodes leverage InfiniBand with eight adapters per node for superior inter-node bandwidth, whereas the Google Cloud servers use Ethernet. The diversity node configurations (8- (8-GPU and 16-GPU nodes) allows us to assess adaptability across different topologies. This evaluation not only highlights the inherent differences between the schemes but also tests their flexibility in various hardware settings.
Compute Capability and Connectivity of GPUs:
The results of evaluation illustrated in Figure We measure throughput in thousands of tokens per second. To better demonstrate how to select the optimal configuration Trail under each condition, we included two configurations. Star-2 and Star-4, in the figure. We omit configurations with lower performance for clarity, As indicated in the figure, in all six settings, at least one configuration of StarTrail achieves higher throughput than Ring Attention, with 2. 14x, 1.414x, 1. 1.629x, 1.425x, 1.360x, 1.199x, 1.77Ix, and 1.346x the throughput of Ring Attention. This advantage primarily due the additional parallel dimension that StarTrail introduces. Unlike Ring Attention, which requires inter-node P2P communication in each iteration, StarTrail's P2P communication is mostly confined intra-node, except for initial data transfers. This experiment clearly demonstrates StarTrail's superior performance across various environments. Another observation is that the optimal configuration for StarTrail may vary depending on the environment, reflecting differences in the computation-con cnpputatioo-commmmiiaatttoo ratio and the trade-off's between collective and P2P communication.

---

DAF A100 0060,3 (TR, 1746)
OPT A110 0030 (6,3 6R1
OPT GPT, Ww..RRRR.... (3R 645,
GFT H105 J.ORRRR PIR 649
Mamony Contumpttio (Nomm))
2.8
L3
11
51
61
Figure 8: The normalized relative memory cost of different configurations of StarTraiil compared with Ring Attention on different clusters.
4.2 Memory Consumption
The memory results, displayed in Figure reveal that for the configurations yielding the highest throughput, StarTrail consumes between 7.9% and 30.79% more GPU memory than RingAttention, while achieving 1.199x to 2.1 14x throughput. Moreover, in scenarios involving larger models, the relative increase in memory consumption due to QKV duplication diminishes. This reduction is explained by equation 6 This phenomenon is further evidenced by the fact that the extra memory ratio for experiments with the 7B model significantly smaller than that for the 3B and 1B model. Considering the substantial throughput gains provided by StarTrail, this tradeoff between memory usage and efficiency is deemed acceptable.
Throgghees Solemnen
method
method
(a) DiT Strong Scaling Nvidia A100 40GB GPUs. All configurations include inter-node com- munication.
(b) GPT Strong Scaling Nvidia H100 80GB
GPUs.
Figure 9; Strong scaling experiments with fixed sequence length of 128K.
4.3 Strong and Weak Scaling
In the scaling tests carry out experiments for both strong and weak scaling. For strong scaling, we fix the sequence length to 128K while increasing the number GPUs from 8 to 64 for the GPT model and from 8 to 32 for the DiT model. For weak scaling, we scale the sequence length from 128k to 512k for the DiT model and from 64k to 512k for the GPT model proportionally increasing the number of GPUs from to 32. As is depicted in Figure and TO StarTrrail shows obvious advantage over Ring Attention as we increase the number of GPUs. The results for strong scaling can also explained by the length on each GPU becomes smaller, and as explained the previous sections, makes it harder to overlap the P2P communication with attention computation. The overall scaling performance is limited by the nature of ring-style communication, but we still consider our improvement over Ring Attention meaningful due the necessity of using Ring-style Parallelisms during training.
ratio. When scaled
GPUs, the local sequence
In summary, StarTrail shows better scalability in both strong and weak scaling experiments, making it better choice for large-scale Transformer model training.
5 Conclusion
StarTrail represents an advanced near-infinite-contex Transformer model training system, featuring communication-opiimizz concentric ring sequence parallelism scheme. Through experiments,

---

Throughhpuee (K Tolante
20
CPU+B
GPL=18
GPU-32
GPU
GPU-18
GPU+32
SPU=B4
(a) DIT Weak scaling on Nvidia A100 40GB GPUs of sequence length from 128K to 512K. All config- urations include inter-node communication.
(b) GPT Weak scaling on Nvidia H100 80GB GPUs of sequence length from 64K to 512K.
Figure 10: Weak scaling Experiments
demonstrate that system only achieves high efficiency across various training environments but also excels under both strong and weak scaling conditions for both CV and NLP models. Current still further improve the co-design
limitations of StarTrail include that although orthogonal, increasingly demanding longer contexts of Star and hybrid parallelism in future works. In an era for both NLP and CV, StarTrail is poised make significant contributions the industry and inspire innovative research in academia.
6 acknowledgement
Yang You's research group is being sponsored by NUS startup grant (Presidential Young Professor- ship), Singapore MOE Tier-1 grant, ByteDance grant, NUS ARTIC grant, Apple grant, Alibaba grant, Google Research and Google grant for TPU usage.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, et al. Gpt-4 technical report, 2023.
[2] Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023.
[3] Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020.
[4] Zhengda Bian. Hongxin Liu, Boxiang Wang, Haichen Huang, Yongbin Li, Chuanrui Wang, Fan Cui. and Yang You. Colossal-ai: unified deep learning system for large-scale parallel training. arXiv preprint arXiv:2110. 140021311..888,,, 2021.
[5] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh, Feb 2024.
[6] Chang Chen, Xiuhong Li, Qianchao Zhu, Jiangfei Duan, Peng Sun, Xingcheng Zhang, and Chao Yang. Centauri: Enabling efficient scheduling for communication- computtationn overlap in large model training via communication partitioning. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS '24, page 178-191, New York, NY, USA, 2024, Association for Computing Machinery.
[7] Shenggan Cheng, Xuanlei Zhao, Guangyang Lu, Jiarui Fang, Tian Zheng. Ruidong Wu, Xiwen Zhang, Jian Peng, and Yang You. Fastfold: Optimizing alphafold training and inference on gpu clusters. In Proceedings of the 29th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, pages 417-430, 2024.
[8] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022.
[9] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024.
[10] Shiqing Fan, Yi Rong. Chen Meng, Zongyan Cao, Siyu Wang. Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, Lansong Diao, Xiaoyong Liu, and Wei Lin. Dapple: A pipelined data parallel approach for training large models, 2020.
[11] Jiarui Fang and Shangchun Zhao. Usp: A unified sequence parallelism approach for long context generative ai, 2024.
[12] W Daniel Hillis and Guy L Steele Jr. Data parallel algorithms. Communications of the ACM, 19(12):1170-1183 1986.
[13] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, Hyoukloong Lee, Jiquan Ngiam, Quoe V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism, 2018.
[14] Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models, 2023.
[15] John M. Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A Kohl, Andy Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David A. Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596:583 589, 2021.

---

[16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[17] Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan. An empirical survey on long document summarization: Datasets, models, and metrics. ACM Computing Surveys, 55:1 - 35, 2022.
[18] Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Moham- mad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models, 2022.
[19] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023.
[20] Dacheng Li, Rulin Shao, Anze Xie, Eric P. Xing, Xuezhe Ma, Ion Stoica, Joseph E. Gonzalez, and Hao Zhang. Distflashattn: Distributed memory-efficient attention for long-context Ilms training. 2024.
[21] Shenggui Li, Fuzhao Xue, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from system perspective. In Anmual Meeting of the Association for Computational Linguistics, 2021.
[22] Shigang Li, Tal Ben-Nun, Salvatore Di Girolamo, Dan Alistarh, and Torsten Hoefler. Taming unbalanced training workloads in deep learning with partial collective operations. In Proceedings of the 25th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 45 65-61,20 2020.
[23] Shigang Li and Torsten Hoefler. Chimera: efficiently training large-scale neural networks with bidirectional pipelines. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-14, 2021.
[24] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context, 2023.
[25] Ziming Liu, Shenggan Cheng, Haotian Zhou, and Yang You. Hanayo: Harnessing wave- like pipeline parallelism for enhanced large model training efficiency. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '23, New York, NY, USA, 2023. Association for Computing Machinery.
[26] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training, 2018.
[27] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catan- zaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron-Im, 2021.
[28] NVIDIA. Nvidia collective communications library, 2020.
[29]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Veto,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.
[30] William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023.
[31] William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023.
[32] Markus N. Rabe and Charles Staats. Self-attention does not need a(n²) memory, 2022.
[33] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2018.
12

---

[34] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models, 2020.
[35] Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024.
[36] Noam Shazeer. Fast transformer decoding: One write-head is all you need, 2019.
[37] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:I909 arXiv:999.008053, 2019.
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023.
[39] Boxiang Wang, Qifan Xu, Zhengda Bian, and Yang You. Tesseract: Parallelize the tensor paral- lelism efficiently. In Proceedings of the 51st International Conference on Parallel Processing, pages 1-11. 2022.
[40] Qifan Xu, Shenggui Li, Chaoyu Gong, and Yang You. An efficient 2d method for training super-large deep learning models. arXiv preprint arXiv:2104.05343, 2021.
[41] Yang You, Zhao Zhang, Cho-Jui Hsich, James Demmel, and Kurt Keutzer. Imagenet training in minutes. In Proceedings of the 47th International Conference on Parallel Processing, pages 1-10, 2018.
[42] Zilin Zhu et al. Ring flash attention, 2024.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We have clearly claimed that we improved the efficiency of ring-style sequence parallelism and provide analysis and experiments the following sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification: As mentioned in the conclusion, although orthogonal to other parallelism, we have not consider the co-design with other parallelism for better performances.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not have any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided the open-source models and GPU configurations we used in the experiments. The algorithm of the parallelism is also provided in this paper.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Our code is not currently available publicly, but we will prepare for open-source if accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nis.cc/public//uust http:///ccggiiee//eeeuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided all the details the experiment sections.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our experiment results are measured by measuring large number of train- ing iterations and taking the average. It is not suitable for our case prove statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All details mentioned in the experiment section.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://s///arrro.ccoo//////////a
Answer:
[Yes]
Justification: We reviewed the NeurlPS Code of Ethics and this research conforms with the guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: impact.
This work is simply improving the training efficiency and has no societal
Guidelines:
The answer NA means that there no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
18

---

Answer:
[NA]
Justification:
We do not introduce any new model or dataset.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have credited the assets properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
19

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our key method is modifying the parallelism technique and is not related with LLM itself.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
A Technical Appendices and Supplementary Material
A.1 Sequence Parallelism Dataloader
20

---

The causal masks present challenge due to the unbalanced computational load across GPUs: sub- sequences at the beginning of sequence require sig- nificantly more computation than those the end. To address this imbalance and achieve load equilibrium among GPUs, we modify the ZigZag scheme intro- duced by [42), illustrated in Figure II The figure illustrates the simplest case of zigzag load-balancing. Notably, the effectiieness of this strategy improves as the number of GPUs increases. This improvement correlates with the expanding difference in compu- tation volume between the first and the last token, which escalates the sequence length extends. This approach ensures that the total workload on each GPU is balanced, eliminating the need for additional communication mechanisms like those employed in DistFlasAAttention 200)
GPUs
GPUs
Team
Team
Toam
8:2
Team
Natve Dataloader
Zizzng Dateloader
Figure 11: A comparison between naive and zigzag dataloader for 8 GPUs with attention parallel dimension of 2. The corresponding initialization ean be found in Figure with the same configuration. The improvement of efficiency from load-balancing increases with the number of GPUs.
A.2 Details in the training process
Apart from the attention process described in Algorithm we would like do provide few other details to be more comprehensive. First, before the concentric ring attention, we need initialize the the Keys and Values and determine each GPU's position within the rings.
Initially, for the setup stage, is essential to establish the sub-rings by rearranging the activation positions. Specifically, during forward propagation, the queries do not require rearrangement; however, the keys and values must be transmitted to their corresponding positions in the ring prior to commencing the loop. As illustrated in Figure 6 and algorithn, this initialization ensures that each team member holds a different shard of keys and values. Moreover, it guarantees that no two teams within the same ring possess identical keys and values. Initially, for the setup stage, it essential to establish the sub-rings by rearranging the activation positions. Specifically, during forward propagation, the queries do not require rearrangement; however, the keys and values must be transmitted to their corresponding positions the ring prior commencing the loop. As illustrated in Figure 6 and algorithm, this initialization ensures that each team member holds a different shard of keys and values. Moreover, it guarantees that no two teams within the same ring possess identical keys and values.
Algorithm 2 get__inn meii__send()
Require: inter-team rank r, intra-team rank r.. inter-team dimension d,, intra-team dimension d, team group size = d, / da
2: target team group rank ra target team = target team group rank team group size + r, // d,
4: target device intra-team rank % d,
5: target global rank target team * d, target device intra-team rank
6: return target global rank
After the initialization of activations, we can set up the rings by providing the GPUs their last and next GPU within their rings, as described Algorithm
A.3 StarTrail Runtime
StarTrail is written in [yTorch[29 and uses the PyTorch torch. autchauuoogaa..oo function and NCCL[28) backend for forward and backward implementation. Trail also employs multiple techniques during runtime to improve its overall training efficiency.
Ingetrate Flash Attention. The StarTrail attention mechanism involves multiple iterations that loop over Keys and Values, with each iteration still using traditional self-attention with corresponding Query, Key, and Value (QKV). This approach enables StarTrail to incorporate flash attention ef- fectively, extending its capability by preserving intermediate states across iterations. Additionally,
21

---

Algorithm 3 Cee__P2P_confige
Require: inter-team rank r,, intra-team rank ra inter-team dimension d,, intra-team dimension d.
1: team group size d, d,
2: self team group rank team group
3: next team in group (rx + 1) 5% team group size + team group size xself team group rank
4: last team in group = (ri- 1)% team group size + team group size xself team group rank
5: next device global rank ra+ next team in group xda
6: last device global rank = T last team in group x xd,
7:
return next device global rank, last device global rank
Table 2: Supported Sequence Length of Ring Attention and StarTrriil on one Nvidia A A100 80GB GPU.
Supported Seq Len 80GB A100 GPU
(K Tokens)
Model Size
Length
Ring Attention
StarTrrail
128
3B
256
512
128
7B
256
512
128
13B
256
512
StarTrail enhances the efficiency of the forward process with the help of torch JIT fuse kernels aside from flash attention.
Overlap communication with computing. In StarTrail attention, P2P communication and self- attention computing are interleaved across iterations, each incurring considerable time. To mitigate this, StaiTrail employs double buffering technique asynchronously exccute communication and computing kernels, effectively overlapping these processes and enhancing GPU utilization.
Save recomputation with checkpoints. StarTrail adopts the checkpointing strategy introduced by DistFlash (20. placing checkpoints at the end of the self-attention phase rather than the FFN of each transformer layer. This checkpoint placement effectively obviates the need to recompute the self-attention forward process during the backward pass, avoiding redundant attention computation.
B Additional Experiment
To comprehensively evaluate the memory consumption of StarTrail and Ring Attention, compared the maximum supported sequence lengths of StarTrail with those reported the Ring Attention paper [24]. As shown in Table although StarTrail requires slightly more memory, it still supports sequence lengths commonly used training tasks.
B.1
Discussion
B.2 Larger Batch Sizes and Models
We utilized small batches and model sizes in our experiments because these choices do not affect the underlying improvements in communication efficiency and poppuaaaion-t-communiaalll StarTrail provides. For larger batch sizes, both communication and computation scale proportionally, leaving the overlapping ability unchanged. Similarly, while larger models involve more layers or
ratio that
22

---

larger hidden sizes, the key attention computations and corresponding ratios remain unaffected. Hence, our conclusions naturally extend to scenarios with larger batches and models.
B.2.1 FlashAttention3 and Hopper GPUs
In addition to the original FlashAttention g used in our experiments, FlashAttention3 35 has been introduced, specifically designed for Hopper and newer Nvidia GPUs. For FP16 precision, which utilized in this paper, FlashAttentiion3 achieves a 1.5-2.0x speedup on Hopper GPUs. As discussed in Section 3] reducing attention computation overhead results in more P2P communication not being overlapped, further emphasizing the need to reduce communication volume. With the increasing adoption of Hopper GPUs, the significance of the StarTrail system will also grow.
B.2.2
StarTrail and Other Parallelisms
Model Parallelism As is well known, tensor parallelism shards activations by attention heads during attention computation, making it easily combinable with StarTraill with minimal effort. However, when combined with tensor parallelism, the need for attention heads limit the scalability of head- beadd sequence parallel methods like DeepSpeed-Ulyssess Pipeline parallelism, on the other hand, divides the model across layers without altering the computation patterns within Transformer blocks, making StarTrail orthogonal to it.
Other Sequence Parallelism StarTrail is orthogonal with other attention-head -haa-saardding-aaaa based sequence parallelism approaches, such as DeepSpeed-Ulysee Ulysses [4]. While DeepSpeed PeepSpeed-llssees distributes attention heads across different devices, StarTrail can independently partition activations along the sequence length dimension. In future work, we can explore combining StarTrail with DeepSpeed-Ulyseee to expand the communication scheduling space, harnessing the scalability of StarTrail alongside the efficiency of DeepSpeed-U
In summary, StarTrail can be seamlessly integrated with other parallel training techniques, enabling the creation of hybrid distributed training system.
B.3 Other Related Works
Attention Optimization. Traditional full attention mechanisms necessitate O(n2) memory for storing the outputs of QKT, leading to significant computational and memory demands. To address these challenges within the GPU, several approaches have been devised to reduce both memory and computational requirements. Memory-effeiint attention 32] introduces a straightforward algorithm that requires only ((1) memory relative to the sequence length, with an extension for self-attention that needs only Olog n) memory. FlashAttention further minimizes I/O overhead and enhances overall efficiency. Additionally, optimization methods specifically tailored for inference, such as ingedAttentionn [19], are also being developed improve the efficiency of attention computations. In this work, we utilize FlashAttention within each iteration to reduce the computation overhead.
Long-Sequence Training Techniques. Sequence Parallelism(211 was initially introduced en- hance the efficiency of parallel long-sequence training. Ring Attention (24) improved communica- tion efficiency through memory-efficient methods (2]. supporting near-infinite sequence lengths. DeepSpeed-Uni Ulysssss [14] employs attention head splitting to achieve high efficiency, though it is constrained by the number of heads. Megatron Sequence Parallelism focuses on reducing memory costs during Tensor Parallelism, while DistFlashAtttention 20 features load-balance scheme and a novel gradient checkpoint method. Our work builds on these innovations, introducing system that supports large-scale training with an efficient communication scheme.
Techniques for Distributed Model Training. Distributed model training encompasses two primary areas: 1) Memory Management: Various techniques aim to conserve GPU memory during dis- tributed training. such mixed precision training 26 and the ZeRO series 34]. In this work, implement ZeRO-2 to manage optimizer states and gradients efficiently. 2) Hybrid Parallelism: Frameworks like Megatron (27) and Colossal AI44 integrate multiple forms of parallelism. There are various existing Parallelism techniques like Pipeline Parallelism[1] [13] [0 23 25 and Tensor Parallelism 337, which can be combined with StaiTrail Parallelism to facilitate large-scale training. We are also considering the integration of additional frameworks such as 6 to enhance overlapping capabilities in future implementations.
23


Reviews:
[{"review_id": "FamzbcRYHq", "reviewer": "Reviewer_LSQf", "summary": "This paper proposes a training system named StarTrail, which uses a concentric ring parallelism scheme to train Transformer models on near-infinite-context sequences. Specifically, the method introduces an additional parallel dimension by grouping GPUs into teams and divides the communication workload into multiple sub-rings. This approach reduces the total peer-to-peer communication volume and avoids bottlenecks caused by slower inter-node connections. The authors conduct experiments on both Natural Language Processing (GPT) and Computer Vision (DiT) models using various hardware clusters equipped with Nvidia A100 and H100 GPUs. The experiments show that StarTrail achieves performance improvements over existing methods without affecting the computational results.", "questions": "N/A", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\nThe proposed method improves training throughput for long-sequence Transformer models. By dividing communication into concentric sub-rings, it reduces P2P communication volume and achieves performance speedups.\n\n**Weaknesses**\n\n1. The primary weakness in terms of significance is that StarTrail is an incremental improvement upon a specific existing method, Ring Attention.  It is a systems-level optimization that re-architects the communication pattern rather than a fundamental breakthrough in attention algorithms.Its scope is confined to this specific paradigm.\n\n2. The core idea involves dividing GPUs into teams and forming sub-rings. While Figures 4 and 6 are helpful, the description of the initialization process and the mapping of ranks to sub-rings and teams can be difficult to follow on a first read. The logic is spread between the main text, figures, and algorithms in the appendix, which could be integrated more smoothly to improve comprehension.", "was_revised": true, "final_justification": "Per previous discussion: I maintain my original score of 5 - Accept.", "rebuttal": "Thank you very much for your thoughtful and constructive feedback, and for recognizing the contributions of our work.\n\nStarTrail is a system-level optimization designed for long-sequence Transformer training. It modifies the order of computation while preserving equivalence with the original attention mechanism. As the paradigm of sequence parallelism has largely converged to Ulysses-style and Ring-style Attention, we focus in this work on optimizing the performance of the Ring-style variant, which is widely applicable in practice. That said, we are also interested in extending StarTrail to other attention mechanisms, such as Linear Attention, and appreciate the suggestion to consider broader applicability.\n\nWe acknowledge that the hierarchical structure of StarTrail may be challenging to follow upon first reading. In response to your comments, we will revise the manuscript to provide clearer and more detailed explanations of the process, and will move essential content from the appendix into the main text to enhance clarity.\n\nOnce again, thank you for your valuable insights and for helping us improve our work."}, {"review_id": "io6yevGHeP", "reviewer": "Reviewer_xk9c", "summary": "In this work, a novel sequence/context parallelism strategy called StarTrail is proposed. By introducing a new parallelism dimension, ring communication can be split into subrings, thereby reducing communication volume. This is particularly beneficial when inter-node communication is a bottleneck due to limited bandwidth. Evaluation results show that StarTrail significantly outperforms the original Ring Attention, achieving higher throughput across various settings.", "questions": "1. How does inter-node bandwidth affect performance? Intuitively, with higher bandwidth, the improvement might be limited since communication is no longer the bottleneck - especially if communication can be overlapped with computation. Some discussion or insight into the relationship between bandwidth and exposed communication would be helpful.\n\n2. Does the evaluation assume pure context parallelism (i.e., CP degree equal to the number of GPUs)? If so, this may not be a practical setup and could result in a weaker baseline, as hybrid parallelism generally yields better throughput or MFU. For example, the open-source training framework NeMo provides a training recipe for LLaMA3-128k [1], where the CP degree is only 8. A total CP degree much higher than 8 may not be realistic.\n\n3. In Figure 8, why is the rightmost subfigure normalized to 0.5 instead of 1?\n\n[1] http://github.com/NVIDIA/NeMo/blob/main/nemo/collections/llm/recipes/llama3_8b_128k.py", "limitations": "See Weaknesses and Questions.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. The motivation and problem setting are reasonable: inter-node bandwidth is clearly a bottleneck for Ring Attention.\n\n2. The analysis and evaluation look promising.\n\n3. Inter-node and intra-node bandwidths differ significantly and can have distinct impacts. The analysis assumes a single bandwidth value W, which may undermine the correctness of the conclusions.\n\n4. The terminology is inconsistent: Team group, ring communication group, and subring appear to refer to the same concept but are used interchangeably.\n\n5. Only values of C = 2 and C = 4 are discussed. With only two data points, it's difficult to assess the full impact of C. Including results for C = 8 (especially for 64 or more GPUs) would be helpful. Additionally, further discussion on why C = 2 or C = 4 performs better across different settings would improve understanding.\n\n6. There is no discussion on convergence. Including a training curve comparison would provide the most straightforward evidence of the correctness of the proposed algorithm.\n\n7. The topology hierarchy is unclear. Based on my understanding, the hierarchy and corresponding communication are as follows:\n\n    Team - C GPUs - all-gather\n\n    Team group - P / C² teams - subring communication\n\n    What is the communication pattern between the C team groups?", "was_revised": true, "final_justification": "Although the authors put significant effort into the rebuttal, the paper still suffers from several issues, including insufficient discussion and evaluation, poor readability, and unclear implementation. For these reasons, I will maintain my score.", "rebuttal": "Thanks for reading our paper and providing constructive suggestions! We sincerely appreciate your feedback and would like to address your concerns as follows.\n\n1. **Bandwidth in the analysis**. In our theoretical analysis, we assumed a single bandwidth $W$ to simplify the presentation. However, we are happy to extend the analysis to incorporate both inter-node and intra-node bandwidths. Specifically, we denote the inter-node bandwidth as $ W_l $ and intra-node bandwidth as $W_h$. For Ring Attention, as illustrated in Figure 2, the overall communication bandwidth is constrained by the bottleneck $W_l$, and the communication overhead can be expressed as $(P - 1)(\\frac{2BNH}{PW_l} + L) = \\frac{2BNH(P-1)}{W_lP} + (P - 1)L$.\n\n   For StarTrail, we consider two possible scenarios. In the first case, all ring-style communication occurs within nodes. The corresponding communication overhead is $(\\frac{P}{C^2}-1)(\\frac{2CBNH}{PW_h} + L)=\\frac{(P-C^2)2BNH}{CPW_h} + (\\frac{P}{C^2}-1)L$. In the second case, where sub-ring communications span across nodes, the bandwidth is again limited by $W_l$, leading to a communication cost of $(\\frac{P}{C^2}-1)(\\frac{2CBNH}{PW_l} + L)=\\frac{(P-C^2)2BNH}{CPW_l} + (\\frac{P}{C^2}-1)L$. As in our main analysis, StarTrail reduces communication overhead by approximately $\\frac{W_h}{W_l}C$ and $C$ in the two cases, respectively.\n\n2. **Topology and related communication**. We would like to clarify several key concepts introduced in the paper:  \n   - **Teams**: Groups of $C$ GPUs that perform all-gather and reduce-scatter operations; for example, 2 GPUs per team as shown in Figure 3(b).  \n   - **Subrings**: Smaller rings used for ring-style communication, each comprising $P/C^2$ GPUs (e.g., 4 GPUs in Figure 3(b)).  \n   - **Team groups**: Collections of subrings that are connected via all-gather and reduce-scatter. In Figure 3(b), there are two team groups: the first includes teams 0–3 and the second includes teams 4–7. Communication between team groups only occurs during the initial P2P communication phase (Figure 6, Section A.2). In subsequent iterations of ring-style and collective communication, no inter-group communication occurs.\n\n3. **The value of C**. As detailed in Section 3.2.1, the parameter $C$ satisfies $1 < C < \\sqrt{P}$. In our experiments with $P=64$, valid values for $C$ are in the range (1, 8). When $C = 1$, StarTrail reduces to Ring Attention. When $C = 8$, rings are eliminated entirely: each GPU gathers Keys and Values from the other 7 teams, performs attention computation, and applies reduce-scatter for final aggregation. This scenario introduces large-scale collective communications that cannot be overlapped with matrix multiplications, leading to performance degradation—hence we did not include it in our main figures. However, we are happy to include this case in the appendix of the revised version. The optimal choice of $C$ depends on system characteristics, and StarTrail achieves the best performance at the sweet point of the tradeoff between the reduction of P2P communication and the collective operations, which can be partially overlapped with matmuls.\n\n4. **Convergence**. StarTrail is mathematically equivalent to FlashAttention (and thus to Ring Attention). As described in Algorithm 1, StarTrail computes attention scores block by block, preserving intermediate values $lse$ and $O$ and updating them iteratively. The only difference is that StarTrail computes $C$ intermediate scores in parallel and merges them using reduce-scatter, rather than updating scores sequentially as in FlashAttention. These approaches are mathematically equivalent. Therefore, StarTrail maintains the same convergence behavior as Ring Attention while improving runtime. That said, we are willing to provide convergence curves in future versions for completeness.\n\n5. **Hybrid Parallelism**. We address this concern from three perspectives.  \n   First, sequence parallelism (particularly Ring-style) operates independently of other forms like tensor or pipeline parallelism and is typically deployed across nodes. This placement makes it more challenging to overlap communication with computation, which further motivates the need for StarTrail.  \n   Second, emerging workloads—such as multi-modal models or long chain-of-thought reasoning—require sequence lengths far exceeding 128k tokens, necessitating higher degrees of sequence parallelism.  \n   Third, even at small degrees of CP(SP) (e.g., 8), StarTrail delivers improved performance with small team sizes (e.g., 2), as demonstrated in Figure 10(a) and (b). Thus, we believe StarTrail is necessary to support the scaling of modern Transformer models with increasingly longer sequences.\n\n6. **Bandwidth in evaluations**. The benefit of StarTrail is related to the communication bandwidth: the lower the bandwidth, the more significant the advantage. So the necessity of optimizing the communication can be weakened if the communication can be well-covered by computation. However, as we have discussed in the last paragraph, sequence parallelisms are usually placed between nodes in real scenarios so that parallelisms that require more communication, like tensor parallelism, can be kept within the nodes. This makes it hard for CP(SP) to be overlapped. To evaluate bandwidth impact, we included two types of interconnects: $8 \\times 400$ Gbps InfiniBand (which is among the highest inter-node connection bandwidths and typically deployed on the newest NVIDIA DGX systems) and 100 Gbps Ethernet (common in cloud A100 setups), as shown in Table 1. Experimental results show that StarTrail significantly outperforms Ring Attention even under high-bandwidth settings like 8*400 Gbps InfiniBand, underscoring its practical effectiveness.\n\n7. **Typo**. We apologize for the typo in Figure 8. The y-axis should indeed be normalized to 1. We will correct this in the revised version.\n\nThanks again for your thoughtful suggestions and careful reading of our paper. We sincerely hope that our responses have addressed your concerns."}, {"review_id": "2PbTWRKPtE", "reviewer": "Reviewer_RamG", "summary": "This paper introduces StarTrail, a new system for training Transformer models on very long sequences. The method improves upon existing ring-based sequence parallelism techniques. StarTrail organizes GPUs into teams and creates multiple smaller communication rings, called sub-rings. This design reduces the total peer-to-peer communication volume, especially across slower network links between compute nodes. The authors demonstrate that StarTrail achieves significant throughput gains over the standard Ring Attention method on both language and vision models. This performance improvement comes at the cost of a modest increase in GPU memory usage.", "questions": "Please see the weaknesses.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The core idea of creating hierarchical communication rings is a very effective solution to a clear problem. The paper correctly identifies that Ring Attention's single large ring is often bottlenecked by slow inter-node communication. By creating sub-rings that can operate within high-speed intra-node networks, the method directly addresses this critical performance issue.\n\n2. The experimental evaluation is extensive and convincing. The authors test their method across multiple hardware types (A100, H100), different cluster topologies, and on two distinct Transformer architectures (GPT and DiT). This thorough testing on various setups provides strong evidence for the general applicability and robustness of their proposed system.\n\n3. The paper presents a clear analysis of the performance trade-offs. The theoretical breakdown of communication and memory costs helps readers understand the fundamental principles behind StarTrail. It clearly explains how the team size parameter C allows a user to balance peer-to-peer communication, collective communication, and memory overhead to optimize for a specific hardware environment.\n\nWeaknesses:\n\n1. The comparison to prior work seems incomplete. The paper does not experimentally compare against other recent and relevant methods for long context training, such as DistFlashAttention.\n\n2. The paper does not fully explore the scalability limitations of its own design. The All-Gather operation within each team could become a new bottleneck at a very large scale. The analysis does not discuss how the cost of this collective communication scales with the team size C or the total number of GPUs, nor does it address the C <= sqrt(P) constraint in depth.\n\n3. The method's performance appears to be highly dependent on the network topology, but this dependency is not explored. The benefits of StarTrail rely on mapping teams and sub-rings to the physical hardware layout, for instance, keeping a sub-ring within a single node. The authors do not discuss how the system would perform on clusters with less favorable or more complex topologies where this mapping is not possible.", "was_revised": true, "final_justification": "This paper presents significant throughput gains.", "rebuttal": "Thanks for reading our paper! We would like to address your concerns as below:\n\n1. **Related Work**. First, we would like to clarify that StarTrail focuses on optimizing the communication pattern of ring-style sequence parallelisms, and the currently most widely used baseline is Ring Attention. Other related works have focused on different aspects of sequence parallelism. For instance, we have discussed DistFlashAttention in Section B.3. DistFlashAttention proposes two main contributions. The first is a modification of the communication pattern to address the load imbalance issue in causal models. However, a more elegant solution, the Zigzag dataloader, has been proposed to address this problem. By simply modifying the data loading process, we can balance the load across GPUs. We have implemented the Zigzag dataloader in StarTrail, as discussed in Section A.1. The second contribution of DistFlashAttention involves reordering recomputation steps, which we have also implemented as a feature in StarTrail. StarTrail demonstrates further performance improvements by incorporating both of these techniques into the baseline. Therefore, we believe it is not necessary to include DistFlashAttention as an evaluation baseline. Nonetheless, we are happy to explore the integration of StarTrail with other optimization techniques in future work.\n\n2. **Collective communication**. We would like to clarify that we have discussed how the cost of collective communication scales with the team size $C$ or the total number of GPUs in Section 3.2.2, line 221. As shown in Equation (2), the additional collective communication volume is much smaller than the reduced P2P communication volume, provided that $C<\\sqrt{P}$. Moreover, we would like to emphasize that the condition $C<\\sqrt{P}$ is a **feature**, rather than a limitation to be addressed. As discussed in Section 3.2.1, the valid range for $C$ is $1 < C < \\sqrt{P}$. In our experiments, with $P=64$, $C$ should be greater than 1 and less than 8. When $C=1$, StarTrail degenerates to Ring Attention. When $C=8$, there are no rings; each GPU in a team first gathers Keys and Values from the other 7 teams, performs attention computation, and applies reduce-scatter to obtain the final result. The optimal choice of $C$ depends on the system configuration, and StarTrail achieves the best performance at the sweet spot that balances reduced P2P communication and overlapping collective operations with matrix multiplications.\n\n3. **Topology of hardware**. We would like to clarify that our evaluation includes four different cluster topologies to demonstrate that StarTrail consistently outperforms the baseline regardless of hardware configuration. As shown in Table 1, our experiments span 2 GPU types, 4 topologies, and 2 types of inter-node connections, covering most common configurations used in modern Transformer training workloads. We would also like to emphasize that StarTrail does not rely on keeping sub-rings within a single node to achieve performance gains. We consider two possible scenarios, and we refer to inter-node bandwidth as $W_l$ and intra-node as $W_h$. In the first, all ring-style communication occurs within nodes. In this case, the communication overhead is $(\\frac{P}{C^2}-1)(\\frac{2CBNH}{PW_h} + L)=\\frac{(P-C^2)2BNH}{CPW_h} + (\\frac{P}{C^2}-1)L$. In the second scenario, where sub-ring communication spans multiple nodes, the communication is bottlenecked by inter-node bandwidth, yielding $(\\frac{P}{C^2}-1)(\\frac{2CBNH}{PW_l} + L)=\\frac{(P-C^2)2BNH}{CPW_l} + (\\frac{P}{C^2}-1)L$. As discussed in the paper, StarTrail reduces communication overhead by approximately $\\frac{W_h}{W_l}C$ and $C$ in these respective cases. As for less favorable environments, it becomes even more difficult to overlap communication with computation, thus making StarTrail more beneficial by reducing communication volume. For more complex topologies, communication is still constrained by the lowest-bandwidth links in the ring, and the corresponding theoretical analysis remains similar. We would also like to clarify that our goal is to address challenges in common long-sequence training settings, while more exotic or complex cluster topologies are beyond the scope of this work.\n\nThanks again for your suggestions, and we sincerely hope we have addressed your concerns."}, {"review_id": "in61ke6fyU", "reviewer": "Reviewer_nsiK", "summary": "This paper proposes StarTrail, a new method to further reduce the communication overhead of training LLM with long sequences when the bandwidth between nodes and within nodes in a GPU cluster is different. Unlike Ring Attention, where each device processes only one input sequence block, StarTrail first groups multiple devices into a team to collaboratively process multiple sequence blocks, and then the intermediate results are aggregated to obtain the final result. The paper theoretically and empirically shows that StarTrail outperforms Ring Attention.", "questions": "The paper shows that StarTrail outperforms Ring Attention when there is a disparity between inter-node and intra-node bandwidths\nin GPU clusters. However, how does StarTrail compare with SoTA methods like LoongTrain and USP which are also robust to network topologies?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The paper studies an important problem.\n* The paper proposes a sequence parallelism method that does not rely on Attention-Head-Sharding.\n\nWeaknesses:\n* The paper does not discuss some important related work. For example, LoongTrain [1] proposes Double-Ring-Attention, which partitions the GPUs into multiple inner rings, to fully utilize available NICs for inter-node communication.\n* The theoretical analysis should not only compare StarTrail with Ring Attention but also include LoongTrain and Unified SP [2] (i.e., the combination of Ring Attention + DeepSpeed-Ulysses).\n* The experiments should not only compare StarTrail with Ring Attention but also include LoongTrain and Unified SP (i.e., the combination of Ring Attention + DeepSpeed-Ulysses).\n* Presentation could be improved.\n  * The order of Figure 2 and Figure 3 is reversed.\n  * There is no explanation for Figure 3 in the paper.\n  * The notations used in Algorithm 1 are different from those listed in Figure 5, which makes it difficult for readers to understand.\n\n[1] LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism  \n[2] USP: A Unified Sequence Parallelism Approach for Long Context Generative AI", "was_revised": true, "final_justification": "After reviewing the authors' responses, I raised my final rating from 3 (Borderline reject) to 4 (Borderline accept). However, if I could, I would give it a 3.5 because the paper is poorly written in its current form. I don't think anyone who isn't an expert in the field would be able to understand it.", "rebuttal": "Thanks for reading our paper. We would like to address your concerns as follows:\n\n1. **About USP**. The main contribution of USP is to efficiently combine Ring Attention and Ulysses. As we discussed in Section 2.1.2, Ulysses is orthogonal to ring-style sequence parallelism, and our work specifically focuses on optimizing the performance of ring-style SP. As such, USP lies beyond the scope of this paper. Nevertheless, we are open to exploring potential combinations of StarTrail and Ulysses in future work on hybrid parallelism.\n\n2. **About LoongTrain**. The most relevant aspect of LoongTrain to our work is its design of Double-Ring Attention. However, Double-Ring Attention pursues a fundamentally different objective. It partitions GPUs into $C$ outer rings and $P/C$ inner rings. The model first passes Keys and Values through each inner ring for one full round, followed by a single iteration through the outer ring. In this setup, the communication volume per inner ring round is $C2BNH$, repeated $\\frac{P}{C}$ times, and the outer ring adds $\\frac{P}{C}2BNH$. Thus, the total P2P communication volume becomes $2CBNH \\times \\frac{P}{C} + \\frac{P}{C}2BNH = 2\\frac{C+1}{C}PBNH$, which is slightly larger than the original Ring Attention ($2PBNH$). By contrast, StarTrail achieves a substantially smaller overall communication volume than Ring Attention. As stated in Section 4.5.2 of the LoongTrain paper, Double-Ring Attention improves computation-communication overlap **at the cost of slightly increased total communication volume**. StarTrail, on the other hand, simultaneously provides **lower communication volume**, **better overlap**, and **improved communication locality**, at the expense of a modest increase in memory usage. In summary, Double-Ring Attention explores a different tradeoff space in ring-style communication and **does not aim to reduce communication volume**. For this reason, we believe it is not appropriate to include it as a baseline, and a direct comparison would not be meaningful.\n\n3. **Writing**. Thank you for your helpful suggestions. We will reverse the order of Figure 2 and Figure 3 and adopt clearer notations to enhance readability. Figure 3 serves as an overview of the entire system, with its components described in detail in Section 3. To further improve clarity, we will add an overview subsection following Section 3.1 to guide readers through the system structure in the revised version.\n\nThanks again for your thoughtful feedback. We sincerely hope we have addressed your concerns."}]

Author rebuttal:

Meta-review: The authors propose StarTrail, a distributed training system for training Transformers on long sequences. Traditional data paradigms like data parallelism, tensor parallelism and pipeline parallelism do not tackle the long sequence length challenge as the sequence length dimension is unchanged. 

The authors propose a systems level contribution that builds upon ring attention, which is a peer to peer communication method that allows for nearly infinite context lengths but has a high communication overhead. The authors propose a more sophisticated peer-to-peer communication scheme by grouping the GPUs into teams and dividing the peer-to-peer communication within these teams.

The authors include both theoretical analysis (Section 3.2.2) as well as extensive experimental results showing their approach outperforms Ring Attention under both strong and weak scaling for a GPT style transformer and a diffusion transformer.

Moreover, the authors have thoroughly sought to address reviewer concerns including both additional theory ( e.g. distinguishing between inter-node bandwidth and intra-node bandwidth) as well as including additional experiments (e.g. a comparison with Double Ring Attention).

Thus I recommend acceptance.
Decision: Accept (poster)