DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers
Authors: Xuanlei Zhao
Shenggan Cheng
Chang Chen
Zangwei Zheng
Ziming Liu
Zheming Yang
Yang You
Venue: ICML 2025 poster (2025)
Keywords: Sequence Parallelism
Sequence Parallel
Long Sequence
High Performance Computing
Distributed System
Multi-Dimentional Transformer
Primary area: optimization->large_scale_parallel_and_distributed

Abstract: Scaling multi-dimensional transformers to long sequences is indispensable across various domains. However, the challenges of large memory requirements and slow speeds of such sequences necessitate sequence parallelism. All existing approaches fall under the category of embedded sequence parallelism, which are limited to shard along a single sequence dimension, thereby introducing significant communication overhead. However, the nature of multi-dimensional transformers involves independent calculations across multiple sequence dimensions. To this end, we propose Dynamic Sequence Parallelism (DSP) as a novel abstraction of sequence parallelism. DSP dynamically switches the parallel dimension among all sequences according to the computation stage with efficient resharding strategy. DSP offers significant reductions in communication costs, adaptability across modules, and ease of implementation with minimal constraints. Experimental evaluations demonstrate DSP's superiority over state-of-the-art embedded sequence parallelism methods by remarkable throughput improvements ranging from 32.2% to 10x, with less than 25% communication volume.

DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers
Xuanlei Zhao'
Shenggan Cheng' Chang Chen' Zangwei Zheng'
Ziming Liu'
Zheming Yang'
Yang You'
Abstract
sequence parallel is the only option. Current sequence par- allelism, such as Megatron-LM (Shoeybi et al., 2019), Ring- Attention (Li et al., 2021; Liu et al., 2023a), Megatron- (Korthikanti et al., 2022), and DeepSpeed-Ulyeeee Ulysssss (Jacobs et al., 2023) are all embedded sequence parallelism meth- ods. As shown in Figure these embedded methods shard along single sequence dimension, which are tailored to the specific pattern and introduce extra communication and complex code modification.
Scaling milti-dimensional transformers to long sequences is important across various domains. The challenges of large memory requirements and slow speed of such sequences require se- quence parallelism. All existing approaches fall under the category of embedded sequence paral- lelism, which are limited to shard along a single sequence dimension, thereby introducing signif- icant communication overhead. However, multi- dimensional transformers involve independent cal- culation across multiple sequence dimensions. To this end, propose Dynamic Sequence Paral- lelism (DSP) as a novel abstraction of sequence parallelism. DSP dynamically switches the paral- lel dimension according to the computation stage with efficient resharding strategy. DSP offers significant reductions communication costs, adaptability across modules, and ease of use with minimal constraints. Experiments demonstrate DSP's superiority over ofate-of-the-art sequence parallelism methods by remarkable throughput improvements ranging from 22 to 10x, with at least 50% communication volume reduction.
However, multi-dimenssonal transformers calculate indepen- dently across multiple sequence dimensions. For instance, for video generation models like OpenSora (Zangwei Zheng, 2024) and Latte (Ma al., 2024), Spatial-Temport Atten- tion (Yan et al., 2021) is adopted which separates attention computations to independent temporal and spatial compu- tation. Therefore, there exists a potential space for new sequence parallelism paradigm.
adapt to the flexible patterns of multi-dimensional trans- formers, we introduce Dynamic Sequence Parallelism (DSP) novel abstraction of sequence parallelism, featured by its elegant design, high effectiveness, and excellent com- patibility. Unlike embedded sequence parallelism, DSP dynamically switches the parallel dimension of sequences during the computation stage with an efficient resharding strategy, completely decoupled from the modules' logic.
1. Introduction
DSP offers several advantages over embedded sequence parallelism: 1) Efficient communication: DSP incurs signifi- cantly lower communication costs due to its simplified com- munication patterns and reduced frequency of exchanges. 2) Adaptability: DSP seamlessly adapts to most modules with- out necessitating specific modifications and imposes few limitations on its usage. 3) Ease of use: DSP is remarkably easy implement, and also provides a simple API for users to enable it effortlessly.
Efficiently scaling multi-dimensional transformers to accom- modate long sequences necessary across diverse domains, including video generation (Singer et al., 2022; Blattmann al., 2023: Ma et al., 2024), image generation (Ramesh al., 2021; Rombach al., 2022; Liu al., 2024), protein structure prediction (Jumper et al., 2021), spatial-temporal information processing (Cong et al., 2021), and beyond. The long length of sequences introduces substantial activation memory costs and notable slowdown for speed, underscor- ing the need for employing parallelism.
Our experiments yield promising results, showcasing DSP's superiority over ffute-of-thh-an embedded sequence par- allelism methods. It achieves an end-to-end throughput improvement ranging from 32.2% to 10x and reduces com- munication volume by at least 75%.
Apart from data parallel and pipeline parallel (Huang al., 2019) which cannot reduce memory cost and inference time,
National University Singapore. Correspondence Yang You <youy wwwp/mos.mm.e.e..eeccocce
We summarize our contributions follows:
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We introduce DSP as a novel abstraction of se- quence parallelism aimed effectively scaling multi-

---

DSP: Dynamic Sequenee Parallelism for Multi-Dimensional Transformers
GPU1
Module
Module
GPU1
Module
Module
Reshord
Reshard
Reshard
Reahand
Reahard
Comm Reshard
Reshard
GPU2
Module
Module
GPU2
Module
Module
Embedded Sequence Parallelism
Dynamic Sequence Parallelism
Figure Comparison Embedded and Dynamic Sequence Parallelism. Reshard means communication to change sequence parallel layout. The blue arrow represents communication. The number and width of the arrows indicate the volume and frequency of communication, respectively.
dimensional transformers. DSP dynamically switches the parallel dimension of sequences during the com- putation stage, offering high effectiveness, elegant for- malism, and excellent compatibility.
Peebles & Xie, 2022). The Transformer is composed of stack of layers, each consisting of a multi-head attention (MHA) and a position-wist feed-forward network (FFN).
Specifically, the MHA comprises independently parame- terized attention heads, formulated as:
By significantly reducing communication volume and frequency, DSP improves pad-to-end throughput by 32.2% 10x and reduces communication volume by at least 50% compared to state-of-the-ar methods.
MHA(z) = Concat(head,,... hea,,heddyy W
(1)
head, = Att Q; K, V.).
(2)
QK
DSP seamlessly integrates with various modifications without requiring specific modifications and imposes few limitations. Its ease of use is highlighted by the minimal code changes needed to incorporate it into existing frameworks with our high-level API.
Att(Q, K, V) softmax
V
(3)
Vd
#MHA = LayerNorm(( + MHA(x)).
(4)
where Att(-) denotes the scaled dot-product attention, Q, K, V are query, key, value projections, and LayerNorm is the layer normalization. The output 7MHA is fed into the FFN, which consists of two linear transformations with ReLU activation in between, computed as:
2. Related Work
Table Meanings of the symbols that used in this paper.
FFN(x) = max W + b,) + b2. Tout = LayerNorm(EMнA + FFN(
(5)
B
The number of batch sizes.
(6)
C
The size of hidden states.
where WT, W2, b1, b2 are the parameters of the FFN.
M
The volume of sequence tensor.
N
The number of GPUs.
Transformer. Multi-dimensional
The n-th GPU.
transformers (Ho et al., 2019; Yang et al., 2022) extend the self-attention mechanism of standard transformers to operate over multiple dimensions beyond just one se- quence. An example of 2D- 2D-Transfommer shown in Fig- ure 2. Let the input multi-dimensional sequence be de- noted as X E (B S₁ , SK are the sequence lengths along K differ- ent sequence dimensions, and C' is the hidden size. Multi- dimensional transformer can be formatted as:
X
The multi- dimensional sequence. The partition of X assigned to GPU n.
X p.n
S
The i-th sequence dimension.
8;
The status that sequence is sharded from dimension S,. The status that sequence is not sharded.
where B is the batch size,
2.1. Background
Xreshape Reshape(X X, [ x Il S, CJ).
(7)
Transformer Architecture. Transformer (Vaswani et al.,
2017) is a type of neural network architecture that has be- come highly influential in natural language processing (De- vlin et al., 2018; Brown et al., 2020; Reid et al., 2024) and other domains (Dosovitskiy et al., 2020; Jumper et al., 2021;
The transformer block operation is then applied along the i-th sequence dimension of Xreshape-
X X transformer block Mockk(Xeshaann Xreshapo).
(8)
2

---

DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers
Bx52
Bx52
Reshops
Reshape 51
Transformer along S2
Transformer along SI
[B, 51, $2, C)
[BxSI, c]
[Bx52, S1,
Figure 2. Illustration of multi-dimensional (2D for this example) transformer. It calculates each dimension of sequence independently the corresponding stage. After the calculation is done in one dimension, will switch to another dimension next stage.
After applying the transformer block operation along all N dimensions, the final output tensor Xout has the same shape Transformer is
attention layers along their hidden dimension, with neces- sary all-reduce operations. This approach enables efficient distributed training while reducing memory and communica- tion overhead compared to data parallelism. It also reduces the tensor size each devices.
as the input tensor X. widely used for applications with multi-dimensional inputs including video data (Xu et al., 2020; He et al., 2021; Geng et al., 2022; Ma et al., 2024), 3D data (Zheng et al., 2021; Chen et al., 2023), protein structure prediction (Jumper et al., 2021; Mirdita et al., 2022), time-series data (Pan et al., 2022; Huang et al., 2022; Deihim et al., 2023) and beyond.
Megatron-SPP Korthikanti et al. (2022) further optimizes activation usage in the attention based on tensor parallelism. To transit between tensor parallelism and sequence paral- lelism in the transformer block, additional all- all-gather and reduce-scatter operations are introduced. But it's limited by the number of attention heads, as self-attention relies on the parallelism of the head dimension of the sequence.
2.2. Related Work
In this section, we discuss the four main parallelism tech- niques employed in deep learning: data parallelism, tensor parallelism, pipeline parallelism, and sequence parallelism.
CeapSpeed-Ulysseee Jacobs et al. (2023) introduces an in- novative approach for training long sequences by utilizing all-to-all collective communication. This method partitions the query, key, and value matrices across attention heads while preserving the original attention computation struc- ture. The process is facilitated by two sets of all-to-all communications that alternate between sequence splitting and attention head splitting. Nevertheless, it is constrained by the number of attention heads as well.
Data parallelism (Hillis & Steele Jr, 1986; Li et al., 2020) is one of the most widely adopted parallelism techniques. The input data is partitioned across devices, each processing subset. Model parameters are replicated, and gradients are summed. ZeRO (Rajbhandari et al., 2019; 2021) optimizes memory by partitioning parameters, states, and gradients across devices, enabling the training of larger models. Ten- sor parallelism (Shazeer et al., 2018; Shoeybi al., 2019), or model parallelism, partitions model parameters across devices. Different model parts assigned to different de- vices. Pipeline parallelism (Huang et al., 2019; Narayanan et al, 2019; Li & Hoefler, 2021; Liu et al., 2023b) partitions the model into stages executed in parallel across devices. Activations are passed between devices in a pipeline style.
Moreover, these sequence parallelism methods designed for parallelism within single sequence dimension. For multi-dimensional transformers, this strategy becomes in- efficient due to unnecessary communication. While spe- cialized parallelism for multi-dimensional sequences has been explored specific domains (Cheng et al., 2024), their applicability remains limited.
Unlike parameter parallelism discussed earlier, sequence parallelism is a technique specifically designed for distribut- ing long sequences and activation across multiple devices. Here are three main methods of sequence parallelism:
3. Dynamic Sequence Parallel
3.1. Problem Definition
Ring Attention. Li et al. (2021) employs an innovative ap- proach to partitioning the sequence dimension using a ring- style peer-to-peer (P2P) communication pattern to transfer keys and values across GPUs. (Liu et al., 2023a) enhance it with an online softmax mechanism, allowing for the compu- tation of attention scores without retaining the full sequence length. However, Ring Attention's 's reliance on P2P commu- nication can be less efficient in high-latency environments.
In sequence parallelism, the objective is to distribute ac- tivation computations across multiple GPUs to reduce the memory overhead caused by long sequences. This approach, however, incurs additional communication costs between GPUs. Our goal is to optimize this trade-off in the context of multi-dimensiiee
transformers.
and set of N GPUs, where Si, Sx are the sequence along K differ- ent sequence dimensions, we aim to partition the computa- tion such that the memory usage per GPU is under capacity while maintaining acceptable communication costs. Let
Given sequence X E R BRS.S...S......
Megatron-LM. Shoeybi et al. (2019) introduces tensor tensor parallelism by partitioning model across devices. The method splits both feed-forward networks and self-
3

---

DSP: Dynamic Sequenee Parallelism for Multi-Dimensional Transformers
Table Definition of Dynamic Primitives DSP. s; denotes sequence sharded from dimension i, while indicates the sequence not sharded. M represents the sequence size, and N signifies the sequence parallel size.
Source Shard
Target Shard
Comm Operation
Comm Volume
Primitives
Freq
8;
8;
/
8i
8j
Switch Split Gather
all-to-all
M/N 0 M
High
Low
8;
8
all- all-gather
Low
Dynamic Switch
Computation Dimension
52
53
all-to-all
[S1/N, S2, 53)
[S1, S2, S3/N]
Figure 3. Illustration of dynamic switch. Through all-to-all communication, the computation sequence is
p.n denote the partition of X assigned to GPU n, where p represents the partition strategy. It can be formulated as:
the operation can be formulated as follows:
Y DynamicSwitch i,j),
(10)
min CommCost(Xp..))
where has the shape
S,..........,,//NNN..... Sx.Cl, Cl, Fur- thermore, Split and Gather operations facilitate smooth tran- sitions between sharded and non-sharded states. Although these operations may involve increased communication com- pared to Switch operations, they are primarily utilized the onset and conclusion of most networks, and also for some global operations in very rare conditions, rendering their costs negligible.
s.t. Memory(Xp.a) p..) < Capacity.
(9)
where Memory Xp, n.) denotes the memory usage of par- tition Xp. GPU CommCost(X,, pX,..) represents the communication cost. We aim to achieve balance that mini- mizes the overall computational overhead while optimizing GPU resource utilization.
3.3. Overview
3.2. Dynamic Primitives
In the realm of multi-dimmesional transformers, computa- tion occurs independently for each sequence dimension. To harness this inherent feature, we introduce Dynamic Sequence Parallelism (DSP), an efficient, adaptive and ease- of-use method for multi-dimensionn transformers.
The key dynamic primitives of DSP are outlined in Table These three primitives form the foundation for adapting DSP to various multi-dimensional transformers.
The first condition is that when there is need to alter sequence parallelism between computation stages, main- tain the shard status of the sequence. This approach sig- nificantly reduces unnecessary communication overhead. However, when it becomes necessary to transit parallelism between dimensions, we employ dynamic switch to effi- ciently transform parallelism. Specifically, as depicted in Figure 3, dynamic switching adjusts the parallelism dimension unrelated to the ongoing computation, utilizing highly efficient all-to-all operations.
To ensure correct computation logic with sequence paral- lelism, embedded methods typically require complex and time-consuming communications within computation mod- ules to change the parallel dimension. As illustrated in Figure 4, the key feature of DSP is its dynamic switch of parallel dimension between computation stages.
By resharding only between computation stages dynami- cally, rather than within them, this approach allows DSP to remain independent of the computation logic within the module. Therefore, DSP climinates numerous unnecessary communications within modules, and able to utilize effi- cient all-to-all operations to switch parallelism dimensions for the intermediate sequence. For operations involving all
Assume X E RRBB,.......//N......... Sx Cl represents the input. The current parallel dimension is so its sequence length is S on each device, where is sequence parallel size. we want to switch the shard dimension from to j,

---

DSP: Dynamic Sequenee Parallelism for Multi-Dimensional Transformers
Computation Stagel
Computation Stage?
Split
Dynamic Switch Shard 52 -53
Gather
[$1. 52, Irputs
[51, S2/N, S3]
[51, 52, S3/N]
53]
outputs
GPU1
ali-to-all
all-gather
Computation Stagel
Comm Volume
Computation Stage2
SLNS2x53/(Nf/(N-11)
Split
Gather
Dynamic Switch
dlong
whard 52w53
[$1, 52, S3) Irputs
(S1, S2/N, S3]
[st. S3/NN)
[S1, S3) outputs
GPU2
dang
Figure 4. System overview of Dynamic Sequence Parallelism. utilizes Split and Gather the beginning and the end of model to separate and collect the complete sequences. In the middle computation, it utilizes Dynamic Switch to change the sharding dimension with efficient all all-to-all communication. that the following computation will not be affected by sharding.
sequence dimensions, including the beginning and end of the model, DSP handles them by Split and Gather operation.
compatible with various distributed frameworks, includ- ing FSDP (Zhao et al., 2023), Accelerate (Gugger et al., 2022), DeepSpeed (Rasley et al., 2020), and Megatron-LM (Shoeybi al., 2019).
Furthermore, we also propose high-level, user-friendly im- plementation of DSP compatible with all distributed frame- works based on PyTorch.
4. Theoretical Analysis
3.4. Adaptability and Flexibility
We choose 2D- TD-Trnssformer as described in Equation 8 as our base model, which is widely employed in real- world applications. Sora (Zangwei Zheng, 2024) variant of 2D- 2D-rransformer, an open-source video generation model, where there are two transformer blocks for two sequence dimensions sep- arately. More details can be found in Appendix A. We choose state-of-the-ar sequence parallel methods including DeepSpeed- Ulysses (Jacobs et al., 2023), Megatron-SS (Ko- rthikanti et al., 2022), Megatron (Shocybi et al., 2019) and RingAttention (Liu et al., 2023a) as baselines.
Given its decoupling from the computation of modules, DSP exhibits remarkable adaptability, making it compatible with wide array of transformer variants such as Cross Attention (Hertz et al., 2022; Ma et al., 2024); specialized kernels like FlashAttention (Dao et al., 2022); special attention mecha- nisms including multi-query attention (Shazeer, 2019) and grouped-query attention (Ainslie et al., 2023); and even beyond like Mamba (Gu& & Dao, 2023) and RWKV (Peng al., 2023). This inherent flexibility enables DSP to seam- lessly integrate into diverse transformers without specific modification. Furthermore, while DeepSpeed- -lysssss and Megatron-SP necessitate attention head splitting, DSP's scal- scal- ability significantly better because shards on sequence length, which is more redundant.
be specific, we use the Open-
4.1. Communication Analysis
The primary advantage of DSP lies in its ability to minimize communication costs and enable scalable communication operations. DSP exploits the inherent characteristies of transformers to eliminate unnecessary communication, compared with embedded approaches such as Mcgatron-LM (Shoeybi et al., 2019), Megatron-ST (Kor- thikanti et al., 2022), RingAttention (Liu et al., 2023a) and DeepSpeed- Ulyssuu (Jacobs et al., 2023). Consider an ac- tivation size of M and a sequence parallel size of N. In 2D-Transformer, there is one transformer block for each sequence dimension per layer, resulting in two transformer
Moreover, DSP's adaptability extends beyond module com- patibility to encompass various parallelism methodologies. From conventional data parallelism to more sophisticated approaches like ZeRO and pipeline parallelism, DSP effort- lessly integrates with diverse parallel computing paradigms, thereby enhancing scalability and performance across dis- tributed computing environments.
By calling just four functions without knowing the detailed implementation, DSP can be enabled on Py Torch and is
5

---

DSP: Dynamic Sequenee Parallelism for Multi-Dimensional Transformers
Table Comparison of DSP with other sequence parallelism methods for 2D-Transformer architectures. M denotes the activation size, and N represents the number of devices. Communication volume refers to the per-layer (two ID blocks) volume per device.
Communication Volume
Activation Memory
Parameter Memory
Ease
Method
of Use
Ring Attention
2M
Megatron-LMN
S.M
Megatron-
8M
O
DeepSpeed- Ulysses DSP (ours)
4M/N
2M/N
O
O
blocks per layer (two 1D blocks). More details are demon- strated in Appendix A.3.
shard all parameters across different devices to ensure low parameter memory footprint.
Megatron-LM & SS both need to gather and scatter the whole sequence. Megatron-LM utilizes 2 all- reduce per block for attention and mlp. while Megatron- SP employs all all-gather and 2 reduce-scatter operations per block. They lead to a total per-device communication volume of 8M for both methods.
5. Experiments
Experiments are conducted on 128 NVIDIA HI00 GPUs, interconnected via NVLink within nodes and InfiniBand across nodes. Our methods and implementations are not dependent on specific hardware architectures and can gen- eralize to other devices, particularly those with less effi- cient interconnects. We follow the same bascline and set- tings as discussed in Section 4, utilizing 720M and 3B size Transformer- 2D models in our experiments. Despite the existence of various 2D-Transformer variants, their archi- tectures are fundamentally similar. We select one model similar to OpenSora (Zangwei Zheng, 2024). The code is implemented using Py Torch (Paszke et al., 2019).
Ring-Aftention needs to communicate the entire key and value in the temporal block only, resulting in total per- device communication volume of 2.M.
DeepSpeed-Ulyee Ulysses incurs 4 all-to-all in temporal block for the query, key, value, and output of attention, resulting in total per-device communication volume of 4M/N.
DSP mitigates communication cost by employing only two all-to-all operations in total two blocks per layer. As shown in Table 4.1, it reduces the communication volume to 2M/N, significantly outperforming other sequence par- allelism techniques. Notably, with merely two all-to-all operations, DSP exhibits efficient scalability even super- large clusters for training and inference on extremely long sequences because the communication volume decreases as the number of nodes increases, rendering DSP an excep- tional choice for large-scale distributed training and infer- ence tasks involving extreme long sequences.
In the following evaluations, we focus on addressing the fol- lowing questions: How DSP's end-to-end performance compared with other SOTA sequence parallelism? 2) How is DSP's scaling ability when scale to many GPUs? What is DSP's memory consumption like in practice?
5.1. End-to-End Performance
In this section, we compare the end-to-end performance of different sequence parallelism methods on 128 NVIDIA H100 GPUs. We use combination of sequence parallelism and data parallelism, with the sequence parallelism set to the minimum size for each method. We evaluate across different sequence lengths ranging from 0.5 million to million tokens, which are common usages for video gener- ation. Details be found in Appendix A.2.2. As shown in Figure 5, DSP is able outperform DeepSpeed-UNy -lysses by 32% to 75%, and other methods by up to 10x due to its communication efficiency. As the sequence length becomes longer and the sequence parallel size increases, as DSP's communication volume decreases as the device number in- creases, our performance's 's advantage over the baselines becomes even more pronounced. When scaling from 0.5M to 4M tokens, our FLOPS drops by most 23%, while other
4.2. Memory Analysis
Regarding activation memory, since we shard every tensor in the transformer, we are theoretically able to achieve the minimum activation cost, similar to DeepSpeed- Ulysses and Ring Attention. In practice, however, our approach requires less reshape and communication overhead, allowing us to further reduce intermediate activation memory compared to other methods. Megatron-SP and Megatron-LMM, on the other hand, needs to hold some entire sequences, resulting in higher memory requirements.
As for parameter memory, as discussed in Section 3.4, we utilize the ZeRO technique (Rajbhandari al., 2019) to
6

---

DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers
End-to-End Performance (2D-Transformer 720M)
End-to-End Performance (2D-Transformer 3B)
165
R Vegaron-ul UV - Hegatron-5P
= DespSpeed Jyyaa
IMd Ring Accerton
2 DSP
250
135
225
105
175
75
125
100
30
50
25
XX
0.5M
1M
2M
4M
0.5M
1M
2M
4M
Accumulated Sequence Langth
Accumulated Sequence Length
Figure 5. End-to-end performance comparison of different sequence parallel methods combined with data parallel 128 H100 GPUs.
The sequence parallel size is set to minimum for each method.
Thruughuu (tamplesg))
1.2
Scaling
(Transformer-2D 720M)
Througheu (samples/s)
Scaling (Transformer-2) 720M)
Megatron-UM
Megatron-SP
DeepSpeed-llyeess
Ring
X
DSP
X
Terouphput (smpless))
Weak Scaling Intra-Node (Transformer-2D 3B)
Weak Scaling Inter-Node (Transformer-20 3B)
Throughput Samplesss))
1.4
1.50
1.25
1.0
1.00
0.75
0.6
0.4
0.25
XXX
0.00
0.0
8
16
32
64
GPU Number
Number
Figure Weak scaling ability evaluation of different methods with sequence parallelism only. "x" denotes out of memory or head, Black boxes represent linear scaling.
methods experience at least 40% drop.
Weak Scaling, In the weak scaling experiments, to main- tain a consistent computational workload for each GPU, the batch size is linearly increased proportional to the number of GPUs, while the sequence length is fixed. As shown in Figures 6, DSP significantly outperforms other methods by more than 80.7%. Moreover, DSP can scale up to 64 GPUs without being limited by the number of attention heads, un- like DeepSpeed-Ulye Ulysses and Megatron-SP. Despite scaling to 64 GPUs, DSP maintains an almost linear throughput increase, with only 15% performance loss from 8 GPUs to 64 GPUs. Additionally, DSP can achieve super-linear scaling for intra-node due to efficient communication.
5.2. Scaling Ability
This section evaluates the scaling ability of DSP from two perspectives: weak scaling and strong scaling. Weak scaling refers to scenarios where the computational workload per device remains constant while incrementally increasing the number of devices. This setup is analogous to the train- ing stage, where the goal is to scale longer sequences over more GPUs. Strong scaling, on the other hand, is more challenging as it requires keeping the total computational workload constant while incrementally increasing the num- ber of devices. In this case, the computation becomes more sparse on each device. Strong scaling is often employed when the objective is to infer an input sequence rapidly across many GPUs for low-latency applications. The experi- ments are divided into intra-node and inter-node evaluations due the different interconnection conditions. Intra-node experiments leverage NVLink interconnect for communica- tion, while inter-node experiments utilize a combination of NVLink and InfiniBand interconnect. More details can be found in Appendix A.2.3,
Strong Scaling. the strong scaling experiments, both batch size and sequence length are fixed. As shown in Figure 7, DSP maintain linear scalability when scaling up 8 GPUs for 720M model and 4 GPUs for 3B model, which covers most practical scenarios. To evaluate the extreme performance capabilities of DSP, we further scale up to 64 GPUs with very little workload per device. Although there an inevitable performance drop, DSP's throughput remains significantly better than the baselines. As shown in Figure our work significantly reduce inference

---

DSP: Dynamic Sequenee Parallelism for Multi-Dimensional Transformers
Strong Scaling Intra-Node (Transformer-2D 720M)
Strong Scaling Inter-Node (Transformer-2D 720M)
0.7
0.90
LM
0.6
0.75
Megatron-SP SP
DeepSpeed-Uytee
0.4
Thrughput
0.45
0.30
DSP
Attention
0.3
0.2
0.15
0.1
0.00
0.0
Strong Scaling Intra-Node (Transformer-2D 3B)
Strong Scaling Inter-Node (Transformer-2) 3B)
Thoughhue (sanelcaa
1.50
0.6
1.25
0.5
0.4
0.3
0.50
0.2
0.25
0.1
0.00
0.0
2
16
32
64
Figure Strong scaling ability evaluation of different methods with sequence parallelism only. "x" denotes of memory or head. Black hoxes represent linear scaling.
Comparison Transformer
Mamory Comparison (2D-Iran 3B)
Negabann Megitnn-5 +p
DuecSoest--yoema Nig Arwitin
Magarron JM
101 Megatron 57
= DeapSpees Clymes
F
Uigg Attentinn
PDt
BSP
70
60
20
SX
20
10
X
X X
x
Number
Figure 8. Inference latency comparison of different sequence paral- Figure Memory comparison different sequence parallelism lelism methods.
methods.
latency compared with baselines with the same workload.
communication costs, 2) adaptability across modules with- out specialized modifications, and 3) remarkable ease of implementation enabled by simple high-level API. Our ex- periments demonstrated DSP's superiority, achieving from 32.2% to 10x higher end-to-end throughput and at least 75% lower communication volume compared to state-of- the-art methods. Its elegance and ease of use make it a promising solution for efficient sequence parallelism across wide range applications.
5.3. Memory Consumption
Figure demonstrates the memory consumption com- parison of different baselines the weak scaling setting. The semi-transparent bar represents cached memory, while the solid bar represents the allocated memory. The total memory usage the sum of them. Our approach exhibits the lowest memory usage, scaling efficiently for longer sequences. Furthermore, DSP's memory usage is compact without excessive cache memory bloat, unlike Ring-Attenn Ring-Attention, Megatron-LM and Megatron-SP.
Limitations. One limitation of this work is that DSP specifically designed for multi- multi-dimensional transformers and may not adapt well to single-dimensions ones like language models. Additionally, while there are global oper- ations that involve all sequence dimensions, which rare in transformer, DSP may not be of optimal efficiency.
6. Conclusion
In this work, introduce Dynamic Sequence Parallelism (DSP), novel sequence parallel abstraction for effectively scaling multi-dimensional transformers to long sequences. Unlike current embedded sequence parallel methods that only shard on single sequence dimension and are tailored to specific patterns, DSP offers general and elegant solution by dynamically switching the parallel dimension during computation, decoupled from the computation module.
Future works. In the future, DSP could expand its scope beyond transformer architectures to architectures including convolution, recurrent, and graph neural networks to utilize its potential across various tasks. Furthermore, automated optimization techniques could enable DSP to dynamically and autonomously determine the most effective switching strategy based on network analysis, thereby optimizing over- all system efficiency and efficacy.
The key advantages of DSP are: 1) substantially reduced
8

---

DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers
Impact Statement
Heigold, G., Gelly, S., et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 11999, 2020.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Geng, Z., Liang, L., Ding, T., and Zharkov, I. Rstt: Real- time spatial temporal transformer for space-time video super-resolution. In Proceedings of the IEEF/CVF Con- ference Computer Vision and Pattern Recognition, pp. 17441-17451, 2022.
References
Ainslie, J., Lee- Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebr' on, F., and Sanghai, S. K. Gqa: Training gener- alized multi-query transformer models from multi-head checkpoints. ArXiv, 0tlb/2305.132453 2023.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. ArXiv, abs/2312.00752, 2023,
Gugger, S., Debut, L., Wolf, T., Schmid, P., Mueller, Z., Mangrulkar, S., Sun, M., and Bossan, B. Accelerate: Training and inference scale made simple, efficient and adaptable., 2022.
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., and Lorenz, D. Stable video diffusion: Scaling latent video diffusion models to large datasets. ArXiv, abs/2311.15122 2023.
He, L.,, Zhou, Q.. Li, X., Niu, L.., Cheng, G., Li, X., Liu, W., Tong, Y., Ma, L.,, and Zhang, L. End-to-end video object detection with spatial-tempon Proceedings of the 29th ACM International Conference on Multimedia, pp. 1507-1516, 2021.
Brown, T., Mann, B., Ryder, N., Subbiah, M.. Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:
transformers. In
1877-1901, 2020.
Hertz, A., Mokady, R., Tenenbaum, J. M., Aberman, K., Pritch, Y., and Cohen-Or, D. Prompt-to-promy image editing with cross attention control. ArXiv, abs/2208.01 01666 2022.
Chen, S., Shu, T., Zhao, H., Zhong, G., and Chen, X. Tem- pee: Temporal-spatial parallel transformer for radar echo extrapolation beyond auto-regression IEEE Transactions on Geoscience and Remote Sensing, 2023.
Hillis, W. D. and Steele Jr, G. L. Data parallel algorithms. Communications of the ACM, (1:1111170-11,, 1986.
Cheng, S., Zhao, X., Lu, G., Fang, J., Zheng, T., Wu, R., Zhang, X., Peng, J., and You, Y. Fastfold: Optimizing alphafold training and inference on gpu clusters. In Pro- ceedings of the 29th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, pp- 417-430, 2024.
Ho, J., Kalchbrenner, N., Weissenborn, D., and Salimans, T. Axial attention in multidimensional transformers. ArXiv, 1ts/1912.1288)) 2019.
Huang. L., Mao, F., Zhang, K., and Li, Z. Spatial-temporal convolutional transformer network for multivariate time series forecasting. Sensors, 22(3):841, 2022.
Cong, Y., Liao, W., Ackermann, H., Yang. M. Y., and Rosen- hahn, B. teatial-temporal transformer for dynamic scene graph generation. 202/ IEEE/CVF International Confer- ence on Computer Vision pp. 16352-16362, 2021.
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in Neural Information Processing Systems, 32, 2019.
Dao, T., Fu, Y., Ermon, S., Rudra, A., and R'e, C. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. ArXiv, abs/2205.1411 2022.
Jacobs, S. A., Tanaka, M., Zhang, C., Zhang, M., Song, L., Rajbhandari, S., and He, Y. Deepspeed ulysses: Sys- tem optimizations for enabling training of extreme long sequence transformer models. ArXiv, abs/2309.14509, 2023.
Deihim, A., Alonso, E., and Apostolopoulou, D. Sttre: A spatio-temporal transformer with relative embeddings for multivariate time series forecasting. Neural Networks, (68:549-559, 2023.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. arXiv preprint arX.v.1810.04805, 2018.
Jumper, J. M., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Zidek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S. A. A., Ballard, A., Cowie, A., Komera-Paredes: B., Nikolov, S., Jain, R., Adler, J.. Back, T., Petersen, S., Reiman, D. A., Clancy, E.,, Zielinski, M., Steinegger, M.,
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M.,

---

DSP: Dynamic Sequenee Parallelism for Multi-Dimensional Transformers
Pacholska, M., Berghammer, T., Bodenstein, S., Silver, D., Vinyals, O., Senior, A. W., Kavukcuoglu, K., Kohli, P., and Hassabis, D. Highly accurate protein structure prediction with alphafold. Nature, 596:583 - 589, 2021.
Pan, X., Wang, L., Wang, Z., and Huang, C. Short-term wind speed forecasting based spatial-temporal graph transformer networks. Energy, 25:: 124095, 2022.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.., Gimelshein, N., Antiga, L.., Desmaison, A., Köpf, A., Yang. E.,, DeVito, Z.,, Rai- son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library. ArXiv, Abs/1912.01703, 2019.
Korthikanti, V. A., Casper, J., Lym, S., McAfee, L. C.,, Andersch, M., Shoeybi, M., and Catanzaro, B. Reduc- ing activation recomputation in large transformer models. ArXiv, abs/2205.05 199, 2022.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000). pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Peebles, W. S. and Xie. S. Scalable diffusion models with transformers. 2023 IEEE/CVF International Con- ference on Computer Vision (ICCV), pp. 4172-4182, 2022. URL /ttts://api... httppppp.........pm/a//mmmmm iesmantisscoolloooo cscholas org/ /ttp//oapaDD::55555100
Li, S. and Hoefler, T. Chimera: Efficiently training large- scale neural networks with bidirectional pipelines. SC21: International Conference for High Performance Comput- ing, Networking, Storage and Analysis, pp. 1-14, 2021.
Peng, B., Alcaide, E.,, Anthony, Q G., Albalak, A., Arcad- inho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., Kranthikiran, G., He, X., Hou, H., Kazienko, P., Kocoú, J., Kong. J., Koptyra, B., Lau, H., Mantri, K. S. I. Mom, F., Saito, A., Tang, X., Wang. B., Wind, S., Wozniak, S., Zhang, R., Zhang, Z., Zhao, Q., Zhou, P.. Zhu, J., and Zhu, R. Rwkv: Reinventing rnns for the transformer era. In Conference on Empirical Methods in Natural Language Processing, 2023.
Li, S., Zhao, Y., Varma, R., Salpekar, O., Noordhuis, P., Li, T., Paszke, A., Smith, J., Vaughan, B., Damania, P., et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.1577 2020.
Li, S., Xue, F., Li, Y., and You, Y. Sequence parallelism: Long sequence training from system perspective. In An- nual Meeting of the Association for Computational Lin- guistics, 2021.
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimization towards training trillion parame- ter models. ArXiv, abs/ abs/1910.02054, 2019.
Liu, H., Zaharia, M., and Abbeel, Ring attention with blockwise transformers for near-infinite context. arXiv 2023a.
preprint arXiv:2310.0f
Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., and He, Y. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. In Proceedings of the Inter- national Conference for High Performance Computing. Networking, Storage and Analysis, pp. 1-14, 2021.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tun- ing. Advances in Neural Information Processing Systems, 36, 2024.
Liu, Z., Cheng, S., Zhou, H., and You, Y. Hanayo: Har- nessing wave-like pipeline parallelism for enhanced large model training efficiency. The International Conference for High Performance Computing, Networking, Storage, and Analysis. pp. 1-13, 2023b.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, Zero-shot text- to-image generation. In International Conference Machine Learning, pp. 8821-8831, 2021.
Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deep- speed: System optimizations enable training deep learn- ing models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
Ma, X., Wang, Y., Jia, G., Chen, X., Liu, Z., Li, Y.-F., Chen, C., and Qiao, Y. Latte: Latent diffusion transformer for video generation. ArXiv, abs/2401 103048,, 2024.
Mirdita, M., Schütze, K., Moriwaki, Y., Heo, L., Ovehin- nikov, S., and Steinegger, M. Colabfold: making protein folding accessible to all. Nature Methods, 19(6):6796682. 2022.
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lil- licrap, T., baptiste Alayrac, J., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., Antonoglou, I., Anil, R., Borgeaud, S., Dai, A., Millican, K., Dyer, E., Glaese, M., Sottiaux, T., Lee, B., Viola, F., Reynolds, M., Xu, Y., Mol- loy, J., Chen, J., Isard, M., Barham, P., Hennigan, T., and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., Gibbons, P. B., and Za- haria, M. Pipedream: generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles. pp. 1-15, 2019.
10

---

DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers
Rombach, R., Blattmann, A., Lorenz, D., Esser, and Ommer, B. High-resolution image synthesis with latent diffuuion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Zheng, C., Zhu, S., Mendieta, M., Yang, T., Chen, C., and Ding, Z. 3d human pose estimation with spatial and temporal transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11656- I 11665, 2021.
Shazeer, N., Cheng, Y., Parmar, N., Tran, D., Vaswani, A., Koanantakool, P., Hawkins, P., Lee, H., Hong, M., Young, C., et al. Mesh-tensee beesh-tensorflow: Deep learning for super- computers. Advances Neural Information Processing Systems, 31, 2018.
Shazeer, N. M. Fast transformer decoding: One write-head is all you need. ArXiv, abs/191 911 1.02150, 2019.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-Im: Training multi- billion parameter language models using model paral- lelism. ArXiv, abs/ 90999999.0000 00003, 2019.
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q. Yang, H., Ashual, O., Gafni, O., Parikh, D., Gupta, S., and Taigman, Y. Make-a-video: Text- to-video generation without text-video data. ArXiv, Abs/2209.14492, 2022.
Vaswani, A., Shazeer, N. M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, Attention is all you need. In Advances Neural Information Processing Systems, 2017.
Xu, M., Dai, W., Liu, C., Gao, X., Lin, W., Qi, G.-J,, and Xiong, H. Spatial-temporal transformer networks for traffic flow forecasting, arXiv preprint arXiv:2001.02908, 2020.
Yan, B., Peng, H., Fu, J., Wang, D., and Lu, H. Learning spatio-temporal transformer for visual tracking. 2021 IEEE/CVF International Conference on Computer Vision, pp. 10428-10437, 2021.
Yang, A., Miech, A., Sivic, J., Laptev, I., and Schmid, C. Tubedetr: Spatio-temporal video grounding with trans- formers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16442- 16453,200
Zangwei Zheng, X. P. Open-sora: Democratizing efficient video production for all, April 2024.
Zhao, Y., Gu, A., Varma, R., Luo, L., chin Huang, C., Xu, M., Wright, I.., Shojanazeri, H., Ott, M., Shleifer, S., Desmaison, A., Balioglu, C., Nguyen, B.. Chauhan, G., Hao, Y., and Li, S. Pytorch fsdp: Experiences on scaling fully sharded data parallel. Proc. VLDB Endow., 16:3848- 3860, 2023.

---

DSP: Dynamic Sequenee Parallelism for Multi-Dimensional Transformers
DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers
Appendix
We organize our appendix as follows:
Section A.1: Model details.
Section A.2: Experiment Settings.
Section A.2.1: Model size.
Section A.2.2: End-to-end performance.
Section A.2.3: Scaling ability.
Section A.3: Parallelism implementation.
A. Appendix
A.1. Model Details
In the theoretical analyses and evaluation section, we use a Transformer-2D model as our base model, similar to OpenSora (Zangwei Zheng, 2024). However, not exactly OpenSora: we have removed its specific cross- atsenttttt module ensure that the performance process two sequence dimensions separately, shown in Figure 10. Specifically, the two dimensions temporal t and spatial for a sequence. Each dimension is processed by a corresponding transformer block, which is common strategy in many applications.
be generalized to other models. Therefore, in each layer, there only two transformer blocks that
A.2. Experiment Settings
A.2.1. MODEL SIZE
Table 4. End-to-end performance parallel settings. Tuple for methods denotes (sequence parallel size, data parallel size).
Model Size
Sequence Length
Temporal Sequence
Spatial Sequence
DeepSpeed Ulysses
Megatron SP
Ring Attention
DSP
0.5M
128
4096
(2, 64)
(2,64)
(2, 46)
(2, 64)
IM
256
4096
(4, 32)
(4,32)
(4, 32)
(4, 32)
720M
2M
512
4096
(8, 16)
(16,8)
(8, 16)
(8, 16)
4M
1024
4096
(16, 8)
(16, 8)
(16, 8)
0.5M
128
4096
(4, 32)
(4, 32)
(4, 32)
(4, 32)
IM
256
4096
(8, 16)
(16,8)
(8, 16)
(8, 16)
3B
2M
512
4096
(16, 8)
(16, 8)
(16, 8)
4M
1024
4096
(32, 4)
(32, 4)
(32, 4)
Table Model settings of 720M and 3B 2D-Transformer.
Model Name
Layers
Hidden States
Attention Heads
Patch Size
720M
28
1152
16
(1, 2, 2)
3B
36
2038
32
(1, 2, 2)
In the experiments, we use 720M and size for DD-Transformer. There specific model settings are shown in Table 5.
12

---

DSP: Dynamic Sequenee Parallelism for Multi-Dimensional Transformers
A.2.2. END-TO-END PERFORMANCE
Here the polished text more formal format without using bullet points: end-to-end performance experiments, 128 GPUs were utilized for all methods. For each method, the minimum sequence parallel size that would not result in out-of-memory errors employed to reduce communication overhead, with data parallelism employed for the remaining size. ZeRO-2 was used for all methods except Megatron-SP. The specific parallel size is detailed in Table 4.
The accumulated sequence length ranged from 0.5M to 4M, which appears significantly larger than typical text lengths. However, such lengths are common for multi-dimennional tasks. In this case, we followed the workload of video generation. The spatial sequence, representing video resolution, was fixed at 1024x 1024. After applying the Variational Autoencoder (VAE) and Patch Embedding, the final length for the spatial sequence 4096. The temporal sequence, representing video length, scales linearly in the test.
A.2.3. SCALING ABILITY
Table 6. Strong scaling experiment settings.
Model Size
Type
Batch Size
Temporal
Spatial
Intra-Node Inter-Node
4096
720M
256
4096
Intra-Node
16
4096
3B
Inter-Node
128
4096
Table Weak scaling experiment settings.
Model Size
Type
GPU Number
Batch Size
Temporal
Spatial
64
4096
2
2
64
4096
Intra-Node
4
4
64
4096
720M
8
8
64
4096
8
1
256
4096
Inter-Node
16
2
256
4096
256
4096
1
16
4096
2
2
16
4096
Intra-Node
16
4096
4096
3B
128
4096
16
2
128
4096
Inter-Node
32
4
128
4096
64
8
128
4096
In weak scaling experiments, shown in Figure 7 we fix the sequence length and linearly increase the batch size, ensuring that the workload on each device remains constant the number of devices scales. In strong scaling experiments, as shown in Figure 6, we fix both the sequence length and batch size, keeping the total computation constant. For each experiment, we set the sequence length to the maximum for the least GPU case to fully utilize the computational resources. Specifically, we use the same spatial sequence length and adjust the temporal sequence length to its maximum for each test and sequence parallel size is GPU number.
A.3. Parallelism Implementation
In Figure 10, we demonstrate the detailed implementation of different sequence parallel methods on 2D- 2D-Transformer. The implementation of SeppSpee--Ulyseee (Rasley et al., 2020), Megatron (Korthikanti et al., 2022) and Megatron-L.M
13

---

DSP: Dynamic Sequenee Parallelism for Multi-Dimensional Transformers
(Shoeybi et al., 2019) are adopted based their official implementation. For Ring-Attention (Liu al., 2023a), adopt unofficial implementation and adapt the TD-Transformer.
Communication Type
Communication Volume
all-reduce
2M
all- all-gather
M
reduce-scatter
M
all-to-all
M/N
Table 8. Per-device communication volume for each operation.
Method
Communication Type
Communication Times Per Layer (Two 1D Blocks)
Time (ms)
DSP
all-to-all
2
914
DS-Ulysses
all-to-all
4
5828
Megatron-LA.
all-reduce
8412
Megatron-SP
all-gather reduce-scatter
51908
Table 9. Comparison of communication methods and performance with 8M length of sequence.
Megatron-SP employs four resource-intensee collective communication operations per transformer block. Specifically, it initiates an all-gather operation to aggregate the entire input æ, succeeded by reduce-scatter operations at the output for both attention and MLP modules, culminating in total communication volume of 8M for one layer (two ID blocks). Note that the communication volume calculated per device.
Similarly, Megatron-LM employs 2 all-reduce per transformer block, culminating 4 all-reduces, in total communication volume of 8.M.
DeepSpeed- -eeespedd-Ulysess adopts the more efficient all-to-all approach. It leverages all-to-all for query, key, value to transform their shard dimension before attention, and all-to-all for output after attention. And it only need to communicate in temporal transformer block. Consequently, the communication volume transmitted per device for an AlltoAll communication of size M across N GPUs is 4M/N.
Ring- Attention is not shown in the figure because it does not require resharding. We implement sequence communication in the temporal transformer as the time axis is split. In the attention module, it needs to pass the key and value to all other devices, resulting in a total communication volume of 2M.
DSP applics dynamic switching between stages to switch the parallel dimension, which involves two AlltAAll operations, totaling 2M/ /N communication.
As shown in Table 8 and 9, we demonstrate the communication volume for each communication method, and their latency for sequence of 8M length.
14

---

DSP: Dynamic Sequenee Parallelism for Multi-Dimensional Transformers
Megatron-LM
Spatial Transformer
All Reduca
All Reduse
10
OKV
Attn
MLP
[b. hn, hd]
[b. hd]
Temporal Transformer
Al Reduce
[b, t. hn, hd]
[b. hrve, hd]
1, hn, hd]
Megatron-SP
Spatial Transformer
Redduee Scotter
Reduce Seatter
All Gather
All Gather
Atti
[b, Mn, S. hn, hd]
[b, 1, hn, hd]
[b, 1, 5, hrin, hd]
[b, t, sh, hn, hd]
Temporal Transformer
Reduce Scatter
Redupe Scatter
Al Gather
Al Gether
CKV
Atti
MLP
bn, hd]
[b. hd]
[b, S, hd]
[b. s, hn, hd]
DS-Ulysses
Spatial Transformer
QKV
Atin
MLP
hd]
[b, hd]
Temporal Transformer
All
MLP
hd)
DSP
Spatial Transformer
Dynamic Switch
Temporal Transformer
Dynomic Switch
QKV
Atin
MLPP
to
All
All
hn,
hal)
[b. s. hn, hd]
b: baich size
L: temporal dimension
spatial dimension hm: atlention head num
hd: attention head dim
Figure 10. Overview of different sequence parallelism methods for 2D- 2D-Transformer.
15


Reviews:
[{"review_id": "ihBsohdALY", "reviewer": "Reviewer_4hHH", "summary": "The authors propose dynamic sequenced parallel (DSP), a model sharding scheme for multi-dimensional transformers. M-D transformers have two or more sequence dimensions unlike just one for the regular transformers. Existing sequence-parallel sharding does not account for this and are sub-optimal. DSP works by assuming the M-D transformer processes each sequence dimension separately (attention only ever looks across one sequence dim at a time). It does sequence parallelism across whichever sequence is being computed along, and switches sharding dim between blocks that are computing along different sequence dims. The switching can be accomplished by a low-cost all-to-all (cost = fraction of all activations).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer 4hHH for the valuable comments and acknowledgement of our work.", "claims_and_evidence": "DSP is designed specifically for multi-dimensional transformers and thus minimizes both communication volume and activation memory per shard. Like all sequence parallelism, DSP is most useful for long sequence lengths - the authors test on 0.5M - 4M lengths which are attainable for high-def video generation.\n\nDSP achieves much higher FLOP utilization that competing sequence parallelism methods - as much as 2x at 4M sequence length. It can also be combined with data parallelism to scale to 128 GPUs. DSP can be used in both training and inference - for inference it reduces latency compared to competing sequence parallelism.", "methods_and_evaluation_criteria": "The evaluation looks fine. The models and scenarios look realistic to me.", "theoretical_claims": "The authors analyze the theoretical communication volume and peak memory for the Open-Sora video generation model. This is a well know text/image-to-video model and their analysis simply looks at how tensors are divided and transferred between GPUs. The analysis is sound.", "experimental_designs_or_analyses": "Fine.", "relation_to_broader_scientific_literature": "It's a new form of sequence parallelism specifically designed for 2D transformers.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths: DSP looks sound and an obvious fit for 2D transformers. The evidence on the claims are good. Paper is well written.\n\nWeaknesses: The idea of DSP seems like low-hanging fruit. It feels like the type of optimization a proper ML engineering team would be able to make with or without a paper on the topic.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "BBcI7Ymp72", "reviewer": "Reviewer_Ux6h", "summary": "The paper introduces a new method, called Dynamic Sequence Parallelism (DSP), to scale multi-dimensional transformers efficiently by dynamically switching the parallel dimension at different computation stages. Unlike existing sequence parallelism techniques (e.g., Megatron-LM, Megatron-SP, DeepSpeed-Ulysses, Ring-Attention), which shard along a single sequence dimension, DSP exploits independent computation across multiple dimensions. By dynamically resharding between computation stages with an efficient all-to-all communication strategy, DSP significantly reduces communication overhead and improves throughput efficiency. The experiments, conducted on 128 NVIDIA H100 GPUs, demonstrate that DSP achieves up to 10x higher throughput while reducing communication volume by at least 50% compared to state-of-the-art methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer Ux6h for the valuable questions and comments. For the concerns and questions, here are our responses.\n\n**Q1: The experiments only use H100 GPUs, leaving performance on other GPUs unknown.**\n\n**A1:** Thanks for pointing out this! We test the performance on A100 GPUs.\n\n**Experiment1:** End-to-end performance on A100 on Transformer-2D 3B with 8 GPUs.\n\n| method      | throughput (samples/s) |\n| :---------- | :--------------------- |\n| DSP         | 1.08                   |\n| DS-Ulysses  | 0.88                   |\n| Megatron-SP | 0.23                   |\n\n**Experiment2:** Runtime breakdown for A100 and H100 on Transformer-2D 3B with 16 GPUs with DSP.\n\n| GPU | communication time | computation time |\n| ------- | ------------------ | ------------------ |\n| H100       | 24.8%               | 75.2%              |\n| A100       | 19.2%               | 80.8%              |\n\n**Analysis:**\n1. The performance improvement slightly decrease for A100 (from 23.5% to 22.7% compared with Deepspeed-Ulysses).\n2. The change is because A100 takes less time ratio to communicate than H100. Because H100 significantly improves computation capbility, therefore taking more time ratio to communicate than A100, making our method more effective.\n\n**Improvement plan:** We have added theroritcal analysis above to the Appendix of our work.\n\n**Conclusion:** For various GPUs, the more time communication takes, the more speedup our work can achieve.\n\n**Q2: Lacks theoretical analysis of complexity, memory, and communication costs.**\n\n**A2:** Thanks for pointing out this problem.\n\n**Communication costs:**\n\n| communication type | communication volume |\n| :----------------- | :------------------- |\n| all-reduce         | 2M                   |\n| all-gather         | M                    |\n| reduce-scatter     | M                    |\n| all-to-all         | M/N                  |\n\nTherefore, the communication volume for each parallel method is\n\n| communication type | communication volume                                  |\n| :----------------- | :---------------------------------------------------- |\n| Megatron-LM        | 4 x all-reduce = 4 x (2M) = 8M                        |\n| Megatron-SP        | 4 x (all-gather + reduce-scatter)= 4 x ( M + M ) = 8M |\n| DS-Ulysses         | 4 x all-to-all = 4 x (M/N)                            |\n| DSP                | 2 x all-to-all = 2 x (M/N)                            |\n\n**Memory cost:**\n\nFor memory, our method usually will not cause much extra memory cost as we show in Figure 9 (Line 447) because of the following reasons:\n\n* The sequence have already been splitted. Therefore there will be only a part of sequence on one device, reducing the overhead to layout change.\n* DSP changes actually less frequent than any other methods, lead to less overhead.\n* Duing training, the layout of a sinlge sequence does not affect overall memory cost because parameter and activation is much larger than it.\n* We use activation checkpointing during training.\n\n**Experimental analysis of memory cost:**\n\n| GPU num | memory overhead |\n| ------- | --------------- |\n| 8       | 0.5%            |\n| 16      | 0.4%            |\n\n**Theoretical analysis of memory cost:**\n\nIn training, everytime we create/change a tensor, it will make a copy, which causes extra memory cost. There are two reshape before and after communication. So let the tensor size be M, we will have activation memory cost of 4M for one communication.\n\nIn 2D-Transformer, every layer has activation memory for communication of 4M size as mentioned above, and other components of 42M. If we consider activation memory per layer only, the memory overhead is 8%.\n\nBut since we use activation checkpoint, the actual memory cost of our method during training will be `Communication Activation / (Communication Activation + Other Activation + Activation Checkpointing * Num Layers + Parameter + Gradient + Optimizer)`, which will be approximately less than 1%. Taking the settings of Transformer-2D 720M into this formula, the memory overhead is 0.67%.\n\n**Improvement plan:** We have added theroritcal analysis above to the Appendix of our work.", "claims_and_evidence": "The claims on the throughput improvements (32.2% to 10×) and communication volume reduction (≥50%) are supported by experiments on 128 H100 GPUs. However, some claims need stronger evidence. My only concern is about efficiency across different hardware as all experiments use H100 GPUs, with no analysis on other GPUs.", "methods_and_evaluation_criteria": "Yes, the evaluation aligns well with the problem, using 128 H100 GPUs to benchmark throughput, communication, and memory across long sequences, with comparisons against Megatron-LM, DeepSpeed-Ulysses, and Ring-Attention. However, a Pareto-optimal trade-off analysis and tests on older GPUs or real-world tasks would strengthen the evaluation.", "theoretical_claims": "The paper lacks a formal theoretical analysis of DSP’s efficiency. While it provides empirical results showing improved throughput and communication efficiency, there are no formal proofs or mathematical derivations to support its claims.", "experimental_designs_or_analyses": "Yes, the experiments are sound and appear correct.", "relation_to_broader_scientific_literature": "This work definitely helps to perform experiments on large language models more efficiently and faster.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n-- Dynamically adjusts the sharding dimension based on the computation stage\n\n-- Significant improvements on throughput by 32.2% to 10× over existing methods.\n\n-- Lower communication cost by at least 50%.\n\n-- It scales efficiently across multiple GPUs.\n\n\nWeaknesses:\n\n-- The paper lacks theoretical analysis of complexity, memory, and communication costs.\n\n-- The experiments only use H100 GPUs, leaving performance on other GPUs unknown.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See the weaknesses!", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XcDuNMfZf2", "reviewer": "Reviewer_qVhs", "summary": "This paper presents DSP (Dynamic Sequence Parallelism) for scaling multi-dimensional transformers. The solution adaptively switches parallel dimensions by reshuffling data with all-to-all communication between multiple GPUs. The evaluation results demonstrate that parallelizing across sequence dimensions can reduce the communication overhead and improve throughput compared to the existing methods that shards the sequence dimension.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer qVhs for the valuable questions and comments, especially about the evalutions which are indeed not detailed enough. \n\n**We have conducted the following experiments to make our evalutions more comprehensive and have added them to the experiment section or appendix or in the latest version.**\n\n**Q1: DSP's scalability according to the model size.**\n\n**A1:** We conduct an end-to-end analysis for DSP with 3B, 7B and 13B using 1M tokens.\n\n| GPU num | model size | TFLOPs per GPU |\n| ------- | ---------- | -------------- |\n| 16      | 3B         | 242.20         |\n| 16      | 7B         | 262.29         |\n| 16      | 13B        | 259.77         |\n\nAnalysis: \n\n* When scale from 3B to 7B, although the communication cost increase, the performance grows because the compuation density increases more.\n* When scale from 7B to 13B, the benefits of denser computation is marginal. The total throughput decrease because of more communication cost, but the loss is still marginal.\n\n**Q2: Breakdown analysis of communication overhead.**\n\n**A2:** We breakdown for communication overhead/actual computation for DSP under weak scaling condition using Transformer-2D 3B:\n\n| GPU num | computation time | communication time |\n| ------- | ---------------- | ------------------ |\n| 8       | 90.1%            | 9.9%               |\n| 16      | 75.2%            | 24.8%              |\n\n**Analysis**: The communication costs grows significantly for inter-node communication even for our method. Highlight the necessity of efficient parallelism.\n\nIf you need more experiments for each parallel or model, please tell us!\n\n**Q3: Cost of layout change/reshape/transpose.**\n\n**A3:** Let's analyze from memory and speed:\n\nFor memory, this usually will not cause much fragmentation as we show in Figure 9 (Line 447) because of the following reasons:\n\n* The sequence have already been splitted. Therefore there will be only a part of sequence on one device, reducing the overhead to layout change.\n* DSP changes actually less frequent than any other methods, lead to less overhead.\n* Duing training, the layout of a sinlge sequence does not affect overall memory cost because parameter and activation is much larger than it.\n\nTo further clarify this, we make experiments about the memory overhead of our method:\n\n| GPU num | memory overhead |\n| ------- | --------------- |\n| 8       | 0.5%            |\n| 16      | 0.4%            |\n\nFor speed, we conduct an experiment to evaluate the time of changing layout and communication:\n\n| GPU num | layout change time | communication time |\n| ------- | ------------------ | ------------------ |\n| 8       | 5.2%               | 94.8%              |\n| 16      | 4.1%               | 95.9%              |\n\n**Analysis:** \n\n* The layout changing time is much less than the communication time. But still takes 5.2% of total time. \n\n* As the sequence and gpu number get larger, the layout change time become less as communication takes more time.\n\n* To fully elimiate the layout change's time, we can overlap communication with layout change.\n\n* We use activation checkpointing in training.\n\n**Conclusion**:\n\n* Memory: The layout changes have little impact on memory cost.\n* Speed: The layout changes will occupy 4.1%-5.2% of communication time, but can be avoided by overlapping. And it will reduce as sequence and gpu becomes larger.", "claims_and_evidence": "The proposed method is mostly clear and makes sense. However, I think there should be an evidence to claim DSP's generalizability and the future work seems to be not directly related to the proposed solution, or it could be too optimistic.", "methods_and_evaluation_criteria": "The experiments techincally sound, but it lacks the detailed analysis in the communication time/computation time, etc. Also, it is not clear that why the input sequence length needs to be fixed in their experiment. I think the experiment should show the scalability according to the sequence length, because it may impact the accuracy of the model, especially the video generation use case.", "theoretical_claims": "Yes, on the complexity analysis.", "experimental_designs_or_analyses": "- Please add an experiment that shows DSP's scalability according to the model size. \n- There should be a breakdown analysis, showing communication overhead. \n- I think memory layout change/reshape/transpose may incur mem copies, so that the memory cost (both capacity and time) could be significant. There should be analysis on this.", "relation_to_broader_scientific_literature": "The proposed solution tackles the emerging problem but its applicability shown in the evaluation seems to be limited.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see questions raised in other sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p4ZyE8ai7e", "reviewer": "Reviewer_o73Q", "summary": "The paper introduces Dynamic Sequence Parallelism (DSP) as a novel approach to sequence parallelism for multi-dimensional transformers.\nIt addresses the limitations of embedded sequence parallelism, which shards along a single sequence dimension and incurs significant communication overhead.\nDSP dynamically switches the parallel dimension of sequences during computation with an efficient resharding strategy.\nThe paper claims that DSP reduces communication costs, enhances adaptability, and is easy to use, offering significant throughput improvements.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer o73Q for the valuable questions and comments. For the concerns and questions, here are our responses:\n\n**Q1: Performance gains are not convincingly demonstrated for long sequences**\n\n**A1**: We thank the reviewer for this important question and have conducted additional experiments to robustly demonstrate DSP’s performance on long sequences.\n\n**Experiment1:** Communication latency comparison.\n\n| method      | latency with 0.5M seq (ms) | latency with 8M seq (ms) |\n| :- | :- | :- |\n| DSP         | 614                        | 2914                     |\n| DS-Ulysses  | 1228                       | 5828                     |\n| Megatron-SP | 5616                       | 51908                    |\n\n**Analysis1:** DSP consistently reduces communication latency across sequence lengths.\n\n**Experiment2:** End-to-end speedup for 8M long sequecnes with two seuqence shapes A:[8192, 1024] and B:[8, 1048576].\n\n| method      | communication time for A (s) | computation time for A (s) | communication time for B (s) | computation time for B (s) |\n| :-- | :-- | :-- | -- | -- |\n| DSP         | 2.8                          | 34.3                       | 2.9                          | 362.1                      |\n| DS-Ulysses  | 5.8                          | 34.3                       | 5.8                          | 362.2                      |\n| Megatron-SP | 51.9                         | 37.2                       | 51.9                         | 365.6                      |\n\n**Analysis2:** DSP significantly reduces communication time for both shapes. However, end-to-end speedup varies: shape A benefits more due to lower computation time, while shape B’s longer single dimension increases attention computation, dominating total time. This shows DSP’s communication efficiency, though overall gains depend on computation complexity.\n\n**Improvement plan:** We have added a detailed discussion of these results in the Appendix of the revised manuscript.\n\n**Conclusion**:\n\n1. DSP delivers consistent communication reductions across sequence.  \n2. End-to-end speedup depends on computation time, which varies with sequence dimensionality.\n\n\n\n**Q2: Is it possible to extend it to other models other than multi-dimensional transformers?**\n\n**A2:** DSP’s simple yet general design extends beyond multi-dimensional transformers to various **multi-dimensional neural networks**, including conv+attention models (e.g., 2D-UNet, 2D-VAE), AlphaFold, and pure conv. \n\nFor example, in 2D-UNet, DSP enables efficient resharding between computations along different dimensions:\n\n$ x -> conv\\\\_dim(1) -> attn\\\\_dim(1) -> DSP\\\\ reshard -> conv\\\\_dim(2) -> attn\\\\_dim(2) -> y$\n\n\n\n**Q3: The paper should include more comprehensive experiments on diverse multi-dimensional transformer architectures.**\n\n**A3:** We thank the reviewer for raising this point. Main stream multi-dimensional transformer architectures typically fall into two categories:  \n\n1. Dimension-by-dimension processing as shown in our paper:\n\n$x -> transformer\\\\_dim1(x) -> transformer\\\\_dim2(x) -> y$\n\n2. Dimension-by-dimension with cross-dimensional interactions:\n\n$x -> transformer\\\\_dim1(x) -> transformer\\\\_dim2(x) -> transformer\\\\_cross\\\\_dims(x) -> y$\n\n**Experiment:** End-to-end performance on the second architectures on Transformer-2D 3B with 8 GPUs.\n\n| method      | throughput (samples/s) |\n| :--- | :--- |\n| DSP         | 1.42                   |\n| DS-Ulysses  | 1.15                   |\n| Megatron-SP | 0.40                   |\n\nThere many be some changes within the transformer (e.g., layernorm, positional embedding) but they don't affect much for the results.\n\n**Improvement plan:** We have added a detailed discussion of these results in the Appendix of the revised manuscript.\n\n**Conclusion:** Our method is able to deliver stable performance on main-stream multi-dimensional transformer architectures.\n\n\n\n**Q4: The authors should provide a more thorough analysis of the trade-off between performance and accuracy.**\n\n**A4:** DSP is designed to preserve accuracy strictly, introducing only negligible errors (1e-7 to 1e-8) due to communication. For example, in video generation tasks, DSP outputs are identical to the originals.\n\n\n\n**Q5: How can DSP be further optimized to provide more substantial speedups in scenarios with small overhead from baseline methods?**\n\n**A5:** While DSP already minimizes communication volume effectively, further speedups are possible by overlapping communication with computation. DSP’s lightweight communication and decoupling from computation make it **ideal for asynchronous execution**, hiding latency behind computation. \n\n\n\n**Q6: Essential references not discussed (GSPMD)**\n\n**A6:** We appreciate the reviewer highlighting this point.\n\n**Improvement plan:** In the revised manuscript, we have added a detailed discussion of GSPMD and other relevant works in the related work section, ensuring a more comprehensive literature review.", "claims_and_evidence": "The paper claims that DSP improves throughput and reduces communication volume for multi-dimensional transformers.  While experiments show improvements, the evidence is limited, primarily focusing on video generation models. The authors mention that \"each dimension is processed by a corresponding transformer block, which is a common strategy in many applications\". Could the authors add citations there and add specific examples?", "methods_and_evaluation_criteria": "The paper evaluates DSP using video generation models, which is a relevant application for multi-dimensional transformers.  The comparison with state-of-the-art sequence parallelism methods (DeepSpeed-Ulysses, Megatron-SP, Megatron-LM, and Ring Attention) is appropriate. However, the evaluation is narrow.", "theoretical_claims": "The paper includes a communication analysis comparing DSP with other sequence parallelism methods.  It claims DSP minimizes communication costs.", "experimental_designs_or_analyses": "The experimental design focuses on demonstrating DSP's superiority in video generation models by varying sequence lengths and comparing throughput and communication volume.", "relation_to_broader_scientific_literature": "The paper discusses relevant prior work, including data parallelism, tensor parallelism, pipeline parallelism, and sequence parallelism.  It positions DSP as a novel abstraction of sequence parallelism for multi-dimensional transformers.", "essential_references_not_discussed": "[1] is an paper that presents the general SPMD intra-layer parallelism, which should be discussed. I think this paper proposes a new sharding annotation for a specific model.\n\n[1] Xu, Yuanzhong, et al. \"GSPMD: general and scalable parallelization for ML computation graphs.\" arXiv preprint arXiv:2105.04663 (2021).", "other_strengths_and_weaknesses": "Strengths\n1. The paper addresses scaling multi-dimensional transformers, an important problem.    \n2. The concept of DSP is clearly presented.    \n3. DSP has good adaptability and flexibility.    \n4. The paper provides a user-friendly API for DSP.    \n\nWeaknesses:\n1. The empirical evaluation is limited.    \n2. Performance gains are not convincingly demonstrated for long sequences.", "other_comments_or_suggestions": "1. The paper should include more comprehensive experiments on diverse multi-dimensional transformer architectures.    \n2. The authors should provide a more thorough analysis of the trade-off between performance and accuracy.", "questions_for_authors": "1. How can DSP be further optimized to provide more substantial speedups in scenarios with small overhead from baseline methods?\n2. Is it possible to extend it to other models other than multi-dimensional transformers?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a new method, called Dynamic Sequence Parallelism (DSP), to significantly reduces communication overhead and improves throughput efficiency. DSP offers significant reductions in communication costs, and the experiments demonstrate its superiority over state-of-the-art sequence parallelism methods. In addition, the authors also successfully convinced reviewers by clarifying some concerns. Thus, acceptance is recommended.
Decision: Accept (poster)