Synergistic Tensor and Pipeline Parallelism
Authors: Mengshi Qi
Jiaxuan Peng
Jie Zhang
Juan Zhu
Yong Li
Huadong Ma
Venue: NeurIPS 2025 poster ()
Keywords: Distributed System
Pipeline Parallelism
Large-Scale Model Training
LLM
Machine Learning System
Primary area: infrastructure
TL;DR: A framework for tensor and pipeline parallelism to reduce TP bubbles.

Abstract: In the machine learning system, the hybrid model parallelism combining tensor parallelism (TP) and pipeline parallelism (PP) has become the dominant solution for distributed training of Large Language Models~(LLMs) and Multimodal LLMs (MLLMs). However, TP introduces significant collective communication overheads, while PP suffers from synchronization inefficiencies such as pipeline bubbles. Existing works primarily address these challenges from isolated perspectives, focusing either on overlapping TP communication or on flexible PP scheduling to mitigate pipeline bubbles. In this paper, we propose a new synergistic tensor and pipeline parallelism schedule that simultaneously reduces both types of bubbles. Our proposed schedule decouples the forward and backward passes in PP into fine-grained computation units, which are then braided to form a composite computation sequence. This compositional structure enables near-complete elimination of TP-related bubbles. Building upon this structure, we further design the PP schedule to minimize PP bubbles. Experimental results demonstrate that our approach improves training throughput by up to 12\% for LLMs and 16\% for MLLMs compared to existing scheduling methods. Our source code is avaiable at https://github.com/MICLAB-BUPT/STP.

Synergistic Tensor and Pipeline Parallelism
Mengshi Qi" Jiaxuan Peng', Jie Zhang, Juan Zhu, Yong Li, Huadong Ma' State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, China {qms, pjx, mhd]-&bupt. .eapppt..dd.... edu. cn
Abstract
In the machine learning system, the hybrid model parallelism combining tensor parallelism (TP) and pipeline parallelism (PP) has become the dominant solu- tion for distributed training of Large Language Models (LLMs) and Multimodal LLMs (MLLMs). However, TP introduces significant collective communication overheads, while PP suffers from synchronization inefficiencies such as pipeline bubbles. Existing works primarily address these challenges from isolated per- spectives. focusing either on overlapping TP communication scheduling to mitigate pipeline bubbles. In this paper, we propose a new synergistic tensor and pipeline parallelism schedule that simultaneously reduces both types of bubbles. Our proposed schedule decouples the forward and backward passes in PP into fine-grained computation units, which are then braided to form a com- posite computation sequence. This compositional structure enables near-complete elimination of TP-related bubbles. Building upon this structure, we further design the PP schedule to minimize PP bubbles. Experimental results demonstrate that our approach improves training throughput by up to 12% for LLMs and 16% for MLLMs compared to existing scheduling methods. Our source code is avaiable at https://gihub..oo/MMCCLBB BUPP/STP
on flexible PP
Introduction
Distributed systems 6 23
38
31
have become the cornerstone for the training of large-scale 20), particularly since model scales have grown
machine learning models 1 35| 28
26
27
commonly employed accelerate training and is most effective for relatively smaller models. As the scale of the model increases, researchers have turned model parallelism
of billions of parameters. Data parallelism [12
15
5
[10 32 21.. which partitions the model into smaller components and distributes them between multiple devices. Model parallelism primarily consists
of two strategies: tensor parallelism (TP) (33) and pipeline parallelism (PP) 7 23 30). Tensor parallelism partitions model weights across devices according to predefined rules, thereby reducing per-device memory requirements. While TP alleviates GPU memory pressure some extent, introduces additional collective communication operations within TP groups, referred to as TP bubbles that exhibit significant growth with increasing TP size, as shown Figure [ For instance, the TP bubbles account for 27.5% of the overall time in the configuration of TP-8 and sequence length of 6,144, 144, which can significantly degrade the performance. the model into chunks along the layer dimension, which are assigned to PP stages according to the PP strategy. Activations and gradients are then transmitted among PP stages during the forward and backward passes. This strategy enables efficient utilization of computational resources but requires careful synchronization between stages to minimize idle time, commonly referred to as PP bubbles.
contrast, pipeline parallelism divides
To mitigate TP bubble issues, existing studies 37
aim to overlap communication with computation at the kernel level, such as customized CUDA kernels or the hardware level,
2
25
"Corresponding author: qms@quu mims@butt.cdu.cn.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Attention
Attn TP Comm.
MLP
MLP TP Comm
Naive
Naive
Naive
Naive
1.0
Normalized Time
Ours
Ours
0.8
Ours
OuiS
0.6
0.4
0.2
0.0
seq=3072, 1p=4 seq=3072, 1p=8 seq=6144, 4p=4
seq 6144, 8p=8
Figure 1: Speedup of overlapping TP communication with computation in PP within a Transformer layer of Qwen2 33 in forward pass. The proportion of TP communications grows significantly with the increased TP size, which are effectively overlapped in our schedule compared with the naive implementation.
where specialized hardware mechanisms like the track-and-trige system [25 are employed. By carefully scheduling paired decomposed communication and computation chunks, these methods effectively reduce overall execution time. As for PP bubbles, GPipe 7 reduces pipeline bubbles by splitting the global batch into microbatches. Subsequently, 1FIB [22] improves upon GPipe by scheduling backward passes earlier, thereby optimizing peak memory footprint. Furthermore,
IFIB-I (23) reduces the bubble rate by introducing virtual stages assigned to individual devices. More recently, Zero Bubble (30 decouples the full backward pass into separate activation and weight gradient computations, enabling more flexible scheduling and achieving significantly lower PP bubble rate.
In practice, hybrid model parallelism, particularly the combination of PP and TP, has become the dominant approach for large-scale model training. However, most of the aforementioned works primarily focus optimization from isolated perspectives, generally yielding limited enhancements. Inspired by studies [17 [44] that integrate two kinds of parallelism strategies for better optimization, work aims integrate PP and TP from synergistic perspective. Specifically, leverages the abundant computation operations inherent in PP to overlap the TP bubbles and employs precise scheduling to mitigate the PP bubbles. Firstly, decouple the forward and backward passes of the model chunk within each stage of PP into fine-grained computation units. By interleaving forward and backward computation units construct execution blocks, achieve near-zero TP idle time as shown in Figure | where TP communication and the fine-grained computation units in PP execute in parallel. Building upon these blocks, we further design a PP schedule with low bubble rate and balanced memory footprints across stages. Additionally, we provide an enhanced variant of our schedule that incorporates activation offloading, serving as a preliminary attempt under limited memory conditions. Our main contributions are summarized as follows:
method
We rethink hybrid model parallelism from a synergistic perspective and decouple the forward and backward passes of PP into fine-grained computation units. which are interleaved together to form braided execution blocks that effectively eliminate TP communication bubbles at the scheduling level.
We design a novel PP schedule that synergistically integrates with TP based on the braided execution blocks. This schedule features a "V"-shape dataflow that achieves balanced memory footprints and a significantly reduced PP bubble rate.
We conduct extensive experiments on popular LLMs and MLLMs of various scales to validate the universality and effectiveness of our method. The results demonstrate that our proposed schedule improves throughput by up to 16% compared to the existing methods.
2
Related Work
Tensor Parallelism. Tensor parallelism 33
partitions tensors across devices to reduce per-device memory usage but introduces communication overhead due to synchronization requirements. This problem becomes more pronounced with TP sizes increase, where the limited
14
37 8 2

---

inter-device bandwidth (e.g., systems without NVLink (24) can severely degrade training efficiency. Several prior works [37 8 2 optimize this kind of collective communication by breaking down both communication and corresponding computation operations into sequences of smaller chunks. Subsequently, they carefully coordinate the execution of these granular units to achieve acceleration, often through the intricate fusion of computation and communication kernels, along with complex hardware control. In contrast, our proposed schedule optimizes TPP communication at the scheduling or software level, providing more accessible and user-friendly approach.
Pipeline Parallelism. Pipeline parallelism [6
18
divides the entire model
into smaller chunks using variety of strategies. GPipe 7 reduces PP bubbles by splitting the
global batch into smaller micro-batches. Building on this, PipeDream [22 improves upon GPipe by
advancing the backward computation to further reduce memory usage. 1FIB-I (23 and Hanayo [18]
further introduce additional virtual stages to lower the bubble rate. More recently, Zero Bubble [30] decouples the backward computation into activation gradient and weight gradient computation to
achieve near-zero PP bubble schedule. However, these existing schedules tend address PP bubbles
in isolation. In this paper, design integrated approach that concurrently minimizes both TP and PP bubbles by synergistically combining these two parallelism techniques.
Attention
M'
8
Pre-Attn
Attn
Pre-MLP
MLP
Figure 2: Illustration of the main computation and communication operation in single TP Trans-
former layer. f is an identity operation in forward while All-Reduce in backward, is opposite to f.
Time
Comp
Attn (F)
MLP Act (B)
|MLP W (B)
MLP (F)
Attn Act (B)
Attn (B)
Stream
Comm
AR
AR
AR
AR
Strcam
(a) Exccution block with forward and full backward.
Comp
Attn (F)
ML.P Act (B)
MLP (F)
Attn Act (B)
Attn (F)
Stream
Comm
AR
AR
AR
AR
AR
Stream
(b) Execution block with forward and activation backward.
Figure 3: Two types of execution blocks that braid the forward (F) and backward (B) computation units to overlap the TP communications (All-Reduce), denoted as AR.
Braided Execution Block
Although TP reduces memory consumption, introduces additional communication overhead after the Attention and MLP computations in both the forward and backward passes, as illustrated in the orange blocks in Figure 2 As shown in the blue blocks of Figure 3al the communication represented by the blocks in the communication stream can be naturally overlapped with the weight gradient computation during the backward pass, where TP communication (All-Reduce) and weight gradient computation run in parallel. However, the communication in the solid blocks during the forward pass cannot be easily overlapped with other operations at software level due to data dependencies, leading to massive TP bubbles. To address this TP bubble issue, we design two types of execution blocks that overlap TP communication with computation through braiding forward and backward computation units, shown in Figure 3 These blocks serve the foundation of our schedule.

---

Specifically, we decompose the entire Transformer layer into several fine- grained computation units: the Pre-Attn unit, Attn unit, Pre-MLP unit, and MLP unit. The units from the forward and backward passes are then interleaved sequentially to form braided execution block, as shown in Figure 3a Moreover, both the Attn and MI.P units backward passes are further split into two components: activation gradient computation and weight gradient computation, following Zero Bubble [30]. When necessary, we activate the separation of weight gradient computations to balance the workload and reduce pipeline bubbles. This separation does not disrupt the execution blocks, forward computation can fill the bubbles introduced by the separation, as shown in Figure 3b The Pre-Attn and Pre-MLP units inserted into the computation stream according to their computational dependencies. All these operations only involve model-level modifications, making our approach more user-friennt swer-ffrendly compared to kernel-level implementations.
the subsequent
Furthermore, considering the residual computations after the Attn and ML.P units that introduce additional data dependencies and increase engineering complexity for backward passes, modify the forward and backward computations to fuse the residual computation into the Attn and MLP units before applying operation g. This modification preserves computational equivalence and does not affect model convergence. The modified forward computation of the Attn unit for a single TP rank can be formulated as follows:
Xin Loyer)Norm(X), Xattn AR(Attentiom(X..) + detach(X)
(1)
where t denotes the tensor parallel size, detach(-) is an operation that stops gradient computation of X, Attention()) the attention computation performed without any communication, and (-)
represents the All-Reduce communication operation. The gradient computation g(X) X) of the hidden state X is then formulated as the following:
@Xattn
axim
g(X) AR
iX
0X
where the term +1 accounts for the gradient contribution from the residual connection. Similarly, the MLP unit adopts comparable computation fusion strategy.
Pipeline Parallelism Schedule
We begin by introducing several key principles that facilitate the construction of the overall schedule Section 4.1 and then introduce the proposed schedule in Section 4.2 Following this, we present a
theoretical analysis of the PP and TP bubbles and peak memory in Section 4.3 Finally, we propose an enhanced variant of our schedule to accommodate scenarios with limited memory Section 4.4
4.1 Building Principles
To ensure an effective pipeline schedule, we outline several guiding principles aimed at optimizing memory efficiency and minimizing bubbles.
Memory
Memory
Rank
Rank 2
1
Rank
Forward for virtual stages
Backward for virtual stages
Parallel
"v"--Shae
Figure 4: Comparison between parallel and "V"-shape pipeline flow for microbatch 1, which shows improved memory balance across stages. That is attributed to the early backward pass device O.
Balanced Memory Footprint: As discussed in 29
18 the IF1B-I 23 often leads memory
bottleneck on the first device due the parallel dataflow of virtual stages, while subsequent stages often have underutilized memory resources. This imbalance results in inefficient resource usage. To

---

utilize a "V""-sape scheduling strategy to achieve more balanced distribution of peak memory across all devices (see Figure We follow this principle by implementing "V""-shae dataflow for our virtual stages to optimize memory efficiency, which also supports the scheduling of braided execution blocks for improved overall performance.
tackle this issue, current scheduling methods [8 17
The braided Section 3] illustrate the arrangement of computation units. The practical construction of these blocks follows two primary patterns: (1) pairing the forward units of one virtual stage with the backward units of another, or (2) braiding the forward and backward units of the same model chunk across different microbatches. The selection of the building pattern is closely tied to the characteristics of virtual stages in PP. Specifically, when partitioning models into virtual stages, it essential to maintain approximately equal execution times across these stages to mitigate pipeline bubbles. For MLLMs, significant disparity often exists between the vision encoder (e.g.. ViT) and the Language Model (LM) due to differences in their hidden sizes. This presents key challenge for multimodal models, where earlier virtual stages typically contain more ViT layers, while later stages consist of fewer LM layers to achieve balanced execution times. However, the imbalance in the number of computation units across virtual stages limits the effectiveness of pattern (1), as mismatched unit counts prevent efficient hiding of TP communication during cross- cross-virtual virtual- -ttge pairings. In contrast, pattern (2), which involves overlapping the forward and backward units of the same chunk, more effectively minimizes the TP bubbles. Therefore, adopt pattern (2) to construct the basic blocks for universality, especially considering the multimodal models.
Bubble Minimization:
potpruattioo-oommmiiciiiiooo execution blocks, as detailed in
12
12
2
6
2
7
3
:
4
8
X1
12
W
overlapped F&B
overlapped F
Figure 5: Synergistic tensor and pipeline parallel schedule with the setting of 4 devices and 12 micro- batches. The dark and light pieces indicate the computation of model chunks and 1, respectively. F, B, and W represent the forward, backward, and backward for weight gradients, respectively.
4.2 Synergistic Tensor and Pipeline Parallel Schedule
Based on the above building principles, we construct a synergistic tensor and pipeline schedule, which implements "V"-shape dataflow across stages for single microbatch, illustrated in Figure 5 Similar to existing schedules 22 23 30], our proposed method comprises three main phases: the warm-up phase, the steady phase. and the cool-down phase. First, the warm-up phase begins with the maximum feasible number of in- in-flight microbatches before the commencement of the first backward pass. This helps to maintain a low PP rate and facilitates the construction of the braided blocks, as the backward passes of carlier microbatches form the blocks in conjunction with the forward passes of subsequent microbatches. The number of microbatches involved in the warm-up phase is determined by the number of stages. During this phase, the overlapped forward and backward (F&B) start from the braided computation of the first and second microbatches, and weight gradient separation is activated to quickly propagate gradients to the next stage, except for the last stage.
After several iterations of overlapped forward and weight (F&W) and F&B, the schedule transitions into the steady phase. In this phase, weight separation is deactivated as long as new microbatches continue to arrive. Under these conditions, the system performs one F&B for model chunk followed by one F&B for chunk O. When the supply of microbatches is exhausted, maintaining full F&B becomes infeasible, and the process degrades into a full backward pass followed by separated F&B. During this degraded phase, weight separation is reactivated to align the time of F&B with that of the full backward pass. During the cool-down phase, the backward passes for all remaining in in-ftight microbatches are completed. The pipeline bubbles are filled with stored weight gradient computations from earlier steps. Moreover, activating weight separation during the steady phase leads to memory accumulation, as the corresponding weight computations remain pending until the final steps. Therefore, the saved activations required for the weight gradients are offloaded to the CPU in parallel with the computation streams and reloaded when necessary.
5

---

4.3 Theoretical Analysis
We define the number PP stages and microbatches as p and m respectively, where p & m. For simplicity, multimodal models are not taken into account in this context. For LLMs, assume the model evenly divided into chunks along with the layer dimension, and each microbatch requires the same activation memory M, for each model chunk across all devices. The computation time for the forward pass one chunk denoted as TF. while the durations of activation and weight gradient computation are TB and Tw, respectively. The TP communication time of one model chunk is represented as TAR, which remains consistent for both forward and backward passes.
Table I:
Comparison between 1FIB-I, Zero Bubble V, and our proposed schedule theoretical PP bubble size, non-overlapped TP communication bubble, and peak activation memory.
Schedule
PP Bubble
TP Bubble
Peak Act. Memory
1FIB-I
(p - 1) (Tr + TAR + TB + Tw)
2 m TAR
(3* 2) * Ma
ZB-V
(p - 1) (Tr AT + TB 27w)
4 m TAR
2 Ma
Ours
(p- 1) (TF TAR + TB Tw)
(2*p + 1) TAR
3 3*p* Ma
As shown in T IFIB-I 23) with 2 virtual stages requires (3*p-2) ()pp2))=2))) Ma peak activation memory for the first device, which progressively decreases as the stage index increases. In the backward pass, the TP communications
overlapped naturally, but the TP communications in the forward pass non-overlapped, resulting in total TP communication time is 2 m TAR. In contrast, Zero Bubble () [29] achieves smaller theoretical bubble size of (p 1) (TF 2TAR + TB 2Tw) and maintains the lower peak activation memory than that of IFIB-I. However, due to the decoupling of the backward into activation and weight gradient computations, the TP communication in both forward and activation backward pass becomes non-overlapped, leading total TP communication time of 4 m * TAR and larger practical bubble size. Our schedule exhibits the approximate bubble size of (p - 1) (Tp +TAR + Ta Tw) compared to the ZB-V LB-V schedule, which is much smaller than that I-I And it achieves substantially reduced TP overheads, benefiting from the overlapped computation and communication, albeit with trade-off in the peak activation memory.
4.4 Enhanced Pipeline Schedule
Considering the practical demands of limited CUDA memory, we attempt to reduce the peak memory footprint of our scheduling strategy. The elevated peak memory usage primarily stems from two phases: the warm-up phase, where the number of in-flight microbatches maximized to maintain a low bubble rate, and the steady phase, where activations from model chunk 0 persist for extended period before being released, denoted as lifespan. Existing methods 44) 34 such as activation checkpoint and offloading have been used to reduce memory. However, checkpointing introduces extra forward pass during backward, increasing execution time. Thus, activation offloading offers better trade-off between memory reduction and e2e performance, when high-speed bandwidth is available between the host and the device.
8
4
8
8
2
5
9
W
F&B
F&W
Offload
Reload
Figure 6: Mustration of our proposed schedule with offloading and reloading for one microbatch in warm-up and steady phase with the setting of PP size 2.
During the warm-up phase Figure where memory usage accumulates rapidly, the activations of chunk I have short lifespan and are thus deprioritized for offload. The offload time T,, determined

---

by the offloading ratio a, is restricted to be less than the forward time Tp to avoid interference with subsequent operations. In the steady phase, chunk 0 activations persist much longer than those in chunk 1, making them the primary target for offloading. The offload ratio in this phase can be set higher than in the warm-up phase due to the braided execution blocks. To minimize PCIe bandwidth contention caused by frequent dual CPU-GPU data transfers, we avoid offloading chunk I activations, ensuring they can be reloaded time without disrupting computation. The ratio a is adjusted based on hardware characteristics such PCIe bandwidth and FLOPs.
5 Experiments
5.1 Setup
We evaluated our proposed schedule on the series of Qwen2 (LLM) [39 and Qwen2-Ve 2LL (MLLM) [36 models, as detailed Table 2 efficiency. Our implementation is built upon the open-source Megatron-Cort project [33] and tested on up 32 NVIDIA A800 SXM4 80G GPUs distributed across nodes. Each experimental result recorded after several warm-up iterations to ensure stability.
Meanwhile, Flash Attention 33
leveraged in all models for
In LLM scenarios, we adopt the approach described in [30], uniformly splitting the model while ensuring that the last stage contains two fewer layers than the other stages, considering the large vocabulary size of 152,064 in Qwen. In MILM scenarios, the ViT encoder is assigned the first virtual stage on device O, and the LM model is uniformly distributed across the remaining virtual stages, and the last virtual stage also contains two fewer layers compared the other stages.
To be specific, our experiments primarily concentrate on the following pipeline parallel schedules: a) interleaved IFIB (1F1B-I), implemented in Megatron-LM 23 which reduces the bubble ratio by interleaving stages among workers but slightly increases memory footprint; b) ZB-V, introduced in [29], which achieves balanced peak memory distribution through a "V""-shape dataflow; and c) our proposed schedule. For consistency, all schedules are configured with two virtual stages per device, and the implementations based open-source codebases.
Table Model configurations evaluated on the experiments.
ViT Layers
ViT Heads
VIT
LM
LM
LM
LM
Model
Scale
Dim.
Layers
Q Heads
KV Heads
Dim.
LLM
12.
30
40
5120
LLM
26.3B
46
56
7168
MILM
14.9B
32
16
2048
33
40
5120
MLLM
28.8B+
26
16
4096
40/43
56
7168
IFIB-I
ZB-V
Ours
seq=3072
seq=3072
seq=6144
6144144
1p=4. 4p=4
tp=8, pp-2
4p=4, pp=4
tp=8, pp=2
10.5
5.0
3.6
7.5
Throughput
10.0
Tri
I
4.8-
3.4
4.6
9.5
6.5
3.2
4.4
9.0
6.0
4.2
3.0
4.0
2.8
64
128
192
64
128
192
64
128
192
64
128
192
microbatches
mierobatches
microbatches
microbatches
Figure 7: Experiment results 12.1B LLM across 16 GPUs in terms of throughput (samples per
second).
5.2
Comparison with Existing Schedules LLM
In Figures 7 and we report experimental results by evaluating Qwen2 on 2-node and 4-node setups, respectively, showing the achieved throughput under various pipeline strategies, sequence lengths,
7

---

IFIB-I
ZB-V
Ours
Scq=2048
soq=2048
40q
Scq=4096
1p 4.
pp-8
1p-8, pp-4
tp 4, pp-8 8
tp-8, pp
13.5
10.0
5.0
6.6
Throughpu
13.0
9.5
11|
4.7
6.4-
4.4
12.5
4.1
12.0
8.5
6.2
3.8
11.5
8.0
6.0
3.5
96
176
256
96
176
256
96
176 microbatches
256
96
176
256
microbatches
microbatches
microbatches
Figure 8: Experiment results on 26.3B LLM across 32 GPUs in terms of throughput.
and model scale configurations. Our proposed schedule outperforms all other strategies across all settings in terms of throughput. As shown in the right part Figure 7 our method achieves a maximum improvement of 12.2% over IFIB-I. This performance gain is attributed the large TP size of 8, the long sequence length of 6, 144, and the low PP size of 2 used in the experiments. As illustrated in Figure [ TP-related bubbles occupy significant proportion under TP size 8, especially for attention units, which are effectively mitigated by our scheduling approach.
However, the performance improvements under PP=8 in Figure g are less pronounced. This is due to the explicit pipeline communication our schedule, which executed immediately after computation and cannot be overlapped with computation, thereby negatively impacting performance. Nevertheless, our scheduling strategy still outperforms the other two methods. The highest throughput improvements for both 12B and 26B LLMs achieved at TP=8, demonstrating that method offers substantial potential when scaling to larger models with TP=8, even TP=16. Notably, ZB-V demonstrates comparable
even worse performance than 1FIB-L. Since V is based the decoupling of the full backward pass, which exposes previously overlapped TP bubbles during the full backward in IFIB-I and introduces additional bubble overheads, thereby eliminating the potential benefit reduced PP bubbles from decoupling.
Furthermore, show the peak activation memory footprints with 4 and 2 PP stages in Figure While our schedule exhibits a slightly higher peak memory footprint, can achieve optimal throughput, especially under eeemory-uncosstraing scenarios. For memory-limiter scenarios, we introduce an enhanced variant that significantly reduces memory footprints while maintaining comparable throughput, detailed in Section 5.4
seq 6144
seq 6144 tp 8, pp
tp 4, pp 4
80
80
Memory (GB)
60
60
40
40-
20
20
1FIB-1
ZB-V
Ours
1F1B-1
ZB-V
Ours
Figure 9: Peak activation memory footprint tested on 12.1B LLM across 2 nodes.
5.3 Comparison with Existing Schedules on MILM
As shown Tables we present experimental evaluations Qwen2-VL on 2-node and 4-node setups, respectively, which report throughput and peak activation memory under various configurations, including ViT lengths, LM lengths, and model scale parameters. For the PP=4, we carefully adjust the ViT and LM sequence lengths to balance computational workload across virtual stages by ensuring the FLOPs of the ViT approximately match those individual virtual stages. contrast, for PP-2 and PP=8, we intentionally maintain disparities in VIT/LM FLOPPs to conduct diverse workload scenarios. Our scheduling approach demonstrates superior performance compared to baseline methods in all balanced computational scenarios. Notably, we observe maximum throughput improvement

---

Table 3: Experimental results for the 14.9B, 28.8B, and 30.3B MLLMs, showing throughput (samples second) and peak activation memory footprint (GB). mbs denotes the number of microbatches.
ViT
LM Length
mbs
Model
TP
PP
Schedule
Memory
Length
64
128
192
IFIB-I
4.36
4.44
4.46
40
1.7B VIT
4
4
ZB-V
4.25
4.30
4.31
30
Ours
4.53
4.63
4.65
56
13.2B LM
3136
5120
IFIB-I
2.42
2.44
2.46
26
16 GPUs
8
2
ZB-V
2.48
2.49
2.49
30
Ours
2.86
2.86
2.87
ViT Length
LM Length
mbs
Model
TP
PP
Schedule
Memory
96
176
256
5.6B ViT
IFIB-I
5.79
5.89
5.85
60
1600
3072
4
8
ZB-V
5.90
5.97
6.01
48
23.2B / 24.7B LM
Ours
5.97
6.19
6.19
IF1B-I
3.37
3.40
3,40
43
2304
5120
8
4
ZB-V
3.51
3.53
3.53
38
32 GPUs
Ours
3.72
3.80
3.80
47
of 11.7% in the configuration with TP=8 and PP=4, which is significantly higher than the 2% improvement in TP=4 and PP=4, compared to IFIB-I. This trend aligns with observations from LM experiments, indicating that larger TP sizes generally yield greater performance gains.
A particularly notable result is the 16.7% performance improvement achieved by our schedule in the PP=2 setting, where the VIT component has relatively lower computational intensity compared to other model chunks. In this case, the lowest peak memory footprint is achieved by IFIB-I in stage 1, where only a few microbatches are in-flight and the memory ViT much smaller compared that in LM. In the PP=8 scenario, where the VIT's FLOPs exceed those of the LM chunks, our schedule and ZB-V V achieve comparable throughputs màs 96. However, as the number of microbatches increases, our approach demonstrates superior scalability and robustness, gaining more improvements compared ZB-V and FFIB-I.
Ours
Ours
4.0
55
Throughput
3.5
Memory (GB)
50
45
3.0
40
35
2.5
30
64
128
192
stage 0
stage
stage 2
stage 3
Figure 10: Experimental results in terms of throughput and peak activation memory footprints over PP stages on the 12. LLM. indicates the enhanced variant with offloading.
5.4 Efficiency of Our Enhanced Schedule
Due to the limited PCIe bandwidth of the A800, where the effectiveness of the offloading variant cannot be fully demonstrated, we perform experiments on 16 NVIDIA H20 GPUs, which are equipped with PCIe Gen interconnection. As shown in Figure 10 the offloading variant incurs a negligible throughput degradation due additional CPU overheads associated with data transmission. However, as demonstrated the right part of Figure TO the memory footprints across all stages are approximately equal, indicating our schedule effectively utilizes the memory resources improve

---

the throughput. And the offloading strategy achieves a peak memory reduction ranging from 10% to 19.2%, with the resulting peak memory footprint being close that of 1FIB-I (40G).
5.5 Results under Maximized Resource Utilization
We conduct experiments on 16 NVIDIA H20 96GB GPUs, adjusting the microbatch size across various parallel configurations to maximize memory utilization and achieve optimal throughput and Model FLOPs Utilization (MFU). As shown in Table[4] our proposed method delivers superior performance under maximized memory utilization. Specifically, under the TP=2, PP=8 configuration, our enhanced schedule achieves slightly higher throughput (2.74 samples/s) and MFU (92.86%) compared to IFIB-I, despite the lower proportion of TP bubbles on H20 GPUs (see the Appendix D for details), while maintaining a lower and more balanced memory footprint of 68 GB.
Table 4: Experiment results on 12.1B 1B LLM under maximized memory utilization, which are evaluated on 16 H20 GPUs. * denotes the enhanced variant of our schedule with activation offloading.
Configurations
Parallelism
mbs
Schedules
Throughput
MFU
Memory
IFIB-I
2.72
92.09
76
TP=2
ZB-V
2.61
88.36
54
pp=8
Ours
OOM
Ours
2.74
92.86
68
IFIB-I
2.47
83.62
53
ZB-V
2.41
81.59
38
TP=4
Ours
2.52
85.32
71
pp=4
IFIB-I
OOM
2
ZB-V V
2.47
83.62
75
Ours*
OOM
mbs=192
IFIB-I
2.06
69.74
40
seq len=
1
ZB-V
2.07
70.08
31
Ours
2.12
71.78
57
IFIB-I
2.14
72.45
TP=8
2
7B-V
2.18
73.81
61
Ours*
OOM
pp=2
IF1B-I
OOM
3
ZB-V
1.74
58.91
90
Ours*
OOM
Conclusion
In this paper, we presented novel hybrid parallel strategy that synergistically integrates tensor and pipeline parallelism for distributed training of LLMs and MLLMs. By decomposing Transformer layers into fine-grained computation units and interleaving forward and backward computations within each PP stage, we constructed execution blocks that effectively overlap TP communication bubbles. Based on these blocks, we designed "V"-ssape PP schedule, which achieves balanced memory across stages and significantly reduces both TP and PP bubbles, albeit at the cost of increased peak memory. Experimental results demonstrated that our approach consistently outperforms existing scheduling methods on both L.Mss and MI.MMs. We also presented an enhanced variant incorporating offloading better accommodate memory-constraie scenarios. In future work, we plan to explore more memory- and throughput-efficieet hybrid schedules in the large-scale machine learning system.
Acknowledgement
This work was partly supported by the Funds for the National Natural Science Foundation China under Grant 62202063, 62572072, U24B20176, and Beijing Natural Science Foundation (L243027).
10

---

References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Language models are few-shot learnrrs. Advances in neural information processing systems, 33:1877-1901, 2020.
[2] Li-Wen Chang. Wenlei Bao, Qi Hou, Chengquan Jiang, Ningxin Zheng. Yinmin Zhong, Xuanrun Zhang, Zuquan Song, Chengji Yao, Ziheng Jiang, et al. Flux: fast software-hased communication overlap on gpus through kernel fusion. arXiv preprint (.Xx::24060068555 2024.
[3] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.00 006911 2023.
[4] Wei Deng, Mengshi Qi, and Huadong Ma. Global-local tree search in vims for 3d indoor scene generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 8975-89888 8984, 2025.
[5] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minihatch sgd: Training imagenet 1 hour, arXiv preprint xrvi: 000.17066000 02677, 2017.
[6] Jun Huang, Zhen Zhang, Shuai Zheng. Feng Qin, and Yida Wang. (DISTMM}: Accelerating distributed multimodal model training. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 1177-1171, 2024.
[7] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Lc, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances neural information processing systems, 32, 2019.
[8] Abhinav Jangda, Jun Huang, Guodong Liu, Amir Hossein Nodehi Sabet, Saced Maleki, Youshan Miao, Madanial Musuvathi, Todd Mytkowic7, and Olli Saarikivi. Breaking the computation and communication abstraction barrier in distributed machine learning workloads. In Proceedings the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Syxtems, pages 402-416. 2022.
[9] Byungsoo Jeon, Mengdi Wu. Shiyi Cao. Sunghyun Kim. Sunghyun Park. Neeraj Aggarwal. Colin Unger. Daiyaan Arfeen, Peiyuan Liao, Xupeng Miao, et al. Graphpipe: Improving performance and scalability of dnn training with graph pipeline parallelism. In Proceedings of the 30th ACM International Conference Architectural Support for Programming Languages and Operating Systems, Volume I, pages 557-571, 2025.
[10] Zhihao Jia, Matei Zaharia, and Alex Aiken. Beyond data and model parallelism for deep neural networks. Proceedings of Machine Learning and Systems, 1:1-13, 2019.
[11] Chenyu Jiang, Zhen Jia, Shuai Zheng, Yida Wang, and Chuan Wu. Dynapipe: Optimizing multi-task training through dynamic pipelines. In Proceedings of the Nineteenth European Conference Computer Systems, pages 542 559, 2024.
[12] Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, and Chuanxiong Guo. A unified architecture for accelerating distributed (DNN} training in heterogencous (GPU/CPU) clusters. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 463-479, 2020.
[13] Tachum Kim, Hyoungjoo Kim, Gyeong-In Yu, and Byung- Chun. Bpipe: Memory-balancee pipeline parallelism for training large language models. In International Conference on Machine Learning, pages 16639-16653. PMLR, 2023.
[14] Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Moham- mad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5:341-355 2023.
[15] Mu Li, David G Andersen, Alexander Smola, and Kai Yu. Communication efficient distributed machinc learning with the parameter server. Advances in neural information processing syxtems, 27, 2014.
[16] Shigang Li and Torsten Hoefler. Chimera: efficiently training large-scale neural networks with bidirectional pipelines. In Proceedings of the International Conference for High Performance Computing. Networking. Storuge and Analysis, pages 1-14, 2021.
[17] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang. Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437. 2024.
11

---

[18] Ziming Liu, Shenggan Cheng, Haotian Zhou, and Yang You. Hanayo: Harnessing wave-like pipeline parallelism for enhanced large model training efficiency. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-13, 2023.
[19] Changsheng Lv, Shuai Zhang, Yapeng Tian, Mengshi Qi, and Huadong Ma. Disentangled counterfactual learning for physical audiovisual commonsense reasoning. Advances in Neural Information Processing Systems, 36: 1446--2248, 2023.
[20] Changsheng Lv, Mengshi Qi, Xia Li, Zhengyuan Yang, and Huadong Ma. Sgformer: Semantic graph transformer for point cloud basd- 3d scene graph generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4035 4043, 2024.
[21] Xupeng Miao, Yining Shi, Zhi Yang. Bin Cui, and Zhihao Jia. Sdpipe: semi-ddcentrallzzed framework for heterogeneity-wwarr pipeline-paralle training. Proceedings of the VLDB Endowment, (0))2334--3333 2023.
[22] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur. Gregory Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM symposium on operating systems principles, pages 1-15, 2019.
[23] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training gpu clusters using megatron-lm. In Proceedings of the international conference for high performance computing, networking, storage and analysis, pages 1-15, 2021.
[24]
NVIDIA Corporation. NVLink & NVSwitch: Fastest HPC Data Center Platform. https: /ww. nvidia
com/ hte///nnus/see can-us/data--ee wwnter/eeteeeeeee wwweerr/vvvlle nvlkik//
[25] Suchita Pati, Shaizeen Aga, Mahzabeen Islam, Nuwan Jayasena, and Matthew D Sinclair. T3: Transparent tracking & triggering for fine-grained overlap of compute & collectives. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. pages 1 1146-1164, 2024.
[26] Mengshi Qi, Weijian Li, Zhengyuan Yang, Yunhong Wang, and Jiebo Luo. Attentive relational networks for mapping images to scene graphs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3957-3966, 2019.
[27] Mengshi Qi, Jie Qin, Yi Yang, Yunhong Wang, and Jiebo Luo.. Semantics-aware spatial-temporal binaries for cross-modal video retrieval. IEEE Transactions on Image Processing, 2009993304, 2021.
[28] Mengshi Qi, Hao Ye, Jiaxuan Peng, and Huadong Ma. Action quality assessment via hierarchical pose- guided multi- stage contrastive regression. arXiv preprint arXiv:250I 003674,, 2025.
[29] Penghui Qi, Xinyi Wan, Nyamdavaa Amar, and Min Lin. Pipeline parallelism with controllable memory. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[30] Penghui Qi, Xinyi Wan, Guangxing Huang, and Min I Lin. 7em bubble (almost) pipeline parallelism. In The Twelfth International Conference on Learning Representations, 2024.
[31] Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed moe: Advancing mixture-of- expertt inference and training to power next-generation ai scale. In JCML 2022, January 2022.
[32] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penpom Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, al. Mesh-tensorflow: Deep learning for supercomputers. Advances in neural information processing systems, 31, 2018
[33] Mohammad Shoeybi. Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint PrXiv:1909.000 2019.
[34] Zhenbo Sun, Huanqi Cao, Yuanwei Wang, Guanyu Feng, Shengqi Chen, Haojie Wang. and Wenguang Chen. Adapipe: Optimizing pipeline parallelism with adaptive recomputation and partitioning. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pages 86-100, 2024.
[35] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 21X 2023.
12

---

[36] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu. Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution. arXiv preprint arXiv:2409.1219, 2024.
[37] Shibo Wang, Jinliang Wei, Amit Sabne, Andy Davis, Berkin Ilbeyi, Blake Hechtman, Dehao Chen, Karthik Srinivasa Murthy, Marcello Maggioni, Qiao Zhang. et al. Overlap communication with dependent computation via decomposition in large deep learning models. In Proceedings of the 28th ACM Interna- tional Conference on Architectural Support for Programming Languages and Operating Systems, Volume
1, pages 93-106, 2022.
[38] Weihu Wang, Yaqi Xia, Donglin Yang. Xiaobo Zhou, and Dazhao Cheng. Accelerating distributed dlrm training with optimized tt decomposition and miero-batchinn In SC24: International Conference for High Performance Computing, Nenworking. Storage and Analysis, pages 1-15. TEEF, 2024.
[39] An Yang. Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang. Guanting Dong. Haoran Wei, Huan Lim, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jianxin Yang. Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keriing Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li. Tianyu Liu, Wenbin Ge, Xiaodong Deng. Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei. Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao. Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. arXiv preprint ar 24v: 10671, 2024.
[40] Tailing Yuan, Yuliang Liu, Xucheng Ye, Shenglong Zhang. Jianchao Tan, Bin Chen, Chengru Song, and Di Zhang. Accelerating the training of large language models using efficient activation rematerialization and optimal hybrid parallelism. In 2024 USENIX Annual Technical Conference (USENIX ATC 24), pages 545-561, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We claim the contribution of our work clearly in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The main limitation of our work is discussed in the last section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
There is no theoretical result in our work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our implementation based on open sourced M-Core, which is accessible from everyone and we describe the setting of experiments clearly.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We would open the source code after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
www.oonoooolylooooooooolooooiill for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli//uuuiu wwwiiicgguie///eeeeeeeeic CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed in Section Experiments. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The results are stable in our clusters.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We list all resources information in Section Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htttps://euur: httpp////lc//tccccccceeeeeeeeeee
Answer: [Yes]
Justification:
The research is conducted with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Our research focus the training of L.MSs and has impact society.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our work is developed based on Megatron- -Core with a license that everyone can work on it.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpsspipetattteeeeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
18

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We will open source the related documents and codes on the camera ready phase.
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
19

---

Answer: [NA]
Justification: The method in this research does not involve LLMs as any important, original, non-standard components, just uses it for training tests.
Guidelines:
The answer NA means that the core method development in this research does not non wwweeurrpssccoooofeeeeeeeeeeeeee /cc/Confference/2222222
involve LLMs as any important, original, non standtadd components.
Please refer to our LLM policy for what should or should not be described.
20

---

Appendix
A
The Warm-Up Phase of the Pipeline Schedule
Memory (GB)
X
3
2
6
(b)
2
4 3
(c)
The construction the warm-up phase: (a) wrong warm-up phase; (b) memory-efficient warm-up phase; (c) throughput-efficient warm-up phase.
Figure 11:
The guiding principle for constructing the warm-up phase is to ensure the correct execution of overlapped forward and backward passes. As illustrated at the top Figure a key requirement that overlapping between forward and backward passes must occur across different microbatches. Specifically, the microbatch index in the forward pass should be greater than that in the backward pass. To satisfy this condition, additional forward pass is required before the overlapped F&B execution begins, as shown in Figure bT... However, this approach necessitates decoupling the backward pass into activation and weight gradient computations, which exposes TP communication and introduces additional PP communication overheads. Please refer to Section R for more details about the memory-efficient warm-up phase.
B Comparison with Other Schedules
compare 1-IB-I 23 ZB-V 29 and our proposed schedule under identical confi configurations: pipeline stages, 2 virtual stages per pipeline stage, and 12 microbatches, as illustrated in Figure [22 Intuitively, both ZB-V V and our schedule have a similar number of PP bubbles, which are significantly fewer than those in IFIB-I. The PP bubbles in IFIB-I primarily result from the warm-up and cool- down phases. In the warm-up phase, the parallel dataflow across virtual stages results in a large number of bubbles. For instance, the long lifespan of virtual stage 2 for microbatch I cannot be efficiently filled without introducing additional forward passes from more microbatches, which would increases the peak memory of the stage 1. Consequently, IF1B-I incurs greater PP bubble overheads during the warm-up phase. Meanwhile, the bubbles in the cool-down phase are inevitable due to inherent data dependencies.
Compared to ZB-V. our schedule results in a small number of PP bubbles the end of the warm-up phase, which are attributed to the relatively long execution time and specific PP dataflow pattern
21

---

PP Stages
Denaflow
Nage
Sagge
Suge
Sage-
XIS
VIS
Stage
Stage
Sage
ZB-VV
Sage7
Stace
VS3
V5.6
(C) Ours
VS
Sage
Sage
(4) Chaec
Virtal Stags
FALH
Figure 12: Synchronous pipeline parallelism schedules compared our work, with 4 pipeline stages and 12 microbatches within training iteration. Dark colors denote the first virtual stage and light colors are the second virtual stage. denotes the tensor and pipeline schedule beginning with the memory--fficient warm-up phase.
of the overlapped F&B blocks. In general, the activation gradient computation time Ta satisfies TB > Tw. where Tw is the weight gradient computation time. As a result, small bubbles also appear the end of the cool-down phase, and their magnitudes are approximately equal for both ZB-V and our proposed schedule.
Moreover, also present schedule Ours (d), which begins with memory- warm-up phase, as illustrated in Figure 12] Although schedule (d) has lower peak memory footprint compared to standard schedule (c), introduces additional PP bubbles towards the end, negatively impacting throughput performance. Therefore, for scenarios involving large-scale model training with large TP sizes and ample memory availability, prefer schedule (c) for its higher performance. Our enhanced schedule variant, which includes activation offloading, is preferable for reducing peak memory usage, assuming sufficient bandwidth. Schedule (d) is suitable only for scenarios with extremely limited memory and large TP overheads.
C Detailed Experimental Results on LLMs
We present detailed experimental results based on the 12.1B and 26.3B Qwen2 models, as summarized in Tables 6 and Notably, the GPU memory footprint reported in Table is slightly higher than the theoretical estimate. We attribute this discrepancy primarily to practical aspects of our engineering implementation, such as the implementation of the fine- grained computation units and the decoupling of weight gradient computation. For example, under TP=8, profiling the activation memory one microbatch virtual stage shows that ZB-V consumes 3.6 GB, while our approach uses 4.3 GB, an increase of nearly 20%. Although this issue has no impact on throughput, we plan to further refine the engineering implementation to reduce this overhead in future work.
22

---

Table Experimental results on 12.1B and 26.3B LLMs terms of peak memory (GB), where the microbatch sizes of samples with sequence length 3072 and 2048 are set to 2.
Sequence Length
Model
TP
PP
IFIB-I
ZB-V
Ours
4
41
30
53
12.1B LLM
3072
8
31
24
43
16 GPUs
6144
4
41
30
54
8
31
24
43
4
8
55
38
72
26.3B L.M
2048
8
43
32
59
32 GPUs
4096
4
55
38
72
8
43
32
59
Table 6: Experimental results on 12. 1B and 26.3B LLMs in terms of throughput (samples per second). mbs represents the number of microbatches.
Sequence Length
Model
TP
PP
mbs
IFIB-I
ZB-V
Ours
64
9.52
9.12
9.87
4
4
128
9.63
9.26
10.1
192
9.66
9.31
10.1
3072
64
6.57
6,42
7.28
8
2
128
6.60
6.46
7.32
12.1B LLM
192
6.60
6.46
7.33
16 GPUs
64
4.51
4.45
4.74
4
4
128
4.57
4.48
4.82
192
4.58
4.49
4.83
6144
64
3.11
3.13
3.46
2
128
3.13
3.13
3.47
192
3.11
3.13
3.49
96
12.3
12.4
13.0
8
176
12.8
12.7
13.2
256
12.7
12.8
13.4
2048
96
8.60
8.71
9.48
4
176
8.67
8.79
9.56
26.3B LLM
256
8.68
8.79
9.61
32 GPUs
96
6.16
6.17
6.33
4
8
176
6.17
6,28
6.49
256
6.28
6.31
6.51
4096
96
4.23
4.26
4.66
8
4
176
4.24
4.28
4.70
256
4.25
4.29
4.72
D Comparison with Other Schedule Methods H20 GPUs
We also conduct experiments on 16 NVIDIA H20 96G GPUs to evaluate the performance of our proposed schedule alongside other baselines, as summarized in Table Our proposed schedule outperforms other approaches by margin, but the performance improvement is far less signifi- cant compared to that achieved on A800 GPUs. This discrepancy can primarily attributed to the characteristics of the H20 GPUs. which have lower BF16 FLOPs but higher communication
23

---

Table 7: Experimental results on 12.1B and 26.3B LLMs in terms of Model FLOPs Utilization (MFU, %).
Sequence Length
Model
TP
PP
mbs
IFIB-I
ZB-V
Ours
64
46.31
44.36
48.01
4
4
128
46.84
45.04
49.13
192
46.99
45.28
49.13
3072
64
31.96
31.23
35.41
8
128
32.10
31.42
35.61
12.1B LLM
192
32.10
31.42
35.65
16 GPUs
64
50.16
49.50
52.72
4
128
50.83
49.83
53.61
192
50.94
49.94
53.72
6144
64
34.59
34.81
38.49
8
2
128
34.81
34.81
38.60
192
34.59
34.81
38.82
96
42.16
42.50
44.56
4
8
176
43.87
43.53
45.24
256
43.53
43.87
45.93
2048
96
29.48
29.85
32.49
8
4
176
29.72
30.13
32.77
26.3B LLM
256
29.75
30.13
32.94
32 GPUs
96
46.32
46.40
47.60
176
46.40
47.23
48.80
256
47.23
47.45
48.95
4096
96
31.81
32.04
35.04
8
176
31.88
32.19
35.34
256
31.96
32.26
35.49
bandwidth. Furthermore, we profile the computation and communication times for the Attention and MLP modules of a single Transformer layer in the 12.1B Qwen2 model, as shown in Figure [3 is evident that the proportion of TP communication is significantly lower than that measured A800 GPUs. As a result, the benefits derived from optimizing TP bubbles have slightly diminished.
Table Experimental results evaluated on 12.1B LLM on 16 H20 GPUs. màs and seg_len denote the number of microbatches and the sequence length, respectively.
Configurations
TP
PP
Schedules
Throughput
MFU
Memory
IFIB-I
3.77
88.34
58
2
ZB-V
3.63
85.06
42
Ours
3.79
88.81
73
mbs=192
IFIB-I
3,39
77.43
41
4
4
ZB-V
3.30
77.32
30
len=E4441
Ours
3.44
80.61
54
IFIB-I
2.80
65.61
31
8
2
ZB-V
2.80
65.61
24
Ours
2.89
67.72
43

---

Attention
Attn TP Comm.
MLP
MLP TP Comm.
Naive
Naive
Naive
1.0
Normalized Time
0.8
0.6
0.4
0.2
seq-
6144,
1p-2
seq 6144, tp
seq 6144, tp
Figure 13: The computation and communication time proportion of Attention and ML.P modules
profiled on H20 GPUs.
E
Evaluations with Existing Training Strategies
E.1 Activation Checkpointing
Our proposed schedule is compatible with activation checkpointing strategies. We conducted the experiment on Qwen2-12.. 1.nn2-12.1BB IB using activation checkpointing (AC) with a batch size of 128 and sequence length of 6k, and report the results in the following table:
Table Performance comparison with different activation checkpointing configurations.
Config
Throughput (samples/s)
Peak Memory (GB)
AC disabled
4.79
56.0
AC enabled in Ours w/ MLP
4.19
44.7
AC enabled in Ours w/ Attn+MLP
3.94
41.5
AC enabled in Ours w/ Att++MLP+Norm
3.75
36.3
As shown Table 9 disabling AC yields the highest throughput 4.79 samples per second but requires peak memory footprint of 56.0 GB. Enabling AC selectively the MLP modules reduces peak memory by 20.2% to 44.7 GB, with a corresponding throughput decrease 12.5% to 4.19 samples/s. Extending AC to both Attention and MLP modules further reduces peak memory to 41.5 GB (a 25.9% reduction), while throughput declines to 3.94 samples/s. The most aggressive configuration, applying AC to all modules, achieves the greatest memory savings, reducing peak memory by 35.2% to 36.3 GB, at the cost of a 21.7% reduction in throughput, resulting in 3.75 samples/s. These results confirm that our scheduling framework is fully compatible with activation checkpointing strategies.
E.2 Data Parallelism and Context Parallelism
To demonstrate the compatibility of our schedule with both data parallelism (DP) and context parallelism (CP), we conduct experiments 12.1B 1B Qwen2 model. As shown in Table [O] under identical tensor parallelism (TP=2) and pipeline parallelism (PP=4) configurations, our method achieves the highest throughput in both settings: 2.71 samples/s with CP (CP=2, màs 128, sequence length 12k) and 9.40 samples/s with DP (DP=2, rabs = 256, sequence length 4k). Our approach consistently outperforms baseline schedules (IFIB-I and ZB-V) across both parallelism paradigms, confirming its robustness and adaptability to different distributed training strategies.
Impact of Communication Overlap on GEMM Execution Efficiency
In distributed training large language models, overlapping communication (e.g., AllReduce) with computation (c.g., GEMM) widely used to improve hardware utilization and reduce training time.
25

---

Table 10: Experimental results on 12.1B LLM compatible with data parallelism and context paral- lelism.
TP
PP
mbs
Seq
Schedules
Throughput
1FIB-I ZB-V
2.64
2
4
128
12k
2.61
Ours
2.71
TP
PP
DP
mès
Seq
Schedules
Throughput
1FIB-I
9.22
2
256
4k
ZB-V
9.16
Ours
9.40
However, this overlap may cause resource contention, particularly for streaming multiprocessors (SMs) which potentially degrade GEMM performance.
To evaluate this effect, we conduct microbenchmarks under two scenarios: (1) GEMM dominates AllReduce, enabling full overlap; and (2) GEMM finishes early, leaving part of the communication exposed. In both cases, we compare overlapped execution against the sequential baseline (GEMM followed by AllReduce) and the standalone GEMM time.
Table Execution times (in milliseconds) for GEMM, AllReduce, and their combinations under sequential and overlapped execution.
Operation
Experiment
Experiment
GEMM
8.605
0.334
AllReduce
3.364
1.643
GEMM + AllReduce (sequential) 11.969
1.977
GEMM with overlapped AllReduce 9.251
1.685
The results, summarized in Table I1 show that in the first scenario, where GEMM takes 8.605 ms and AllReduce 3.364 ms. the overlapped execution completes in 9.251 ms, only 7.5% slower than the GEMM alone and 22.6% faster than the sequential execution (11.969 ms). This indicates that the communication effectively hidden with minimal interference.
In the second scenario, where GEMM is very short (0.334 ms) and AllReduce dominates (1.643 ms), the overlapped execution finishes in 1.685 ms, which is just 2.6% overhead over the AllReduce time and still 1 14.8% faster than the sequential baseline (1.977 ms). Notably, the GEMM itself is not measurably slowed down; rather, the slight increase in total time stems from the unavoidable tail of communication that cannot be hidden.
26


Reviews:
[{"review_id": "4Dmgu2rFiw", "reviewer": "Reviewer_EjFt", "summary": "This paper proposes a novel approach to large-scale Transformer training by better coordinating tensor and pipeline parallelism. It aims to reduce the \"tensor parallel bubble\"—the communication overhead—by overlapping it with the backward computation of the pipeline. The experiments demonstrate improved throughput, albeit with some memory overhead.", "questions": "- How would the proposed scheduling handle highly imbalanced stages, such as those caused by MoE layers or custom modules?\n- Is the system compatible with activation checkpointing strategies commonly used in large-scale models?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strength**\n- The paper provides a clear analysis of how tensor and pipeline parallelism interact and where current methods fall short.\n- The proposed scheduling design is practical and accounts for the potential memory challenges introduced by this method.\n- The implementation considers real hardware constraints, such as memory usage and communication overhead, making the method more applicable to real-world training setups. Also the experiments are well-executed across multiple large models (including LLMs and multimodal models), providing solid evidence of scalability and general applicability.\n\n**Weaknesses**\n- I am not very familiar with pipeline parallelism, so I found some figures (e.g. Figure 3) a bit challenging to fully understand.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your support! And we sincerely thank the reviewer for the valuable feedback. Below, we address each point raised in your review.\n\n> I am not very familiar with pipeline parallelism, so I found some figures (e.g. Figure 3) a bit challenging to fully understand.\n\nSorry for that. Figure 3 presents the execution timelines of the computation units. \"F\" and \"B\" denote the forward and backward passes of these units. Taking the Attn unit as an example, it completes computation in the computation stream and then immediately activates the communication stream to execute the `AllReduce` (AR) operation. The forward computation in the MLP unit must wait for the completion of that AllReduce due to the inherent data dependency. Therefore, the computation stream would remain idle during the AR execution in the naive implementation. \n\nIn our schedule, we interleave the forward and backward passes of the units to reduce idle time. And that AR operation is naturally overlapped with the computation of backward passes in Figure 3 (a). Figure 3 (b) presents the timeline in which the weight gradient computation is separated from the full backward pass. We will revise Figure 3 to make it easy to understand in the final version.\n\n> How would the proposed scheduling handle highly imbalanced stages, such as those caused by MoE layers or custom modules?\n\nThanks for your valuable question. The imbalance among stages indeed negatively affects performance, especially for multimodal models, custom modules, and so on. In this work, we indeed try to address this issue, although it is not emphasized in the paper, as our focus is primarily on the implementation of the proposed PP and TP schedules. \n\nSpecifically, we first profile the execution time of the forward and backward passes of each unit, as well as the memory footprint. Then, we design a simple algorithm to split the consecutive unit sequence into balanced PP stages by ensuring that the total forward and backward time per stage is approximately equal and ensuring that the memory footprint of each PP stage stays within memory limits.\n\n> Is the system compatible with activation checkpointing strategies commonly used in large-scale models?\n\nYes! Our schedule is compatible with activation checkpointing strategies. We conduct an experiment on Qwen2-12.1B using activation checkpointing (AC) with a batch size of 128 and a sequence length of 6k, and report the results in the following table:\n Config   | Throughput (samples/s) | Peak Memory (GB)\n---|---|----\nAC disabled | 4.79  | 56.0\nAC enabled in Ours w/ MLP  | 4.19 | 44.7 \nAC enabled in Ours w/ Attn+MLP | 3.94  | 41.5 \nAC enabled in Ours w/ Attn+MLP+Norm  | 3.75  | 36.3 \n\n\nAs shown, disabling AC yields the highest throughput of 4.79 samples per second but requires a peak memory footprint of 56.0 GB. Enabling AC selectively on the MLP modules reduces peak memory by 20.2% to 44.7 GB, with a corresponding throughput decrease of 12.5% to 4.19 samples/s. Extending AC to both Attention and MLP modules further reduces peak memory to 41.5 GB (a 25.9% reduction), while throughput declines to 3.94 samples/s. The most aggressive configuration, applying AC to all modules, achieves the greatest memory savings, reducing peak memory by 35.2% to 36.3 GB, at the cost of a 21.7% reduction in throughput, resulting in 3.75 samples/s. These results confirm that our scheduling framework is fully compatible with activation checkpointing strategies."}, {"review_id": "Pq7A1sdTBc", "reviewer": "Reviewer_P8vN", "summary": "Model training at scale often uses tensor and pipeline parallelism. Tensor parallelism can suffer from high communication overhead. This paper offers a way to overlap communication with computation to improve training speed.", "questions": "- Figure 3 doesn't make full sense to me. For example, in Figure 3a, is the green AR in the communication stream for a different microbatch than the green \"Attn (F)\"? If they are for the same microbatch, how does \"Attn (F)\" complete before its communication?\n- How much is computation slowed down by the overlapped communication (since communication probably uses some SMs that can't be used by GEMMs)?\n- Is there a way to apply these ideas to PP=1?\n- What are the throughputs in Figure in TFLOP/s/GPU? That is, how fast on an absolute scale are the baselines and \"Ours\"?", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Important problem (improving efficiency of LLM and and MLLM training at scale).\n- Solid contributions: the modification to the interleaved pipelining schedule is simple but effective, and the idea to overlap compute and communication for different microbatches makes sense.\n- Results are good.\n\nWeaknesses:\n- Some of the writing can be made a bit more precise. For example: a) I don't think TP communication significantly grows with TP size as long as you're able to run collectives at the same bandwidth; of course compute-communication ratio decreases. b) What does \"at the kernel or hardware levels\" mean when discussing communication overlapping with computation? c) Clarify hardware setup in statements like \"where limited inter-device bandwidth can severely degrade training efficiency\".\n- I'm not sure how \"synergistic\" the modification to the interleaved pipelining schedule is, can't the proposed memory improvements be used for pipeline parallelism in isolation?", "was_revised": true, "final_justification": "The paper studies an important problem, provides novel contributions, and presents good results. The authors addressed weaknesses and questions I raised during the review process. As a result, I rate it 5 and keep the paper as accept.", "rebuttal": "Thank you for your support! And we sincerely thank the reviewer for the valuable feedback and suggestions for improved clarity. Below, we address each point raised in your review.\n\n> Some of the writing can be made a bit more precise. For example: a) I don't think TP communication significantly grows with TP size as long as you're able to run collectives at the same bandwidth; of course compute-communication ratio decreases. \n\nSorry for misleading. The original statement in our paper was \"The proportion of TP communications grows significantly with the increased TP size\". Here, \"proportion of TP communications\" refers to the ratio of TP communication time to the total time, not the absolute TP communication time. \n\nWe agree that if the bandwidth remains constant, the absolute communication time may not grow significantly. However, as TP size increases, the compute-to-communication ratio decreases, leading to TP communication becoming a larger fraction of the overall runtime, which is we intended to highlight.\n\nTo avoid misunderstanding, we will revise the sentence to clarify this point in the final version.\n\n> b) What does \"at the kernel or hardware levels\" mean when discussing communication overlapping with computation? \n\nThanks for your question. The statement of \"at the kernel or hardware levels\" means that the overlapped communication is implemented based on the well-designed and complex CUDA kernels, such as `all-gather+gemm` kernel [1] (kernel level), or on some specific hardware, such as track and trigger in [2] (hardware level).\n\nThanks for your feedback! We will revise this sentence in our final version to make it clearer.\n\n[1] Chang, Li-Wen, et al. Flux: Fast software-based communication overlap on gpus through kernel fusion. arxiv.\n[2] Pati, Suchita, et al. T3: Transparent tracking & triggering for fine-grained overlap of compute & collectives. ASPLOS 2024.\n\n> c) Clarify hardware setup in statements like \"where limited inter-device bandwidth can severely degrade training efficiency\".\n> \nThank you for the insightful suggestion. We clarify the sentence as follows:\n\n\"where limited inter-device bandwidth (e.g., systems without NVLink) can severely degrade training efficiency.\"\n\n> I'm not sure how \"synergistic\" the modification to the interleaved pipelining schedule is, can't the proposed memory improvements be used for pipeline parallelism in isolation?\n\nYes! The proposed memory improvements can be used for pipeline parallelism in isolation, but they are not synergistic with the interleaved pipelining schedule actually. The modification of memory is just designed for coping with limited memory scenarios. \"Synergistic\" here is to reflect in the execution timeline of overlapped F&B block in the schedule, where we interleave the forward and backward passes in the same PP stage to overlap the TP communication in forward and backward passes.\n\n> Figure 3 doesn't make full sense to me. For example, in Figure 3a, is the green AR in the communication stream for a different microbatch than the green \"Attn (F)\"? If they are for the same microbatch, how does \"Attn (F)\" complete before its communication?\n\nSorry for the misunderstanding here. In Figure 3a, the green \"Attn (F)\" and the green AR both correspond to the same microbatch. \"Attn (F)\" refers solely to the computation of the attention forward pass, while AllReduce (AR) represents the associated communication operation. And those two constitute a complete attention operation. The green AR is initiated only after the corresponding computation (Attn (F)) is completed. Therefore, the forward pass of the attention module is only considered fully complete once both the computation and the communication have finished.\n\n> How much is computation slowed down by the overlapped communication (since communication probably uses some SMs that can't be used by GEMMs)?\n\nThank you for your insightful question. We understand your concern that overlapped communication might occupy certain GPU resources (e.g., SMs), potentially slowing down GEMM computations. To examine the issue, we conduct experiments to measure GEMM performance with and without overlapping AllReduce (AR) operation. The execution time of GEMM is set to be larger than that of AllReduce in Experiment 1, which is opposite in Experiment 2. The execution times are reported as follows:\nOperation|  Experiment 1|Experiment 2\n-|-|-\nGEMM|8.605ms|0.334ms\nAllReduce|3.364ms|1.643ms\nGEMM+AR|11.969ms|1.977ms\nGEMM with overlapped AR|9.251ms|1.685ms\n\nIn Experiment 1, the communication is effectively hidden behind computation, resulting in minimal overhead (approximately 7.5%) for GEMM and substantial time savings compared to sequential execution (GEMM+AR, 11.969ms).\n\nIn Experiment 2, although the GEMM operation completes earlier, leaving a portion of the communication time unhidden, the overall execution time remains close to the theoretical lower bound (i.e., the communication durations) and is significantly shorter than sequential execution (1.977 ms).\n\nThese results demonstrate that overlap is beneficial in both scenarios, with minimal interference and improved overall efficiency.\n\n\n> Is there a way to apply these ideas to PP=1?\n\nThe idea of leveraging the backward passes to overlap TP communication can be valid to a certain extent. Specifically, when PP=1, our method degenerates into pure TP training with gradient accumulation. In this case, where gradient accumulation is enabled, the backward pass for microbatch 1 can be interleaved with the forward pass for microbatch 2. Therefore, the overlap is valid for the majority of microbatches.\n\n> What are the throughputs in Figure in TFLOP/s/GPU? That is, how fast on an absolute scale are the baselines and \"Ours\"?\n\nThank you for the question. Due to the absence of specific numbers, we present some results in TFLOP/s/GPU of Figures 7 and 8.\n\n| Methods          | 1F1B-I | ZB-V | Ours |\n| --------------- | ----- | --- | --- |\n| TP=4, Seq=6144, PP=4, bs=192 | 158.8  | 155.7 | 167.5 |\n| TP=8, Seq=6144, PP=2, bs=192 | 107.8  | 108.5 | 121.1 |\n| TP=4, Seq=4096, PP=8, bs=256 | 147.3  | 148.0 | 152.7 |\n| TP=8, Seq=4096, PP=4, bs=256 | 99.7   | 100.7 | 110.8 |\n\nAs shown in the table, in the TP=4, Seq=6144, PP=4, bs=192 setup, our approach achieves an absolute improvement of 8.7 TFLOPS/s/GPU over 1F1B-I. Furthermore, in the TP=8, Seq=4096, PP=4, bs=256 configuration, we achieve an absolute improvement of 10.1 TFLOPS/s/GPU compared to ZB-V. These results underscore the consistent and significant gains in computational efficiency delivered by our scheduling method across diverse parallel and workload settings."}, {"review_id": "5xhkjJKMWi", "reviewer": "Reviewer_nDzM", "summary": "This paper introduces a pipeline parallel schedule that incorporates communication computation overlap to reduce tensor parallel bubbles by \"fusing\" backwards and forwards tasks. The schedule is introduced after some relevant background. An alternative schedule with offloading to reduce memory consumption is also introduced. Evaluations against some prior baselines are provided.", "questions": "### Major\n\nCan you clarify equation 1?\n\nCan you clarify whether Figure 5 should actually have smaller boxes for the combined F & B parts?\n\nWhy not consider long context baselines?\n\nWhy not evaluate against previous TP overlap work?\n\nWhy not consider a memory apples to apples comparison?\n\nWhy not evaluate against more recent work already cited as inspiration, for example, Hanayo?\n\n### Minor\n\nDid you try any experiments without 2 virtual stages per device? Any difference in performance for the baselines?\n\nAny idea how this performs with context parallelism?", "limitations": "Some limitations (such as performance PP = 8) discussed in lines 272-273.  The authors would be encouraged to make the limitations clearer.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I would first like to thank the authors for submitting their work to NeurIPS 2025.\n\n### Pros\n\nThe background treatment of PP and TP is fairly comprehensive overall, covering GPipe, 1F1B, Interleaved, Zero Bubble, Flux, etc. In addition, the authors correctly describe the current paradigm of distributed model training with a combination of TP and PP. \n\nThe idea of overlapping the TP computation \"braiding” seems to be sound and makes sense. So does the pipeline schedule itself (i.e. no overlapped critical-path tasks).\n\nSection 4.2 is written well, and provides some well-needed intuition for the schedule and its design. The theoretical analysis in section 4.3 seems to check out and offers a good comparison with some well known prior works.\n\nOffloading is also a nice extension to the basic schedule as well.\n\nThe evaluation section contains a lot of experimental results against 2 solid baselines, and the results seem to be promising.\n\n### Cons\n\nIn section 3, it’s stated nowhere what the Pre-Attn, Attn, Pre-MLP, and MLP consist of. Figure 2 has Attn and MLP annotated, but there should be a pointer to this, and Pre-Attn and Pre-MLP should be clearly annotated.\n\nEquation 1 is not clear at all for the most part, it should be made clear that this is for a single TP rank. Also, I’m not convinced that this formula is correct as shown. Where is $\\frac{detach(X)}{t}$ coming from? What exact operations does this $Attention$ actually perform?\n\nFigure 4 is likewise not motivated. It’s clear for someone that’s aware of interleaved pipeline parallelism, but there’s no annotation of what the colours represent, or any annotation of device, or any markings for the memory.\n\nIn Figure 5, I don’t see the concept of TP bubbles illustrated clearly. My understanding is the overlapped computations should have less latency overall. Perhaps an illustration of the time difference compared to a pipeline schedule without any TP bubble reduction would make this clearer, in the style of Figure 4 from [1].\n\nTable 1 could benefit from including parameter memory as well.\n\nIn lines 202-203 where it is stated that activation offloading offers a better E2E performance trade-off compared to checkpointing. This is a strong statement that is not strictly true in all situations to the best of my knowledge. In the case where the device to host bandwidth is poor, checkpointing might actually enable larger batch sizes and more efficient execution overall.\n\nIn terms of the evaluation, I have four issues.\n\n1. Although TP overlap methods are discussed in line 35, there’s no comparison done, for example against Flux. I see that as one of the most appropriate baselines given one of the main topics of this paper is reducing TP bubbles as well.\n2. I see there is an evaluation of the memory footprint of offloading compared to the default method. But given that Figure 9 shows that the other baselines take less memory, would they not be more efficient by being able to support a larger microbatch size?\n3. What about long context experiments? At longer contexts, the all-reduce latency may no longer be overlapped with compute causing the bubble to reappear.\n4. Why are only 1F1B and ZB the baselines? Some other newer works that are cited earlier as inspiration (line 75) are not compared to at all.\n\nI also don’t see any discussion of asynchronous vs synchronous pipeline parallelism, which is one of the main topics covered in Zero Bubble Pipeline Parallelism [2]. Seems to be that the proposed schedule is synchronous but ultimately it is being compared against asynchronous schedules like ZB-V [2] which should at least have some discussion.\n\n---\n\nOverall, while this paper introduces an interesting new schedule for pipeline parallelism which incorporates “TP codesign”, introducing yet another pipeline schedule is of limited novelty from a research perspective - The main insight here is that backwards and forwards tasks can be fused to provide comp/comm overlap. I am not convinced that the evaluation is sufficient to prove that the method is significant. There are also clarity and overall polish/quality issues.\n\n#### Other Comments\n\nNits:\n\nFigure 1 already includes results in the introduction. I feel this is more appropriate for the results section where the experimental setup can be explained.\n\nIn Figure 3, state that green is forward, and blue is backward.\n\nEquation 1: Use the full form of LayerNorm.\n\nFigure 5: It’s hard to see what’s an overlapped F & B vs F followed by B.\n\nTable 1: Activation Memory -> Peak Activation Memory\n\nTable 3: Include “throughput” and units here for clarity.\n\n\n[1] Narayanan et al., Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM, SC 2021, https://dl.acm.org/doi/10.1145/3458817.3476209\n\n[2] Qi et al., Zero Bubble Pipeline Parallelism, Arxiv, https://arxiv.org/pdf/2401.10241", "was_revised": true, "final_justification": "Overall, the reviewers addressed a lot of my concerns. Overall, the paper's results are good, and the idea itself is interesting and novel. It is an interesting addition to the area of pipeline parallel training.\n\nHowever, I still believe there are some issues with clarity and quality of the work with respect to figures and equations. The other overall decision factor here is I am not sure if a new pipeline parallelism schedule is of enough significance.", "rebuttal": "We thank the reviewer for the valuable feedback. Below, we provide responses to each point.\n> Equation 1 is not clear ... Where is $\\frac{detach(X)}{t}$ coming from? Attention actually perform?\n\nThanks for your suggestion. We would like to note that Equation 1 is for a single TP rank.\n\nThe standard process (LayerNorm + Attention) before MLP is formulated as:\n\n$X_{ln}=LayerNorm(X),$\n\n$X_{attn}=Attention(X_{ln}),$\n\n$X_{AR}=AR(X_{attn}),$\n\n$Y=X+X_{AR},$\n\nwhere $Attention$ is to calculate the attention output for a single TP rank, then $AR$ is to get the output for all TP ranks.\n\nDue to the residual operation executing after AllReduce, it introduces a data dependency from LayerNorm, increasing engineering complexity for backward passes. Therefore, we remove this dependency by detaching $X$, which no longer calculates gradients during backward. \n\n$\\frac{detach(X)}{t}$ arises from this modification, that the residual operation is removed before AllReduce to make sure it does not stall waiting for AllReduce completion: $Y=AR(X_{attn}+\\frac{detach(X)}{t})$, where $t$ is the number of TP ranks. And it keeps computational equivalence in forward as $X=AR(\\frac{X}{t})$. In backward, we add $1$ gradient for $X$ to compensate for the detach operation of residual operation.\n> In Figure 5, ... TP bubbles illustrated clearly ...\n\nFigure 5 is intended to illustrate the pp structure without explicitly visualizing TP bubbles. In our schedule, TP bubbles are implicitly hidden within the overlapped F&W blocks, which indeed have smaller latency than the sum of the individual F and W blocks. We chose to align all blocks in Figure 5 for visual clarity, as misaligned blocks, caused by shrinking the overlapped F&W blocks, could make it harder for readers to trace the communication patterns across stages. That said, we appreciate your feedback, and we will explore improved visualizations that more clearly reflect TP bubble reduction while maintaining readability.\n> Table 1 ... parameter memory ...\n\nThe methods listed in Table 1 are not bidirectional schedules, which have the same parameter memory when not considering the embedding and head layer. Thanks for the suggestion and we will update Table 1 in the final version.\n> In lines 202-203 where it is stated that activation offloading ...\n\nThank you for pointing this out. We will revise the sentence to make it more precise and reflective of the conditions under which the statement holds. We also appreciate your suggestion and will carefully review the rest of the paper to ensure similar statements are appropriately qualified.\n> ... there’s no comparison done, for example against Flux ...\n\nThere are several works designed to improve TP communication, including Megatron-LM and Flux. Their implementations are primarily at the kernel level by fusing computation and communication. In contrast, our approach is implemented entirely at the software level, without relying on custom kernels, making it more flexible and easier to integrate into various training pipelines without requiring custom kernel development.\n\nMoreover, in Megatron-LM, TP overlapping cannot be enabled independently—it requires coupling with sequence parallelism (SP), which introduces an additional parallel strategy. Similarly, Flux implements only several fused kernels, `all-gather+gemm` and `gemm+reduce-scatter`, which require additional parallel to fit these predefined patterns. In contrast, our method is synergistic with widely adopted pipeline and tensor parallelism strategies in large-scale model training and requires no kernel fusion, custom CUDA extensions, or modifications to core operators, enabling easier deployment across diverse architectures. Due to these constraints, we did not combine these works with baselines in our main experiments.\n\nDue to insurmountable technical challenges in integrating Flux kernels into our experimental environment despite extensive efforts, we conducted experiments combining TP overlap in Megatron-LM with pipeline schedules, which embodies a similar overlap as Flux. The results are listed below, where * indicates TP overlap kernels are enabled.\nMethods|Throughput (samples/s)\n-|-\n1F1B-I*|7.13\nZB-V*|7.06\nOurs|7.27\n\nIn the Qwen2-12B with a sequence length of 4k, our schedule achieves a throughput of 7.27 samples/s, outperforming 1F1B-I and ZB-V. This represents an improvement of approximately 0.14 samples/s over the best baseline. The results indicate that our synergistic schedule enables more effective cooperation between TP and PP, achieving superior performance compared to simply combining conventional TP overlap techniques with standard pipeline schedules.\n>  ...  consider a memory apples to apples comparison?\n\nSince different schedules have varying memory footprints, we ensured that all schedules could be executed under the same configuration to enable a fair comparison. This constraint inevitably led to suboptimal memory utilization for some schedules. The primary goal of these experiments was to compare the efficiency of different schedules, rather than to assess their memory usage.\n\nWe were also fully aware of this and conducted a related experiment (Table 5) in the appendix. This experiment explored the optimal throughput that can be achieved by each schedule through maximizing the utilization of memory under the given batch size 192 and sequence length 8k. The results showed that our schedule realized the best throughput (2.74 samples/s), better than 1F1B-I (2.72 samples/s) and ZB-V (2.61 samples/s).\n\n> What about long context experiments?\n\nWe increase the sequence length to 16k and reduce the model dim and layer number to simulate this scenario. And the detailed results are as follows:\nMethods|Throughput (samples/s)\n-|-\n1F1B-I|4.24\nZB-V|4.21\nOurs|4.36\n\nAs shown in the Table, our schedule outperforms other baselines in the setup of longer context and smaller model scale, even though AR may not be fully overlapped. This indicates that our schedule is more robust to the varying sequence length.\n\n> Why are only 1F1B and ZB the baselines? ... are not compared to at all, like Hanayo.\n\nWe chose 1F1B-I and ZB-V as baselines because 1F1B-I is a classical and effective method introducing virtual stages, while ZB-V is a more recent method published at NeurIPS 2024. Other methods such as Hanayo are indeed promising; however, they are not open-sourced, and we cannot ensure faithful reproduction, especially regarding communication optimizations they incorporate.\n\n> discussion of asynchronous vs synchronous pipeline parallelism ...\n\nWe would like to correct a mistake here: ZB-V, cited in our paper, is [1], rather than the reference you mentioned. And *asynchronous* is not the topic in [1]. Therefore, such an issue is entirely irrelevant to our comparison. Moreover, in practice, synchronous schedules are more popular, considering the potential convergence issue in asynchronous schedules. Therefore, evaluating all methods in a synchronous setting is reasonable.\n\n[1] Qi et al. Pipeline Parallelism with Controllable Memory. NeurIPS 2024.\n\n> the position of Fig.1; full form of LayerNorm in Eq. 1; visualization of overlapped F & B in Fig. 5; Activation Memory -> Peak Activation Memory in Tab. 1; throughput's units in Tab. 3; color annotations in Fig. 3; clear discussions in line 272-273; a pointer to Pre-Attn and Pre-MLP in Fig. 2; annotation in Fig. 4\n\nThanks for your valuable comments! We will revise our paper in the final version according to your suggestions.\n> ... experiments without 2 virtual stages per device? \n\nNo! When virtual stages are disabled, our schedule is unavailable as we leverage the virtual stages to realize the overlap.\n> ... perform with context parallelism\n\nOur method is compatible with context parallel. Here, we conduct a corresponding experiment as follows:\nQwen2 12.1B|1F1B-I|ZB-V|Ours\n-|-|-|-\nseq=12k, TP=2, PP=4, CP=2, bs=128| 2.64|2.61|2.71\n\nAs shown, our method achieves a throughput of 2.71 samples/s, outperforming both 1F1B-I (2.64 samples/s) and ZB-V (2.61 samples/s). This demonstrates that our scheduling strategy remains effective and superior even when integrated with context parallelism to handle longer sequences.\n\nPlease refer to the rebuttals for Reviewers EjFt and MuRP to get more results when it is integrated with more parallelism and strategies.\n\n> ... pipeline schedule is of limited novelty from a research perspective ...\n\nWhile overlapping communication and computation are common in LLM training, our method ingeniously combines TP and PP together without any custom CUDA kernels, making them synergistically work. And all experiments in the paper and appendix consistently demonstrate that our schedule is effective.\n\nFurthermore, the value of our approach has been acknowledged by multiple reviewers. Reviewer EjFt noted that \"The proposed scheduling design is practical,\" Reviewer P8vN highlighted its \"solid contributions,\" stating that \"the modification to the interleaved pipelining schedule is simple but effective, and the idea to overlap compute and communication for different microbatches makes sense,\" and Reviewer MuRP described the scheme as a \"thoughtful and non-trivial approach.\" These comments affirm that our work offers a meaningful and well-justified advancement in the design of efficient training schedules."}, {"review_id": "cnhBWRggdP", "reviewer": "Reviewer_MuRP", "summary": "This paper proposes a novel hybrid model parallelism strategy that jointly optimizes tensor parallelism (TP) and pipeline parallelism (PP) for efficient large-scale training of LLMs and MLLMs. The authors introduce a “braided” scheduling technique that interleaves fine-grained forward and backward computation units to overlap TP communication with PP execution. In addition, they develop a “V”-shaped pipeline schedule that balances memory footprint across stages while minimizing pipeline bubbles. The proposed method demonstrates up to 12% and 16% throughput improvement on LLMs and MLLMs, respectively, compared to existing methods like 1F1B-I and Zero Bubble. The method is implemented atop Megatron-Core and evaluated across multiple model scales and hardware settings.", "questions": "Please see the weakness part.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper is well-motivated and targets practical training efficiency issues: pipeline bubbles and GPU idleness during TP communication.\n2. The scheduling scheme is a thoughtful and non-trivial approach. It includes a detailed breakdown of computational units, a V-shaped scheduling strategy, and a memory-aware enhancement, all supported by theoretical analysis of memory footprint and bubble rates.\n3. The implementation is based on Megatron-Core and intended to be released, which supports reproducibility.\n\nWeakness:\n1. Despite that the evaluation part includes several pipeline overlapping baselines, the paper omits a comparison with some recent TP overlapping frameworks (Megatron-LM, Flux). It remains unclear whether a combination of TP overlapping and ZB-V will outperform the proposed work? \n2. Meanwhile, the throughput of Fig. 7 and 8 begins at a high y-axis baseline, which visually exaggerates the perceived speedup.\n3. The method increases peak activation memory compared to some baselines, which may limit applicability in memory-constrained environments if offloading is not viable.\n4. How well does the scheduling scheme integrate with other parallel overlapping schemes such as data parallelism and context parallelism? It would be valuable to understand the generality and composability of the approach.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your support! We sincerely thank the reviewer for the valuable feedback and suggestions. Below, we address each point raised in your review.\n\n> Despite that the evaluation part includes several pipeline overlapping baselines, the paper omits a comparison with some recent TP overlapping frameworks (Megatron-LM, Flux). It remains unclear whether a combination of TP overlapping and ZB-V will outperform the proposed work?\n\nThank you for the valuable question. Megatron-LM and Flux improve TP communication primarily at the kernel level by fusing computation and communication. In contrast, our approach is implemented entirely at the software level, without relying on custom kernels, making it more flexible and broadly applicable.\nMoreover, in Megatron-LM, TP overlapping cannot be enabled independently—it requires coupling with sequence parallelism (SP), which introduces an additional parallel strategy. Similarly, Flux implements only several fused kernels, `all-gather+gemm` and `gemm+reduce-scatter`, which require one more parallel to fit these predefined patterns. In contrast, our method is synergistic with widely adopted pipeline and tensor parallelism strategies in large-scale model training and requires no kernel fusion, custom CUDA extensions, or modifications to core operators, enabling easier deployment across diverse architectures. Due to these constraints, we did not combine these frameworks with pipeline parallelism in our main experiments.\n\nHere, we conducted additional experiments combining TP overlap in Megatron-LM with pipeline schedules. The throughput results are summarized below, where * indicates TP overlap kernels are enabled.\nMethods|Throughput (samples/s)\n-|-\n1F1B-I*|7.13\nZB-V*|7.06\nOurs|7.27\n\nIn the Qwen2-12B configuration with a sequence length of 4k, our schedule achieves a throughput of 7.27 samples/s, outperforming both 1F1B-I and ZB-V. This represents an improvement of approximately 0.14 samples/s over the best baseline. The results indicate that our synergistic schedule enables more effective cooperation between TP and PP, achieving superior performance compared to simply combining conventional TP overlap techniques with standard pipeline schedules.\n\n> Meanwhile, the throughput of Fig. 7 and 8 begins at a high y-axis baseline, which visually exaggerates the perceived speedup.\n\nWe appreciate the reviewer’s valuable feedback. We agree that starting the y-axis at a high baseline may exaggerate the visual perception of speedup. To address this, we will adjust the y-axis scale in Figures 7 and 8, ensuring a more accurate and fair visual comparison of throughput improvements.\n\n> The method increases peak activation memory compared to some baselines, which may limit applicability in memory-constrained environments if offloading is not viable.\n\nWe appreciate the reviewer’s observation. We acknowledge that our method increases activation memory compared to the baselines, as it prioritizes efficiency over memory usage. To assess this trade-off, we conducted an experiment (Table 5 in Appendix) that maximizes memory utilization under a fixed batch size and sequence length. The results show that our schedule achieves the highest throughput (2.74 samples/s), outperforming 1F1B-I (2.72 samples/s) and ZB-V (2.61 samples/s). This suggests that despite a higher memory footprint, our method remains more efficient in practical settings.\n\n> How well does the scheduling scheme integrate with other parallel overlapping schemes such as data parallelism and context parallelism? It would be valuable to understand the generality and composability of the approach.\n\nThank you for the thoughtful question. Our method is compatible with data parallelism and context parallelism, and we have evaluated its performance under these settings. Specifically, we conduct experiments using the Qwen2 12.1B model and report the throughput (samples per second) under different parallelism configurations:\n\nConfig | 1F1B-I| ZB-V| Ours\n-|-|-|-\ncontext parallel: seq=12k, TP=2, PP=4, CP=2, bs=128| 2.64|2.61|2.71\ndata parallel: seq=4k, TP=2, PP=4, DP=2, bs=256| 9.22|9.16|9.40\n\nThese results demonstrate that our approach integrates well with context parallelism and data parallelism, and provides improved throughput over the baselines in both settings. This supports the generality and composability of our scheduling scheme across multiple parallelism strategies."}]

Author rebuttal:

Meta-review: The authors proposes a novel approach to large-scale Transformer training by better coordinating tensor and pipeline parallelism, which targets at reducing the "tensor parallel bubble"—the communication overhead—by overlapping it with the backward computation of the pipeline.

The reviewers (through review and rebuttal) recognize the significance of the work, specifically the scheduling strategy in efficiently overlapping communication and computation with tensor and pipeline parallelism. 

During the rebuttal, the authors clarify the reviewers’ detailed comments, such as the imbalanced stages in MoE layers, clarification of the formulations, evaluations against previous TP overlap work, memory-throughput trade-offs, and so on. And some additional results demonstrate the generalization ability of the approach.

Most of the reviewers express that their comments are clearly addressed, and express their positive attitude towards the acceptance of the paper. To me, the idea is interesting and of sufficient novel. The resutls have demonstrate the superiority of the proposed  method. I would recommend acceptance of the paper. The authors are encouraged to incorporate the comments from the reviewers into the final version the manuscript.
Decision: Accept (poster)