Title: Synergistic Tensor and Pipeline Parallelism
Venue: NeurIPS 2025
Accepted: yes
Keywords: Distributed System

Pipeline Parallelism

Large-Scale Model Training

LLM

Machine Learning System
Abstract: In the machine learning system, the hybrid model parallelism combining tensor parallelism (TP) and pipeline parallelism (PP) has become the dominant solution for distributed training of Large Language Models~(LLMs) and Multimodal LLMs (MLLMs). However, TP introduces significant collective communication overheads, while PP suffers from synchronization inefficiencies such as pipeline bubbles. Existing works primarily address these challenges from isolated perspectives, focusing either on overlapping TP communication or on flexible PP scheduling to mitigate pipeline bubbles. In this paper, we propose a new synergistic tensor and pipeline parallelism schedule that simultaneously reduces both types of bubbles. Our proposed schedule decouples the forward and backward passes in PP into fine-grained computation units, which are then braided to form a composite computation sequence. This compositional structure enables near-complete elimination of TP-related bubbles. Building upon this structure, we further design the PP schedule to minimize PP bubbles. Experimental results demonstrate that our approach improves training throughput by up to 12\% for LLMs and 16\% for MLLMs compared to existing scheduling methods. Our source code is avaiable at https://github.com/MICLAB-BUPT/STP.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel hybrid model parallelism strategy that jointly optimizes tensor parallelism (TP) and pipeline parallelism (PP) for efficient large-scale training of LLMs and MLLMs. The authors introduce a “braided” scheduling technique that interleaves fine-grained forward and backward computation units to overlap TP communication with PP execution. In addition, they develop a “V”-shaped pipeline schedule that balances memory footprint across stages while minimizing pipeline bubbles. The proposed method demonstrates up to 12% and 16% throughput improvement on LLMs and MLLMs, respectively, compared to existing methods like 1F1B-I and Zero Bubble. The method is implemented atop Megatron-Core and evaluated across multiple model scales and hardware settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper is well-motivated and targets practical training efficiency issues: pipeline bubbles and GPU idleness during TP communication.\n2. The scheduling scheme is a thoughtful and non-trivial approach. It includes a detailed breakdown of computational units, a V-shaped scheduling strategy, and a memory-aware enhancement, all supported by theoretical analysis of memory footprint and bubble rates.\n3. The implementation is based on Megatron-Core and intended to be released, which supports reproducibility.\n\nWeakness:\n1. Despite that the evaluation part includes several pipeline overlapping baselines, the paper omits a comparison with some recent TP overlapping frameworks (Megatron-LM, Flux). It remains unclear whether a combination of TP overlapping and ZB-V will outperform the proposed work? \n2. Meanwhile, the throughput of Fig. 7 and 8 begins at a high y-axis baseline, which visually exaggerates the perceived speedup.\n3. The method increases peak activation memory compared to some baselines, which may limit applicability in memory-constrained environments if offloading is not viable.\n4. How well does the scheduling scheme integrate with other parallel overlapping schemes such as data parallelism and context parallelism? It would be valuable to understand the generality and composability of the approach.\n\nQUESTIONS:\nPlease see the weakness part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a pipeline parallel schedule that incorporates communication computation overlap to reduce tensor parallel bubbles by \"fusing\" backwards and forwards tasks. The schedule is introduced after some relevant background. An alternative schedule with offloading to reduce memory consumption is also introduced. Evaluations against some prior baselines are provided.\n\nSTRENGTHS AND WEAKNESSES:\nI would first like to thank the authors for submitting their work to NeurIPS 2025.\n\n### Pros\n\nThe background treatment of PP and TP is fairly comprehensive overall, covering GPipe, 1F1B, Interleaved, Zero Bubble, Flux, etc. In addition, the authors correctly describe the current paradigm of distributed model training with a combination of TP and PP. \n\nThe idea of overlapping the TP computation \"braiding” seems to be sound and makes sense. So does the pipeline schedule itself (i.e. no overlapped critical-path tasks).\n\nSection 4.2 is written well, and provides some well-needed intuition for the schedule and its design. The theoretical analysis in section 4.3 seems to check out and offers a good comparison with some well known prior works.\n\nOffloading is also a nice extension to the basic schedule as well.\n\nThe evaluation section contains a lot of experimental results against 2 solid baselines, and the results seem to be promising.\n\n### Cons\n\nIn section 3, it’s stated nowhere what the Pre-Attn, Attn, Pre-MLP, and MLP consist of. Figure 2 has Attn and MLP annotated, but there should be a pointer to this, and Pre-Attn and Pre-MLP should be clearly annotated.\n\nEquation 1 is not clear at all for the most part, it should be made clear that this is for a single TP rank. Also, I’m not convinced that this formula is correct as shown. Where is $\\frac{detach(X)}{t}$ coming from? What exact operations does this $Attention$ actually perform?\n\nFigure 4 is likewise not motivated. It’s clear for someone that’s aware of interleaved pipeline parallelism, but there’s no annotation of what the colours represent, or any annotation of device, or any markings for the memory.\n\nIn Figure 5, I don’t see the concept of TP bubbles illustrated clearly. My understanding is the overlapped computations should have less latency overall. Perhaps an illustration of the time difference compared to a pipeline schedule without any TP bubble reduction would make this clearer, in the style of Figure 4 from [1].\n\nTable 1 could benefit from including parameter memory as well.\n\nIn lines 202-203 where it is stated that activation offloading offers a better E2E performance trade-off compared to checkpointing. This is a strong statement that is not strictly true in all situations to the best of my knowledge. In the case where the device to host bandwidth is poor, checkpointing might actually enable larger batch sizes and more efficient execution overall.\n\nIn terms of the evaluation, I have four issues.\n\n1. Although TP overlap methods are discussed in line 35, there’s no comparison done, for example against Flux. I see that as one of the most appropriate baselines given one of the main topics of this paper is reducing TP bubbles as well.\n2. I see there is an evaluation of the memory footprint of offloading compared to the default method. But given that Figure 9 shows that the other baselines take less memory, would they not be more efficient by being able to support a larger microbatch size?\n3. What about long context experiments? At longer contexts, the all-reduce latency may no longer be overlapped with compute causing the bubble to reappear.\n4. Why are only 1F1B and ZB the baselines? Some other newer works that are cited earlier as inspiration (line 75) are not compared to at all.\n\nI also don’t see any discussion of asynchronous vs synchronous pipeline parallelism, which is one of the main topics covered in Zero Bubble Pipeline Parallelism [2]. Seems to be that the proposed schedule is synchronous but ultimately it is being compared against asynchronous schedules like ZB-V [2] which should at least have some discussion.\n\n---\n\nOverall, while this paper introduces an interesting new schedule for pipeline parallelism which incorporates “TP codesign”, introducing yet another pipeline schedule is of limited novelty from a research perspective - The main insight here is that backwards and forwards tasks can be fused to provide comp/comm overlap. I am not convinced that the evaluation is sufficient to prove that the method is significant. There are also clarity and overall polish/quality issues.\n\n#### Other Comments\n\nNits:\n\nFigure 1 already includes results in the introduction. I feel this is more appropriate for the results section where the experimental setup can be explained.\n\nIn Figure 3, state that green is forward, and blue is backward.\n\nEquation 1: Use the full form of LayerNorm.\n\nFigure 5: It’s hard to see what’s an overlapped F & B vs F followed by B.\n\nTable 1: Activation Memory -> Peak Activation Memory\n\nTable 3: Include “throughput” and units here for clarity.\n\n\n[1] Narayanan et al., Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM, SC 2021, https://dl.acm.org/doi/10.1145/3458817.3476209\n\n[2] Qi et al., Zero Bubble Pipeline Parallelism, Arxiv, https://arxiv.org/pdf/2401.10241\n\nQUESTIONS:\n### Major\n\nCan you clarify equation 1?\n\nCan you clarify whether Figure 5 should actually have smaller boxes for the combined F & B parts?\n\nWhy not consider long context baselines?\n\nWhy not evaluate against previous TP overlap work?\n\nWhy not consider a memory apples to apples comparison?\n\nWhy not evaluate against more recent work already cited as inspiration, for example, Hanayo?\n\n### Minor\n\nDid you try any experiments without 2 virtual stages per device? Any difference in performance for the baselines?\n\nAny idea how this performs with context parallelism?\n\nLIMITATIONS:\nSome limitations (such as performance PP = 8) discussed in lines 272-273.  The authors would be encouraged to make the limitations clearer.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nModel training at scale often uses tensor and pipeline parallelism. Tensor parallelism can suffer from high communication overhead. This paper offers a way to overlap communication with computation to improve training speed.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Important problem (improving efficiency of LLM and and MLLM training at scale).\n- Solid contributions: the modification to the interleaved pipelining schedule is simple but effective, and the idea to overlap compute and communication for different microbatches makes sense.\n- Results are good.\n\nWeaknesses:\n- Some of the writing can be made a bit more precise. For example: a) I don't think TP communication significantly grows with TP size as long as you're able to run collectives at the same bandwidth; of course compute-communication ratio decreases. b) What does \"at the kernel or hardware levels\" mean when discussing communication overlapping with computation? c) Clarify hardware setup in statements like \"where limited inter-device bandwidth can severely degrade training efficiency\".\n- I'm not sure how \"synergistic\" the modification to the interleaved pipelining schedule is, can't the proposed memory improvements be used for pipeline parallelism in isolation?\n\nQUESTIONS:\n- Figure 3 doesn't make full sense to me. For example, in Figure 3a, is the green AR in the communication stream for a different microbatch than the green \"Attn (F)\"? If they are for the same microbatch, how does \"Attn (F)\" complete before its communication?\n- How much is computation slowed down by the overlapped communication (since communication probably uses some SMs that can't be used by GEMMs)?\n- Is there a way to apply these ideas to PP=1?\n- What are the throughputs in Figure in TFLOP/s/GPU? That is, how fast on an absolute scale are the baselines and \"Ours\"?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper proposes a novel approach to large-scale Transformer training by better coordinating tensor and pipeline parallelism. It aims to reduce the "tensor parallel bubble"—the communication overhead—by overlapping it with the backward computation of the pipeline. The experiments demonstrate improved throughput, albeit with some memory overhead.

STRENGTHS AND WEAKNESSES:
**Strength**
- The paper provides a clear analysis of how tensor and pipeline parallelism interact and where current methods fall short.
- The proposed scheduling design is practical and accounts for the potential memory challenges introduced by this method.
- The implementation considers real hardware constraints, such as memory usage and communication overhead, making the method more applicable to real-world training setups. Also the experiments are well-executed across multiple large models (including LLMs and multimodal models), providing solid evidence of scalability and general applicability.

**Weaknesses**
- I am not very familiar with pipeline parallelism, so I found some figures (e.g. Figure 3) a bit challenging to fully understand.

QUESTIONS:
- How would the proposed scheduling handle highly imbalanced stages, such as those caused by MoE layers or custom modules?
- Is the system compatible with activation checkpointing strategies commonly used in large-scale models?

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-09