PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
Authors: Xinyi Wan
Penghui Qi
Guangxing Huang
Min Lin
Jialin Li
Venue: ICML 2025 poster (2025)
Keywords: LLM Training
Pipeline Parallelism
Offloading
Memory Optimization
Primary area: optimization->large_scale_parallel_and_distributed

Abstract: Pipeline parallelism (PP) is widely used for training large language models (LLMs), yet its scalability is often constrained by high activation memory consumption as the number of in-flight microbatches grows with the degree of PP. In this paper, we focus on addressing this challenge by leveraging the under-explored memory offload strategy in PP. With empirical study, we discover that in the majority of standard configurations, at least half, and potentially all, of the activations can be offloaded with negligible overhead. In the cases where full overload is not possible, we introduce a novel selective offload strategy that decreases peak activation memory in a better-than-linear manner. Furthermore, we integrate memory offload with other techniques to jointly consider overall throughput and memory limitation. Our experiments proves that the per-device activation memory effectively reduces with the total number of stages, making PP a stronger alternative than TP, offering up to a 19\% acceleration with even lower memory consumption.

PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
Xinyi Wan 12
Penghui Qi
Guangxing Huang' Min Lin Jialin Li²
Abstract
remains constrained by the activation memory. Increasing the number of PP stages reduces layers per device but ne- cessitates more in-flight microbatches to minimize pipeline bubbles. This trade-off leaves overall activation memory demands unchanged.
Pipeline parallelism (PP) is widely used for train- ing large language models (LLMs), yet its scala- bility is often constrained by high activation mem- ory consumption as the number of in in-ftight micro- batches grows with the degree of PP. In this paper, focus addressing this challenge by leverag- ing the under-explored memory offload strategy in PP. With empirical study, we discover that in the majority of standard configurations, at least half, and potentially all, of the activations can be offloaded with negligible overhead. In the cases where full overload is not possible, we introduce novel selective offload strategy that decreases peak activation memory in a better-than-linear manner. Furthermore, we integrate memory of- fload with other techniques to jointly consider overall throughput and memory limitation. Our experiments proves that the per-device activation memory effectively reduces with the total number of stages, making PP stronger alternative than TP, offering up to 19% acceleration with even lower memory consumption. The implementation is open-sourced at this url.
In this work, we address this memory limitation of PP by offloading memory the host. While memory offload is widely adopted in data parallelism (DP) (Goyal et al., 2017; Ren et al., 2021), its potential in PP remains largely unex- plored. PP is particularly suited for memory offload because the gap between the forward pass and the backward pass cre- ates a natural window for offloading and reloading activation memory without interfering with other computations. This contrasts sharply with activation rematerialization (Chen et al., 2016) which introduces significant recomputation overhead. Memory offload, when properly scheduled and overlapped with other computation, can be free lunch.
Formally, for single transformer layer, we define T, as the round-trip time for its activation memory to be moved from device host (D2H) and then host to device (H2D), Te the time for its total forward and backward compute, then the ratio between them, denoted as k: To/To, is an important indicator on the proportion of the activation mem- ory that can be offloaded without introducing significant efficiency drawbacks. Full activation memory offload is possible if k s 1, as the offload operations can be fully overlapped by compute. Without considering the trivial lay- ers which can be recomputed with negligible overhead (e.g. Dropout, GeLU, LayerNorm), we can estimate the offload cost for the rest of the layers and subsequently estimate k. With sequence length (s), hidden size (h), PCI-E duplex bandwidth (B,) and GPU compute bandwidth (Bz), is computed as: (Narayanan et al., 2021; Korthikanti et al., 2023)
1. Introduction
As modern large transformer models (Vaswani et al., 2017) scale towards trillions of parameters, model parallelism be- comes essential for distributing model parameters across multiple devices. Compared to ZeRO (Rajbhandari et al., 2020) and tensor parallelism (Shoeybi et al., 2019), pipeline parallel (PP) (Huang et al., 2019: Harlap et al., 2018) has lower communication volume and higher arithmetic in- tensity. However, while PP shards layers across devices to reduce parameter memory requirements, its scalability
To 10 T-
B. B.
k
(1)
3(6h + +s)
'Sea Al Lab National University of Singa- pore. Correspondence to: Min Lin <limmin @ sca.com>, Jialin Li call@comm.nu..duu.
Equual contribution
Note that the value of A decreases model size or sequence length increases. Figure 1 (left) demonstrates that the value of k is surprisingly small under typical hidden dimension and sequence size settings, indicating that a significant por- tion activation memory can be offloaded. We observe that & I 1 when the hidden dimension exceeds 8k or the
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
Theoretical k
Slowdown of Fully Offload (%)
4096
1.70
1.46
1.15
0.80
4096
31.1%
20.0%
6.2%
-7.4%
30
Model Dimens on
1.50
5120
1.40
1.23
1.00
0.72
5120
15.0%
6.9%
0.2%
+3.8%
20
6144
1.19
1.07
0.88
0.66
1.25
6144
2.9%
0.5%
0.3%
0.2%
7168
1.03
0.94
0.80
0.61
1.00
7168
0.6%
0.5%
0.2%
-4.%%
100
8192
0.92
0.84
0.72
0.56
0.75
8192
1.0%
0.8%
0.8%
0.1%
10240
0.74
0.69
0.61
0.49
10240
0.3%
0.1%
0.2%
-2.6%
0.50
4096
8192
16384
32768
4096
8192
16384
32768
Sequence Length
Sequence Length
Figure 1. Ablation studies on offload overhead on NVIDIA A100 GPUs. On the left, & values were estimated using Formula with B. = 220 TFLOPS/s and B, = 15 GB/s. On the right, the reduction in throughput due to offload was measured through experiments. The experiments are performed utilizing the fully offloaded IFIB schedule outlined in Figure A, involving 8 PP devices and 32 microbatches. The number of transformer layers was chosen to ensure that the bascline, without offload, does not OOM. It is important to note that some values in the second graph are negative, as the haseline experiences frequent CUDA malloc/dealloc operations due to high memory usage.
sequence length is greater than 16k; under these conditions, all activation memory can be offloaded with negligible overhead, as shown in Figure (right).
Interleaved 1F18
Activodton Womory
Stage
Stage ocal
When h >> 1, offloading all activation memory would lead to a decrease in throughput. In this scenario, we resort to a partial offload where a subset of the activations are of- floaded. Similar to the rematerialization approach where trivial operations are preferred as they impose less compu- tation burden, we perform selective offload that prioritize activations that yield the greatest reduction in peak mem- ory usage. We introduce general guideline for selective offload that always prefers activations with longer lifespan, i..., longer gaps between forward and backward passes. In- tuitively, the longer an activation remains in flight, the more contributes to peak memory.
Time PipeOffeaad
Adivatton Momory
Figure 2. Memory pattern of different schedules. We plot the acti- vation memory of each stage separately and show their contribution to the total activation memory. In Interleaved IFIB, offloading stage 0 results in only 50% reduction in peak activation memory, despite stage 0 having a longer lifespan. Contrastingly, PipeOf- fload the bottom distributes activation memory uniformly across time, offering better- better-than-ae than-lineat memory savings if stage 0 is of- floaded.
widely adopted strategy to reduce pipeline bubbles in- volves placing multiple stages on the same device, as demon- strated in works such as (Narayanan et al., 2021). (Qi et al., 2024), and (Liu et al., 2023). Notably, different stages have varying lifespans, potentially resulting in a more efficient, seater-than-linees reduction in peak memory usage. The efficiency of selective offload also depends on the mem- ory usage pattern of the pipeline schedule. For instance, when visualizing the memory usage patterns of interleaved IFIB (Narayanan et al., 2021) and our PipeOffload method in Figure 2, it is apparent that offloading stage 0 in our method yields a 3/4 reduction in peak memory, while in the interleaved IFIB, at most a 1/2 reduction can be achieved.
In the rest of this paper, we detail the selective offload strat- egy in Section 2, introduce a family of pipeline schedules that trade-off between throughput and peak memory in Sec- tion 3, elaborate implementation details in Section 4, and finally evaluate and compare the methods in Section 5,
2. Selective Offload
When full activation offload not feasible (k: >> 1) without overhead on the throughput, an efficient selective offload strategy becomes crucial. In this section, we explore con- siderations for a selective offload strategy when multiple pipeline stages are placed into each device as in interleaved IFIB (Narayanan et al., 2021).
When memory offload applied in practice, we need to further consider its interplay with other factors, especially the trade-off between memory and throughput. We tend the interleaving strategy into generalized form. of- fering smooth memory reduction with minimal throughput loss. This approach provides flexibility in optimizing per- formance based on specific system needs.
Following (Qi et al., 2024), we decompose a pipeline sched- ule into repeating a building block, which describes how a
2

---

PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
102%
Selective
85%
Pandenwing
Pua Adbo an
Unifor
40%
Reyworing
22%
Figure 4. The building block (top) describes the pattem for each mi- crobatch, where F represents forward and B represents backward. Both the interlcaving (middle) and uniform repcating (bottom) strategies adhere to this building block. Although sharing the same peak memory, the contributions of pipeline stages differ these two strategies. We emphasize the contributions of the first pipeline stages with bold borders.
0%.
Number sicelne stagen atflaccingg dextce
Figure 3. The memory reduction ratio of stage-level offload under 8 PP devices and 16 stages.
single microbatch should be scheduled in the pipeline (as in Figure 4). By ensuring each microbatch adheres to this pat- ter,, the peak memory usage is approximately proportional to the summed lifespan of each pipeline stage. Intuitively, the longer activations remain in flight, the more memory PP schedule consumes. Based on this insight, preferring offloading pipeline stages with longer lifespan is natural selective strategy for more memory reduction.
Based on these observations, we propose prioritizing the offloading of pipeline stages with longer lifespans. The uniform repeating strategy should be favored over the inter- leaving strategy for its superior memory reduction efficiency within the same offload budget. Figure 3 presents the the- oretical peak memory curve under various offload budgets using the uniform repeating strategy. It demonstrates better- than-linear efficiency in memory reduction. Notably, by offloading only half of the pipeline stages, peak memory can be reduced to approximately one-quarter in scenarios with an 8 PP degree and 16 pipeline stages per device.
However, lifespan is not the sole factor influencing a stage's contribution peak memory. The strategy for organizing microbatches within a pipeline schedule also significantly impacts how stages contribute to peak memory. We evaluate two strategies for building pipeline schedules: the inter- leaving strategy (as in (Narayanan et al., 2021)) and the uniform repeating strategy (as in (Qi et al., 2024)). The interleaving strategy employs a bi-level repeating pattern: the outer loop interleaves pipeline stages, while the inner loop uniformly repeats set number of microbatches. In contrast, the uniform repeating strategy consistently sched- ules the next microbatch after a fixed offset. As shown in Figure 4, although both strategies share the same build- ing block and peak memory, their contributions to peak memory differ. Specifically, offloading or recomputing the first pipeline stage (indicated by white numbers) in rank O reduces peak memory by 3 activations in the interleaving strategy, whereas the uniform repeating strategy achieves reduction of 4 activations. This demonstrates greater memory reduction with the uniform repeating strategy than with interleaving srategy.
3. Trading off Memory and Throughput
In PP, there is trade-off between activation memory and pipeline bubbles. While the uniform repeating strategy of- fers superior memory reduction efficiency through offload, it may also lead to an increased number of pipeline bubbles compared to the interleaving strategy, as illustrated Fig- ure 4. In scenarios where memory pressure not a primary concern, the interleaving strategy is preferable due to its ability to maintain higher throughput. In this section, we ex- tend the interleaving strategy into generalized form. This approach aims to achieve smooth memory reduction while minimizing throughput loss, providing a flexible solution that can be tailored to specific system requirements.
3.1. Free Lunch for Interleaved 1F1B
Zero-Bubble Strategy We use the zero-bubble strategy from (Qi et al., 2023) to reduce pipeline bubbles without any trade-offs. Specifically, the backward pass is split into activation gradient computation (B) and weight gradient computation (W). Unlike ZB-HI and ZB-H2 schedules in (Qi et al., 2023), we delay W passes to further reduce bubbles, as this complicates our strategies. Since our focus is mainly on memory reduction, we simply keep their origi- nal schedules with split backward pass (see the middle of Figure 5).
Figure 2 illustrates the contribution of each pipeline stage to peak activation memory across 8 devices. It is evident that in both strategies, stage 0 (with longer lifespan) contributes more than equal to stage (with shorter lifespan) to peak memory. Notably, the uniform repeating strategy, where each stage's 'ss  contribution to peak memory is roughly propor- tional to its lifespan (Qi et al., 2024), tends to offer greater memory reduction compared to the interleaving strategy under the same budget.
3

---

PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
Stage Siage
Forward
Stage Stage
Badkward
Backward
PP san:1
7
one
Stage 0 Stage
Forward
Adjust Wirmsp
Stage Stage
Activction Grodient
Stage
Weight Gradionn
F
Figure 5. Top: vamilla interleaved IFIB; Middle: with split backward; Bottom (GIS): after adjusting warmup.
Building block with shorter lifespan
2
Staggn O Stage
Forward
Stage Stage
Acthation Gradient
Stage
Weight Grndinnt
Squease
#Pram:
PP
Figure 6. Top: the building block of GIS-H; Bottom: GIS-H schedule with less activation memory.
Adjust Warmup We modify the vanilla interleaved IFIB schedule to make it more memory efficient, shown in Figure 5. Specifically, we reduce the number of forward passes in the warmup phase from d(v - 1) + 2(d - i) to d(v- - 1) + d - for PP rank i, where d is the number of devices and u is the number of pipeline stages per device. This change reduces the maximum peak memory (occurring in rank 0) from du + d to du, which is a prominent improvement especially for small u. Importantly, this modi- fication achieves memory savings without introducing any additional pipeline bubbles.
portunity to further reduce activation memory. We design building blocks with shortened lifespans (Figure 6) and organize microbatches into pipeline schedule by repeating these building blocks. Similar to interleaved IFIB, we use bi-level repeating pattern: the outer loop interleaves pipeline stages, and the inner loop uniformly repeats a number of microbatches (denoted as g). Notably, GIS is special case where equals d. We can adjust g to control the lifespan, with minimal value of [ to satisfy dependencies be- tween stages. Note that reducing the lifespan incurs extra pipeline bubbles during the warmup phase. For any value of ( 5g S d), the maximum peak activation memory is g(u - 1) + d (in rank O), and the size of extra pipeline bub- bles is (d- g) * (v 1). By selecting the largest value of that fits within the memory limit, we can achieve an optimal schedule with minimal throughput loss. In the extreme case where g [ (Figure 6), it results in peak memory usage that is about half of the vanilla interleaved 1FIB. We refer to this specific schedule as GIS-H.
For convenience, we refer to the schedule after splitting backward and adjusting warmup as GIS.
3.2. Memory Reduction by Shortening Lifespan
We generalize the interleaving strategy to accommodate smaller memory constraints with minimal efficiency loss, while maintaining a IF1B pattern. Inspired by (Qi et al., 2024), we reduce peak memory usage by shortening the lifespan of the building block.
By uniformly repeating the building block (as described in Section 2) of GIS-H, with minor modifications to prevent collisions (Qi et al., 2024), we present schedule called PipeOffload with similar peak memory. In Pipeoffload, offloading half of the pipeline stages is referred to PO-H. while offloading all stages is termed PO-F. We compare the
From Figure 5, we observe redundant lifespans for each microbatch. For instance, in GIS, although backward passes have tight dependencies within each stage, the waiting time between stages excessively long. This presents an op-

---

PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
bubble and activation memory in Table 1.
can lead to significant latency fluctuations (see Appendix D), which can result in notable overhead due to computation passes waiting for offloading or reloading to complete. In contrast, we use single stream for both offloading and reloading. By sharing the stream, we stabilize the lateney of offload and reload passes, simplifying the scheduling and enhancing system robustness and performance.
Table 1. Comparing activation memory and bubble rate of differ- ent schedules. Additional notations: Activation memory of the entire model (A Time of F. B. W passes of a single stage on a device (T', TB. Tw). Note that for PO-F, the activation memory proportionally ll decreases with vd.
Schedole
Rubbic
When scheduling the offload and reload passes based on uni- form repeating strategy, shown in Figure 7, maintain pipeline bubbles after repeating the building block. For a given model and its training configurations, we first calcu- late k as defined in Formula 1. Then, we allocate separate slots in the stream using a one-offload--oe ierooad-one-rlladde one-reload pattern. Given the pipeline stages to offload, we process them one by one from left to right, finding the earliest available offload slot after the forward pass and placing the offload there. For reloading, we move from right to left, identifying the latest available reload slot before the backward pass and placing the reload there. Although some slots may remain unoccupied and the schedule is not squeezed, it can be op- timized automatically when running on devices. For A100 GPUs, where PCI-E is often shared by two adjacent devices, we stagger the offload streams and insert synchronization events across the two devices avoid the same operation occurs simultaneously (see Appendix D for more details).
Manory
1F18
M
vid 1 + Ip + Tw)
1F194
(d 1) + Ty + Tw)
GIS
M
(d 1)TT - Ta)
GIS-F1
2v M
id 1)(Tp + Tix) + 3" (Tp + Tn Tw)
Po-11
By M
<v(d- ()(TF TB + Tw)
PO-F
O mi)
< v(d - 1)TF + TB + Tw)
4. Offload Implementation
In this section, we elaborate the implementation details of our offload strategy, particularly highlighting the differences with the approach described in (Yuan et al., 2024). As we aim to reduce memory usage with minimal overhead, we primarily focus on leveraging the "free lunch" opportunity of offload, which necessitates that the offloading and reload- ing processes can be fully overlapped with computation, thereby avoiding any additional overhead for the original pipeline.
4.3. A on Other Hardware Platforms
In Figure 1 we showed that k is relatively small on A100 GPUs. While the value of k is reliant on the hardware architecture, we anticipate that H100 will exhibit a similar k to A100, given that H100 boasts 2x PCI-E bandwidth following an upgrade from PCI-E v4 to v5, along with a 3x increase in compute bandwidth. Furthermore, the model flops utilization (MFU) typically reported by the community is lower for H100s (43% in Llama 3 (Dubey et al., 2024)) in contrast to A1 (approximately 60% in our evaluations), thereby narrowing the bounds even further.
4.1. Improve Offload Efficiency
To reduce the offload constraint and enable greater offload capabilities, we adopted the subsequent approaches: a) Em- ploying direct recomputation on sntivatiin-heayy but com- putationally lightweight layers like GeLU to diminish acti- vation memory per layer, leading to a notable 40% decrease in activation while maintaining throughput efficiency. b) Guaranteeing stable and swift PCI-E bandwidth through strategy. c)
the utilization of hardware-topooooy-awee Decreasing host-side memory capacity overhead by lever- aging continuous buffers. Furthermore, it is important to highlight that all these methods contribute to reducing k; which signifies the number of stages that can be offloaded. For more detailed explanation of these techniques, please refer to Appendix C.
4.4. Caveats of Offload
Though a carefully implemented offload strategy brings only negligible overhead to the compute throughput, there're 're some moderate issues. Firstly, the host memory capacity is another notable bound for the offload. However, host mem- ory is usually several times larger than the total GPU mem- ory installed on the host and is usually extensible with much lower cost compared to GPU. Secondly, achieving "free lunch" offload scenario becomes unattainable if the time interval between matching forward and backward passes is shorter than the round-trip time of offload. A low time interval implics a short lifespan, leading to negligible con- tribution to peak activation memory. In practice, skipping offloading for these passes has been observed not to im-
4.2. Offload Scheduling
When integrating offloading and reloading into a pipeline, careful scheduling alongside computation passes is essential. (Yuan et al., 2024) employs fixed scheduling strategy, initiating offloading immediately after the forward pass and reloading at the start of the last backward pass. Offloading and reloading are placed into separate streams to enable overlap. However, in practice we find that separate streams
5

---

PipcOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
Stage 0
Forward
Stage
Actination Gradiert
Stage
Weight Grudient
Cfllised Sixr.
Stage
Relood
Figure 7. The offload scheduling based on uniform repeating with one pipeline stage offloaded per device and &: = 1. We use a single stream with one-offload-om -oa-a--oo-oaaadddd pattern, where "-" means empty slot.
pact peak activation memory. Lastly, the PCI-E traffic for data movement between host and device may interfere with cross-node P2P communication protocols such as Infiniband or RoCE, potentially slowing down communication if not scheduled prudently. Notably, P2P communication does not significantly impact the utilizable PCI-E bandwidh due to two reasons: a) The volume of P2P communication is substantially lower than offload. For each transformer layer, the total activation memory is times greater compared to the layer output. b) Most P2P communication occurs within node on NVLINK, bypassing PCI-E, which mitigates the impact on offload constraints.
modern hardware, such as H100.
The pipeline schedules we evaluate include: a) IFIB (Har- lap et al., 2018) and vanilla Interleaved IFIB (referred to as IFIB-I) (Narayanan et al.. 2021) implemented in Megatron- LM; b) Our generalized interleaved schedule, GIS and GIS- H. detailed Section c) Our better-than-linear offload techniques outlined in previous Sections. We concentrate on two key configurations. PO-H and PO-F, where either half([]]]
full(u) stages are selectively offloaded. No- tice that we skip the PO-F settings if the corresponding k > 1. For all schedules except 1FIB, we set the number of stages on each device to the maximum possible value so that each stage has at most 1 transformer layer, unless explicitly specified.
5. Experimeents
We evaluate our methods on GPT-3-like models based Megatron-LM LM (Narayanan al., 2021). In most cases, one transformer layer is removed from both the first and last pipeline stages to address imbalances caused by vocabulary layers, similar to Llama 3 (Dubey et al., 2024) and Deepseek v3 (Liu et al., 2024). The models used are listed in Table Our primary metrics are throughput, measured as model flops utilization (MFU), and activation memory, defined the difference between peak and iteration-start memory. The reported activation memory refers to the maximum peak activation memory observed across all devices.
5.1. Activation Memory Reduction with Similar Throughput
In Figure we present a comparative analysis of activation memory and throughput across various methods. Compared to /FIB-I, our GIS-H method effectively halves the acti- vation memory, while PO-H achieves a more substantial reduction, decreasing it to at most. PO-F further mini- mizes activation memory, providing a solution for scenarios where other methods might encounter out-of-memory is- sues.
In terms of throughput, all methods generally perform well, with pipeline bubbles only occurring during the warmup and cooldown phases. Our GIS method outperforms /FIB-I by offering both higher throughput and reduced activation memory. Although PO-I and PO-F show slightly reduced throughput compared to GIS and GIS-H, H, due to additional pipeline bubbles, they still surpass FFI in throughput (ex- cept for cases which runs PO-F with k exactly equals to 1). For detailed numerical results and explanations of any missing data points, please refer to Figure 15 in appendix.
Table list of models used in experiments. For all models turn on GQA (Ainslie et al., 2023) with number of query group set
Model
Layers
Attention Heads
Hidden Size
Batch Size
GPUs
5.8B
32
32
4096
32
2-32
10.5B
38
40
5120
64
18. 1B
46
48
6144
128
16
42.9B
62
64
8192
256
32
66.6B
62
80
10240
256
5.2. Better-Thand Linear Selective Offload
83.8B
78
80
10240
256
32
In Figure we illustrate the impact of memory savings when implementing stage-level offload across various sched- ules. The results indicate that our pipeline offload (PO) schedules with uniform repeating demonstrate memory sav- ings that surpass linear scaling, unlike GIS-H, HH, aligning with
Our experiments run on up to 32 NVIDIA A100 80G GPUs on nodes interconnected by RoCE RDMA network. As mentioned Section 4.3, though the à is hardware- dependent, we focus on 100 because it similar for other
6

---

PipcOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
k on 10.58 Model 8 GPUS
k on 18.18 Model 16 GPUS
on 42 98 Model 32 GPUs
1.5
Bound of PG-F
Bound P0-
1.0
www. Bound PF-F
1.0-
1.0
0.5
0.5
0.5
8192
16384
32768
16384
32768
8192
16384
32768
Sequence Length
Sequence Length
Sequance Langth
Activation Memory (GB) GPUs
10.58 Model
Activation Memory Model
Activation Memory (GB) on 42.98 Model 32 GPUs
16
Activaton (GB)
8.0
Activation Mamory (GB)
80
Activation Mamory (G9)
80
1F18-1
GIS 615-
PC-H1
1118-1
G15
PG-1
191 B-
R25
PD-1
60-
1F18
PO-
60-
1F18
PO
60
1F1B
P
80
20
20-
20
10394
Soquance Longth
Soquence Length
Sequence Longth
MFU (%) 10.58 Model GPUs
MFU (%) 18.18 Madel
MFU (56 42.98 Model
16 GPUs
32 GPUs
70
NFU (N)
NFU (16)
60
60
60
NFU
50
1F1B-1
GIS
PO-H
50
1F1B-1
G1S
PO-H
50
1F1B:1
G15
PO-H
6096
8192
16384
32768
4096
1G384
32708
4096
8192 Sequence Length
16384
Secuence
Sequence Length
Figure& 8. Memory and Throughput Comparison of Different Methods. For detailed numbers, please refer to Figure 15 in appendix.
the analysis in Section 2. It's important to note that across all schedules, there exists a consistent overhead caused by temporary activation memory when offloading all stages.
unattainable without our techniques, with interleaved IFIB combined with 8 TP (together with sequence parallelism in (Korthikanti et al., 2023)). The results depicted in Figure 11 showcase a notable 12%- 19% acceleration in training, attributed to the elimination of TP, which typically incurs significant communication overhead. We also highlight that PO-F method not only exhibits higher throughput but also consumes less activation memory compared to IFIB-I com- bined with the maximum TP degree. This finding suggests that in scenarios where PO-F is applicable (k < 1). pure PP should be preferred.
5.3. Activation Memory Scaling Study
We delve into the strong scaling of activation memory by analyzing per-device activation memory using a fixed 5.8B model across various total numbers of stages e x d. The results depicted in Figure 10 reveal that the per-device ac- tivation memory (left figure) of PO-H and PO-F exhibits superior scaling compared to other methodologies, primar- ily due to the reduction in the number of in flight activations (as indicated in the right figure). Notably, the number of in- flight activations remains constant for PO-F, matching the analysis in Table 1. This observation implies that in the most common scenarios where each stage has only I transformer layer, each device essentially maintains an activation mem- ory equivalent a small constant number (approximately 4 in our experiments) of transformer layers, irrespective of the total number of layers and pipeline devices.
5.5. Comparing With Other Pipeline Parallel Methods
We compare our approach with existing pipeline parallelism (PP) methods, particularly those aimed at reducing activa- tion memory, including V-Min and V-Half from Qi et al. (2024), as well as activation re-materialization combined with 1F1 (shown as FFIB-I-R). As shown in Figure 12, our methods achieve more favorable Pareto frontier in the trade-off between memory usage and throughput. Through detailed analysis, we attribute the suboptimal performance of the V-Shaped schedule proposed in Qi et al. (2024) to significant disparities among Tr. TB. and T Tw at large se- quence lengths. These disparities violate the core assump- tion in Qi et al. (2024) that these timings are approximately equal, leading to degraded efficiency.
5.4. Comparing With Tensor Parallelism
The high activation memory volume is one of the biggest concern for scaling PP to more devices. Commonly, stan- dard settings such as Llama3 (Dubey et al., 2024) often employ a maximum TP degree, typically set at 8, to re- duce the activation memory per device. By leveraging our methods to save activation memory on PP, we now compare the performance of using pure PP, which was previously
7

---

PipcOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
Activation Memory on 10.58 Model
Activation Memory 18.18 Model
Activation Memory on 42.90 Model
GPUs
GPUs
32 GPUs
Sequence Length 16384
Sequence Length 16384
Sequence Length 8192
Activation (GB)
30
Activatton Memory (GB)
50
Activatian Memory (GB)
50
25
40
PO
40
PO
20-
30
30
15
20
20
10
Offioaded Stages
Offioaded Stages
Offieaded Stages
Figure 9. Better-than- selective offload. We gradually increase the number stages offload two schedules: GIS-H defined in Section 3 and Pipeline Offload (PO), the uniformly repeated schedule introduced in Section 2
Activation Nemory
Number of In-iight Adtivattioon 011
42.9B Model Seq 16384
66.68 Model Seq 16384
83.88 Model Seq 8192
37:34
0651
70
PO F
H
NFU (86)
70
Pg
70
PO-F
FO-H
60
TP8
TP8
MFU
60
TP8
50
50
50
25
50
25
50
25
50
Activation Memory (GB)
Activation Memory (GB)
Activation Memory (GB
Figure 10. Per-device activation memory when training a 5.8B model using different total number of stages. The left figure shows the activation memory in GB while the right figure shows the number of in-ghtttt microbatches (different stages of the same mi- crobatch are counted multiple times). If there're re multiple settings for the same d, the setting with minimum activation memory is reported. The amount of activation memory is estimated by running the scheduler and count the in-ftight activation memory on GPU. Detailed data on this experiment is shown in Figure 18 in appendix.
Figure II. Comparison of pure PP using our methods with hybrid parallelism using PP+TP, PO-F and PO-H runs 32-way pure PP while /FIB-I runs with TP8xPP4.
Convergence Comparison on 5.4B model
PO-H
PO-
IF181
8-
5.40 Mode: 342
4b Model
6-
304 Saris
199
wox 10-11
200
400
600
800
1000
iteration
Activation Hnenrryra 1ox
Acthondis Mamoty
Figure 13. Loss curves proves algorithmic correctness of our meth- ods.
Hestll CPus deg B192
GPub Seg 4026
371448
Tunta Forder
Pomes frwcer
exhibit nearly identical loss trajectories, providing strong evidence for convergence correctness. Notably, our methods are designed as exact algorithms with no compromises in accuracy.
Activiiis Manicy
Figure 12. Pareto frontier compared with other methods.
6. Related Work
Pipeline Parallelism Various pipeline schedules have been developed to reduce activation memory usage in PP. An early notable work is 1FIB (Fan et al., 2021; Harlap et al., 2018), which uses a www--ooowrd-o---bbaeee mitigate the high memory usage of GPipe (Huang et al.,
5.6. Convergence Experiment
We compare the loss curves of our PO-H and PO-F imple- mentations with IFIB-I to validate correctness on a 5,4B model using GPUs. As shown in Figure 13, the results
pattern to
8

---

PipcOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
2019). BPipe (Kim al., 2023) was later introduced ad- dress the memory imbalance issue of 1FIB by transferring activations across devices. Leveraging the auto-regressive property of causal transformers, token-level pipeline sched- ules have been proposed by (Li et al., 2021; Sun et al., 2024), showing promising memory reduction results, espe- cially for long-context training. Vocabulary parallelism was recently proposed by (Yeung et al., 2024) to address the memory imbalance caused by vocabulary layers, alleviating the memory bottleneck of PP. (Qi et al., 2024) introduced general framework showing that peak memory can be di- rectly controlled by lifespan of the building block. Based on this insight, they proposed a memory- meeooryballnnce V-Shape schedule, reducing peak activation to at most 1/3 compared IFIB.
scalability of PP and PP becomes feasible in cases they not possible previously. We demonstrate that the enhanced PP can serve as a compelling alternative to other distributed training methods, such as tensor parallelism.
Impact Statement
This paper presents work whose goal is to advance the field of Large Language Model Training Systems. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F, and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305.13245. 2023.
Activation Rematerialization and Offload Activation rematerialization was first proposed by (Chen et al., 2016) which trades computation for memory. To alleviate its over- head, selective strategies have been developed (Korthikanti et al., 2023; Yuan et al., 2024), focusing on recomputing operations with high memory-to-computatiin ratio.
Chen, T., Xu, B., Zhang, C., and Guestrin, C. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.00 16004061744 2016.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint 0rXiv:2407.21788 2024.
Offload techniques have also been explored to address mem- ory constraints in training LLMs, but prior work (Ren al., 2021; Rajbhandari al., 2021) often focuses on model states, leading to poor overlap between data transfer and computation and resulting in high overhead. A recent work (Yuan et al., 2024) on offloading activation memory in PP is the most related work to ours. However they draw an opposite conclusion than ours, that activation offload causes significant overhead and should be avoided if possible. They emphasize memory reduction, allowing offload delay computation, which often results in brittle schedule in- troducing significant pipeline overhead. In contrast, we focus on improving the memory reduction efficiency and minimizing overhead by fully overlapping offload with com- putation. We deliver a different insight that offload can be a free lunch in PP, and full activation often feasible make PP scalable.
Fan, S., Rong, Y., Meng, C., Cao, Z., Wang, S., Zheng, Z., Wu, C., Long, G., Yang. J., Xia, L.. et al. Dapple: A pipelined data parallel approach for training large models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pp. 431-445 2021.
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 026777 2017.
Harlap, A., Narayanan, D., Phanishayee, A., Seshadri, V., Devanur, N., Ganger, G., and Gibbons, P. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint 18Xiv:1806.03377, 2018.
7. Conclusion
In this work, we present PipeOffload, novel pipeline sched- ule that incorporates multiple innovative techniques to sig- nificantly decrease the activation memory requirements of PP. Through evaluation, we demonstrate that PO-H can reduce activation memory to less than a quarter of that re- quired by interleaved 1FIB schedules while maintaining similar throughput across a wide range of real-world mod- els. PO-F proves particularly impactful for larger models or extended sequence training tasks, where activation memory can be further reduced to that of a small constant number of transformer layers. These methods greatly improved the
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
Kim, T., Kim, H., Yu, G.-I,, and Chun, B.-G. Bpipe: Memory-balancee pipeline parallelism for training large language models. International Conference on Ma- chine Learning, pp. 16639-16653, PMLR, 2023.

---

PipcOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
Korthikanti, V. A., Casper, J., Lym, S., McAfee, L.,, Ander- sch, M., Shoeybi, M., and Catanzaro, B. Reducing activa- tion recomputation in large transformer models. Proceed- ings of Machine Learning and Systems, 5, 2023.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper,
J., and Catanzaro, B. Megatron-Im: Training multi- billion parameter language models using model paral- lelism. arXiv preprint arXiv:1909 08053, 2019.
Li, Z., Zhuang. S., Guo, S., Zhuo, D., Zhang, H., Song, D., and Stoica, I. Terapipe: Token-level pipeline parallelism for training large-scale language models. In Interna- tional Conference on Machine Learning, pp. 6543-6552. PMLR, 2021.
Sun, A., Zhao, W., Han, X., Yang, C.,, Zhang, X., Liu, Z., Shi, C., and Sun, M. Seq Efficient sequence-level pipeline parallelism for large language model training. arXiv preprint arXiv:2406. 0222244000348888 2024.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19451 2024.
Yeung, M. T., Qi, P. Lin, M., and Wan, X. Balancing pipeline parallelism with vocabulary parallelism. arXiv preprint arXiv:2411.0228 2024.
Liu, Z., Cheng, S., Zhou, H., and You, Y. Hanayo: Harnessing wave-like pipeline parallelism for en- hanced large model training efficiency. The Inter- national Conference for High Performance Comput- ing, Networking, Storage, and Analysis, pp. 1-13, 2023. URL https: stmp/pppsssmmiicscchl....... (t:CCoppuII::1133300000
Yuan, T., Liu, Y., Ye, X., Zhang, S., Tan, J., Chen, B., Song, C., and Zhang, Accelerating the training of large lan- guage models using efficient activation rematerialization and optimal hybrid parallelism. In 2024 USENIX Annual Technical Conference (USENIX ATC 24), pp. 545-561, 2024.
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Pat- wary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. Efficient large- scale language model training on gpu clusters using megatron- Im. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-15, 1-15,2021. 2021.
Qi, P., Wan, X., Huang, G., and Lin, M. Zero bubble pipeline parallelism. In The Twelfth International Conference on Learning Representations, 2023.
Qi. P., Wan, X., Amar, N., and Lin, M. Pipeline par- allelism with controllable memory. arXiv preprint arXiv:2405. 15362, 2024.
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Per- formance Computing, Nenworking, Storage and Analysis, pp. 1-16. IEBE, 2020.
Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., and He, Zero- ieroniiffiitt: Breaking the gpu memory wall for extreme scale deep learning. In Proceedings of the international conference for high performance computing, networking, storage and analysis, pp. 1-14, 2021.
Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang. S., Zhang, M., Li, D., and He, Y. [Zero-offload]: Democratizing (billion-scale) model training. In 2021 USENIX Anmual Technical Conference (USENIX ATC 1). pp. 551-564, 2021.
10

---

PipcOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
A. 1F1B schedule with Fully Activation Offload
PPoekk
PPank
Barkworrd
Offrced
Relrad
Figure 14. Schedule of applying fully activation offload to IFIB. Note that in this schedule we also use the topology-aware offload which synchronizes offloading and reloading adjacent devices.
B. More Details on Memory and Throughput Comparison between Different Methods
In Figure 15, we show more details and numbers the experiment introduced in 5.1
C. Implementation
In this section, we show more implementation details on how we reduce the offload bound and save memory on both host and devices.
Continuous Host Buffer Bins We notice that pytorch round up the size of host memory to nearest power of 2, resulting in at most Ix waste of host memory. To mitigate this for each offloading pass, we use bigger continuous buffer bins with sizes which are power 2. We run a heuristic-basee search to find a solution that a) All activation memory of a single offloading pass can these bins b) At most 3 bins are used e) The total size of the bins is minimum. In practice we find this method reduces the waste of host memory negligible amount. During reloading we move the continuous buffers directly to device and construct individual activation memory tensors on them.
Deterministic Device Memory Management We allocate all offloddrrelated buffers on the compute stream and use CUDA events to synchronize the usage of buffers between the offload stream and compute stream. This method circumvents the issue of non-deterministic buffer deallocation times that could prompt frequent cudaMalloes if one were follow more simplistic approach using Tensor.record_streel
Selective Recomputation with Negligible Overhead Using methods similar to (Chen al., 2016), we simply recompute LayerNorm and GeLU layers in the backward pass to reduce the activation memory that is subject to offload. We also implement customized dropout that preserves the random seed during the forward pass and reconstructs the dropout mask in the hackward pass. Recomputing these layers reduces the activation memory for single layer from 34bsh (as (Korthikanti et al., 2023)) to 206sh. Our ablation study results demonstrate a 40% reduction activation memory with negligible throughput overhead (around 1-2% performance impact). We conduct an ablation experiment as in Figure 16
Topology-awand Offflodd Scheduling achieve optimal performance, we implement a synchronized interleaved memory transfer schedule where two devices within the same NUMA node collaboratively execute alternating H2D and D2H transfers using cross-device event synchronization. The scheduling of memory transfer will be studied in Appendix D. Our approach carefully co-designs computation and memory transfer schedules ensure transfer stability and efficiency while maintaining interdependencies.
Enhanced Node Assignment In many parallel processing (PP) schedules, there exists imbalance in activation memory across stages, leading to a linear decrease in activation memory from the initial to the final rank. This imbalance places varying loads on the host memory capacity of different nodes. To optimize host memory utilization, we consolidate the PP stages by grouping lower-ranked stages with higher-ranked stages on the same node. For instance, in a PP setup with 16 devices, ranks 0-3 and 12-16 are assigned to node 0, while ranks 4-11 are allocated to node 1. is important to note that this approach results doubling of tross-node communications.

---

PipcOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
on Model
on 18 Model
k on 42.98 Model GPUs
8 GPUs
16 GPUs
Socuerie Length
4096
1.4
1.4
1.4
1.4
1.4
1.4
4096-
1.2
1.2
1.2
12
1.2
1.2
4096
0.9
0.9
0.9
0.9
0.3
0.9
8192
0.8
Sesuercee
16384
1.0
1.0
1.0
1.0
16384
0.9
0.9
0.9
18384
0.7
0.7
0.7
32768
0.7
0.7
0.7
0.7
0.7
0.7
0.7
0.7
0.7
0.7
32768
0.6
0.6
0.6
0.6
371M
1F1B
MFU (%6) on 10.58 Model
MFU (%) en 18.18 Model
MFU (%) an 42. .8 Model 32 GPUs
8 GPUs
16 GPUs
Sequence Longah
4096
68%
65%
6.3%
SKIP
57%
Scquence Lengah
4006
6.%
6.3%
50%
SKIP
58%
4096
6.5%
6-%%
64%
62%
5.%
55%
Sequence
16384
0OM
M00
68%
8.3%
59%
62%
16384
00M
00M
59%
64%
01.6
D0M
18384
00M
00M
00M
63%
65%
OOM
32768
00M
64%
GOM
00M
63%
DOM
00M
00M
0.0M
COMY0
6194
177
17bb:
015-41
1F3B
Activation Memary on 10.58 Model
Activation Memory on 18.18 Model
Activation Mem ary an 42.98 Model
8 GPUs
16 GPUs
32 GPUS
Sequarce Langes
4096
15
13
SKIP
12
Sequeric Lorgth
4006
24
19
13
4
SKIP
18
4006
49
34
25
10
33
8192
00M
Sequarce
16384
DOM
00M
32
12
9
48
16384
DCM
COM
51
15
9
DOM
16384
DCM
00M
DOM
41
COM
32768
GOM
14
COM
32768
OOM
32768
00M
00M
D0M
OOM
G5
G6-H
PC
PO-F-
1770
171
005-11
po
No-F
1P18
1F18
171
Figure 15. Detailed Data of Throughput and Memory Comparison of Different Methods. SKIP indicates the PO-F method is skipped
because > 1. While OOM indicates GPU out-of-memory error, OOM(H) represents an OOM on host, which only happens on the largest model with largest sequence length.
Reduced Activation Memory (%)
Slowdown(%)
2.0
Baseline
Baseline
4096
43.4%
43.6%
43.8%
00M
4096
2.2%
2.0%
1.9%
00M
Model Dimension
5120
43.2%
43.6%
43.7%
Baseline
00M
43
5120
1.7%
1.5%
1.6%
Baseline
00M
1.5
Baseline
Baseline
6144
42.4%
42.9%
43.0%
Baseline
00M
42
6144
1.6%
1.6%
1.1%
Baseline
00M
1.0
7168
42.7%
43.1%
43.1%
00M
7168
1.6%
1.6%
1.8%
00M
Baseline
Baseline
8192
40.1%
40.8%
40.8%
00M
41
8192
0.8%
1.1%
0.6%
00M
0.5
Baseline
Baseline
10240
40.4%
40.9%
40.9%
0OM
10240
0.8%
0.9%
1.0%
00M
40
0.0
4096
8192
16384
32768
4096
8192
16384
32768
Sequence Length
Sequence Length
Figure 16. Ablation study on applying recomputation for Layernorm, GeLU and dropout layers. Approximately 40% activation is reduced with around 1-2% slowdown.
12

---

PipcOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
D. Topology-aware Offload Scheduling
G
GPU 4
PCI-E Switch
PCI-E Switch
GPU
GPU 5
CPU
CPU
GPU 2
GPU 6
PCI-E Switch
PCI-E Switch
GPU 3
GPU 7
Figure 17. GPU server topology of our A100 GPU server. Adjcent GPUs share the same PCI-E switch hence potentially interfere with each other.
Designing the offload schedule based the GPU server's topology is crucial. For instance, in testing of the A100 server, illustrated in Figure 17, observed that Host-to-Deviee (H2D) or Device-to-Host (D2H) transfers between GPU pairs on the same PCI-E switch could potentially interfere with each other. Therefore, coordinating the offload operations among GPU pairs on the same PCI-E switches becomes critical consideration.
In our investigation, we assessed the throughput and stability of these transfers using various co-scheduling methods. We organized H2D and D2H operations into sequential groups, with operations within each group executing consecutively. To mimic real-world scenarios where two devices sharing the same PCI-E switch might trigger transfers at different times during training iterations, introduced random inter-group delays. Each experiment repeated multiple times, and we generated the distribution of per-device bidirectional throughput in Table 3.
It's worth noting that while our experimental results favor the synchronized interleaved method, server topologies can differ. Therefore, in practice, it's essential to select co-scheduling methods that are most compatible with a specific hardware configuration.
E. Detailed Data on Activation Memory Scaling Study
In Section 5.3 and Figure 10, we've shown how activation memory scales with d. In this section show detailed data on how the activation memory scale with u and d along, shown in plot in Figure 18
13

---

PipcOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
Table Comparing co-scheduling methods for offload operations on a pair of GPUs connected to the same PCI-E switch. Results shows that the syncronized interleaved schedule is both stable and fast.
Schedule
Sketch
Bandwidth Histogram
Parallel Both devices under
NUMA node execute alter- nating H2D and D2H trans- fers parallelly.
Device
Interleaved Similar to par- allel H2D-D2H, but devices under each NUMA node be- gin with complementary op-
Devke
10
15
20
25
erations.
Dual Stream Both devices simultaneously execute H2D
Denkcr
and D2H operations using separate CUDA streams.
Device
Synchronized Parallel Par- allel H2D-D2H with cross- device synchronization Desicn
Device
before each memory transfer.
10
15
20
Synchronized Interleaved Interleaved H2D-D2H with cross-device event synchro- nization before each memory transfer.
Desice
15
Synchronized Dual Stream
Dual-Stream H2D-D2H with cross- dross-device event synchro- nization before each memory transfer.
Device
10
15
20
Number of In-Fiight ictivations Activation Memory (GB)
Memory (GB)
Number of in- Flight Activattons Activatton Activaaionsttiiitti Memory (GB)
Activation Memory (GB)
d=4
d=16
d=4
d=16
20-
20
Stages per Device (v)
Stages per Device (v)
Number of In-flight Activations of PO-H
Number of In-flight Activations of PO-F
d=2
d=8
d=4
- d-16
d=2
d-8
Stages per Device (v)
Stages per Device (v)
Figure 18, Scaling of activation memory under different v and
14


Reviews:
[{"review_id": "KFq4yiEE98", "reviewer": "Reviewer_s5WZ", "summary": "In this work, the authors introduced PipeOffload, a pipeline parallelism strategy that offloads activations with negligible overhead. The evaluation results demonstrate that PipeOffload uses less memory than the 1F1B method, enabling LLM training with larger model sizes and longer sequence lengths.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedbacks. Please see individual points below.\n> Including learning curves could enhance the understanding of convergence and correctness.\n\nWe checked that our methods produces identical loss curves compared to vanilla 1F1B-I. We'll include this in the PDF.\n[Convergence Graph](https://i.imgur.com/bVgxTQb.png)\n\n\n> Incorporating comparisons with Chimera and DualPipe\n\nOur primary contribution is memory savings with minimal overhead. Since DualPipe, Chimera, and 1F1B share identical peak memory consumption, comparing against 1F1B and 1F1B-I sufficiently supports our claim.\n\nAdditionally, DualPipe's advantage lies in expert-parallel (EP) pipeline scheduling, where it fuses forward and backward passes to overlap EP communication with computation. As our work does not focus on EP integration, a comparison with DualPipe with EP is beyond our scope.\n\nMoreover, we have added experiments comparing our approach with state-of-the-art memory-saving methods, including the V-Shape PP schedules from [2] and 1F1B-I with rematerialization (recompute). The results, shown in the image below, demonstrate that our methods push the Pareto frontier of the throughput-memory trade-off.\n\n[Pareto Frontier Figure](https://i.imgur.com/GiQXeSB.png)\n\nIn this figure, 1F1B-I-R represents Interleaved 1F1B with full rematerialization, while V-Min and V-Half are from [2].\n\n\n> It is unclear how bandwidth affects the performance of the proposed method. What is the minimum memory bandwidth required to hide the offload overhead? Is this method applicable on systems with only PCIe?\n\n\nCommon hardware, including PCIe-only systems, already provides sufficient bandwidth for activation offloading. Our method remains effective under these conditions.  \n\nWe extensively analyzed this in the introduction section. Formula 1 and Figure 1 demonstrate that typical hardware can offload a large portion (at least half, possibly all) of activations. In Formula 1, $k$ describes the ratio between offload time and compute time, and we see an important term $B_o$ which is the offload bandwidth. We plotted practical $k$ values based on A100 GPUs with PCIe v4 on Figure 1 (left) and have shown that a $k$ is actually small. Section 4.3 further discusses $k$ across different hardware, confirming similar trends.  \n\nNotably this only relies on the host-device PCIe bandwidth, so it also works on systems with only PCIe.\n\nTo clarify this emphasis on bandwidth, we will update the PDF to highlight the role of Formula 1 and Figure 1.  \n\n\n> it is unclear in what case the proposed pipeline parallelism method can outperform tensor parallelism.\n> Discussion on being integrated with other parallelism strategies\n\nThanks for raising this important discussion on TP vs. PP. In short, PP should be preferred when memory allows, as it generally provides higher throughput.\n\nOur results in Figure 11 align with prior work [1][2], showing PP outperforms TP due to TP’s high communication overhead. However, when memory is limited, TP can still help reduce activation memory. This highlights our contribution—reducing PP memory usage so that less TP is needed.\n\nOn implementation, PipeOffload is orthogonal to other parallelization strategies and can be integrated into hybrid parallelism.\n\n\nA summary of a detailed analysis of TP vs PP that we'll add to the PDF:\n\n- Activation Memory: When using $d$ devices, For TP, each GPU has approximately $\\frac{1}{d}$ of the activation memory of the entire model. For our PipeOffload, with selective offloading each device consumes approximately $\\frac{1}{4}$ for a selective offload and $O(\\frac{1}{vd})$ for a fully offload. Considering in practical scenarios $d\\le8$ for TP, PipeOffload has a similar, or even situationally (when $k\\lt1$, see Figure 1 in our paper) lower per-device activation memory compared to TP.\n- Throughput: Prior studies [2][3] show that TP with 8 devices incurs a 20–30% throughput overhead. For PipeOffload, the bubble rate is $\\lt\\frac{d-1}{n+d-1}$, where n is the number of microbatches. In most practical cases, the bubble rate of PipeOffload is lower than the overhead of TP.\n\n\n> Other methods like 1F1B-I may achieve better MFU if sufficient memory is available.\n\nWe agree that when memory is sufficient, throughput should be prioritized. However, in many practical cases, \"sufficient memory\" is only achieved by using TP, which incurs significant throughput overhead, or by applying activation rematerialization. Our work provides an effective alternative for acceleration in such scenarios.\n\nReferences\n\n[1] BPipe: Memory-Balanced Pipeline Parallelism for Training Large Language Models\n\n[2] Pipeline Parallelism with Controllable Memory\n\n[3] Megatron-LM: Training Multi-Billion Parameter Language Models using GPU Model Parallelism.", "claims_and_evidence": "The evidence provided is generally sufficient to support the claims.", "methods_and_evaluation_criteria": "The chosen baselines and datasets are generally appropriate for the study.", "theoretical_claims": "The theoretical claims appear to be correct.", "experimental_designs_or_analyses": "The evaluation is generally satisfactory. However, including learning curves could enhance the understanding of convergence and correctness.", "relation_to_broader_scientific_literature": "This work introduces a novel pipeline parallelism approach that reduces memory usage without additional overheads. This is significant for modern LLM training and could impact parallelization strategy design.", "essential_references_not_discussed": "The provided references are appropriate and sufficient. However, incorporating comparisons with recent pipeline parallelism approaches, such as Chimera and DualPipe, could further strengthen the work and enhance its impact.", "other_strengths_and_weaknesses": "1. It is unclear how bandwidth affects the performance of the proposed method. What is the minimum memory bandwidth required to hide the offload overhead? Is this method applicable on systems with only PCIe? \n\n2. Additionally, it is unclear in what case the proposed pipeline parallelism method can outperform tensor parallelism. \n\n3. In modern LLM training, hybrid parallelism is commonly employed, combining data parallelism, tensor parallelism, pipeline parallelism, and sequence parallelism. Since using pipeline parallelism alone is uncommon, further discussion may be needed on how this method can be integrated with other parallelism strategies to ensure efficient scalability and performance.", "other_comments_or_suggestions": "The proposed method seems more competitive for training larger models or longer sequences with limited memory. Other methods like 1F1B-I may achieve better MFU if sufficient memory is available.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ArOY93JRpD", "reviewer": "Reviewer_bsbd", "summary": "This paper proposes a method for optimizing large language model (LLM) training by leveraging pipeline parallelism (PP) focusing on selectively offloading activations. It identifies the opportunity to overlap computing and data transfer during the forward and backward passes, and proposes selective offloading based on layer lifespan. The authors compare several schedules (1F1B, 1F1B-I, GIS, GIS-H, PO-H, and PO-F) analyzing memory/throughput trade-offs. The experiments demonstrate improvements in activation memory and Model FLOPS Utilization (MFU) compared to other PP methods, or 12–19% training speed-up compared to tensor parallelism (TP) when using the Megatron-LM model and scaling across 2–32 A100 80G GPUs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments. We respond to individual points from your review below.\n\n> The paper lacks clarity regarding its novel contributions ... the distinction between established and novel concepts, such as Selective Offload and Zero-Bubble Strategy, is not adequately delineated ...\n> The design section could benefit from clarifying the originality ...\n\nOur main contributions are:\n* For the first time, we identify the oppotunity that all activation memory of PP can be offloaded with negligible overhead, which is critical but overlooked for the scalability of PP.\n* We propose a novel selective offload strategy in PP, which can achieve better-than-linear memory reduction. To the best of our knowledge, this is the first selective strategy for offload.\n* We extend the popular interleaved 1F1B into a generalized form for the first time, which can smoothly trade memory with minimal throughput loss, providing a flexible solution that can be tailored to specific system requirements.\n* We provide implementation details in Section 4 and Appendix, which is critical to achieve high performance for offload in PP.\n\nAs referenced in the paper, Zero-Bubble was proposed by [1]. We apply this strategy as a preliminary step in our generalized interleaving strategy (Section 3.2). We don't claim it as our contribution, as it's trivial.\n\n> It would be helpful to address the fundamental differences against the existing work [2]\n\nAlthough we are inspired by some concepts (building-block, lifespan) from [2], our work has fundamentally different contributions and insights compared to [2]. Specifically, we focus on selective offloading (Section 2), designing a novel scheduling methodology to facilitate offloading (Section 3), and implementing optimized offload techniques (Section 4). These contributions are centered on offloading, whereas [2] aims to reduce activation memory through schedule-based methods, and does not mention offloading at all. \n\n> Is GIS(-H) an existing method or an entirely new design? ...\n\nThe generalized interleaving strategy (Section 3.2) is entirely new. Both GIS and GIS-H can be seen as special cases of tuning $g$ to $g=d$ or $g=\\frac{d}{2}$. While GIS happens to resemble Figure 17\\(c\\) in [2], it is not identical. To the best of our knowledge, GIS-H ($g=\\frac{d}{2}$) and all GIS schedules with $g \\lt d$ are entirely novel.\n\n> acceleration factor is only measured against TP instead of other PP methods.\n> what is the acceleration of PipeOffload compared to SOTA PP methods (rather than TP ones)?\n> Given PP being a research hotspot, expanding the comparison baselines to include methods like BPipe, GPipe, and PipeFill ... if any of them is open-source.\n\nOur primary focus is memory reduction to expand the applicability of pipeline parallelism (PP). Like all memory-saving methods, ours does not improve PP throughput; rather, it trades throughput for memory savings. This makes comparisons with tensor parallelism (TP) relevant, as TP is often preferred due to PP’s high memory requirements.\n\nBPipe and [2] are important works sharing the same target of memory saving with us, while only [2] is open-source. We conducted experiments comparing our approach with [2] and 1F1B-I with rematerialization which are the SoTA memory saving PP schedule. The results, shown in the image below, demonstrate that our methods (PipeOffload and GIS schedules) push the Pareto frontier of memory and throughput.\n\n[Pareto Frontier Figure](https://i.imgur.com/GiQXeSB.png)\n\nIn this figure, 1F1B-I-R represents Interleaved 1F1B with full rematerialization, while V-Min and V-Half are from [2].\n\n> Regarding Figure 2, what is the cause of the asymmetry between Stages 0 and 1?\n\nAre you asking why the trends of Stage 0 and Stage 1 are reversed? Intuitively, after merging both stages, the final schedule follows a one-forward-one-backward pattern, ensuring a constant total activation memory across the two stages. As a result, when the memory usage of one stage increases, the other decreases.\n\n> Regarding Figure 3, how is the curve modeled and calculated? What does the kink near x=13 mean?\n\nWe measure the percentage of memory reduction at $x \\in \\{0, 1, \\ldots, 16\\}$ and connect these points to form a curve. The kink near $x=13$ is simply because the evaluation points are discrete ($x$ must be an integer). \n\n> ... It would be advantageous to also cross-validate certain equations, for example, memory and bubble amount ...\n\nAs mentioned in the caption of Figure 8, we provide detailed numbers in Figure 13.\n\n>It would be beneficial to explain why Megatron-LM is selected as the evaluation target if that is not common knowledge.\n\nWe believe it's a common knowledge that Megatron-LM is SOTA and a common choice.\n\n[1] Qi, P., Wan, X., Huang, G., and Lin, M. Zero bubble pipeline parallelism.\n\n[2] Qi, P., Wan, X., Amar, N., and Lin, M. Pipeline parallelism with controllable memory.", "claims_and_evidence": "I do not notice any issue regarding unsupported claims. For example, the claim about selective offload is well supported by both theoretical prediction and experimental measurements.", "methods_and_evaluation_criteria": "It would be helpful to further improve the evaluation by considering these issues:\n1. The acceleration factor is only measured against tensor parallelism methods instead of other pipeline parallelism methods.\n2. Given pipeline parallelism being a research hotspot, expanding the comparison baselines to include methods like BPipe, GPipe, and PipeFill could provide valuable benchmark comparisons and strengthen the paper's claims if any of them is open-source.", "theoretical_claims": "The equations presented in Section 3 appear reasonable to me.\nIt would be advantageous to also cross-validate certain equations, for example, memory and bubble amount, against experimental measurements.", "experimental_designs_or_analyses": "I reviewed the experimental designs and analyses and, while I did not identify any immediately apparent issues, I am not fully confident whether they are sound. Please find my questions below.", "relation_to_broader_scientific_literature": "The paper lacks clarity regarding its novel contributions relative to existing research. Specifically, if this paper includes a concise listing of key contributions, I would be able to determine its novelty more precisely.\nFurthermore, the distinction between established and novel concepts, such as Selective Offload and Zero-Bubble Strategy, is not adequately delineated. The paper should clearly identify which aspects are pre-existing and which are original, and elaborate on their differences.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "+ The paper includes vibrant figures to help understanding.\n- The design section could benefit from clarifying the originality of each optimization technique discussed.", "other_comments_or_suggestions": "1. Introducing abbreviations (e.g., 1F1B, GIS, ZB-H2) before their first usage would enhance readability.\n2. It would be beneficial to explain why Megatron-LM is selected as the evaluation target if that is not common knowledge.\n3. Minor typo: Line 327, “similar to.”", "questions_for_authors": "Thanks for submitting to ICML’25. This is solid work and I generally enjoyed my reading. My overall rating of this paper is somewhat neutral. It would be extremely helpful if the authors could properly address these questions:\n\nNovelty:\n\n1. It would be helpful to address the fundamental differences against the existing work “Pipeline Parallelism with Controllable Memory. Qi et al, 2024.”\n2. Is GIS(-H) a realization of an existing method or an entirely new design? If existing, outlining the differences between PipeOffload and GIS(-H) would be helpful. If new, adding one more state-of-the-art baseline in the evaluation might be worth considering.\n\nEvaluation:\n\n3. Regarding the main evaluation, what is the acceleration of PipeOffload compared to state-of-the-art *pipeline* parallelism methods (rather than tensor parallelism ones)?\n\nTechnical details:\n\n4. Regarding Figure 2, what is the cause of the asymmetry between Stages 0 and 1?\n5. Regarding Figure 3, how is the curve modeled and calculated? What does the kink near x=13 mean?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "2e4sbTWmL1", "reviewer": "Reviewer_UJJy", "summary": "This paper proposes PipeOffload, a novel selective offload strategy to decrease peak activation memory in pipeline parallelism training. PipeOffload greatly improves the scalability of pipeline parallelism, and makes pipeline parallelism a stronger alternative than tensor parallelism.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedbacks. See response to individual points below\n\n> The selective offload strategy introduces additional complexity in managing activation lifespans, which could complicate implementation and maintenance.\n\nEvery new feature requires some implementation, however, we try our best to make implementation simple. The offloading majorly leverages the existing PyTorch ```Tensor.to()``` function to do D2H and H2D transfers. The scheduling code handles offloading as PP native passes and uses simple heuristics to determine the ordering between them.\n\n> The effectiveness of the proposed strategies may vary across different hardware configurations, potentially limiting their general applicability.\n\nWe agree that the offload time ratio $k$ is dependent on hardware, particularly the host-to-device bandwidth. Our implementation allows for adjustable scheduling to accommodate different hardware configurations.\n\nAdditionally, mainstream GPUs are rapidly improving in PCIe bandwidth (e.g., A100: 64 GB/s, H100: 128 GB/s, B100: 256 GB/s, Grace Hopper: 900 GB/s). Given this trend, we expect our method to remain generalizable to future devices.\n\n> How seamlessly can PipeOffload be integrated into existing machine learning frameworks that utilize 3D parallelism\n\nOur implementation is based on Megatron-LM, one of the most popular LLM training frameworks. PipeOffload is orthogonal to other parallelism schemes and requires minimal additional effort when integrated into a hybrid parallelism strategy.\n\n> is it feasible to apply PipeOffload in conjunction with 1F1B-I to mitigate this issue?\n\nYes, the same method can be applied to 1F1B-I. However, as discussed in Section 2 of our paper, selective offloading in 1F1B-I only achieves linear memory savings. This setup is still preferable when higher throughput is the priority.\n\n> in Figure 7 of your paper, how did you determine the durations for offloading/reloading and forward/backward computations? Could you elaborate on the methodology used to decide these lengths?\n\nIn our current implementation, for simplicity, we use the ratio $k$ between offload time and compute time, as calculated by Formula 1, while assuming F/B/W passes run for identical time. Experiments show that this approach works well. However, we anticipate that using more accurately profiled timings for compute and offload passes will lead to even better performance.\n\n> Can you elaborate on the memory imbalance across different stages? Does PipeOffload need to offload/reload the same amount of activations for different stages?\n\nCould you clarify whether this imbalance refers to the different activation memory for different stages on the same device, or different stages on different devices? If it's about different stages on the same device, the imbalance arises from the varying lifespans of activations across stages. For example, in the forward pass, stage 0 computes before stage 1, while the backward pass is computed after stage 1. This results in stage 0’s activations living longer than those of stage 1, contributing to higher peak memory for stage 0.\nIf multiple stages are offloaded, the number of activations to be offloaded will be the same across stages, but the peak memory reduction will differ, with stage 0 providing a greater reduction than stage 1.\n\n> How does PipeOffload's performance compare to that of bidirectional pipeline parallelism? Is it possible to combine PipeOffload with bidirectional pipeline parallelism strategies?\n\nThe same insight of selective offloading can also be applied to bidirectional PP, as in bidirectional PP there're also multiple stages on the same device.\n\n> How to decide the offload schedule for different topologies?\n\nThe topology influences how offload/reload passes are co-scheduled across different devices. For instance, if two GPUs are connected to the same PCIe switch, we schedule them in a \"Synchronized Interleaved\" pattern, as described in Table 3. If the GPUs have independent PCIe connections, we schedule offload/reload passes independently on each device.", "claims_and_evidence": "Claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Methods and evaluation make sense.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "The experimental design is sound.", "relation_to_broader_scientific_literature": "This paper helps improve the LLM training efficiency with pipeline parallelism.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n1. The paper is well-written and clearly motivated.\n2. Address an important problem.\n3. The paper presents a novel approach to reduce peak memory consumption in PP and achieves good performance.\n\nWeaknesses\n1. The selective offload strategy introduces additional complexity in managing activation lifespans, which could complicate implementation and maintenance.\n2. The effectiveness of the proposed strategies may vary across different hardware configurations, potentially limiting their general applicability.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. How seamlessly can PipeOffload be integrated into existing machine learning frameworks that utilize 3D parallelism (combining data, tensor, and pipeline parallelism), and what challenges might practitioners face during such integration?\n2. Given that PipeOffload introduces a higher bubble overhead compared to the 1F1B-I (One-Forward-One-Backward Interleaved) scheduling strategy, is it feasible to apply PipeOffload in conjunction with 1F1B-I to mitigate this issue?\n3. In Figure 7 of your paper, how did you determine the durations for offloading/reloading and forward/backward computations? Could you elaborate on the methodology used to decide these lengths?\n4. Can you elaborate on the memory imbalance across different stages? Does PipeOffload need to offload/reload the same amount of activations for different stages? \n5. How does PipeOffload's performance compare to that of bidirectional pipeline parallelism? Is it possible to combine PipeOffload with bidirectional pipeline parallelism strategies?\n6. How to decide the offload schedule for different topologies?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "08rAUbrrus", "reviewer": "Reviewer_7WYn", "summary": "In this paper, the authors present the memory offload strategies in pipeline parallel training. The authors suggest that the variables with long lifespan should be offload, and then introduce a selective offload strategy that decreases peak activation memory in a better-than-linear manner. Moreover, that offload strategy can jointly considered with other techniques. Experiments illustrate the offload strategy can introduce 19% acceleration with lower memory consumption.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedbacks. See response to individual points below.\n\n> Whether the proposed strategy is suitable for other pipeline parallel strategy like DualPipe? The authors can present some analysis theoretically and empirically.\n\nTL;DR: Yes, but different devices will have different memory deduction.\n\nThe selective offload strategy is applicable to bi-directional schedules like Chimera and DualPipe, as they involve multiple stages with different directions on the same devices. In these discussions, we refer to the stage on device $i$ with a forward pass from device $i$ to device $i+1$ as stage 0, and the reverse stage as stage 1.\n\nIn DualPipe, stages on the same device have varying lifespans. For instance, on device 0, stage 0 has a longer lifespan, while stage 1 has a negligible one. This creates an opportunity for better-than-linear memory reduction, especially since DualPipe uses a \"Uniform Repeat\" strategy, as mentioned in Section 2.\n\nHowever, the lifespan distribution varies across devices. For example, stage $\\frac{d}{2}$ has nearly equal lifespans for both stages, and selective offloading leads to linear memory reduction. In a rough analysis, offloading one stage on devices 0 and $d-1$ results in approximately 100% activation reduction, while on device $\\frac{d}{2}$, it results in about 50%. The memory reduction for other devices can be linearly interpolated.\n\nFinally, we emphasize that DualPipe's main advantage is in expert-parallel (EP) pipeline scheduling, where it fuses forward and backward passes to overlap EP communication with computation.\n\n> Whether the authors can provide the core codebase of the experiments? It may make the experiments reproducible.\n\nWe are going to open-source the code base for our implementation and experiments for a broader impact in the community. We did not upload it as supplementary material because we were not confident in anonymizing the code sufficiently to comply with double-blind review rules.", "claims_and_evidence": "To the reviewer's best knowledge, the claims and evidence are all supported by clear evidence. There is not any fatal mistake or overclaim in this paper.", "methods_and_evaluation_criteria": "The proposed method is effective in the pipeline parallel training according to the development of the method and the experimental results. It makes sense for the pipeline parallel training.", "theoretical_claims": "The reviewer has taken a quick check on the theoretical claims. There is not any fatal issue.", "experimental_designs_or_analyses": "The experiments can validate the benefits of the proposed offload strategies as the proposed method can achieve a better performance with the model size varying from 5.8B to 83.8B.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "**Strength**\n1. The writing is excellent. It is easy for readers who has little basic knowledge in pipeline parallelism to understand the manuscript.\n2. The idea is simple but effective. It is easy to follow.\n3. The authors take a thorough experiment under model with model size varying from 5.8B and 83.8B. And the proposed method can achieve a better performance.\n\n**Weakness**\n1. The offload strategy is only based on 1F1B strategy. It is unknown that how the proposed strategy in other pipeline parallel strategy like DualPipe [1].\n2. The authors fail to provide the code of the proposed method, which may decrease the reliability of the experimental results.\n\n[1] DeepSeek-V3 Technical Report", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. Whether the proposed strategy is suitable for other pipeline parallel strategy like DualPipe? The authors can present some analysis theoretically and empirically.\n2. Whether the authors can provide the core codebase of the experiments? It may make the experiments reproducible.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new pipeline parallelism approach for efficient training. Concretely, it proposes to selectively offload activations to the host such that the latency from data movement can be effectively hidden. On a 32 A100 node setup, this approach makes pipeline paralleism an attractive alternative to tensor parallelism across various model sizes.

This is a solid paper that presents a sensible method backed up by empirical experiments, and thus I am recommending that this paper be accepted. (It would have been nice to consider training setups with even a greater number of GPUs, though I acknowledge that this may not be feasible on an academic budget.)
Decision: Accept (poster)