Group-based Interleaved Pipeline Parallelism for Large-scale DNN Training
Authors: PengCheng Yang
Xiaoming Zhang
Wenpeng Zhang
Ming Yang
Hong Wei
Venue: ICLR 2022 Poster (2022)
Keywords: Model parallelism
Pipeline parallelism
Distributed training

Abstract: The recent trend of using large-scale deep neural networks (DNN) to boost performance has propelled the development of the parallel pipelining technique for efficient DNN training, which has resulted in the development of several prominent pipelines such as GPipe, PipeDream, and PipeDream-2BW. However, the current leading pipeline PipeDream-2BW still suffers from two major drawbacks, i.e., the excessive memory redundancy and the delayed weight updates across all stages. In this work, we propose a novel pipeline named WPipe, which achieves better memory efficiency and fresher weight updates. WPipe uses a novel pipelining scheme that divides model partitions into two groups. It moves the forward pass of the next period of weight updates to the front of the backward pass of the current period of weight updates in the first group, retains the order in the second group, and updates each group alternatively. This scheme can eliminate half of the delayed gradients and memory redundancy compared to PipeDream-2BW. The experiments, which train large BERT language models, show that compared to PipeDream-2BW, WPipe achieves $1.4\times$ acceleration and reduces the memory footprint by 36%, without nearly sacrificing any final model accuracy.

Published as a conference paper at ICLR 2022
GROUP-BASED INTERLEAVED PIPELINE PARALLELISM FOR LARGE-SCALE DNN TRAINING
Pengcheng Yang, Xiaoming Zhang, Wenpeng Zhang, Ming Yang, Hong Wei Ant Group, China
(rnggc615@gmmnn 1. com, xiaominglan, htt.ligglllaaaaaanaaaaagggggne zhang@antgrruup.com zhangMenpeng com, vincent. www.ntgrou.nncccooommmmmmm com com weihong36466hhtmmal com
ABSTRACT
The recent trend of using large-scale deep neural networks (DNN) to boost per- formance has propelled the development of the parallel pipelining technique for efficient DNN training, which has resulted in the development of several prominent pipelines such GPipe, PipeDream, and P However, the current leading pipeline PipeDream-2BW still suffers from two major drawbacks, i.e., the excessive memory redundancy and the delayed weight updates across all stages. In this work, we propose a novel pipeline named WPipe, which achieves better memory efficiency and fresher weight updates. WPipe uses novel pipelining scheme that divides model partitions into two groups It moves the forward pass of the next period of weight updates the front of the backward pass of the current period of weight updates in the first group, retains the order in the second group, and updates each group alternatively. This scheme can eliminate half of the delayed gradients and memory redundancy compared to PipeDream-2BW. The experiments, which train large BERT language models, show that compared to PipeDream - BWW, WPipe achieves 1.4x acceleration and reduces the memory footprint by 36%, without nearly sacrificing any final model accuracy.
INTRODUCTION
Several recent lines of research (Liu al., 2019 Yang et al. 2019 Lan et al. 2019 Raffel et al. 2019 Brown et al.) 2020 Lin et al.) (021) on various application domains have collectively demonstrated that larger DNN models can yield better performance. This creates an emerging trend in scaling up the number of model parameters, resulting in very large DNNs, the memory footprint of which will be beyond the limit single accelerator.
To resolve the aforementioned problem, researchers and practitioners have focused on model paral- lelism, which allows to further scale up model parameters significantly by partitioning a large model multiple accelerators/workee. inter-layer model parallelism (Narayanan et al., (202)) and intra-layer model parallelism (Shoeybi) al. 2019), either suffers from low resource utilization or high communication overhead. Recently, some studies proposed the pipeline-parallel technique to accelerate conventional model-parallel training. The pipelining technique can effectively improve the utilization of computing resources by scheduling different workers to consume different mini- batches in parallel. However, naive pipelining cannot be directly used for DNN training due to two problems: (1) staleness of weight updates (Harlap et al., 2018), which is caused by the inconsistency of the weight versions used in forward pass and its corresponding backward pass, and (2) excessive in-flight activations, which are heaped up by continuous forward activations waiting for their corresponding backward passes in pipelining.
However, the conventional model parallelism, which includes
To address the staleness problem, some workable pipelines have been proposed with different emphases on low memory footprint and high throughput. GPipe proposes the periodic pipeline flushing technique, which maintains only one weight version. This technique can achieve a low memory footprint and almost the same weight update semantics data parallelism. but at the cost of introducing bubble overhead (Huang et al., 2019), thus limiting throughput. PipeDream proposes the weights stashing (Harlap et al. 2018) technique eliminate bubble overhead, resulting in higher

---

Published as a conference paper at ICLR 2022
throughput; however, the multiple weight versions incur high memory footprint. To further reduce the weight versions maintained in PipeDream, PipeDream- 2BW was proposed in which the double- buffered weight update technique (Narayanan al. 2021) is used, which largely reduces the number of weight versions and enables similar weight update semantics as that in data parallelism.
To reduce in- in-flight activations, GPipe and PipeDream-2B8) adopt activation recomputation (Chen et al.] 20166] Jain et al. 2019). Recomputation enables the forward pass to execute without retaining the internal activations, which are recomputed in the corresponding backward pass. This method of being recomputed when needed can effectively alleviate the stacking up of in-flight activations and greatly reduce the memory footprint. Although both GPipe and PipeDream-22 recomputation technique, PipeDream- BB has fewer in flight activations than GPipe owing to the more timely execution of the backward pass in its pipeline. Regarding Pipedream, unfortunately. doesn't 't take effective measures on this issue, which prevents it from training larger model.
utilize the
Currently, PipeDream-22 BBW the pipeline-paree parallel system with the best overall performance. However, it still suffers from two main problems: delayed weight updates and excessive memory redundancy on weight versions and activations (Narayanan et al. 2021). In this paper, we propose a novel pipeline, named WPipe, to solve these two problems. In W WPipe, model partitions are divided into two groups, namely Go and G1, which share the same workers. In the execution pipeline, the forward pass of the next update period is moved to the front of the backward pass of the current update period in Go. and the execution order in G1 is retained, and then, Go and G are updated alternatively. We named the alternate update technique for the grouping weights double-grouper weight updates (2GW). Compared to 2BW, 2 GW has two main improvements: (I) 2GW realizes the same weight update semantics as that in data parallelism and only maintains one weight version without introducing bubble overhead in G1. (2) 2GW reduces half of the in-flight activations since only half of the weights are involved in the training at a time.
Additionally, we incorporate some conventional but effective communication optimization techniques including the overlap of computation and communication, hybrid parallelism, and heterogeneous communication to further reduce communication overhead. Finally, our throughput experiments show that WPipe achieves acceleration of 7.1 compared that of PipeDream and 2.1 x compared to that of GPipe for Bert192 with 1.4 billion parameters. When training models with up to 5.5 billion parameters, which cannot be trained by GPipe and PipeDream, WPipe is 1.4x faster than PipeDream 2BW. Memory experiments show that compared to PipeDream 2BW, WPipe reduces the memory footprint by 35.8% for Bert96. Convergence experiments show that WPipe can achieve similar final accuracy to data parallelism.
2
BACKGROUND
In this section, we briefly introduce the related techniques of model parallelism.
Model parallelism. Model parallelism (Chen et al. 2018 Chilimbi et al. 2014 Dean et al. 2012
Jia et al. 2018) partitions large model into multiple parts and assigns them different workers. Each worker is responsible for their own weight updates and sending and receiving intermediate activations and gradients. The conventional model parallelism includes inter-layer model parallelism and intra-layer model parallelism (Shoeybi et al. 2019). The former suffers from underutilized resources as only one worker is active at a time, as shown Figure [p. The latter requires all-to-all for each layer. The all-to-all communication
aggregation with intermediate octivations/gradienn overhead, which grows in proportion to the number of layers, makes difficult to expand to deeper layers, especially in heterogeneous network interconnections.
Forward Pass
Backward Pass
Idle
WorkerO
WorkerO
Worker1
Worker1
Time (a)
Time
(b)
Figure 1: The execution timelines of inter-layer model parallelism and naive pipeline parallelism.
2

---

Published as a conference paper at ICLR 2022
Pipeline parallelism effectively improve the utilization of computing resources of inter-layer model parallelism by scheduling different workers to consume different mini-batches simultaneously. However, naive pipeline parallelism suffers from staleness issue for weight updates. As shown in Figure Db. when the backward pass of the latter mini-batch is executed, its corresponding weights could be updated by the prior mini-batch. For example, for mini-batch the weights used in the backward pass were updated by mini-batch 1.
Pipeline Parallelism.
Existing solutions tackle the issue by trading off throughput and memory footprint. GPipe proposes the periodic pipeline flushing technique (Huang et al. 2019). As shown in Figure 2h. it splits mini- batch into multiple smaller pipelining. Within each pipeline period, GPipe can accumulate gradients across micro-batcher and flush synchronously. this way, it achieves the same weight update semantics as data parallelism and only maintains one weight version. However, in GPipe, bubble overhead (Huang et al. 2019) limits throughput, especially when large mini-batch cannot be supported. PipeDream-fluse essentially the same GPipe (Narayanan et al. 2021), and its main improvement is to move forward the backward pass, thereby releasing in-flight activations as soon as possible and reducing memory footprint.
which are injected into the model continuously to achieve
Forward Pass
Backward Pass
Update
WorkerO
Upddan
WorkerO
Worker1
Upstex
Worker1
W?
Time (a)
Time
(c)
Weight stashing
WorkerO
WorkerO
Worker1
Worker1
W:
Time
(b)
Time
(d)
2BW
Figure 2: Timelines of various dipeline-paralley executions: (a) GPipe updates the weights
mini-batch. (b) PipeDream-ffush moves backward pass forward compared to GPipe. (c) PipeDream implements weight stashing to update the weight immediately by default. (d) PipeDream-2F88 realizes periodic updates through gradient accumulation. For example, when the update period is 2, micro-batches and 4 update weights in the figure.
PipeDream uses the weight stashing technique to tackle the staleness issue without introducing the bubble overhead, and thus achieves higher throughput (Harlap et al. (018). However, it must maintain a large number of weight versions. From Figure 2b worker i(2 = 0, 1,...) comprises. N i weight versions, where N is the number of model partitions. This results in high memory footprint, especially when N is large. Moreover, in PieeDream, the weight update semantics has different delay terms different stages.
PipeDreemm- BB (Nrraynnan et al. 2021). as an upgraded version of PipeDream, has higher through- put and more memory efficiency. As shown in Figure 22. it uses double-buffered weight updates (2BW), which is combined with gradient accumulation, to reduce effectively the number of weight versions maintained each worker from N - 2. Notably, in 2BW, its weight update semantics has only one weight delay term at each stage (Narayanan et al. 2021)
3 SYSTEM DESIGN OF WPIPE
In this section, we first analyze the problems of 2BW. Then, we propose double-grouped weight updates (2GW) and analyze 2GW from three aspects: model partitions grouping, effective learning, and memory footprint. Finally, we introduce several communication optimization techniques.
3.1 INTRODUCTION
To simplify the description, we denote the naive model-paralled timeline as Vz (where "2" represents the number of model partitions), shown in Figure Ta. Then denote the process of cross-splicing V,s to form pipeline as P(nV2) (where n represents the number of V2), as shown in Figure Пp. Analyzing the execution regular of the pipeline in Figure Tb, can find that P(nV2) can ensure that
3

---

Published as a conference paper at ICLR 2022
Forward Pass
Backward Pass
Idle
Introduced Idle
Figure 3: Separate two adjacent update periods in PipeDream 2BW by introducing idle time blocks.
the backward pass is executed immediately after the forward pass is completed that the forward activations can be released in time to alleviate the stacking up of the activations. However, the P(nV2) cannot be used directly due the inconsistency issue discussed above. To address the issue, 2BW uses two weight versions and combines gradient accumulation. Why does 2BW work effectively? The point is that 2BW effectively handles the intersection of adjacent update periods. As shown in Figure 2 the forward pass for micro-batch 3 is executed before the backward pass for micro-batch 2, but backward pass is executed after the backward pass for micro-batel necessary to continue to use the old weight version in the period of cecro-batchhss and 4, and the new weight version (updated by micro-bart batro- 2) will be used in the next period and 6). Thus, if we want to further reduce the number of weight versions, need to eliminate the intersections of micro-batches and 3, and 5, etc. To achieve this, we temporarily introduce idle time blocks, shown in Figure 3 (its pipeline is the same as DeeDDarramfuhhii
At this time, is
these idle time blocks cannot be eliminated, can we fill them? Through the statistical analysis, we found that for pipeline of PipeDream-2BW with model partitions, least the idle time blocks shaped like P(rV.), > (n 1) are required to eliminate the intersections. In addition, a pipeline accumulation period has at least execution time blocks shaped like P(=V,), >= Obviously, when x >= n, they have the same shape. Thus, can we fill these idle time blocks with the pipelining of another part of the weights of the same model?
3.2 DOUBLE-GROUPH WRIGHT UPDATES (2GW)
In this subsection, we introduce the 2GW technique from two points: model partitions grouping and weight update semantics.
Gg Forward Pass
Backward Pass
Forward Pass
Backward Pass
Idle
G F&B
F,&B,&B
F2&B2&B
Fr.F
B
(a)
(b)
1
3
3
(c)
Figure 4: Derivation WPipe pipeline: (a) Further partitioning and grouping, (b) moving the Go forward pass of the next step to the front of the backward pass of Go of the current, (c) expansion of the pipelining for (b).
Model Partitions Grouping. As shown in Figure 44, based on Figure D we further split each model partition into two and obtain twice the number of model partitions. Then, we divide them into two groups, Gg and G1, and train them on the same devices. For G1, is typical 1/2 structure, which can realize continuous pipeline expansion through P. But for Go. since its forward pass and backward pass are separated, continuous pipeline cannot be directly achieved by cross-splicing (P). Therefore, we first need to convert it into 1/ structure. By analyzing the execution rules of the pipeline after grouping, we found that moving the forward pass of G2 of the next step to the front of the backward pass of Go of the current step form Vg. shown in Figure 4p. After doing this, we can cross- cross-splice Go and G respectively to obtain P(G,) and P(G)) and then continue to cross-splice their results to obtain P(P(Go), P(G1)). For G1, its micro-batches and 3 can be well
4

---

Published as a conference paper at ICLR 2022
Table Details of the memory footprint of pipeline systems of GPipe, PipeDream, PipeDream-2B%) and WPipe, where we ignore the intermediate activations for recomputation.
BUBBLE RATIO
THE SIZE OF ALL BUFFERS
ACTIVATIONS (XON-RECOMPPTATION)
ACTIVATIONS (RECOMPUTATION)
PIPELINE
GPIPE
MS. 2M N+1 S.
S. NONE S, 0.5S,
PIPEDREAM
0
N+1 Sm
2 S. 4M N+1 S.
PIPEDREAM-25
25.
WPIPE
Sm
separated by the pipelining of Go (P(Go)), shown in Figure 4c. This also answers the question mentioned in section 3.1. The pipelining of Go can fill in the idle time blocks in Figure 3
Staged
Stage1
Stage2
Staga3
6. Forward Pass
Gg Baskward Pass
G, Forward Pass
G Backward Pass
Idle
Figure 5: Timeline of execution of 2GW, where only Go needs double weight versions.
Weight Update Semantics. As shown in Figure 4r. G1 only maintains one weight version and has the same weight update semanties as data parallelism. For Go, due to the movement operation, the intersections naturally exist, such as micro- batro-bbtccess 3, 4, and in Figure Nevertheless, Go only needs to maintain weight versions and has only delay term in the weight update semanties, which the same as 2BW. If using SGD optimizer, the weight update semantics of 2GW are as follows:
++1 w', W
ff G,2 ..... W., (t-1+)) (t-1+4)
(1-1+)) ) 10,
Other optimizers can be similarly analyzed. Figure 5irres]ssowws a more detailed pipeline-paralle timeline of 2GW. each pipeline stage, the model partitions Go and G₁ are executed alternately without introducing bubbles after the system is warmed up.
3.3 MEMORY FOOTPRINT ANALYSIS
The memory footprint of most pipelines is mainly divided into four parts, i.e., naive model weights, weight buffers (saving history weight versions), in-flight activations, and optimizers. Since the naive model weights and optimizers are roughly the same for each pipeline method, mainly analyze weight buffers and in in-flight activations. In WPipe, G1 has no weight buffer, and the size of Go's 's weight buffer is 2. We use Sp represent the size of the entire model parameters, Mc, and Mc, represent the sizes of Go and G,, respectively. Supposing that the model is evenly divided (Mc, MG,),, the size of weight buffer of WPipe is M 2Mc, MG, + Mc, Sp- For in-flight activations, we use Sa to indicate the activations the entire model for a micro-batel From Figure 5
S. we statistically conclude that the in-flight activation amount of stage, equal to M 2N S, for G1, and of all stages is equal to: for Go and (M- - i) 2N
N=
S.
4M N
Σ
(2M
Sa.
(2)
2N
4
where the N and M indicate the number of pipeline stages and micro-batches, respectively, and M >= N. The same statistical method is used for GPipe, PipeDream, and PipeDream--285 2- obtain the size of all weight buffers and in-fhthht activations, respectively, as shown Table T
With Activation Recomputation. From Table the in-flight activations increase linearly with the number of pipeline stages, which will become a memory bottleneck when training a large model. Thus, pipeline system must take measures reduce in-flight activations. The activation recomputation is
5

---

Published as a conference paper at ICLR 2022
an effective method widely accepted, especially for 2GR. It can eliminate the stacking up of historical in- flight activations, leaving only the activations in an active state in the system, as shown in Table From the table, can summarize the following points: (a) compared to PipeDream-2BW, the weight buffer size of WPipe reduced by half, and the superiority is more obvious compared to PipeDream. Although GPipe has weight buffer, has introduce M+N-1 bubble overhead; (b) recomputation significantly reduce in n--fiight activations for GPipe, PipeDDramm230h and especially WPipe.
NV1
3.4 COMMUNICATION OPTIMIZATION
We use C₂ to represent the cost of intermediate activations//radiints in one communication, and then the model-parallel communication overheads of different pipeline systems are CwPipe 2(2N,, 1)Ca. C2aN = 2(NV - 1)Ca. and CGPipe = 2(N, - 1)Ca, respectively, where N, indicates the number of stages. They show that with the same number of stages, the communication overhead of WPipe is twice that GPipe and PipeDream-22 2 BW. Nevertheless, in most cases, C2 is small and can be ignored, especially when training some language models, such as Transformer (Vaswani et al. 2017). When C. is large, We have the following countermeasures:
Combination with Data Parallelism. Nor- mally, too large Ca is caused by too large micro-batch. we can proportionally reduce the depth of the pipeline while reducing the size of the micro-batch, and then we proportionally in- crease the width of data parallelism to maintain the same global batch size, as shown in Fig- u As result, the size of the micro-batch be- comes smaller, and the C. also decreases, while the number of accelerators and the global batch size remain unchanged. Of course, we need to weigh the communication overhead between model parallelism (Cwripe) and data parallelism (Cpe) to choose the appropriate ratio of depth (d) and width (10). When Cwpipe is greater than CDP, we reduce the value of d w, d * w NaPU- Otherwise, we increase the value of d w.
stage0
Gcc
Gsa
Goo
Gac
Goe
Guy
Goe
Gue
stage1
stage2
12
Gup
Gyz
Gua
G/2
Goc
stage3
Model Parallelism Depth
Data Paralleliem
Figure 6: Layouts of model parallelism and data parallelism.
Overlap of Computation and Communication & Heterogeneous Network Communication. The former almost offset the overhead of activation recomputation. The latter can effectively heterogeneous networks to balance communication overhead. Please refer to the appendix for details. In addition, some communication compression techniques are also shown in the appendix.
EXPERIMENTS
WPipe is implemented with PyTorch-1.4 1.4 (Edward Z. Yang 2 and executes on two environments, ie.. a single machine with eight 16-GB VI00 GPUs (Env-1) and private cluster with x 8V100 GPUs (Env-2).
4.1 QUALITY OF CONVERGENCE
In this section, we compare the convergences of WPipe and PipeDream-2B4 by comparing the accuracy when training the same model on the same dataset with the same hyperparameters.
Text Classification. We finetuned BERT (Devlin et al., 2018) and BERTLARGE DDelln et al. 2018) for WPipe, PipeDream-2E4 and data parallelism on the QQP and MNLI tasks (Wang et al. 2018). We used respectively bert bartt-baeeee www---auaaassssseeeeeaaeaaa--s-e uncase and bert-large weights from trans
pre-training 5. 0 (Wolf et al., 2020). We used Adam optimizer, a learning rate of 8 x 10 10 = 8 x 10 10-5) with 1000 steps warmup(ws = 1000) and a mini-batcl size of 4 10-5 with for BERTLARGE From Table 2 WPipe, PipeDream 2BW, and data parallelism have similar final accuracy. For further analysis, we continue to perform image classification experiments.
256(b 256) for BERT BAsE and the same optimizer,
2000 and 128

---

Published as a conference paper at ICLR 2022
Table The results of the convergence experiment WPipe. We train the models three times with different seeds. Then, we calculated the means and standard deviations of the results. Where DP represents data parallelism and PD-2BW represents PipeDream-2B4
TASKS
MODEL
METRIC
DP
WPIPE
PD-2HW
ACC
87.66 + 0.06
87.68 + 0.06
87.63 + 0.03
BERTBASE
F1
0.02 83.39
0.02
83.39+ 0.04
83.34 0.05
QQP
ACC
97.63+0.00
87.59=
87.38 =
BERTLARGE
F1
83.26 0.08
83.21
ACC
82.81
83.051 0.19
82.981 0.14
BERT BASE
M-ACC
$3.16.10.00 0.03
83.0
82.82 0.26
MNLI
ACC
8.11
86.25 1
86.29 + 0.12
BERTLARGE
M-ACC
86.05#0.26
86.08 = 0,24
OXFORD
RESNEXT55._2X4D
ACC
99.55
99.47=
99.51 0.12
FLOWERS102
RESNEXTION 1013888
ACC
-9.39
99.19
99.19 0.19
HESNEXX55_32244
ACC
97.15+
97.28 0.20
97.26+ 0.09
CIFARIO
RESNEXTIO1 _32x8D
ACC
98.15
98.18 0.07
98.15 0.07
RESNEXT5O 32x4D
ACC
85.67 0.12
85.62+ 0.09
85,80 + 0.21
CIFAR100
RESNEXTIO1 32x8D
ACC
87.90 + 0.30
88.00 0.21
87.93 0.18
Image Classification. We finetuned, respectively, the ResNeXt50 (32x4d) (Xie et al. 2017) and ResNeXt101 (32x8d) (Xie et 2017) for WPipe. PipeDream-22W, and data parallelism on the three datasets of CIFAR- 10 (Krizhessky et al. 2009). CIFAR- 100 (Krizhevsky et al. 2009), and Oxford 102 Flowers (Nilsback & Zisserman 2008). We used the pre-training weights from the torchvis on (Francisco Massa 2021), D, SDG optimizer, v = 1 x 10-2 with 0.05 warmup ratio and b 256. From Table there is still obvious difference in their final accuracy. Thus, continue to analyze the changes in loss of the three methods during training, while supplementing the training experiments from scratch.
although WPipe, PipeDream-22 BBW and data parallelism have similar final loss and accuracy, the curve of WPipe closer to data parallelism. Among them, the loss of data parallelism drops the fastest, and its accuracy rises the fastest, and WPipe second only data parallelism, which benefits from the fact that the weight update delay of WPipe is reduced by half, compared to PipeDream-22Ht For more loss curves and accuracy curves, please refer the appendix.
Intermediate Metrics Analysis. As shown in Figures 7a 7c
0.2
0.8
0.7
0.5
0,3
0.4
0.2
UP pe
0.1
Popo remm 259
0.0
Duta Farullsiom
5
1.0
15 Epechs
23
25
35
Epocha
Epochs
(a) QQP:BERTIARGE
(b) MNLI-BERTLAGGE
(c) Cifar10:ResNeet50.
Figure 7: Part of the training loss from Table and the accuracy when training ResNeXt50 from scratch with WPipe, PipeDream--2Ht and Data Parallelism (SGD with a fixed learning rate of 0.01).
4.2 THROUGHPUT
In this section, we measure the throughputs WPipe through training the large-scale DNN models on Env-1 and Env-2 and compare them with other pipeline systems. The main factors that affect throughput are batch size (B), the ratio of model parallelism to data parallelism (d w), and the possibility to recompute (S.) as presented below:
T mTx xT x Sma); B b M w, M >= d; S. (True, False]; Saw = (d w)ld+ * = No SM = {d, 1odd++1... 1,...:
(3)
7

---

Published as a conference paper at ICLR 2022
where T is the optimal result in optimization space S. Sdw SM. H w) represents all isomers in the same : w. Sas represents the gradient accumulation period. With regards to the optimal solution of T, the study in (Narayanan et al.] [021) gives workable solution, we do not do further research. Note that since B directly affects the accuracy of the model, it is often a given value.
4.2.1 COMPARISON WITH PIPELINED APPROACHES
As shown in Figure 8 throughput experiments are carried out through training different layers of BERT and ResNeXt models on the Env-1 and Env-2. For more throughput experiment data, please refer the appendix. In general, the communication overhead of data parallelism is larger for BERT, and model parallelism is larger for ResNeXt.
Comparison to PipeDream-2BW. In general, the suitable batch size is between 2 and 2 it too large or too small, it is not conducive to the convergence of the model. However, in most cases, when training large models, some methods cannot reach this batch size, shown in Figure 8a When in this range, as shown in Figure 8c. when batch = 2", WPipe is 1.4 times faster than PipeDream-2BW for Bert768 with 5.5 billion parameters. Compared to 2BW, WPipe not only reduces the cost of switching between weighted versions by half but also has a larger solution space when the batch is larger. As shown in Figure 8d] when batch 29. WPipe can run in the configuration of : w 4 : 16 in Env-2, and 2BW due to memory limitations, the model-parallel depth is at least 8, that is, d >= 8, so in this case, the throughput of WPipe higher (for convolutional networks, in general, C. is larger and CDp is smaller, so d w. needs to be smaller improve throughput). From Figures and and [8d] it can be concluded that after communication optimization, even on a convolutional network with a larger C.. WPipe has throughput not lower than PiprDream-2BB.
Comparison to GPipe. For GPipe, its biggest problem is that there are idle time blocks on its execution timeline. In addition, its forward activations cannot be released in time. These factors directly or indirectly affect its throughput. Compared to GPipe, from Figure Sa 3d WPipe is 2.1x faster for Bert192 and x.2x faster for ResNeXt302.
Comparison to PipeDream. Due to poor memory efficiency, the larger models cannot be trained using PipeDream, such Bert384 and Bert768. Practice shows that for ResNeXt200, ResNeXt302, ResNeXt500, and ResNeXt800, PipeDream cannot automatically split them. According to existing data, WPipe is 7.1x faster for B9rt192.
WPisc
100
50
25W
150
Thrograst Gange (20
Thooyhoott Gaare/ 10
60-
4.5
30
40-
10
15
Gobal aisi-aati stoe 0:21
minibates (1+2)
Obobal minibates uze (h3)
(a) 1100)).)8V
(b) ResNeXX332(8V1000)
(c) Bert768(64V 100s).
200
Thoopppp
302
Throughpr
350
289--
2810
Throughpe
502
402
WP
202
200
102
52
100
10
10 Global misi
Global www.immiiiitaihhccco 0:21
(d) ResNeXt800064v1000 1000s.
(c) Ben:384(1::4, 64V100s).
(f) BesNeXt500064V1005
Figure 8: Optimal throughput for different batches in the Env-1 and Env-2, Where SM.N {2
4, 4 2,8 13 in Env-1, SM.N = (4: 16, 8 8, 16 : 4,32 2, 64 : 1} in Env-2. Se 8f show the throughput changes with different configurations.
8

---

Published as a conference paper at ICLR 2022
4.2.2 COMMUNICATION OPTIMIZATION ANALYSIS
Recomputation. The activation recomputation can greatly reduce memory footprint and expand mini-batch size for pipeline systems. As shown in Figure Bel PipeDream-2BW can expand the maximum mini-batch by 16 times, and WPipe can even expand by 44 times. WPipe has a more obvious memory advantage with activation recomputation. Although the batch size is not large as possible, usually when training large models, the batch size must be within a certain range to have better convergence. In many cases, due memory limitations, the effective range of the batch size cannot be reached. At this time the advantage of WPipe is crucial.
The Ratio between Model and Data parallelism. As shown in Figure[8ij as M N decreases, the corresponding throughput increases for ResNeXt500. This happens because the commu- nication overhead of data parallelism is smaller than that of model parallelism in the convolutional network. In this case, M : N can be adjusted more widely, which is more conducive to improving throughput, e.g. batch = 2 in Figure 88
Table The impact of network communication on throughput.
Env-2(4:16)
ResNeXt500
WPipe-R(A) WPipe-R(BB 497.3/3328
607.8/3328
Heterogeneous Network Settings. We change the communi- cation placement of model parallelism and data parallelism in the same M N to achieve higher throughput. A means that data-parallel communication takes place between machines and model-parallel communication takes place inside machines, while the opposite. As shown in Table in the case of A. the throughput of WPipe is higher, because the model- paraalel communication is the bottleneck for ResNeXt. Compared with the inter-machine, the bandwidth inside the machine is higher and the communication speed is faster. The model-pane model-parallel communication bottleneck has been better alleviated.
4.3 MEMORY OPTIMIZATION ANALYSIS
In the throughput experiment, WPipe can exe- cute on a larger batch, which also reflects the ad- vantage of the lower memory footprint of WPipe.
Here, we conduct a more detailed analysis: (1)
Manoty 0ostarme.G.
12
The results of experiments show that the mem- ory footprint of various sipeline-paralle meth-
ods is significantly reduced with activations re-
computation, especially for WPipe. As shown in Figure D. WPipe reduces the memory footprint by 65% for Bert96 on the GPr micro-batch of 64, and 56% for Rt20 on the per- GPU micro-batch of 32. They are 47% and 29% respectively for PipeDreemm-22 BW; (2) with the in-
Bart)6(R)
ResNo0:200:33))
PipnDrrearr pe
PipxCrear 225W-R
WPipe-R
Figure 9: The Bert96 and ResNeX1200 memory footprint vary with batch size. We set M : N as 8 1 for Bert96 and M N as 2 4 for ResNeXt120,, which the fastest configuration. We measured the average maximum memory footprint per GPU.
crease of batch size, the effect of recomputation on WPipe-R is more significant. From Figure 9 for Bert96, WPipe-R R is reduced by -14 com- pared to GPipe-R on the per-GPU micro-batch of 8 (because GPipe has weight buffer, and its initial memory footprint is lower), and on the Per-GPU micro-batch of 64, it increases to 28%. Compared to PipeDream-2Bv-R under the same conditions, WPipe-R increases from 21% to 36%.
5 CONCLUSIONS
In this work, we proposed and implemented WPipe, system for group-based interleaved pipeline- parallel training. Compared to the state-of-the-art approach, PipeDream-2Bel WPipe achieves better memory efficiency, higher throughput, and fresher weight updates through the double-grouped weight updates. Specifically, (1) throughput: WPipe achieves 1.4x acceleration; (2) memory footprint: WPipe-R reduces the memory footprint by 36%; and (3) convergence: although WPipe and PipeDream- 2BW have similar final accuracy when training, WPipe has a weight update semantics closer to data parallelism.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work is supported by the Computational Intelligence Department of Ant Group. We thank Jiyan Jiang for his helpful discussions. We would like to thank the Aliyun EFLOPS team for their substantial support in designing the industry-leading training platform to facilitate fast trials in this work. We also thank anonymous reviewers for their insightful and valuable comments.
REFERENCES
Nvlink. URL
www. www..nvidia.c http:://wwww.....a.a...o///oattr
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Chi-Chung Chen, Chia-Lin Yang, and Hsiang- Yun Cheng. Efficient and robust parallel dnn training through model parallelism on multi-gpu platform. arXiv preprint arXiv: 02839, 2018.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: 1604.06174, 2016.
Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. Project adam: Building an efficient and scalable deep learning training system. In //th [USENIX) Symposium on Operating Systems Design and Implementation ({OSDI) 14), pp. 571-582, 2014.
Thor Johnsen. Christian Sarofeen. Nvidia/apex. https://giihub..cc////DDD /aae////////p/aaaaaeaaaaa 2021.
Jeffrey Dean, Greg S Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V Le, Mark Z Mao, Marc' Aurelio Ranzato, Andrew Senior, Paul Tucker, et al. Large scale distributed deep networks. 2012.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rX00:18100008805, 2018.
Soumith Chintala. Edward Z. Yang. pytorch/pytorch. pytorch 2021.
http:://gtthub..cm///ttttr.
NicolasHug. Francisco Massa. pytorch/vision. httpas///gthhuu.cc//////rrrrr//s 2021.
Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprins arXiv: 03377, 2018.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le, Yonghui Wu, al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pp. 103-112, 2019.
Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Kurt Keutzer, Ion Stoica, and Joseph Gonzalez. Checkmate: Breaking the memory wall with optimal tensor rematerialization. arXiv preprint 190.0053, 2019.
Zhihao Jia, Matei Zaharia, and Alex Aiken. Beyond data and model parallelism for deep neural networks. arXiv preprint arXiv.1807.05333 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1909. 1942, 2019.
10

---

Published as a conference paper at ICLR 2022
Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding. Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, et al. M6: A chinese multimodal pretrainer. arXiv preprint arXiv:2103.00823, 2021.
Yinhan Liu, Myle Out Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeitlemoyer and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1rXiv:1907.11666 2019.
Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. Memory-efficiee pipeline-parallel dnn training. In International Conference on Machine Learning, pp. 7937-7947. PMLR, 2021.
M-E. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In Proceedings of the Indian Conference Computer Vision, Graphics and Image Processing, Dec 2008.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Img Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv: 19000000, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:180.007661, 2018
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ar natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pp. 38-45, 45, Online, October 2020. Association for Computational Linguistics. URL httph: www. ac web. www.oraatgoogg/////...o
emn
6.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08.. 08237, 2019.

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 OTHER EXPERIMENTAL DATA OF CONVERGENCE.
Figures 10a 10h show the training loss curves of the remaining experiment in Table 2. Figure 10i shows the Fl curve of BERT BASE when training from scratch. Analyzing the trend of the curve, found that WPipe and data parallelism can converge normally, but PipeDream- 2BW does not.
Parallelism PlpDDraam-2WW
Parallelism Rip:Dream-2BW
Data Parallelism
0.5
0.6
PlpdDraam-2WW WPipe
WPipe
WPipe
0.3
0.4
Loos
0.2
0.0
0.0
23
95
Epochs
Epachs
Eoochs
(a) QQP:BERTRASE
BASE. BASE-
(b) MNLI-BERTBACK BASE.
(c) Flowerss00::ResNexxst
Data Parallelism
Data Paralielism PlpuDream-2BW
2.0
Data Paralletism
PipeDream-2WW
PipeDream-2BW
WPipp
1.5
WPipe
1.5
2-
1.0
0.5
0
23
47 Epochs
73
95
17 Epachs
26
35
0
17 Epachs
26
35
(d) Flowerst02..esNext100..
(e) Cifar10.ResNeXt50..
(f) Cifarl0:ReNNeX1101..
Data Parallelism
Data Parallelism 2Bc
0.750
SpeDream-2BW
0.725
WPipe
0.200
0.975
2-
0.650
0.635
pa
26
26
Epochs
Epochs
(g) Cifar100:ResNext50..
(h) Cifar100:ReNNext00...
(i) QQP:BERT BASE.
Figure 10: The remaining part of the training loss from Table and the Fl when training BERTBASE from scratch with WPipe, PipeDrcam-2E8 and DataParallel (Adam with a fixed learning rate of 5 x 10-6).
A.2 TIROUGHPPU
In the experiment, our settings of models are hidden-size=7ee seq beq.len=12 of all Bert models, and groups=32, width per group=4 of all ResNeXt mod- els. Regarding cluster configuration, there are 8 machines in our private cluster, and each machine has GPUs with memory size of 16G, (nten()) Platinum 8163 CPU, 512GB of RAM with 25Gbps Ethernet interface, and 300GBps NVLink (nvl). which is 96 times the Ethernet bandwidth. In addition, the version PyTorch we used was 1.4.
As shown in Figure IIa 1ld we continue to train Bert96 and ResNeX1200 on single machine with WPipe, PipeDream- 2BW, Dream, and GPipe, and train Bert384 and ResNeXt500 on multiple machines with WPipe and PipeDream-2Ei 2BW. The overall conclusion is the same the above. WPipe has a more obvious acceleration effect on Transformer series networks (Bert, GDP, etc.), but has a slight acceleration convolutional networks, compared to PipeDreem-22W.
12

---

Published as a conference paper at ICLR 2022
100
2BW
20W
2B0
Throudoue
FrxxDream
Throughosk
703
159
minkkaatch
(a) Bert96(8V100))
(b) 1000)
(c) Ben138466AV1004
503
19P
Wric KAPC
AHHORRAFC
201V
WPipe NaraAFC
100
WHite NonoPC
Throughtt
403
Throughout BEwcRIn
302
PppDream 2BWW-A
Throughss Goanolcc000m
60
TpsGream-2BW.
60
202
202
Gebai mini-ottch size 0:21
Gibaa misicatch size 0:21
Ghotal misibatch sze th2
(d) ResNeXt500664V100))
(c) ResNcX118044:16, 64V100s).
(f) Bert768(4:16, 64V100s).
Figure 11: Optimal throughput for different batches the Env-1 and Env-2. Where SM.N = 2 2,8 1} in Env-1, SM:N {4 16, 8, 16 4, 64 1} in Env-2. Figures Ilc 1lf
show the effect of compressed model-parallel communication on different models.
A.3 COMMUNICATION OPTIMIZATION
In this section, we use a specific example to analyze the effectiveness of the model-parallel communi- cation compression. In addition, we add detailed description of overlap and heterogeneous network communication.
Communication Compression. Referring to the automatic mixed precision algorithm of the apex (Christian Sarofeen, 2021), we im- plemented the automatic precision compression technique for intermediate activations/graddentts which reduced the communication overhead by nearly half across the pipeline. This offsets the communication overhead introduced by further splitting model partitions. As shown Fig- ure 12, the sender divides the intermediate acti- /ations/gradients by appropriate scolers (pow- of 2) and converts them into half-precision tensors. If the conversion succeeds, the sender will transmit the half-precision tensors together with the scalers the receiver. After receiving the data, the receiver restores their original pre- cision by the scaler. If the conversion fails, the sender will transmit the full precision tensors.
Activations/Gradient
Activations/Gradients
Dynamic Scale
lensor fp16, scaler
Unscale
Sender
Receiver
Figure 12: The communication process of the inter- mediate activations and gradients using automatic precision compression.
Automatic Precision Compression (APC). As shown in Table 4 WPipe is 1.17x faster with APC when training ResNeXt500 and the batch size equals 3328. The loss (the negative value) of accuracy, which is brought by APC, does not exceed 0. %, as shown in Table 5 However, when the communication overhead is not large, the use of APC will not speed up but will be slower. As shown in Figure IIf when batch <= 211, C2 is small, and the acceleration benefit cannot exceed the compression overhead. When batch 2 there is positive feedback. For convolutional networks, C2 is large, and APC has always had a positive effect, but the effect only obvious when the batch size is large enough, as shown in Figure 11e In summary, when
Table 4: The impact of network commu- nication throughput.
2(4:1)
ResNeX1500
WPipe-R-APC WPipe-R-NonAPE
732.6/3328
607.8/3328
13

---

Published as a conference paper at ICLR 2022
Table The accuracy difference between using APC and using APC with the same hyperparam- eters. We set V 8 x 10-b, b = 32 8, ws 200, epoch for BERT and 4 x 10 b 16 8, ws 100, epoch for BERTLARGE and 0.01, b wr 0.05 for ResNeXt50 32x4d and ResNeXt101
BERT
BERTLARGE
F1
-0.0003 + 0.0011 0.0035 = 0.0127
RESNEXT50
RESNEXT101
V ACC
-0.0008+0.0014
0.0012+ 0.0012
the batch is large, we use the communication compression technique to have positive benefit, but in many cases, we do not need to use such a large batch size to train the model, so most cases, model-parallel communication will not be bottleneck.
Overlap of Computation and Communication. As shown in Figure 13 communication overlap with the forward pass, backward pass, and activation recomputation. Especially for activation recomputation, its time overhead can offset mostly.
the activations or gradients
Gradient Recv
Activation Recv
Forward
Recompute Forward
Backward
Forward
Activation
Gradient
Send
Send
Figure 13: The overlap of model execution and activations/gradients.
Heterogeneous Network Communication. Generally, the intra- macranmachine bandwidth (NVLink) is higher than the inter-machine bandwidth. In WPipe, the communication tasks mainly include model-paralle communication and data-paralle communication. Thus, WPipe allows tasks with large communication overhead use the higher intra-machine bandwidth, and tasks with small communication overhead to use inter-machine bandwidth, to balance communication overhead.
A.4 MODEL PARTITIONS GROUPING
Model partitions
P1
P2
P3
P4
Go
P1
P2
G
P3
P4
Pipeline stages
Stage0
P1
P3
Stage1
P2
P4
P2
Figure 14: The relationship between model partitions and pipeline stages.
A.5 EXPANSION
GPipe Grouping. As shown in Figure [5] the pipeline grouping technique can also be applied to GPipe, thereby reducing GPipe's bubbles. In Figure 15 the model partitions further divided
14

---

Published as a conference paper at ICLR 2022
into four and divided into two groups. At this time, the bubble is reduced by half. For further discussion, when the number of groups is N, the bubble will reduced to N. but the model-parallel communication overhead will increase by N times. However, if NVLink can be used, the impact of increased communication overhead will be greatly reduced.
F10
F11
Bubble
B11
B10
F2o
F21
B21
B20
F10
F11
F30
F31
Bubble
B31
3
B11
B10
F20
F21
F F40
F41
B41
B40
B21
B20
Figure 15: The pipeline grouping applied to GPipe.
15


Reviews:
[{"review_id": "pa7drHi_Opl", "reviewer": "Reviewer_rrtJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The proposed pipeline strategy is novel and effective.\n2. Clean analysis of the proposed strategy in terms of memory footprint, communication overhead, bubble time, and update staleness.\n\nWeakness:\nThere is no strong weakness of this paper. For other minor clarification and writing issues, see the comments below.\n\nDetailed comments:\n1. The term \"model parallelism\" should be used more precisely. There are three types of model parallelism: pipeline parallelism, operator placement (which is referred by you as \"conventional model parallelism\" or \"naive model parallelism\"), and operator partition (e.g., the strategy in Megatron-LM v1). In your paper, it seems you totally ignore the third type operator partition. I suggest renaming all \"naive model parallelism\", \"conventional model parallelism\" in your paper to \"operator placement\", and adding some description of the operator partition.\n2. The memory footprint analysis with activation recomputation seems wrong. I think all numbers in the last column of Table. 1 are wrong. Take GPipe for example, activation recomputation only reduces the memory footprint of a micro-batch on a stage by a constant factor. It cannot reduce the footprint by a factor of \"M\", because anyway, each stage has to keep the input activation of the \"M\" in-flight micro-batches. The same analysis applies to other pipeline schemes.\n3. What will happen if we split the weights into more groups instead of just 2 groups?\n4. Please provide the exact settings of the models and the cluster. For example, provide the hidden_size, seq_len, num_heads of all BERT models. Provide the network bandwidth of your private cluster.\n5. How do you split the models into pipeline stages? For BERT, you can assign an equal number of transformer layers to each stage. How do you split the ResNext?\n", "summary_of_the_paper": "This paper introduces a novel pipeline training strategy WPipe. WPipe divides model partitions into two groups and updates each group alternatively, which eliminates half of the delayed gradients and memory redundancy compared to Pipedream-2BW. \n\nThe experimental results show that WPipe can achieve higher throughputs and reduce memory footprint with similar final model accuracy.", "summary_of_the_review": "This paper introduces a novel pipeline training strategy and comprehensively evaluates its performance.\nAlthough the writing can be improved, I recommend accepting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gT8bkYYJBb", "reviewer": "Reviewer_VWfQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has following strengths. \n1. It tackles an important problem of how to fit rapidly growing model sizes into slower growing device memory. Pipeline parallelism has been a useful approach for this problem, and this paper provides some improvements in that space. \n2. It proposes an interesting approach to address key issues surrounding memory efficiency and weight update sematics in existing pipeline parallelism approach\n\nThe weakness include:\n1. The overall appears narrow and incremental in the sense that WPipe is fixing a memory efficiency bug of PipeDream-2BW relative to PipeDream-flush. In other words, WPipe is trying to deliver the best-of-both-worlds for these two existing approaches. As a result, it is not obvious how the results applicable beyond the PipeDream* line of pipeline parallelism.\n\n2. The throughput results (4.2) are difficult to understand because they are based on unfamiliar models (e.g., Bert768 and ResNetXt500) which are not previously defined and different from the familiar models (e.g., Bert_base, ResNetXt50) used for the convergence quality results (4.1). It seems that the models in 4.2 were constructed to show the best case throughput benefits of WPipe, but don't appear to be real world models. \n\n3.  I feel the writing could be greatly improved as I found multiple portions difficult to follow. Although I am not hands-on with pipeline parallelism, I feel I have a pretty solid understanding at the high-level, but yet this draft was less accessible than expected. Below are some specific examples of confusing portions and questions for the authors:\n\n  a) The idea (stated in 3.2) of splitting the model partitions to double the number of partitions and obtain two groups (G0 & G1) is quite confusing, especially since memory footprint analysis in 3.3 assumes the model is evenly divided so that G0 and G1 are half of the model.  It would be helpful to illustrate the before/after of model partitions with WPipe relative to PipeDream-2BW, independent of mini- or micro-batches. This should be possible since model partitions are simply mappings of model parameters to devices. \n\n  b)  It seems the cells in Figures 1 & 2 are mini-batches and micro-batches respectively. It would be helpful to clarify this detail since these two Figures are often discussed together. Also, it seems Figure 2(c) is missing the update cell. Is this a typo?\n\n  c) Much of the material in 3.4 is prior work and so could be written more briefly and systematically. ", "summary_of_the_paper": "The paper proposes, WPipe, a technique for reducing the memory overheads of weight versioning and improving freshness of weight updates in pipeline parallelism through a novel scheduling of the the pipeline stages. It presents evaluation results which show that WPipe does not harm model quality, and that the memory efficiency can improve the throughput compared to prior pipeline approaches. ", "summary_of_the_review": "WPipe is basically a memory efficiency optimization of PipeDream-2BW, but the paper does not report throughput benefits on real world models. So it is not clear how practical the optimizations are. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AmMfoBOmLBr", "reviewer": "Reviewer_fotX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There is some novelty in the new pipeline scheme, but I feel it is incremental. On the practical side, the method has the potential to be more impactful to large scale DNN transformer models such as BERT, GPT etc. The empirical results are very encouraging to me. The presentation and clarity in the paper could be improved - the authors should re-use terminology used earlier in the pipeline parallelism literature to make the presentation consistent and less confusing.\n\nPros:\n- Simple idea that seems to work very well empirically. Can be readily adapted and used in existing training pipelines.\n- Memory Footprint Analysis is provided for the method\n- Empirical results are very encouraging\n\nCons:\n- Novelty factor is bit limited\n- Presentation could be improved\n- Reproducibility can be improved (it would be benefit readers if the authors could provide hyper-parameters and experimental setup in a Table summarized for all of their runs)\n\nComments:\n- In the abstract, authors mention - \"It moves the forward pass of the next period ...\" -> period is not defined at this point and is too vague a term to use without defining what it exactly refers to. \n\n- \"model partitions\" -> do the authors mean pipeline stages? Strongly recommend using standard terminologies from model parallelism literature and explain things more consistently. I feel the paper is written a little hastily.\n\n- \".. raw pipeline parallelism ... suffers from inconsistency issue\" -> what does inconsistency mean here? It still is a bit vague term to me.\n\n- How does WPipe react to varying the varying number of pipeline stages? What is the maximum number of pipeline stages the authors tested the method with? It would be useful to understand this scalability.\n\n\n\n", "summary_of_the_paper": "The paper presents a novel pipeline parallelism technique for training large DNN models named as WPipe. The method aims to improve upon existing pipeline methods such as Pipedream-2BW by having a better memory efficiency and more fresh weight updates. Key idea is to divide the model partitions into two parts groups and make the forward pass of the second part to reduce the delayed gradients. The method is tested empirically and performs better than proposed current state of the art baselines.", "summary_of_the_review": "The novelty factor in the proposed work is incremental and I also feel the paper can benefit from some more polishing of the draft due to the presentation issues I outlined.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2LRFHHArijz", "reviewer": "Reviewer_5TDo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Training very large DNNs models is a hot topic of practical interest. Data parallelism cannot help when the models are so big that they cannot fit in a single accelerator. In this case, model-parallel techniques can be applied. There are several existing pipelines, such as GPipe, PipeDream, PipeDream-2BW that improve various aspects of the naive execution. However, they typically suffer from (1) staleness of the weight updates, caused by inconsistencies in the versions of the weights used in the forward and backward pass, and (2) large number of in-flight activations, caused by the forward passes waiting for their backward passes in the pipeline. \n\nThe paper proposes a new pipeline called WPipe, that builds on the existing ones, but addresses the previously mentioned two issues. A careful construction is proposed, based on two groups of weights. A key feature of the new scheme is the placement of the forward pass of the next step for one of the groups (G0) at the front of the backward pass of G0 of the current step. This elegant construction, allows group G1 to only maintain one weight version and G0 to maintain only two weight versions, thus significantly reducing the memory overhead. \n\nA theoretical analysis of the memory footprint is provided. WPipe can also benefit from activation recomputation techniques, that can further reduce the in-flight activations at the expense of more computation. Furthermore, the communication overhead is analyzed, and some mitigation strategies are suggested: data parallelism, overlap of computation and communication, heterogeneous network communication. \n\nA significant part of the paper is dedicated to experimental analysis, including text classification (BERT models) and image classification (ResNeXt models). The experiments show that compared to state-of-the-art PipeDream-2BW, WPipe is more memory efficient (by36%) and has higher throughput (1.4x). Although the final accuracy is similar for the two schemes, WPipe is shown to have weight update semantics closer to data parallelism.", "summary_of_the_paper": "The paper addresses the problem of very large scale deep neural network training through model parallelism. A new model parallel pipeline called WPipe is proposed, that builds on previously existing schemes while targeting (1) lower memory redundancy and (2) fresher weight updates. Experimental evaluation shows that WPipe can achieve significant acceleration (1.4x) and lower memory footprint (36% smaller) when compared to state-of-the-art PipeDream-2BW.", "summary_of_the_review": "The paper proposes a new model-parallel pipeline called WPipe, for DNN training. A careful construction is proposed, based on two groups of weights. A key feature of the new scheme is the placement of the forward pass of the next step for one of the groups (G0) at the front of the backward pass of G0 of the current step. This elegant construction, allows group G1 to only maintain one weight version and G0 to maintain only two weight versions, thus significantly reducing the memory overhead. Theoretical analysis and an extensive experimental evaluation on text and image data show that WPipe outperforms state-of-the-art schemes, being 1.4 times faster while using 36% less memory.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new pipeline-parallel training method called WPipe. WPipe works (on a very high level) by replacing the two-buffer structure of PipeDream-2BW with a two-partition-group structure, allowing resources to be shared in a similar way to PipeDream-2BW but with less memory use and less delays in weight update propagation across stages. The 1.4x speedup it achieves over PipeDream-2BW is impressive.

In discussion, the reviewers agreed that the problem WPipe tries to tackle is important and that the approach is novel and interesting. But there was significant disagreement among the reviewers as to score. A reviewer expressed concern about the work being incremental and difficult to follow. And while these were valid concerns, and the authors should take note of them when revising their paper, I do not think they should present a bar to publication, both based on my own read of the work and also in light of the fact that other reviewers with higher confidence scores did not find novelty to be a disqualifying concern. As a result, I plan to follow the majority reviewer opinion and recommend acceptance here.
Decision: Accept (Poster)