# Pipelined Training with Stale Weights in Deep Convolutional Neural Networks

Authors: Lifu Zhang; Tarek S. Abdelrahman
Journal: Applied Computational Intelligence and Soft Computing
 (2021)

---

## Abstract

The growth in size and complexity of convolutional neural networks (CNNs) is forcing the partitioning of a network across multiple accelerators during training and pipelining of backpropagation computations over these accelerators. Pipelining results in the use of stale weights. Existing approaches to pipelined training avoid or limit the use of stale weights with techniques that either underutilize accelerators or increase training memory footprint. This paper contributes a pipelined backpropagation scheme that uses stale weights to maximize accelerator utilization and keep memory overhead modest. It explores the impact of stale weights on the statistical efficiency and performance using 4 CNNs (LeNet-5, AlexNet, VGG, and ResNet) and shows that when pipelining is introduced in early layers, training with stale weights converges and results in models with comparable inference accuracies to those resulting from nonpipelined training (a drop in accuracy of 0.4%, 4%, 0.83%, and 1.45% for the 4 networks, respectively). However, when pipelining is deeper in the network, inference accuracies drop significantly (up to 12% for VGG and 8.5% for ResNet-20). The paper also contributes a hybrid training scheme that combines pipelined with nonpipelined training to address this drop. The potential for performance improvement of the proposed scheme is demonstrated with a proof-of-concept pipelined backpropagation implementation in PyTorch on 2 GPUs using ResNet-56/110/224/362, achieving speedups of up to 1.8X over a 1-GPU baseline.


---

## Body

## 1. Introduction

Machine learning (ML), in particular convolutional neural networks (CNNs), has advanced at an exponential rate over the last few years, enabled by the availability of high-performance computing devices and the abundance of data. Today, CNNs are applied in a variety of fields, including computer vision [1], biological and medical science [2], social media [3], image analysis and classification [4, 5], and urban planning [6] to name a few.However, modern CNNs have grown in size and complexity to demand considerable memory and computational resources, particularly for training. This growth makes it sometimes difficult to train an entire network with a single accelerator [7–9]. Instead, the network is partitioned among multiple accelerators, typically by distributing its layers among the available accelerators, as shown in Figure 1 for an example 8-layer network. The 8 layers are divided into 4 computationally balanced partitions, P0,…,P3, and each partition is mapped to one of the 4 accelerators, A0,…,A3. Each accelerator is responsible for the computations associated with the layers mapped to it.Figure 1
Partitioning of layers.However, the nature of the backpropagation algorithm used to train CNNs [10] is that the computations of a layer are performed only after the computations of the preceding layer in the forward pass of the algorithm and only after the computations of the succeeding layer in the backward pass. Further, the computations for one batch of input data are only performed after the computations of the preceding batch have updated the parameters (i.e., weights) of the network. These dependences underutilize the accelerators, as shown by the space-time diagram in Figure 2; only one accelerator can be active at any given point in time.Figure 2
Schedule of computations.The underutilization of accelerators can be alleviated by pipelining the computations of the backpropagation algorithm over the accelerators [7–9, 11, 12], that is, by overlapping the computations of different input batches on the multiple accelerators. However, this overlap causes an accelerator to potentially use weights that are yet to be updated by an accelerator further down in the pipeline. The use of such stale weights can negatively affect the statistical efficiency of the network, prevent the convergence of training, or produce a model with lower inference accuracy [7–9, 11, 12].Existing pipelined training approaches either avoid the use of stale weights (e.g., with the use of microbatches [8]), constrain the training to ensure the consistency of the weights within an accelerator (e.g., using weight stashing [9]), utilize weight adjustments (e.g., weight prediction [11]), or limit the use of pipelining to very small networks (e.g., [13]). However, these approaches underutilize accelerators [8], inflate memory usage to stash multiple copies of weights [9], or are unable to handle large networks [13].In this work, we explore pipelining that allows for the full utilization of accelerators while using stale weights. This results in a pipelining scheme that, compared to existing schemes, is simpler to implement, fully utilizes the accelerators, and has lower memory overhead. We evaluate this pipelining scheme using 4 CNNs: LeNet-5 (trained on MNIST), AlexNet, VGG, and ResNet (all trained on CIFAR-10). These CNNs are commonly used in the literature for the evaluation of pipelined training, and they represent models with a wide range of parameter sizes and complexity. We analyze the impact of weight staleness and show that if pipelining is introduced in early layers in the network, training does converge and the quality of the resulting models is comparable to that of models obtained with nonpipelined training. For the 4 networks, the drop in accuracy is 0.4%, 4%, 0.83%, and 1.45%, respectively.However, inference accuracies drop significantly when the pipelining is deeper in the network, up to 12% for VGG and 8.5% for ResNet-20. This drop makes the pipelined-trained models inferior to ones trained without pipelining. On the one hand, limiting pipelining to early layers is often not a limitation since the early convolutional layers in the network typically contribute to the bulk of the computations and thus are the ones to use and benefit from pipelining. On the other hand, we also address this drop in accuracy by a hybrid scheme that combines pipelined and nonpipelined training to maintain inference accuracy while still delivering performance improvements.We demonstrate the potential of our approach to pipelined training using ResNet-56/110/224/362 trained on CIFAR-10 and CIFAR-100 with PyTorch on a 2-GPU system. We show that our pipelined training delivers a speedup of up to 1.8X with only a drop of no more than about 2-3% in inference accuracy.Thus, this work makes the following contributions:(1)
It proposes and evaluates a pipelined training scheme that uses stale weights. It studies when the use of such stale weights can result in models with comparable prediction accuracies to those produced by nonpipelined training. The result is a simpler and less memory-intensive pipelined training scheme.(2)
It proposes and evaluates a hybrid pipelined training scheme that combines pipelined and nonpipelined training to mitigate loss of prediction accuracies with deeper pipelining.(3)
It presents a theoretical analysis that shows that our pipelined training scheme converges.The remainder of this paper is organized as follows. Section2 briefly describes the backpropagation for training of CNNs. Section 3 reviews the current literature on pipelined training. Section 4 details our pipelining scheme and how nonpipelined backpropagation and pipelined backpropagation are combined. Section 5 highlights some of the implementation details. Experimental evaluation is presented in Section 6. Finally, Section 7 gives concluding remarks and directions for future work. A set of appendices provide the training hyperparameters, more detailed results on memory usage, and a proof of convergence for our scheme.


## 2. The Backpropagation Algorithm

The backpropagation algorithm [10] consists of two passes: a forward pass that calculates the output error and a backward pass that calculates the error gradients and updates the weights of the network. The two passes are performed for input data one minibatch at a time.In the forward pass, a minibatch is fed into the network, propagating from the first to the last layer. At each layerl, the activations of the layer, denoted by xl, are computed using the weights of the layer, denoted by Wl. When the output of the network (layer L) xL is produced, it is used with the true data label to obtain a training error e for the minibatch.In the backward pass, the errore is propagated from the last to the first layer. The error gradients with respect to preactivations of layer l, denoted by δl, are calculated. Further, the error gradients with respect to weights of layer l, ∂e/∂Wl, are computed using the activations from layer l−1 (i.e., xl−1) and δl. Subsequently, δl is used to calculate δl−1. When ∂e/∂Wl is computed for every layer, the weights are updated using the error gradients.In the forward pass, the activations of the layerl, xl, cannot be computed until the activations of the previous layer, i.e., xl−1, are computed. In the backward pass, ∂e/∂Wl can only be computed once, and xl−1 and δl have been computed. Moreover, δl depends on δl+1. Finally, for a given minibatch, the backward pass cannot be started until the forward pass is completed and the error e has been determined.The above dependences ensure that the weights of the layers are updated using the activations and error gradients calculated from the same batch of training data in one iteration of the backpropagation algorithm. Only when the weights are updated can the next batch of training data be fed into the network. These dependences limit parallelism when a network is partitioned across multiple accelerators and allow only one accelerator to be active at any point. This results in underutilization of the accelerators. It is this limitation that pipelining addresses.


## 3. Literature Review

There has been considerable work that explores parallelism in the training of deep networks. In data parallelism [14–19], each accelerator has a copy of the model. The accelerators process different minibatches of training data simultaneously in iterations, aggregating gradients to update weights at the end of each iteration. This is done synchronously [14, 17] or asynchronously [16]. More related to our work is model parallelism [16, 20–23] in which a large model is partitioned into different accelerators, each responsible for updating the weights for its portion of the model. The data dependences, described in Section 2, allow only one accelerator at a time to be active, resulting in underutilization. Pipelined parallelism addresses this underutilization and is the focus of our work. Below, we review salient work on pipelined parallelism in training.Early work on pipelined training focuses on small networks and does not study pipelined parallelism in detail. Petrowski et al. [24] introduced the idea of pipelined backpropagation in neural network training. However, they realized the idea for only a 3-layer perceptron on a torus of 16 processors. Mostafa et al. [13] implemented a proof-of-concept validation of pipelined backpropagation training for a 3-layer fully connected binary-state neural network with truncated-error FPGA. However, the implementation does not have the coarse-grained layer-wise pipelined parallelization.More recently, PipeDream [9] implemented pipelined training for large neural networks such as VGG-16, Inception-v3, and S2VT across multiple GPUs. It limits the usage of stale weights by a technique referred to as weight stashing. The technique keeps multiple versions of the weights during training, to ensure that the correct (i.e., nonstale) weights are used in each pipeline stage. This technique results in high inference accuracies and high utilization of the accelerators but increases the memory footprint of training.GPipe [8] implements a library in TensorFlow to enable pipelined parallelism for the training of large neural networks. It pipelines microbatches within each minibatch to keep the gradients consistently accumulated. This eliminates the use of stale weight during training but at the expense of “pipeline bubbles” that degrade performance. GPipe utilizes these bubbles to reduce memory footprint by recomputing forward activations instead of storing them during the backward pass of training. The approach results in high inference accuracies with no increase in memory footprint, but the pipeline bubbles underutilize the accelerators, resulting in lower performance.Huo et al. [12] implemented decoupled backpropagation (DDG) using delayed gradient updates. They showed that DDG guarantees convergence through a convergence analysis. Similar to PipeDream, DDG uses multiple copies of the weights, thus increasing memory footprint. Further, DDG pipelines only the backward pass of training, leaving forward pass unpipelined, which underutilizes resources. Huo et al. [25] followed up by proposing feature replay (FR) that recomputes activations during backward pass, similar to GPipe, reducing memory footprint and improving inference accuracy over DDG. Nonetheless, also similar to GPipe, the recomputations lower speedups.Chen et al. [11] introduced weight prediction to mitigate weight staleness. Although their pipelined training shows improvement in throughput, they trained their networks for only 5000 iterations and it is not clear if their method can achieve standard model quality; their resulting model accuracies are much lower than typical for the models they train.Guan et al. [26] presented XPipe, which combines elements of GPipe and PipeDream implementations of pipelined training to improve efficiency by allowing the overlapping of the pipelines of multiple microbatches from different minibatches. Nonetheless, they avoid the use of stale weights using weight prediction.Kosson et al. [27] extended weight prediction in a fine-grained pipelined scheme that inserts pipeline registers between every pair of layers and limits the minibatch size to 1, aiming for a hardware implementation. They used a weight adjustment scheme to tackle weight staleness.Park et al. [28] described HetPipe that combines data parallelism in the form of virtual workers with the pipelined parallelism of PipeDream, targeting heterogeneous clusters of GPU workstations. Jia et al. [29] proposed FlexFlow, a framework that explores data and model parallelism in the training, but they did not consider pipelined parallelism. Li et al. [30] proposed Pipe-SGD that pipelines computation and communication as opposed to the forward and backward passes. The model is not partitioned across the accelerators. Instead, the pipelining is used to overlap communication of weight updates and compute to hide communication time and control the staleness at only 1 cycle. Therefore, large models may not fit on an accelerator.A common theme to the above body of work is that it employs various techniques to avoid the use of stale weights. These techniques introduce either computational inefficiencies or memory footprint increases. In this work, we propose the use of stale weights and study their impact on the quality of trained models. We show that when pipelining is implemented in the early network stages or when hybrid training is used, we can train models with high prediction accuracy, smaller memory footprint, and higher performance.For example, in contrast to PipeDream and DDG, we do not maintain multiple copies of weights, reducing memory footprint. In contrast to GPipe and Huo et al. [25], our approach has no pipeline bubbles and does not replicate computations, resulting in better performance. Further, compared to Chen et al. [11], our pipelined training can produce models with a final quality that is comparable to the standard model quality for VGG-16 and ResNet with different depths on CIFAR-10/CIFAR-100 datasets.


## 4. Proposed Pipelined Training Method

### 4.1. Pipelined Backpropagation

We illustrate our pipelined backpropagation implementation with theL layer network shown in Figure 3, using conceptual pipeline registers. Two registers are inserted between layers l and l+1, one register for the forward pass and a second for the backward pass. The forward register stores the activations of layer lxl. The backward register stores the gradients δl+1 of layer l+1. This defines a 4-stage pipelined backpropagation. The forward pass for layers 1 to l forms forward stage FS1. The forward pass for layers l+1 to L forms forward stage FS2. Similarly, the backward pass for layers l+1 to L and 1 to l forms backward stages BKS1 and BKS2, respectively.Figure 3
Pipelined backpropagation algorithm.The forward and backward stages are executed in a pipelined fashion on 3 accelerators: one forFS1, one for both FS2 and BKS1, and one for BKS2 (we combine FS2 and BKS1 on the same accelerator to reduce weight staleness, as will become evident shortly). In cycle 0, minibatch 0 is fed to FS1. The computations of the forward pass are done as in the traditional nonpipelined implementation. In cycle 1, layer l activations xl are fed to FS2 and minibatch 1 is fed to FS1. In cycle 2, the error for minibatch 0 computed in FS2 is directly fed to BKS1, the activations of layer lxl are forwarded to FS2, and minibatch 2 is fed to FS1. This pipelined execution is illustrated by the space-time diagram in Figure 4 for 5 minibatches. The figure depicts the minibatch processed by accelerator cycles 0 to 6. At steady state, all the accelerators are active in each cycle of execution.Figure 4
Illustration of pipelined computations of each cycle.The above pipelining scheme utilizes weights inFS1 that are yet to be updated by the errors calculated by FS2 and BKS1. At steady state, the activations of a minibatch in FS1 are calculated using weights that are 2 execution cycles old or 2 cycles stale. This is reflected in Figure 4 by indicating the weights used by each forward stage and the weights updated by each backward stage. The weights of a forward stage are subscripted by how stale they are (negative subscripts). Similarly, the weights updated by a backward stage are subscripted by how delayed they are (positive subscripts).Further, since the updates of the weights byBKS2 require activations calculated for the same minibatch in FS1 for all layers in the stage, it is necessary to save these activations until the error gradients with respect to the weights are calculated by BKS2. Only when the weights are updated using the gradients can these activations be discarded.In the general case, we useK pairs of pipeline registers (each pair consisting of a forward register and a backward register) inserted between the layers of the network. We describe the placement of the register pairs by the pipeline placement vector, PPV=p1,p2,…,pK, where pi represents the layer number after which a pipeline register pair is inserted. Such a placement creates K+1 forward stages, labeled FSi,i=1,2,…,K+1, and K+1 backward stages, labeled BKSi,i=1,2,…,K+1. Forward stage FSi and backward stage BKSK−i+2 correspond to the same set of layers. Specifically, stage FSi contains layers pi+1 to pi+1, which are inclusive. We assign each forward stage and each backward stage to an accelerator, with the exception of the FSK+1 and backward stage BKS1, which are assigned to the same accelerator to reduce weight staleness by an execution cycle. In total, 2K+1 accelerators are used.We quantify weight staleness as follows. A forward stageFSi and backward stage BKSK−i+2 use the same weights that are 2K−i+1 cycles old. Further, a forward stage FSi must store the activations of all layers in the stage for all 2K−i+1 cycles which are used for the corresponding backward stage BKSK−i+2. We refer to these saved activations as intermediate activations. We define the degree of staleness as 2K−i+1. For each pair of stages FSi and BKSK−i+2, let there be Ni weights in their corresponding layers. The layers before the last pipeline register pairs always use stale weights. Thus, we define percentage of stale weight as ∑i=1KNi/∑i=1K+1Ni.On the one hand, the above pipelined execution allows a potential speedup of2K+1, using as many accelerators, over the nonpipelined implementation, keeping all the accelerators active at steady state. On the other hand, the use of stale weights may prevent training convergence or may result in a model that has an inferior inference accuracy. Further, it requires an increase in storage for activations. Our goal is to assess the benefit of this pipelined execution and the impact of its downsides.Appendix C presents an analytical proof of the convergence of our pipelined training scheme.


### 4.2. Hybrid Pipelined/Nonpipelined Backpropagation

Hybrid training combines pipelined training with nonpipelined training. We start with pipelined training and after a number of iterations, we switch to nonpipelined training. This can address drops in inference accuracy of resulting models because of weight staleness, but it reduces the performance benefit since during nonpipelined training, the accelerators are underutilized.The extent of the speedup obtained by hybrid training with a given number of accelerators is determined by the number of iterations used for pipelined and nonpipelined training. Assume thatnnp iterations are used to reach the best inference accuracy for nonpipelined training, and that in hybrid training, np iterations np≤nnp are pipelined followed by nnp−np iterations of nonpipelined training to reach the same inference accuracy as nonpipelined training. The speedup of hybrid training with respect to the nonpipelined training with 2K+1 accelerators is nnp/np/2K+1+nnp−np. For large K, the speedup approaches an upper bound of nnp/nnp−np.


## 4.1. Pipelined Backpropagation

We illustrate our pipelined backpropagation implementation with theL layer network shown in Figure 3, using conceptual pipeline registers. Two registers are inserted between layers l and l+1, one register for the forward pass and a second for the backward pass. The forward register stores the activations of layer lxl. The backward register stores the gradients δl+1 of layer l+1. This defines a 4-stage pipelined backpropagation. The forward pass for layers 1 to l forms forward stage FS1. The forward pass for layers l+1 to L forms forward stage FS2. Similarly, the backward pass for layers l+1 to L and 1 to l forms backward stages BKS1 and BKS2, respectively.Figure 3
Pipelined backpropagation algorithm.The forward and backward stages are executed in a pipelined fashion on 3 accelerators: one forFS1, one for both FS2 and BKS1, and one for BKS2 (we combine FS2 and BKS1 on the same accelerator to reduce weight staleness, as will become evident shortly). In cycle 0, minibatch 0 is fed to FS1. The computations of the forward pass are done as in the traditional nonpipelined implementation. In cycle 1, layer l activations xl are fed to FS2 and minibatch 1 is fed to FS1. In cycle 2, the error for minibatch 0 computed in FS2 is directly fed to BKS1, the activations of layer lxl are forwarded to FS2, and minibatch 2 is fed to FS1. This pipelined execution is illustrated by the space-time diagram in Figure 4 for 5 minibatches. The figure depicts the minibatch processed by accelerator cycles 0 to 6. At steady state, all the accelerators are active in each cycle of execution.Figure 4
Illustration of pipelined computations of each cycle.The above pipelining scheme utilizes weights inFS1 that are yet to be updated by the errors calculated by FS2 and BKS1. At steady state, the activations of a minibatch in FS1 are calculated using weights that are 2 execution cycles old or 2 cycles stale. This is reflected in Figure 4 by indicating the weights used by each forward stage and the weights updated by each backward stage. The weights of a forward stage are subscripted by how stale they are (negative subscripts). Similarly, the weights updated by a backward stage are subscripted by how delayed they are (positive subscripts).Further, since the updates of the weights byBKS2 require activations calculated for the same minibatch in FS1 for all layers in the stage, it is necessary to save these activations until the error gradients with respect to the weights are calculated by BKS2. Only when the weights are updated using the gradients can these activations be discarded.In the general case, we useK pairs of pipeline registers (each pair consisting of a forward register and a backward register) inserted between the layers of the network. We describe the placement of the register pairs by the pipeline placement vector, PPV=p1,p2,…,pK, where pi represents the layer number after which a pipeline register pair is inserted. Such a placement creates K+1 forward stages, labeled FSi,i=1,2,…,K+1, and K+1 backward stages, labeled BKSi,i=1,2,…,K+1. Forward stage FSi and backward stage BKSK−i+2 correspond to the same set of layers. Specifically, stage FSi contains layers pi+1 to pi+1, which are inclusive. We assign each forward stage and each backward stage to an accelerator, with the exception of the FSK+1 and backward stage BKS1, which are assigned to the same accelerator to reduce weight staleness by an execution cycle. In total, 2K+1 accelerators are used.We quantify weight staleness as follows. A forward stageFSi and backward stage BKSK−i+2 use the same weights that are 2K−i+1 cycles old. Further, a forward stage FSi must store the activations of all layers in the stage for all 2K−i+1 cycles which are used for the corresponding backward stage BKSK−i+2. We refer to these saved activations as intermediate activations. We define the degree of staleness as 2K−i+1. For each pair of stages FSi and BKSK−i+2, let there be Ni weights in their corresponding layers. The layers before the last pipeline register pairs always use stale weights. Thus, we define percentage of stale weight as ∑i=1KNi/∑i=1K+1Ni.On the one hand, the above pipelined execution allows a potential speedup of2K+1, using as many accelerators, over the nonpipelined implementation, keeping all the accelerators active at steady state. On the other hand, the use of stale weights may prevent training convergence or may result in a model that has an inferior inference accuracy. Further, it requires an increase in storage for activations. Our goal is to assess the benefit of this pipelined execution and the impact of its downsides.Appendix C presents an analytical proof of the convergence of our pipelined training scheme.


## 4.2. Hybrid Pipelined/Nonpipelined Backpropagation

Hybrid training combines pipelined training with nonpipelined training. We start with pipelined training and after a number of iterations, we switch to nonpipelined training. This can address drops in inference accuracy of resulting models because of weight staleness, but it reduces the performance benefit since during nonpipelined training, the accelerators are underutilized.The extent of the speedup obtained by hybrid training with a given number of accelerators is determined by the number of iterations used for pipelined and nonpipelined training. Assume thatnnp iterations are used to reach the best inference accuracy for nonpipelined training, and that in hybrid training, np iterations np≤nnp are pipelined followed by nnp−np iterations of nonpipelined training to reach the same inference accuracy as nonpipelined training. The speedup of hybrid training with respect to the nonpipelined training with 2K+1 accelerators is nnp/np/2K+1+nnp−np. For large K, the speedup approaches an upper bound of nnp/nnp−np.


## 5. Implementation

We implement pipelined training in two ways: simulated in Caffe [31] (version 1.0.0), where the whole training process is performed on one process with no parallelism, and actual with parallelism across accelerators in PyTorch [32] (version 1.0.0.dev20190327).The simulated implementation is used to analyze statistical convergence, inference accuracy, and impact of weight staleness, for a large number of stages/accelerators, unconstrained by parallelism and communication overhead. In contrast, the actual implementation reports real performance and serves as a proof-of-concept implementation that demonstrates the performance potential of pipelined training with stale weights. PyTorch is used instead of Caffe to leverage its support for collective communication protocols and its flexibility in partitioning a network across multiple accelerators. The versions of Caffe and PyTorch we use have no support for pipelined training. Thus, both were extended to provide such support.We develop a custom Caffe layer in Python, which we call a Pipeline Manager Layer (PML), to facilitate the simulated pipelining. During the forward pass, a PML registers the input from a previous layer and passes the activation to the next layer. It also saves the activations for the layers connected to it to be used in the backward pass. During the backward pass, a PML passes the appropriate error gradients. It uses the corresponding activations saved during the forward pass to update weights and generate error gradients for the previous stage, using existing weight update mechanisms in Caffe.To implement actual hardware-accelerated pipelined training, we partition the network onto different accelerators (GPUs), each running its own process. Activation and gradient data are communicated among accelerators using an asynchronous send/receive communication protocol, but all communication must go through the host CPU, since point-to-point communication between accelerators is not supported in PyTorch. This increases communication overhead. Similar to the PMLs in Caffe, the activations computed on one GPU are copied to the next GPU (via the CPU) in the forward pass and the error gradients are sent (again via the CPU) to the preceding GPU during the backward pass. The GPUs are running concurrently, achieving pipeline parallelism.


## 6. Evaluation

### 6.1. Setup, Methodology, and Metrics

Simulated pipelining is evaluated on a machine with one Nvidia GTX1060 GPU with 6 GB of memory and an Intel i9-7940X CPU with 64 GB of RAM. The performance of actual pipelining is evaluated using two Nvidia GTX1060 GPUs, each with 6 GB of memory, hosted in an Intel i7-9700K machine with 32 GB of RAM.We use a number of CNNs and datasets in our evaluation. For the simulated implementation, we use LeNet-5 [33] trained on MNIST [34], AlexNet [35], VGG-16 [36], and ResNet [1], all trained on CIFAR-10 [37]. We evaluate simulated pipelining with these networks/datasets to manage simulation time. For the actual implementation, we experiment with progressively larger ResNet depths: 56, 110, 224, and 362, trained on CIFAR-10 and CIFAR-100 [38]. In both the simulated and actual implementations, we train the networks following their original setting [1, 33, 35, 36], with minor variations to the hyperparameters, as described in Appendix A and Appendix B.We elect to use the above CNNs for two reasons. First, they are commonly used in the evaluation of pipelined training (e.g., VGG in PipeDream [9] and ResNet in GPipe [8], which we compare to in our evaluation). Second, these networks have increasing sizes, ranging from the small LeNet to the large VGG and the progressively larger ResNets. This range in size allows us to effectively assess the impact of stale weight on pipelined training. We leave the use of larger networks, such as BERT [39] or DLRM [40] to future work.We evaluate the effectiveness of pipelined training in terms of its training convergence and its Top-1 inference accuracy, compared to those of the nonpipelined training. We use the speedup to evaluate performance improvements. The speedup is defined as the ratio of the training time of the nonpipelined implementation on single communication-free GPU to the training time of the pipelined training.


### 6.2. Training Convergence and Inference Accuracy

Pipelined training is done using 4, 6, 8, and 10 pipeline stages. Table1 shows where the registers are inserted in the networks using their PPVs (defined in Section 4). Pipeline registers are inserted among groups of convolutional layers, resulting up to 8 pipeline stages for AlexNet and ResNet-20 and 10 pipeline stages for LeNet-5 and VGG-16.Table 1
Pipeline placement vectors for CNNs.
CNNNumber of layers4-Stage6-Stage8-Stage10-StageLeNet-55(1)(1, 2)(1, 2, 3)(1, 2, 3, 4)AlexNet8(1)(1, 2)(1, 2, 3)—VGG-1616(2)(2, 4)(2, 4, 7)(2, 4, 7, 10)ResNet-2020(7)(7, 13)(7, 13, 19)—Figure5 shows the improvements in the inference accuracies for both pipelined and nonpipelined training as a function of the number of training iterations (each iteration corresponds to a minibatch). The figure shows that for all the networks, both pipelined training and nonpipelined training have similar convergence patterns. They converge in more or less the same number of iterations for a given number of pipeline stages, albeit different inference accuracies. This indicates that our approach to pipelined training with stale weights does converge, similar to nonpipelined training.Figure 5
Inference accuracy curves for simulated pipelined training. (a) LeNet-5 on MNIST. (b) AlexNet on CIFAR-10. (c) VGG-16 on CIFAR-10. (d) ResNet-20 on CIFAR-10.
(a)(b)(c)(d)Table2 shows the inference accuracies obtained after up to 30,000 iterations of training. For LeNet-5, the inference accuracy drop is within 0.5%. However, for the other networks, there is a small drop in inference accuracy with 4 and 6 stages. AlexNet has about 4% drop in inference accuracy, but for VGG-16, the inference accuracy drop is within 2.4%, and for ResNet-20, the accuracy drop is within 3.5%. Thus, the resulting model quality is generally comparable to that of a nonpipelining-trained model.Table 2
Inference accuracy for simulated pipelined training.
CNNNonpipelined (%)4-Stage (%)6-Stage (%)8-Stage (%)10-Stage (%)LeNet-599.0098.6498.6298.6198.47AlexNet82.5178.4778.3278.47—VGG-1691.3690.5388.9683.7379.85ResNet-2091.5090.0588.0083.01—However, with deeper pipelining (i.e., 8 and 10 stages), inference accuracies significantly drop. There is a 12% and a 8.5% inference accuracy drop for VGG-16 and ResNet-20, respectively. In this case, the model quality is not comparable to that of the nonpipelined training. This result confirms what is reported in the literature [9] and is attributed to the use of stale weights.


### 6.3. Impact of Weight Staleness

We wish to better understand the impact of the number of pipeline stages and their location in the network on inference accuracy. We focus on ResNet-20 because of its relatively small size and regular structure. It consists of 3 residual function groups with 3 residual function blocks within each group. In spite of this relatively small size and regular structure, it enables us to create pipelines with up to 20 stages by inserting pipeline register pairs within residual function blocks.We conduct two experiments. In the first, we increase the number of pipeline stages (from earlier layers to latter layers) and measure the inference accuracy of the resulting model. The results are shown in Table3, which gives the inference accuracy of pipelined training after 100,000 iterations, as the number of pipeline stages increases. The 8-stage pipelined training is created by a PPV of (3, 5, 7), and the subsequent pipeline schemes are created by adding pipeline registers after every 2 layers after layer 7. Clearly, the greater the number of stages is, the worse the resulting model quality is.Table 3
Fine-grained pipelining inference accuracy.
StagesInference accuracy (%)Nonpipelined91.50890.281088.371288.731487.941687.301886.232079.09The number of stale weights used in the pipelined training increases as the number of pipeline stages increases. Thus, Figure6 depicts the inference accuracy as a function of the percentage of weights that are stale. The curve labeled “increasing stages” shows that the drop in inference accuracy increases as the percentage of stale weights increases.Figure 6
Inference accuracy vs % of stale weights.In the second experiment, we investigate the impact of the degree of staleness described in Section4. Only one pair of pipeline registers is inserted. The position of this register slides from the beginning of the network to its end. At every position, the percentage of stale weights remains the same as in the first experiment, but all stale weights have the same degree of staleness. The result of this experiment is shown by the curve labeled “sliding stage” in Figure 6. The curve shows the inference accuracy also drops as the percentage of stale weights increases. However, it also indicates that the drop of inference accuracy remains more or less the same as in the first experiment in which the degree of staleness is higher. Thus, the percentage of stale weight appears to be what determines the drop in inference accuracy and not the degree of staleness of the weights.The percentage of stale weights is determined by where the last pair of pipeline registers are placed in the network. It is the position of this pair that determines the loss in inference accuracy. Therefore, it is desirable to place this last pair of registers as early as possible in the network so as to minimize the drop in inference accuracy.While at first glance this may seem to limit pipelining, it is important to note that the bulk of computations in a CNN is in the first few convolutional layers in the network. Inserting pipeline registers for these early layers can result in a large number of stages that are computationally balanced. For example, our profiling of the runtime of ResNet-20 shows that the first three residual functions take more than 50% of the training runtime. This favors more pipeline stages at the beginning of the network. Such placement has the desirable effect of reducing the drop in inference accuracy while obtaining relatively computationally balanced pipeline stages.


### 6.4. Effectiveness of Hybrid Training

We demonstrate the effectiveness of hybrid training, also using ResNet-20. Figure7 shows the inference accuracy for 20 K iterations of pipelined training followed by either 10 K or 20 K iterations of nonpipelined training. This inference accuracy is compared to 30 K iterations of either nonpipelined or pipelined training with PPV (5, 12, 17). The figure demonstrates that hybrid training converges in a similar manner to both pipelined and nonpipelined training. Table 4 shows the resulting inference accuracies. The table shows that the 20 K + 10 K hybrid training produces a model with accuracy that is comparable to that of the nonpipelined model. Further, with an additional 10 K iterations of nonpipelined training, the model quality is slightly better than that of the nonpipelined model. This demonstrates the effectiveness of hybrid training.Figure 7
Hybrid training convergence.Table 4
Hybrid training inference accuracy.
Inference accuracy (%)Baseline (30 K)91.50Pipelined (30 K)88.2920 K + 10 K hybrid90.7120 K + 20 K hybrid91.72


### 6.5. Pipelined and Hybrid Training Performance

Our evaluation using simulated pipelining explored pipelines with up to 20 pipeline stages (up to 10 accelerators). In this section, we implement and evaluate a proof-of-concept implementation with actual pipelining. The goal is to demonstrate that pipelined training with stale weights, with and without hybrid training, does deliver performance improvements.Specifically, we implement 4-stage pipelined training for ResNet-56/110/224/362 on a 2-GPU system. Each GPU is responsible for one forward stage and one backward stage. Thus, the maximum speedup that can be obtained is 2. We train every ResNet for 200 epochs for CIFAR-10 dataset and 300 epochs for CIFAR-100 dataset. Tables5 and 6 show the inference accuracies with and without pipelining, for the CIFAR-10 and CIFAR-100 datasets. They also show the speedups of pipelined training over the nonpipelined one. The tables indicate that the quality of the models produced by pipelined training is comparable to those achieved by the simulated pipelining implementation. The tables also show that speedup exists for all networks. Indeed, for ResNet-362, the speedup is 1.8X. This is equivalent to about 90% utilization for each GPU. Finally, the tables reflect that as the networks get larger, the speedup improves. This is because for larger networks, the ratio of computation to communication overhead is higher, leading to better speedups.Table 5
Inference accuracy and speedup of actual pipelined/hybrid training for CIFAR-10.
ResNetPPVAccuracyTime (seconds)SpeedupNonpipelined (%)Pipelined (%)Hybrid (%)NonpipelinedPipelinedHybridPipelinedHybrid-56(19)92.6392.8992.756,7454,0905,4291.65X1.24X-110(37)93.5992.8893.5513,1507,57010,4521.73X1.26X-224(75)92.7791.3993.3327,23114,99821,2451.81X1.28X-362(121)93.4690.5393.9844,81424,64034,8141.82X1.29XTable 6
Inference accuracy and speedup of actual pipelined/hybrid training for CIFAR-100.
ResNetPPVAccuracyTime (seconds)SpeedupNonpipelined (%)Pipelined (%)Hybrid (%)NonpipelinedPipelinedHybridPipelinedHybrid-56(19)72.3070.2572.3410,0686,0408,1131.67X1.24X-110(37)72.6371.5273.0319,62411,16215,5911.76X1.26X-224(75)73.0070.9072.6940,80722,56132,1321.81X1.28X-362(121)71.7771.0372.0966,97737,00851,6881.81X1.29XMoreover, we combine the 4-stage pipelined training described above with nonpipelined training to demonstrate the performance of hybrid training. We train every ResNet using pipelined training for 100 epochs and 150 epochs and follow it up by 100 epochs and 150 epochs of nonpipelined training for CIFAR-10 and CIFAR-100, respectively. Because the maximum speedup for the pipelined training is 2 and only half the training epochs is accelerated, the maximum speedup for this hybrid training iss=t/t/2+t/4=1.33, where t is the training time of nonpipelined training. Tables 5 and 6 also show the inference accuracies and speedup of the hybrid training for each ResNet and validate that hybrid training can produce a model quality that is comparable to the baseline nonpipelined training while speeding up the training process. Indeed, hybrid training can sometimes produce models with slightly better inference accuracies. Similar to pipelined training, as network size grows, the speedup of hybrid training reaches 1.29X, approaching the theoretical limit of 1.33X.


### 6.6. Memory Usage

Pipelined training requires the saving of intermediate activations, as described earlier in Section4, leading to an increase in memory footprint. This increase in memory is a function of not only the placement of the pipeline registers but also of the network architecture, input size, and the minibatch size. We calculate the memory usage of the 4-stage pipelined ResNet training above to show that this increase is modest for our pipelining scheme. Specifically, we use torchsummary in PyTorch to report memory usage for weights and activations for a network and calculate the additional memory required by the additional copies of activations. Assuming a batch size of 128, the percentage increase in memory usage is no more than 58% for ResNet-56/110/224/362 (see Table 7). More analysis of memory increase appears in Appendix D.Table 7
Memory usage of 4-stage pipelined ResNet training.
CNNPPVActivationsWeights (MB)IncreaseIncrease percentage in upper boundResNet-56(19)10.87 MB × batch size3.256.32 MB × batch size58ResNet-110(37)21.43 MB × batch size6.5912.35 MB × batch size57ResNet-224(75)43.70 MB × batch size13.6425.07 MB × batch size57ResNet-362(121)70.67 MB × batch size22.1740.50 MB × batch size57


### 6.7. Comparison to Existing Work

We compare our pipelined training scheme with two key existing systems: PipeDream [9] and GPipe [8]. We believe that PipeDream and GPipe are representative of existing key approaches that implement pipelined training, including decoupled backpropagation (DDG) [12] and feature replay (FR) [25] (discussed in Section 3). We compare on the basis of three aspects: the pipelining scheme, performance, and memory usage.Our pipelining scheme is simpler than that of PipeDream and GPipe in that we do not require weight stashing nor do we divide minibatches into microbatches. This leads to less communication overhead and is amicable to rapid realization in machine learning framework such as PyTorch or in actual hardware such as Xilinx’s xDNN FPGA accelerators [41].Our pipelining scheme, as PipeDream, eliminates bubbles that exist in the pipeline leading to better performance. For example, we obtain a speedup of 1.7X for ResNet-110 using 2 GPUs in contrast to GPipe that obtains a speedup of roughly 1.3X for ResNet-101 using 2 TPUs. We also obtain similar performance compared to PipeDream for similar networks. When the number of pipeline stages grows, pipeline bubbles exhibit more negative effect on performance shown in GPipe on a 4-partition pipelined ResNet-101 using 4 TPUs as its bubble overhead doubled compared to that of the 2-partition pipelined ResNet-101.Our scheme uses less memory compared to PipeDream, although it introduces more memory overhead compared to GPipe. PipeDream saves intermediate activations during training, as we do. However, it also saves multiple copies of a network’s weights for weight stashing. The memory footprint increase due to weight stashing depends on the total weight memory compared to activation memory, the number of active minibatches in the training pipeline, the minibatch size, and the training dataset. In some cases, weight stashing can have a significant impact on memory footprint. For example, for AlexNet trained on CIFAR-10 with a minibatch size of 128 using a 4-stage pipelined training, in which the weight memory is much larger than the activation memory, PipeDream’s memory footprint increase is 177% more than ours. A more detailed memory usage comparison is presented in Appendix D.


## 6.1. Setup, Methodology, and Metrics

Simulated pipelining is evaluated on a machine with one Nvidia GTX1060 GPU with 6 GB of memory and an Intel i9-7940X CPU with 64 GB of RAM. The performance of actual pipelining is evaluated using two Nvidia GTX1060 GPUs, each with 6 GB of memory, hosted in an Intel i7-9700K machine with 32 GB of RAM.We use a number of CNNs and datasets in our evaluation. For the simulated implementation, we use LeNet-5 [33] trained on MNIST [34], AlexNet [35], VGG-16 [36], and ResNet [1], all trained on CIFAR-10 [37]. We evaluate simulated pipelining with these networks/datasets to manage simulation time. For the actual implementation, we experiment with progressively larger ResNet depths: 56, 110, 224, and 362, trained on CIFAR-10 and CIFAR-100 [38]. In both the simulated and actual implementations, we train the networks following their original setting [1, 33, 35, 36], with minor variations to the hyperparameters, as described in Appendix A and Appendix B.We elect to use the above CNNs for two reasons. First, they are commonly used in the evaluation of pipelined training (e.g., VGG in PipeDream [9] and ResNet in GPipe [8], which we compare to in our evaluation). Second, these networks have increasing sizes, ranging from the small LeNet to the large VGG and the progressively larger ResNets. This range in size allows us to effectively assess the impact of stale weight on pipelined training. We leave the use of larger networks, such as BERT [39] or DLRM [40] to future work.We evaluate the effectiveness of pipelined training in terms of its training convergence and its Top-1 inference accuracy, compared to those of the nonpipelined training. We use the speedup to evaluate performance improvements. The speedup is defined as the ratio of the training time of the nonpipelined implementation on single communication-free GPU to the training time of the pipelined training.


## 6.2. Training Convergence and Inference Accuracy

Pipelined training is done using 4, 6, 8, and 10 pipeline stages. Table1 shows where the registers are inserted in the networks using their PPVs (defined in Section 4). Pipeline registers are inserted among groups of convolutional layers, resulting up to 8 pipeline stages for AlexNet and ResNet-20 and 10 pipeline stages for LeNet-5 and VGG-16.Table 1
Pipeline placement vectors for CNNs.
CNNNumber of layers4-Stage6-Stage8-Stage10-StageLeNet-55(1)(1, 2)(1, 2, 3)(1, 2, 3, 4)AlexNet8(1)(1, 2)(1, 2, 3)—VGG-1616(2)(2, 4)(2, 4, 7)(2, 4, 7, 10)ResNet-2020(7)(7, 13)(7, 13, 19)—Figure5 shows the improvements in the inference accuracies for both pipelined and nonpipelined training as a function of the number of training iterations (each iteration corresponds to a minibatch). The figure shows that for all the networks, both pipelined training and nonpipelined training have similar convergence patterns. They converge in more or less the same number of iterations for a given number of pipeline stages, albeit different inference accuracies. This indicates that our approach to pipelined training with stale weights does converge, similar to nonpipelined training.Figure 5
Inference accuracy curves for simulated pipelined training. (a) LeNet-5 on MNIST. (b) AlexNet on CIFAR-10. (c) VGG-16 on CIFAR-10. (d) ResNet-20 on CIFAR-10.
(a)(b)(c)(d)Table2 shows the inference accuracies obtained after up to 30,000 iterations of training. For LeNet-5, the inference accuracy drop is within 0.5%. However, for the other networks, there is a small drop in inference accuracy with 4 and 6 stages. AlexNet has about 4% drop in inference accuracy, but for VGG-16, the inference accuracy drop is within 2.4%, and for ResNet-20, the accuracy drop is within 3.5%. Thus, the resulting model quality is generally comparable to that of a nonpipelining-trained model.Table 2
Inference accuracy for simulated pipelined training.
CNNNonpipelined (%)4-Stage (%)6-Stage (%)8-Stage (%)10-Stage (%)LeNet-599.0098.6498.6298.6198.47AlexNet82.5178.4778.3278.47—VGG-1691.3690.5388.9683.7379.85ResNet-2091.5090.0588.0083.01—However, with deeper pipelining (i.e., 8 and 10 stages), inference accuracies significantly drop. There is a 12% and a 8.5% inference accuracy drop for VGG-16 and ResNet-20, respectively. In this case, the model quality is not comparable to that of the nonpipelined training. This result confirms what is reported in the literature [9] and is attributed to the use of stale weights.


## 6.3. Impact of Weight Staleness

We wish to better understand the impact of the number of pipeline stages and their location in the network on inference accuracy. We focus on ResNet-20 because of its relatively small size and regular structure. It consists of 3 residual function groups with 3 residual function blocks within each group. In spite of this relatively small size and regular structure, it enables us to create pipelines with up to 20 stages by inserting pipeline register pairs within residual function blocks.We conduct two experiments. In the first, we increase the number of pipeline stages (from earlier layers to latter layers) and measure the inference accuracy of the resulting model. The results are shown in Table3, which gives the inference accuracy of pipelined training after 100,000 iterations, as the number of pipeline stages increases. The 8-stage pipelined training is created by a PPV of (3, 5, 7), and the subsequent pipeline schemes are created by adding pipeline registers after every 2 layers after layer 7. Clearly, the greater the number of stages is, the worse the resulting model quality is.Table 3
Fine-grained pipelining inference accuracy.
StagesInference accuracy (%)Nonpipelined91.50890.281088.371288.731487.941687.301886.232079.09The number of stale weights used in the pipelined training increases as the number of pipeline stages increases. Thus, Figure6 depicts the inference accuracy as a function of the percentage of weights that are stale. The curve labeled “increasing stages” shows that the drop in inference accuracy increases as the percentage of stale weights increases.Figure 6
Inference accuracy vs % of stale weights.In the second experiment, we investigate the impact of the degree of staleness described in Section4. Only one pair of pipeline registers is inserted. The position of this register slides from the beginning of the network to its end. At every position, the percentage of stale weights remains the same as in the first experiment, but all stale weights have the same degree of staleness. The result of this experiment is shown by the curve labeled “sliding stage” in Figure 6. The curve shows the inference accuracy also drops as the percentage of stale weights increases. However, it also indicates that the drop of inference accuracy remains more or less the same as in the first experiment in which the degree of staleness is higher. Thus, the percentage of stale weight appears to be what determines the drop in inference accuracy and not the degree of staleness of the weights.The percentage of stale weights is determined by where the last pair of pipeline registers are placed in the network. It is the position of this pair that determines the loss in inference accuracy. Therefore, it is desirable to place this last pair of registers as early as possible in the network so as to minimize the drop in inference accuracy.While at first glance this may seem to limit pipelining, it is important to note that the bulk of computations in a CNN is in the first few convolutional layers in the network. Inserting pipeline registers for these early layers can result in a large number of stages that are computationally balanced. For example, our profiling of the runtime of ResNet-20 shows that the first three residual functions take more than 50% of the training runtime. This favors more pipeline stages at the beginning of the network. Such placement has the desirable effect of reducing the drop in inference accuracy while obtaining relatively computationally balanced pipeline stages.


## 6.4. Effectiveness of Hybrid Training

We demonstrate the effectiveness of hybrid training, also using ResNet-20. Figure7 shows the inference accuracy for 20 K iterations of pipelined training followed by either 10 K or 20 K iterations of nonpipelined training. This inference accuracy is compared to 30 K iterations of either nonpipelined or pipelined training with PPV (5, 12, 17). The figure demonstrates that hybrid training converges in a similar manner to both pipelined and nonpipelined training. Table 4 shows the resulting inference accuracies. The table shows that the 20 K + 10 K hybrid training produces a model with accuracy that is comparable to that of the nonpipelined model. Further, with an additional 10 K iterations of nonpipelined training, the model quality is slightly better than that of the nonpipelined model. This demonstrates the effectiveness of hybrid training.Figure 7
Hybrid training convergence.Table 4
Hybrid training inference accuracy.
Inference accuracy (%)Baseline (30 K)91.50Pipelined (30 K)88.2920 K + 10 K hybrid90.7120 K + 20 K hybrid91.72


## 6.5. Pipelined and Hybrid Training Performance

Our evaluation using simulated pipelining explored pipelines with up to 20 pipeline stages (up to 10 accelerators). In this section, we implement and evaluate a proof-of-concept implementation with actual pipelining. The goal is to demonstrate that pipelined training with stale weights, with and without hybrid training, does deliver performance improvements.Specifically, we implement 4-stage pipelined training for ResNet-56/110/224/362 on a 2-GPU system. Each GPU is responsible for one forward stage and one backward stage. Thus, the maximum speedup that can be obtained is 2. We train every ResNet for 200 epochs for CIFAR-10 dataset and 300 epochs for CIFAR-100 dataset. Tables5 and 6 show the inference accuracies with and without pipelining, for the CIFAR-10 and CIFAR-100 datasets. They also show the speedups of pipelined training over the nonpipelined one. The tables indicate that the quality of the models produced by pipelined training is comparable to those achieved by the simulated pipelining implementation. The tables also show that speedup exists for all networks. Indeed, for ResNet-362, the speedup is 1.8X. This is equivalent to about 90% utilization for each GPU. Finally, the tables reflect that as the networks get larger, the speedup improves. This is because for larger networks, the ratio of computation to communication overhead is higher, leading to better speedups.Table 5
Inference accuracy and speedup of actual pipelined/hybrid training for CIFAR-10.
ResNetPPVAccuracyTime (seconds)SpeedupNonpipelined (%)Pipelined (%)Hybrid (%)NonpipelinedPipelinedHybridPipelinedHybrid-56(19)92.6392.8992.756,7454,0905,4291.65X1.24X-110(37)93.5992.8893.5513,1507,57010,4521.73X1.26X-224(75)92.7791.3993.3327,23114,99821,2451.81X1.28X-362(121)93.4690.5393.9844,81424,64034,8141.82X1.29XTable 6
Inference accuracy and speedup of actual pipelined/hybrid training for CIFAR-100.
ResNetPPVAccuracyTime (seconds)SpeedupNonpipelined (%)Pipelined (%)Hybrid (%)NonpipelinedPipelinedHybridPipelinedHybrid-56(19)72.3070.2572.3410,0686,0408,1131.67X1.24X-110(37)72.6371.5273.0319,62411,16215,5911.76X1.26X-224(75)73.0070.9072.6940,80722,56132,1321.81X1.28X-362(121)71.7771.0372.0966,97737,00851,6881.81X1.29XMoreover, we combine the 4-stage pipelined training described above with nonpipelined training to demonstrate the performance of hybrid training. We train every ResNet using pipelined training for 100 epochs and 150 epochs and follow it up by 100 epochs and 150 epochs of nonpipelined training for CIFAR-10 and CIFAR-100, respectively. Because the maximum speedup for the pipelined training is 2 and only half the training epochs is accelerated, the maximum speedup for this hybrid training iss=t/t/2+t/4=1.33, where t is the training time of nonpipelined training. Tables 5 and 6 also show the inference accuracies and speedup of the hybrid training for each ResNet and validate that hybrid training can produce a model quality that is comparable to the baseline nonpipelined training while speeding up the training process. Indeed, hybrid training can sometimes produce models with slightly better inference accuracies. Similar to pipelined training, as network size grows, the speedup of hybrid training reaches 1.29X, approaching the theoretical limit of 1.33X.


## 6.6. Memory Usage

Pipelined training requires the saving of intermediate activations, as described earlier in Section4, leading to an increase in memory footprint. This increase in memory is a function of not only the placement of the pipeline registers but also of the network architecture, input size, and the minibatch size. We calculate the memory usage of the 4-stage pipelined ResNet training above to show that this increase is modest for our pipelining scheme. Specifically, we use torchsummary in PyTorch to report memory usage for weights and activations for a network and calculate the additional memory required by the additional copies of activations. Assuming a batch size of 128, the percentage increase in memory usage is no more than 58% for ResNet-56/110/224/362 (see Table 7). More analysis of memory increase appears in Appendix D.Table 7
Memory usage of 4-stage pipelined ResNet training.
CNNPPVActivationsWeights (MB)IncreaseIncrease percentage in upper boundResNet-56(19)10.87 MB × batch size3.256.32 MB × batch size58ResNet-110(37)21.43 MB × batch size6.5912.35 MB × batch size57ResNet-224(75)43.70 MB × batch size13.6425.07 MB × batch size57ResNet-362(121)70.67 MB × batch size22.1740.50 MB × batch size57


## 6.7. Comparison to Existing Work

We compare our pipelined training scheme with two key existing systems: PipeDream [9] and GPipe [8]. We believe that PipeDream and GPipe are representative of existing key approaches that implement pipelined training, including decoupled backpropagation (DDG) [12] and feature replay (FR) [25] (discussed in Section 3). We compare on the basis of three aspects: the pipelining scheme, performance, and memory usage.Our pipelining scheme is simpler than that of PipeDream and GPipe in that we do not require weight stashing nor do we divide minibatches into microbatches. This leads to less communication overhead and is amicable to rapid realization in machine learning framework such as PyTorch or in actual hardware such as Xilinx’s xDNN FPGA accelerators [41].Our pipelining scheme, as PipeDream, eliminates bubbles that exist in the pipeline leading to better performance. For example, we obtain a speedup of 1.7X for ResNet-110 using 2 GPUs in contrast to GPipe that obtains a speedup of roughly 1.3X for ResNet-101 using 2 TPUs. We also obtain similar performance compared to PipeDream for similar networks. When the number of pipeline stages grows, pipeline bubbles exhibit more negative effect on performance shown in GPipe on a 4-partition pipelined ResNet-101 using 4 TPUs as its bubble overhead doubled compared to that of the 2-partition pipelined ResNet-101.Our scheme uses less memory compared to PipeDream, although it introduces more memory overhead compared to GPipe. PipeDream saves intermediate activations during training, as we do. However, it also saves multiple copies of a network’s weights for weight stashing. The memory footprint increase due to weight stashing depends on the total weight memory compared to activation memory, the number of active minibatches in the training pipeline, the minibatch size, and the training dataset. In some cases, weight stashing can have a significant impact on memory footprint. For example, for AlexNet trained on CIFAR-10 with a minibatch size of 128 using a 4-stage pipelined training, in which the weight memory is much larger than the activation memory, PipeDream’s memory footprint increase is 177% more than ours. A more detailed memory usage comparison is presented in Appendix D.


## 7. Concluding Remarks

We propose and evaluate a pipelined execution scheme of backpropagation for the training of CNNs. The scheme uses stale weights, fully utilizes accelerators, does not significantly increase memory usage, and results in models with comparable prediction accuracies to those obtained with nonpipelined training.The use of stale weights has been recognized in the literature to significantly affect prediction accuracies. Thus, existing schemes avoid or limit the use of stale weights [7–9, 12]. In contrast, we explore the impact of stale weights and demonstrate that it is the placement of the last pair of pipeline registers that determines the loss in inference accuracy. This allows us to implement pipelining in the early layers of the network with little loss to accuracy while reaping computational benefits. Limiting pipelining to such early layers is not a disadvantage since the bulk of computations is in the early convolutional layers. Nonetheless, when deeper pipelining is desired, we introduce hybrid training and show that it is effective in mitigating the loss of prediction accuracy for deep pipelining, while still providing computational speedups. Our scheme has the advantage of simplicity and low memory overhead, making it attractive when accelerator memory is constrained, in particular for specialized hardware accelerators.Our evaluation using several CNN networks/datasets confirms that training with our scheme does converge and does produce models with inference accuracies that are comparable to those obtained with nonpipelined training. Our proof-of-concept implementation on a 2-GPU system shows that our scheme achieves a speedup of up to 1.82X, demonstrating its potential.This work can be extended in a number of directions. One direction is to evaluate the approach with a larger number of accelerators since pipelined parallelism is known to scale naturally with the number of accelerators. Another is to evaluate the approach on larger datasets, such as ImageNet. Finally, our pipelining scheme lends itself naturally to hardware implementation due to its simplicity. Thus, another direction for future work is to evaluate pipelined parallelism using Field Programmable Gate Array (FPGA) or ASIC accelerators.



---

*Source: 3839543-2021-09-22.xml*