# Pipelined Stochastic Gradient Descent with Taylor Expansion
## 2 Problem Definition and Related Work
GPipe [19] executes the forward and backward passes by the pipeline method at a level of micro-batch, which a mini-batch is further divided. However, though the problem of weight inconsistency is solved, its convergence and parallelization efficiencies may decrease depending on the size of the mini-batch and the number of computing nodes because the backward passes start after all the forward passes have been finished, i.e., some computing nodes may not participate in the computation during the forward-to-backward and backward-to-forward transition periods. DAPPLE [20] uses a gradient accumulation method in which the gradients of mini-batches are accumulated and the model parameters are updated once using the accumulated gradients. Though it avoids the weight inconsistency problem, this method has the effect of increasing the effective mini-batch size, thereby reducing the random walk phenomenon and lowering the accuracy of the DNNs. Figure 1: Pipelined parallelism. \(F_{n}^{t}\) and \(B_{n}^{t}\) represent the forward pass and backward pass, respectively, at computing node \(n\) for mini-batch index \(t\). SpecTrain [21] tries to prevent the weight inconsistency by predicting the future model parameters using the momentum technique and employing them in the forward and backward passes. Linear weight prediction [22] predicts the future model parameters using the momentum to circumvent the weight inconsistency problem as in SpecTrain. However, in these techniques, the higher the number of computing nodes, the more inaccurate the predicted model parameters become, resulting in limited parallelization efficiency. PipeMare [23] tries to mitigate the weight inconsistency problem by using estimated forward pass model parameters at the time of the backward pass and alleviates the delayed gradient problem by using smaller values of the learning rate. However, it does not completely solve the weight consistency problem, so the more computing nodes there are, the more unstable the model convergence becomes. Spike compensation [22] increases the contribution of the latest gradient in pipelined SGD with momentum. It has the effect that the overall contribution of each gradient matches the case of original SGD where weight inconsistencies do not occur. However, the training becomes unstable as the number of computing nodes increases. FTPipe [24] is a pipeline parallelism proposed for fine-tuning pre-trained models. It does not mitigate the weight inconsistency problem with the assumption that the pre-trained models are less sensitive to weight staleness. However, not taking the staleness into account at all leads to an accuracy drop. A potential solution for the weight inconsistency problem involves the creation of the same number of model replicas as that of time delays in the delayed gradient for each computing node and updating these replicas during the backward pass [25]. For example, PipeDream [26; 27] uses this method, with the name of weight stashing. Furthermore, the same number of model replicas as the largest number of delay occurrences at all computing nodes can be created such that all computing nodes can use the same previous time step parameters during the backward passes. It is called vertical sync in PipeDream [26]. This method can be considered as the combination of model parallelism and data parallelism. Therefore, additional synchronization of model parameters is required.