# Pipelined Stochastic Gradient Descent with Taylor Expansion
## 4 Experiments  
Experiments were conducted to analyze the efficiency of the proposed pipelined SGD method by comparing it with the conventional parallel SGD algorithms in terms of the accuracy and training speed on an image classification task using two datasets: CIFAR-10 and CIFAR-100. VGG-16 [16] and ResNet-34 [9] were used as the DNN models for the image classification experiments. The training algorithms used in the comparative study include the conventional sequential SGD (denoted as SGD), the naive pipelined SGD that condones the weight inconsistency (denoted as NaivePipe), PipeDream [26], SpecTrain [21], and TaylorPipe. In order to analyze the effect of the weight inconsistency and delayed gradient problems only and compare the conventional methods with the proposed methods fairly, PipeDream and SpecTrain used the same work scheduling and model replicas as TaylorPipe, which were explained in Section 3.1. Our implementation of PipeDream computes the master model parameters by calculating the average of the local model parameters at every model synchronization period. Since this effectively becomes one of the SSGD methods, the delayed gradient problem does not occur. We implemented the aforementioned algorithms using the DNN library provided by the Compute Unified Device Architecture and compared their performance under the same hardware conditions. The performance of each algorithm was evaluated with a varying number of GPUs (one, \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Method**} & **Computation Complexity** & **Computation Complexity** & \multirow{2}{*}{**Communication Complexity**} \\  & **(Computing Node)** & **(Parameter Server)** & \\ \hline SSGD & \(\mathcal{O}(H^{2}LMT/N)\) & \(\mathcal{O}(H^{2}LT/\kappa)\) & \(\mathcal{O}(H^{2}LT/\kappa)\) \\ ASGD & \(\mathcal{O}(H^{2}LMT/N)\) & \(\mathcal{O}(H^{2}LT)\) & \(\mathcal{O}(H^{2}LT)\) \\ TaylorPipe & \(\mathcal{O}(H^{2}LMT/N)\) & - & \(\mathcal{O}(HMT)\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The computation and communication complexities of SSGD, ASGD, and TaylorPipe for training a fully connected DNN model using \(N\) computing nodes. \(H\) denotes the number of neurons in each layer, \(L\) represents the number of layers in the model, \(M\) indicates the number of instances in each mini-batch, and \(T\) is the number of mini-batches in the training data. two, four, and eight) on a server with eight NVIDIA RTX 2080 Ti GPUs. We conducted experiments five times with a different random seed and reported the best case. An image augmentation technique was applied to prevent overfitting. The hyperparameter settings for experiments can be found in Appendix A. The test error rate curves can be found in Appendix B. Table 2 shows the image classification error rate and training time of each training method using the CIFAR-10 dataset and the VGG-16 model for various numbers of GPUs. SGD shows the error rate of 6.73%. NaivePipe failed to converge because it does not cope with the weight inconsistency and delayed gradient problems. Both of the conventional pipelined SGD methods (PipeDream and SpecTrain) and the proposed pipelined SGD method (TaylorPipe) show similar error rates to SGD when two GPUs are used. However, the error rates increase as the number of GPUs used increases. Since PipeDream uses one of the SSGD methods, which increases the effective mini-batch size by the number of computing nodes, the enlarged effective mini-batch size will degrade the DNN model training. SpecTrain predicts the future model parameters to prevent the weight inconsistency, but the greater the number of computing nodes, the more inaccurate the predicted model parameters become, resulting in limited DNN model training. However, TaylorPipe shows stable performance regardless of the number of GPUs used. We suspect that the simple averaging method and the momentum-based future model prediction scheme are not as effective as the Taylor expansion-based future gradient prediction, in mitigating the delayed gradient problem.