# Pipelined Stochastic Gradient Descent with Taylor Expansion
## 4 Experiments
Another important factor that affects the overall training time is statistical efficiency, i.e., the number of epochs required to reach a desired level of accuracy. Since NaivePipe failed to converge, it finished early. In the eight-GPU experiments, SpecTrain and PipeDream are the slowest and the second slowest among the parallel algorithms and their error rates are much higher than the proposed methods. TaylorPipe not only shows the comparable error rate with SGD but also is 2.6 times faster than SGD. Table 3 shows the averaging training time per epoch for each method using the CIFAR-10 dataset and the VGG-16 model for various numbers of GPUs. For all parallel algorithms, the average training time per epoch decreases in proportion to the number of GPUs used. NaivePipe is the fastest because it does not perform any extra work to handle the weight inconsistency and delayed gradient problems. PipeDream is faster than SpecTrain because the synchronization period was set to be somewhat large. TaylorPipe is faster than SpecTrain since the number of operations for gradient prediction is smaller than the number of operations for weight prediction, i.e., TaylorPipe predicts the gradients during the backward pass only whereas SpecTrain predicts the weights both for the forward and backward passes. For all parallel algorithms, as the number of GPUs used increases, the effect of extra computation time to handle the weight inconsistency and delayed gradient problems diminishes and the overhead of communication time overwhelms, leading to comparable training time per epoch in the last column of Table 3. \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline \multirow{2}{*}{**Algorithm**} & \multicolumn{4}{c}{**Error Rate (\%)**} & \multicolumn{4}{c}{**Training Time (Hours)**} \\ \cline{2-9}  & **1-GPU** & **2-GPU** & **4-GPU** & **8-GPU** & **1-GPU** & **2-GPU** & **4-GPU** & **8-GPU** \\ \hline SGD & 6.73 & - & - & - & 11.96 & - & - & - \\ NaivePipe & - & 72.07 & 74.18 & 81.70 & - & 0.10 & 0.06 & 0.05 \\ PipeDream & - & 6.61 & 6.93 & 7.95 & - & 7.89 & 4.95 & 4.64 \\ SpecTrain & - & 7.03 & 7.07 & 7.43 & - & 7.86 & 5.30 & 4.65 \\ TaylorPipe & - & 6.64 & 6.63 & 6.70 & - & 8.35 & 5.26 & 4.51 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Image classification error rate (%) and training time (in hours) of each method using the CIFAR-10 dataset and the VGG-16 model for various numbers of GPUs. Table 4 shows the image classification error rate and training time of of each method using the CIFAR-100 dataset and the VGG-16 and ResNet-34 models on eight GPUs. In the VGG-16 experiment, the training time for PipeDream to reach the minimum error rate is the slowest among the pipelined SGD algorithms. SpecTrain is the one of the slowest algorithms and shows the worst error rate. The proposed method, TaylorPipe, shows comparable error rates to SGD, with about 2.7 times faster convergence speed. In the ResNet-34 experiment, PipeDream shows the slowest training time to reach the minimum error rate among the parallel algorithms. Since the synchronization period of PipeDream was set to 1, PipeDream should flush the pipeline more frequently to synchronize the master model with local models. It not only increases the training time per epoch but also uses computing nodes inefficiently. These are the major drawbacks of the pipelined SGD using the SSGD method. SpecTrain shows the second slowest training time and achieves the worst error rate of 31.05%. TaylorPipe shows the fastest training time among the parallel algorithms. Compared to SGD, the training time of TaylorPipe is 2.6 times faster. Furthermore, TaylorPipe recorded a similar error rate to that of SGD. Based on our experiments, the proposed method, TaylorPipe, shows robust performance in terms of both image classification error rate and the training time to reach the minimum error rate. Therefore, it can be inferred that TaylorPipe not only overcomes the weight inconsistency problem but also more efficiently alleviates the delayed gradient that occurs in the pipelined SGD.