# Pipelined Stochastic Gradient Descent with Taylor Expansion
## 5 Conclusions  
In this study, a novel pipelined parallel SGD algorithm, TaylorPipe, is proposed to mitigate the weight inconsistency and delayed gradient problems that occur when implementing the inherently sequential SGD algorithm as pipeline parallelism to improve the training speed. It generates multiple model replicas to address the weight inconsistency problem and predicts gradients by using Taylor expansion to alleviate the delayed gradient problem. The experimental results confirmed that the convergence rate does not decrease even if the number of computing nodes increases, unlike in the conventional parallel SGD algorithms. Therefore, the proposed method is expected to train DNN models more efficiently in a large-scale parallel processing environment. Furthermore, the attempted combination of pipeline parallelism and data parallelism in this paper will serve as an example advising diverse approaches to address issues that arise during parallelizing deep learning. However, the proposed method \begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{**Algorithm**} & \multicolumn{2}{c}{**Error Rate (\%)**} & \multicolumn{2}{c}{**Training Time (Hours)**} \\ \cline{2-5}  & **VGG-16** & **ResNet-34** & **VGG-16** & **ResNet-34** \\ \hline SGD & 29.14 & 27.42 & 12.20 & 22.77 \\ PipeDream & 29.38 & 28.04 & 4.70 & 13.11 \\ SpecTrain & 30.50 & 31.05 & 4.60 & 9.28 \\ TaylorPipe & 29.46 & 27.09 & 4.43 & 8.69 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Image classification error rate (%) and training time (in hours) of each method using the CIFAR-100 dataset and the VGG-16 and ResNet-34 models on 8 GPUs. \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Algorithm** & **1-GPU** & **2-GPU** & **4-GPU** & **8-GPU** \\ \hline SGD & 293 & - & - & - \\ NaivePipe & - & 187 & 118 & 107 \\ PipeDream & - & 194 & 123 & 111 \\ SpecTrain & - & 213 & 129 & 112 \\ TaylorPipe & - & 200 & 126 & 110 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Average training time (in seconds) per epoch for each method using the CIFAR-10 dataset and the VGG-16 model for various numbers of GPUs. has a disadvantage in terms of memory efficiency because it creates multiple model replicas. For future work, we believe that applying memory optimization techniques in deep learning, such as gradient accumulation, gradient checkpointing, or mixed precision methods, and so on, may be explored to effectively address this problem. **Author Contributions:** [PERSON], I.Y. and [PERSON] contributed equally to the work. All authors have read and agreed to the published version of the manuscript. **Funding:** This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF), funded by the Ministry of Science, ICT, and Future Planning (NRF-2017R1E1A1A01078157). Also, it was supported by the NRF under project BK21 FOUR. **Institutional Review Board Statement:** Not applicable. **Informed Consent Statement:** Not applicable. **Data Availability Statement:** Not applicable. **Conflicts of Interest:** The authors declare no conflict of interest. * [PERSON] et al. (2023) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] YOLOV4_CSPBi: Enhanced Land Target Detection Model. _Land_**2023**, _12_, 1813. [CrossRef]
* [PERSON] (2004) [PERSON] Solving large scale linear prediction problems using stochastic gradient descent algorithms. In Proceedings of the Twenty-First International Conference on Machine Learning, Banff, AB, Canada, 4-8 July 2004; p. 116. * [PERSON] and [PERSON] (2018) [PERSON]; [PERSON] A Bayesian Perspective on Generalization and Stochastic Gradient Descent. In Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada, 30 April-3 May 2018. * [PERSON] et al. (2016) [PERSON]; [PERSON]; [PERSON]; [PERSON] Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27-30 June 2016; pp. 770-778.