# Collective Communication Performance Evaluation for Distributed Deep Learning Training
## 6 Experiments in the PyTorch Environment
### Deep Learning Experiment  
Based on the results measured for each architecture, experiments were conducted at the deep learning application stage. The time taken to train a ResNet-18 model on the Cifar-10 dataset with a batch size of 32 for 10 epochs was measured. The measured contents include the time taken for each collective communication function call and parameter averaging, as  
Figure 21: PyTorch ring all-reduce latency. Figure 20: PyTorch parameter server gather latency. well as the total training time. In the parameter server approach, both broadcast and gather operations were performed in a single iteration, hence the time for each was measured. Specifically, for gather, it includes the time taken to compute the average of aggregated parameters from all nodes at the parameter server. Furthermore, since NCCL communicates based on GPUs, the number of worker trainings at the parameter server is N-1 instead of N. As a result, you can see the difference in step size in Table 5.  
Table 5 presents the results of the parameter server architecture. In all environments, MPI outperformed GLOO, and NCCL demonstrated excellent performance in each collective communication function and computational process, considering the differences in architectures. A notable observation is that, similar to the trends observed in previous experiments, in the cross-docker environment, GLOO showed approximately 0.77 times lower latency compared to bare metal, resulting in approximately 0.82 times longer training time. In contrast, in NCCL, the improvement in latency in the cross-docker environment compared to bare metal was more than 1.4 times, showing a higher rate of increase than other libraries. Table 6 shows the results of deep learning performance in the all-reduce architecture. In this case, all three libraries have the same architecture. Similarly, NCCL, MPI, and GLOO demonstrate superior performance in all environments. In the multi-GPU per container environment, they all exhibit lower latency compared to bare metal, while in the single GPU per container environment, higher latency is observed. Particularly noteworthy is that NCCL shows a significant performance degradation, with an all-reduce time that is 1.76 times higher than bare metal. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Server** & **Backend** & **Steps** & **All-Reduce Time (s)** & **Training Time (s)** \\ \hline \multirow{3}{*}{Bare metal} & MPI & 391 & 291 (-) & 384 (-) \\  & GLOO & 391 & 531 (-) & 650 (-) \\  & NCCL & 391 & 119 (-) & 186 (-) \\ \hline \multirow{3}{*}{Singularity} & MPI & 391 & 282 (\(\times\)0.96) & 374 (\(\times\)0.97) \\  & GLOO & 391 & 505 (\(\times\)0.95) & 623 (\(\times\)0.95) \\  & NCCL & 391 & 96 (\(\times\)0.80) & 162 (\(\times\)0.86) \\ \hline \multirow{3}{*}{Single Docker} & MPI & 391 & 284 (\(\times\)0.97) & 378 (\(\times\)0.98) \\  & GLOO & 391 & 488 (\(\times\)0.91) & 607 (\(\times\)0.93) \\  & NCCL & 391 & 94 (\(\times\)0.79) & 164 (\(\times\)0.88) \\ \hline \multirow{3}{*}{Cross Docker} & MPI & 391 & 355 (\(\times\)1.21) & 451 (\(\times\)1.17) \\  & GLOO & 391 & 617 (\(\times\)1.16) & 750 (\(\times\)1.15) \\ \cline{1-1}  & NCCL & 391 & 210 (\(\times\)1.76) & 283 (\(\times\)1.51) \\ \hline \hline \end{tabular}
\end{table} 
Table 6: All-reduce architecture deep learning latency.