# Collective Communication Performance Evaluation for Distributed Deep Learning Training
## 7 Summary of Experimental Results  
### Linux Shell  
In Figure 22, the results of collective communication executed in the Linux shell for each environment are presented. In the case of MPI_Send, used for communication between the chief worker and the parameter server in the NCCL architecture, only the measured time for OpenMPI was displayed in the overall summary because the time required for MPICH and OpenMPI differed by less than 0.1 s.  
When examining the broadcast and gather (allgather) subroutines in the parameter server method, it is evident that MPI's performance is lower in bare metal and single-container setups compared to NCCL in broadcasting. However, the latency of NCCL increases rapidly in the cross-docker environment, resulting in higher latency than MPI. In the case of gather (allgather) measurements, MPI outperforms NCCL in all environments, with the gap widening further in cross docker, where it exhibits an 118% higher latency than OpenMPI. For all-reduce, it is noticeable that NCCL's performance is significantly better than that of MPI, with a substantial difference of 78% when compared to MPICH in the bare metal setup, representing the largest gap in performance among the tested scenarios. In comparing MPI and CUDA-aware MPI, it is evident that there is not a significant discrepancy in latency between the two architectures. This suggests that, when using a single server, the impact of latency appears to be more influenced by hardware architecture than direct transmission support.  
Table 7 summarizes the best-performing environment and library for each subroutine. In the broadcast subroutine, NCCL_Bcast performed in a single-docker environment achieved the best latency at 0.76 s, outperforming other scenarios. For the gather subroutine, the shortest latency was achieved using the CUDA-aware library in a bare metal environment, with a measurement of 2.225 s. In the ring all-reduce method, NCCL performed exceptionally well in a singularity environment, showing the best performance with a latency of 2.09 s. Overall, it is worth noting that the performance degradation is minimal in multi-GPU allocation environments. When comparing bare metal to single-container setups, all libraries exhibit similar or lower latencies.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Experiment** & **GPU Allocation Type** & **Environment** & **Library** & **Latency (s)** \\ \hline Broadcasting & multi-GPU per container & single docker & NCCL & 0.76 \\ Gathering & local server & bare metal & CUDA-aware & 2.22 \\ All-reduce & multi-GPU per container & singularity & NCCL & 2.09 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Best results of each experiment (Linux shell).  
Figure 22: Linux shell communication library architecture latency.  
Table 8 highlights the lowest performing environment and library for each subroutine. In both the broadcast and gather (allgather) subroutines of the parameter server method, NCCL performed in a cross-docker environment exhibited low performance, with latencies of 2.384 s and 5.135 s, respectively. This indicates that the performance of NCCL is significantly reduced in the single-GPU environment per container configuration of the parameter server method. For the all-reduce subroutine, the highest latency was measured in the MPICH bare metal environment, with a latency of 3.877 s. In contrast, NCCL performed much better in the same environment, with a latency of 0.892 s, showcasing a significant difference of 334%.