# Collective Communication Performance Evaluation for Distributed Deep Learning Training
## 5 Experiments in the Linux Shell Environment
### Linux Shell Function Execution Times  
In the Linux shell experiments, the latency for each execution step under the criterion of utilizing all available resources reflects the combined results of function calls from collective communication libraries and cudaMemcpy. The latencies for each process are shown in Tables 2-4 as follows. Table 2 displays the execution times of each function during the broadcast operation. In terms of bare metal benchmarks, for MPICH, the broadcast function takes 0.945 s, occupying 59% of the total time, while cudaMemcpy accounts for 0.653 s, representing 41%. Conversely, OpenMPI requires 1.235 s for the broadcast function, constituting 65% of the total time. Since CUDA-aware MPI does not include cudaMemcpy time, the broadcast \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Server** & **Backend** & **Broadcast Time (s)** & **CudaMemcpy Time (s)** & **MPI Send Time (s)** & **Total Latency (s)** \\ \hline \multirow{3}{*}{Bare metal} & MPICH & 0.945 (59\%) & 0.653 (41\%) & - & 1.598 \\  & OpenMPI & 1.235 (65\%) & 0.650 (35\%) & - & 1.885 \\  & CUDA-aware MPI & 1.840 (100\%) & - & - & 1.840 \\  & NCCL & 0.453 (45\%) & 0.171 (17\%) & 0.384 (38\%) & 1.008 \\ \hline \multirow{3}{*}{Singularity} & MPICH & 0.439 (40\%) & 0.649 (60\%) & - & 1.088 \\  & OpenMPI & 0.439 (40\%) & 0.650 (60\%) & - & 1.089 \\  & CUDA-aware MPI & 0.911 (100\%) & - & - & 1.091 \\  & NCCL & 0.438 (55\%) & 0.166 (20\%) & 0.196 (25\%) & 0.800 \\ \hline \multirow{3}{*}{Single Docker} & MPICH & 1.262 (66\%) & 0.650 (34\%) & - & 1.912 \\  & OpenMPI & 1.268 (66\%) & 0.647 (34\%) & - & 1.915 \\  & CUDA-aware MPI & 1.920 (100\%) & - & 1.920 \\  & NCCL & 0.420 (55\%) & 0.167 (22\%) & 0.173 (23\%) & 0.760 \\ \hline \multirow{3}{*}{Cross Docker} & MPICH & 1.257 (65\%) & 0.658 (35\%) & - & 1.915 \\  & OpenMPI & 1.277 (65\%) & 0.654 (35\%) & - & 1.931 \\ \cline{1-1}  & CUDA-aware MPI & 1.973 (100\%) & - & - & 1.973 \\ \cline{1-1}  & NCCL & 1.941 (81\%) & 0.167 (7\%) & 0.276 (12\%) & 2.384 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Linux shell function execution times (broadcast). Figure 17: Linux shell MPI CUDA-aware all-reduce latency. Figure 18: Linux shell NCCL all-reduce latency. function is used as a single process. NCCL performs operations through processes such as broadcast, cudaMemcpy, and MPI send, which transfer data from the parameter server to the chief worker. The time breakdown for NCCL is 0.453 s (45%) for the broadcast function, 0.171 s (17%) for cudaMemcpy, and 0.384 s (38%) for MPI send. When comparing against the server environment, in singularity, the overall function calls and cudaMemcpy times decrease, while in a cross-docker environment, the NCCL backend shows over four times longer broadcast function execution times. Table 3 shows the execution times of functions during the gather (allgather) operation. Both MPICH and OpenMPI consist of gather functions and cudaMemcpy. For MPICH, the gather subroutine takes 1.806 s (70% of the total time), while for OpenMPI, it takes 1.641 s (67%). In both cases, cudaMemcpy accounts for approximately 0.7 s (30%). CUDA-aware MPI takes 2.565 s for the entire process. NCCL demonstrates a lower execution time of 0.466 s (13%) on bare metal, but in a cross-docker environment, it takes 1.925 s, constituting 37% of the total time. Table 4 shows the execution times of functions in the all-reduce architecture. In MPI libraries, this includes the time for communication between CPU memories and the memory copy between the host (CPU) and the device (GPU). For other libraries, the time for cudaMemcpy is not included due to direct communication between GPUs. In CPU communication MPI (MPICH, OpenMPI), except for MPICH having 1.3 times longer all-reduce function time than OpenMPI in bare metal environments, overall, both libraries show similar latency. Additionally, in all libraries, including CUDA-aware MPI, there is a tendency for the all-reduce function call time to decrease in singularity. NCCL demonstrates superior overall performance compared to other libraries.