# Collective Communication Performance Evaluation for Distributed Deep Learning Training
## 8 Related Works  
Section 8 of the paper discusses and highlights the related research papers and their explanations, emphasizing the differences between those papers and the current study. One of the papers examined in this section focuses on analyzing latency in various communication libraries in both inter-node and intra-node environments [25]. It delves into the collective communication functions commonly used in distributed deep learning, providing a detailed investigation of each library's performance. However, this paper is limited to conducting experiments solely on bare metal, without considering performance analysis in a virtualized environment. As emphasized in our study, it is crucial to analyze performance not only in bare metal but also on cloud virtual servers, reflecting the real-world scenarios where distributed deep learning takes place. Another paper discussed in this section evaluated the performance of collective communication within a container environment [26]. This paper specifically measured and analyzed latency by replicating distributed deep learning architectures within docker container environments through Linux shell commands. However, it is important to note that measuring latency within the Linux shell environment may have limitations, as it differs from measuring collective communication performance in actual deep learning workloads. In contrast, our study addresses this limitation by analyzing communication latency in practical distributed deep learning scenarios, encompassing the collective communication library used in PyTorch alongside the Linux shell environment. This approach provides a more comprehensive understanding of real-world distributed deep learning communication performance. \begin{table}
\begin{tabular}{c c c c c} \hline
**Experiment** & **GPU Allocation Type** & **Environment** & **Library** & **Latency (s)** \\ \hline Broadcasting & single GPU per container & cross docker & MPI & 363.35 \\ Gathering & single GPU per container & cross docker & MPI & 1084.23 \\ All-reduce & single GPU per container & cross docker & GLOO & 617.26 \\ \hline \end{tabular}
\end{table} 
Table 12: Worst results of each experiment (PyTorch deep learning). In another study, GLOO and NCCL, communication libraries provided by PyTorch, were applied to a model like ResNet50 and Bert to evaluate PyTorch's data parallel learning acceleration [27]. This study demonstrated the potential for near-linear scalability in the architecture implemented in the NCCL backend. However, it is important to note that this study did not consider each method of the parameter server and all-reduce, which are data parallelism architecture methods, and it did not experiment with container environments such as docker containers and singularity. There are also studies focusing on performance changes when using deep learning APIs within containers [28]. One study benchmarked the performance changes when configuring a deep learning software framework within a docker container compared to a local server. While this study found no noticeable downsides to running deep learning APIs in docker containers and suggested encapsulation and dissemination as a feasible solution, it did not analyze communication overhead, which is a limitation. Another study analyzed the performance of singularity and udocker [29] through deep learning infrastructure benchmarking, showing performance that was equivalent to that of a local server [30]. However, this study did not perform a comparative analysis of various collective communication libraries. Furthermore, there is a study that analyzed the communication of deep learning workloads in an HPC environment [31]. This study profiled the performance impact of high-performance interconnects like InfiniBand, Omni-Path, PCIe, and NVLink when integrated into a heterogeneous HPC system with GPUs. It conducted an analysis of MPI microbenchmarks and DNN training workloads using Horovod.