# Collective Communication Performance Evaluation for Distributed Deep Learning Training
## 1 Introduction
They are chosen for their scalability and ease of management, with market spending on cloud virtualization reaching USD 63.7 billion in the first quarter of 2023 [8]. These cloud services provide flexibility and cost benefits, making it easier to set up and manage distributed deep learning training. However, when working in a cloud environment, it is crucial to consider the performance of collective communication in distributed deep learning. Efficient communication greatly affects the performance and scalability of such training. While previous research has primarily focused on assessing collective communication performance in bare metal [9] environments, our study takes into account both bare metal and containerized environments [10], including docker [11] and singularity [12] containers. This comprehensive approach ensures a thorough evaluation of distributed deep learning training performance in the cloud. Distributed deep learning can be divided into two main categories: data parallelism [13] and model parallelism [14]. Data parallelism involves multiple nodes with identical models processing a single dataset, while model parallelism distributes one model across multiple nodes for training. In this study, we focus on data parallelism, which is commonly used in distributed deep learning, and examine the communication methods between nodes. Two representative communication architectures for data parallelism are the parameter server [15] and ring all-reduce [16]. The parameter server follows a centralized approach, where a central server manages model parameters while client nodes process data to compute gradients and update the server. In contrast, ring all-reduce is a decentralized approach where each node owns part of the model, and gradients are exchanged and combined with neighboring nodes to update the model. These data parallelism methods are widely employed in distributed deep learning research and play a significant role in designing efficient deep learning workloads. For instance, when optimizing heterogeneous server environments, a combination of parameter server and all-reduce methods is often considered for decision-making [17]. Additionally, in the fully sharded data parallel (FSDP) approach [18], which involves dividing parameters and gradients into smaller portions and assigning them to different nodes for parallel processing, collective communication is also utilized. Analyzing these approaches contributes to enhancing the structures and efficiency of distributed deep learning and the deep learning workload design. In this study, we conducted an in-depth examination of intra-node communication methods in the context of data parallelism. Our primary objective was to assess the performance of collective communication in distributed deep learning environments. We conducted a comprehensive analysis across various scenarios, encompassing both bare metal and containerized environments, utilizing both the Linux shell and the PyTorch framework. The ultimate objective of our research was to identify effective strategies for intra-node communication in data parallelism and propose the most suitable communication approaches for distributed deep learning environments. Here is a summary of the contributions:  
* Virtualization environment (container vs. bare metal): We assessed the collective communication libraries' performance in diverse server environments within a multi-GPU cluster. Our investigation revealed discernible variations in collective communication performance between bare metal and container setups, prompting a detailed analysis of the performance changes exhibited by each library. Furthermore, we evaluated different types of containers within the well-established docker and singularity container environments, tailored for high-performance computing (HPC) applications. Our findings indicated a reduction in collective communication time within the single-container (multi-GPU) environment for the MPI library, while an increase in latency was observed in cross-container communication using NCCL.