# Collective Communication Performance Evaluation for Distributed Deep Learning Training
## 3 Distributed DL Communication: Libraries and Architectures Compared
### PyTorch Environment  
PyTorch has several communication options like MPI, GLOO, and NCCL. MPI uses the OpenMPI CUDA-aware library, GLOO by Facebook is for distributed deep learning using CPU memory, and NCCL by NVIDIA focuses on GPU-to-GPU communication. Figure 8 shows the aggregate communication library architecture in the PyTorch environment. In a parameter server setup, MPI and GLOO handle CPU-GPU communication, but NCCL is exclusively for GPU-GPU communication. To use NCCL with the parameter server, you need to add a data duplication step between the CPU and GPU using CUDA functions. All three libraries have the same architecture for ring all-reduce. When performing actual distributed deep learning, the parameter server architecture integrates broadcast and gather routines. Additionally, since the architecture of NCCL and other communication libraries is different, it can be seen that different processes are performed during model learning. Likewise, the execution process for all-reduce is the same for the three integrated communication libraries. In the process of collecting actual model parameters, both architectures must perform average calculations to obtain the values of the actual collected parameters.  
In the PyTorch environment, additional experiments were conducted in practical distributed deep learning settings. This included configuring parameter servers and all-reduce architectures for three collective communication libraries to train local models on real datasets and aggregate updated model parameters through inter-node communication. Figure 9 illustrates the processes for both architectures.  
Figure 9a illustrates the distributed deep learning process with the parameter server architecture. The process involves loading the Cifar-10 dataset and conducting training according to specified epochs and batch sizes. Different procedures are followed depending on whether the NCCL backend is used during the distributed process. Firstly, when using CPU communication-based methods such as MPI or GLOO, the gather function is invoked to collect local training parameters from workers into the parameter server. Subsequently, the parameters are averaged. The parameter server then receives the averaged parameters, broadcasts them to the workers, and proceeds to the next iteration. On the other hand, if NCCL is utilized, the allgather function gathers worker parameters into the parameter  
server's GPU memory. Following this, a CPU memory copy is performed, and parameter averaging takes place. The averaged parameters are then allocated to the GPU memory and broadcast to workers for subsequent training iterations.  
Figure 8(b) illustrates the all-reduce architecture, which follows a uniform procedure across three collective communication libraries. After each worker completes local training, the all-reduce function is invoked to synchronize parameters. Then, parameter averaging is performed by each worker to advance the training iteration.  
Figure 8: PyTorch flowchart.  
Figure 9: PyTorch deep learning flowchart.