## 3 Three-Dimensional FDTD Parallel Algorithm on the CUDA Platform
### Data Transmission Scheme Based on NCCL
#### 3.2.2 GPUs Communicate Directly Based on NCCL  
The NCCL technology can directly transmit data between multiple GPU devices without the help of a CPU. A data transmission scheme based on the NCCL technology is shown in Figure 5. The GPU in the node can carry out point-to-point data interaction through the NVLink. For inter-node GPUs, this paper uses the NCCL technology combined with an Infiniband network card to exchange data between the GPUs directly. Figure 4: GPUs communicate indirectly through the CPU. When applying the NCCL technology in GPU-parallel FDTD methods, the NCCL needs to be initialized first. Specifically, we use _ncclGetUniqueld_ to create a unique communication identifier _ncclUniqueld_, and initialize the communicator using this identifier for each process through _ncclComlInitRank_. After the initialization of the communicator is completed, the environment variable export _NCCL_P2P_EVEL = NVL_ can be set to specify that the NCCL prioritizes the NVLink when building communication paths, ensuring that communication is conducted through the NVLink as much as possible. In GPU clusters with an NVLink interconnection, this setting can significantly improve the communication efficiency and reduce data transmission bottlenecks, especially suitable for bandwidth-sensitive scenarios, such as boundary data exchange in distributed FDTD methods. For data exchange between FDTD subdomain segmentation sections, the commonly used communication functions are _ncclSend_ and _ncclRecv_, which are blocking communication methods that can ensure the orderliness of data exchange and naturally meet the synchronization requirements of FDTD time step advancement. To improve communication efficiency, multiple communication operations can be packaged into one group communication through _ncclGroupStart_ and _ncclGroupEnd_. For example, boundary data exchange operations from the X-, Y-, and Z-directions can be merged into the same communication group to reduce communication calls and improve bandwidth utilization. In addition, to further optimize performance, the boundary data buffers in the three directions can be packaged into a continuous memory block at the sending end, and sent to the target GPU through only one transfer operation, and then unpacked at the receiving end. This method significantly reduces the frequency and overhead of data transmission, improving the overall communication efficiency. At the end of the simulation or at a specific moment when the data from each GPU need to be aggregated, _ncclAllGather_ can be used to collect the extrapolated cross-sectional data from the various devices into the video memory of each GPU, facilitating subsequent result processing and visualization. During the initialization, _ncclBroadcast_ can also be used to synchronize the simulation parameters of the main process with the other GPUs, ensuring that all the processes perform their calculations under the same configuration. It should be noted that the NCCL can only transmit continuous memory regions, while the FDTD's subdomain data are usually continuous when divided along the X-direction, resulting in discontinuous data access when divided along the Y/Z-direction. Therefore, for achieving efficient communication, we have designed an auxiliary buffering mechanism. A continuous buffer region (the purple area in Figure 5) is opened for each subdomain in the boundary direction, which is responsible for packaging and unpacking the boundary data. This mechanism effectively avoids the unnecessary overhead of manual  
Figure 5: GPUs communicate directly based on NCCL. packaging before each communication, improving the overall efficiency and maintainability of communication. The time-domain stepping process of the GPU-based FDTD parallel algorithm is as follows:  
1. Send the magnetic field **H** data of the yellow region to the purple data buffer of the adjacent subdomain;
2. Electric field **E** iteration;
3. Send the electric field data of the yellow region to the purple data buffer of the adjacent subdomain;
4.