# Multi-GPU Acceleration for Finite Element Analysis in Structural Mechanics
## 2 Multi-GPU Architecture and Communications  
Using multiple graphics cards incorporates another level of parallelism to the GPU system, namely, task-level parallelism. This additional level permits us to distribute the workload of applications between the multiple GPUs and then exploit the data parallelism for which we design these massive parallel architectures. Such distributed memory systems provide an efficient way to increase flexibility in the implementation and scale up the performance of heavy computing applications. Figure 1 shows the typical configuration of a GPU cluster, where we equip the nodes with many GPUs, namely, multi-GPU systems. We install the graphics cards in motherboards with multiple PCIe slots and/or PCIe slots expansion boards. We can observe that GPU clusters and multi-GPU systems are distributed memory systems, where the computational units use their local memory. This flexible computing architecture with different memory spaces requires an efficient mechanism for sharing data between the computational nodes of the High-Performance Computing (HPC) system.  
Figure 1 shows that the communication mechanism can use the PCI-e data bus and the network, including Ethernet or Infiniband. These technologies have different memory bandwidths operating at distinct transfer rates. Communications also introduce latency when data transfer requires computing several operations, such as the operating system (OS) management and the computation of communication protocols. Remote Direct Memory Access (RDMA) technology provides data exchange between devices without involving the OS and CPU. Such devices include GPUs, network interface cards (NICs), and storage adapters. This technology permits boosting the network and host performance with lower latency, lower CPU load, and higher bandwidth. We eliminate communication overheads and bottlenecks since we bypass the host CPU and memory, which results in minimal impact on host resources. We can also use RDMA over networks because RDMA is natively supported in Infiniband interconnect technology and can be used over Converged Ethernet (RoCE) in Ethernet networks. Both technologies share a user API but have different physical and link layers.  
Communication frameworks facilitate using these efficient communication mechanisms. One popular option is using the Unified Communication X (UCX) framework, which is cross-platform and performance-oriented for low overheads in the communication path. It supports the devices of most GPU vendors using GPUDirect RDMA technology and ROCm for Nvidia and AMD products, respectively. The most popular implementations of the standard Message Passing Interface (MPI) adopt this framework, which we can use to build applications that scale in distributed memory systems. The latest versions of the most popular MPI implementations support CUDA-aware MPI computing, such as OpenMPI from the v1.7.0 release and MPICH from the v4.0 one. This compatibility facilitates the development of scalable software in multi-core and many-core computing architectures.  
We adopt these functionalities in the iterative solver preconditioned with a smooth aggregation-based AMG method using a subdomain strategy for the implementation. In particular, we divide the coefficient matrix of the linear system of equations and the vectors needed for the iterative solver and the preconditioner. We optimize the operations between partitioned vectors and the coefficient matrix using sparse-matrix representation dividing the local and remote operations. The former uses the Boost.Compute library [52], which is a GPU/parallel-computing library for C++ based on OpenCL that can operate using sparse-matrix representations. The latter uses the communication framework mentioned above, which optimizes the communications depending on the network topology. The MPI communications use point-to-point communication patterns to avoid bandwidth problems.