# Multi-GPU Acceleration for Finite Element Analysis in Structural Mechanics
## 4 Parallel Approach  
The partition of complex models into smaller and more manageable pieces is a common approach to exploit distributed computational resources and/or to address large-scale problems. The rapid development in multi-core and many-core architectures has increased the interest in these techniques, the so-called domain decomposition methods (DDMs). They permit us to divide the computational burden and the memory requirements across the computational resources of the computational system. The underlying idea consists of partitioning the domain into a set of subdomains, in which we assign individual resources to address the global problem in parallel by communicating to the resources in charge of the other subdomains. This strategy increases the overall performance by performing the operations in parallel and distributing the memory requirements. The load balancing of the calculation required by the subdomains is a key to increasing computing performance. DDMs are iterative in nature and usually require communication between the computational resources. Minimizing such communications is very important for incrementing the computing performance. This work makes use of a Global Subdomain Implementation (GSI) [58], in which we assume that the coefficient matrix \(K_{l}\) for each level \(l\) of (2) is distributed across \(p=\{1,\ldots,n_{p}\}\) processes (with \(n_{p}\) the number of processes) by contiguous blocks of rows as follows:  
\[K_{l}=\left( \begin{array}{c}K_{l}^{0}\\ \vdots\\ K_{l}^{p-1}\end{array} \right), \tag{3}\]  
where we perform the computation of each block submatrix \(K_{l}^{p-1}\in\mathbb{R}^{n^{p-1}\times n}\) by one single processor \(p\) with \(n^{p-1}\) number of rows of the block submatrix, and \(n\) number of columns of \(K_{l}\). The block submatrices use the global row indices \(\{\mathbf{n}^{0},\ldots,\mathbf{n}^{p-1}\}\) to facilitate operating between subdomains. We store these submatrices as local \(K_{loc}^{p-1}\) and remote parts \(K_{rem}^{p-1}\). The former is a square matrix including the \"local\" \(n^{p-1}\) unknowns, whereas the latter is a matrix containing coefficients with global column indices stored in other processors. This strategy allows us to differentiate between the \"local\" and \"distributed\" computations. Local computation is performed with the data stored in the own process \(p\), whereas distributed computation requires some communication mechanism. The communications make use of the standardized and portable MPI mechanism. When we initialize communications, each process \(p\) calculates the global columns required from both the own process and other processes. Data exchange consists of receiving ghost values from the processes sharing unknowns (global column indices) and then sending the data to the processes that require them to form the gather vectors. Since each processor \(p\) knows its receive and send processors, data exchange is performed directly between such processes. This strategy reduces the computational complexity and the storage requirements because the number of neighbors and the amount of data are independent of the number of \(p\) processors. The GSI requires an efficient mesh partitioning into several non-overlapping subdomains [59]. Ideally, the number of interface elements of these subdomains should be minimized, thus reducing data exchange between processes. The efficient implementation of mesh partitioning is key for addressing large-scale problems and/or with several subdomains because the partition method is memory intensive. The process consists of the generation of a dual graph (each finite element becomes a graph vertex) from the geometric mesh of the finite element model, and we then use a multilevel k-way partitioning [60] method to define the subdomains considering optimization criteria, such as the minimization of the resulting subdomain connectivity graph and the contiguous partitions enforcing. All these functionalities are available using modern high-level open-source libraries.