# Multi-GPU Acceleration for Finite Element Analysis in Structural Mechanics
## 5 Numerical Experiments
### Numerical Results  
The numerical experiments evaluate the GPU performance by solving sets of finite element models that we obtain by modifying the number of unknowns of the problem with the _div_ parameter. We increase the _div_ parameter until we reach the hardware limitations of the multi-GPU system using the four graphics cards due to insufficient device memory. For all the experiments, we show the wall-clock time using multi-core and many-core architectures, the device memory used for the corresponding problem size, and the speedup from the use of both one (serial execution) and eight (maximum number of cores) CPUs of the multi-core system. Figure 4: Single-arch dam experiment: (**a**) geometric configuration, (**b**) boundary conditions, and (**c**) mesh parameterization and partitioning into four subdomains. We solve the 2D simply supported beam experiment using as a preconditioner the AMG method with aggressive coarsening and parallel smoother SPAI-0 for relaxing the solver iterations. Figure 6a shows the wall-clock time for solving the finite element problem with different unknowns. We solve this model with more than 73 M of Degrees of Freedom (DoF) using the serial implementation, which takes about 52 h. We can accelerate the resolution of this problem using four cores of the multi-core system, which takes less than 15 h. This result shows good scalability by obtaining a speedup of 3.5 from the serial implementation. We cannot solve the problem of more than 73 M of DoFs using eight cores due to the memory required by the parallel partitioning approach, limiting the problem size to almost 57 M of DoFs for eight partitions with the 128 GB of RAM available in the desktop computer. We have to remark that the number of CST finite elements have a similar order to the number of unknowns for this type of finite element model. Thus, dual graphs from the geometric mesh (generated for each subdomain) also have high device memory requirements. Figure 6b shows the amount of device memory used to solve the 2D simply supported beam experiment using GPU computing with different numbers of unknowns. By incrementing the number of graphics devices, we can increase the size of the problem that we can solve and also accelerate the resolution of the system of Equation (1). We can address finite element models of up to 18 M of DoF using one GPU, whereas we can solve finite element models of up to 73 M of DoF using four graphics devices. Concerning the performance, Figure 6c,d show the speedup using multi-core and many-core computing from solving the problem using one and eight CPUs, respectively. They also show the number of levels used by the AMG preconditioner. We can observe that the acceleration obtained using GPU computing is of a different order of magnitude concerning the speedup obtained using multi-core computing. The speedup using GPU computing increases with the problem size through the higher bandwidth of such massive architectures. The speedup curve using one GPU is more regular than the other curves using multiple GPUs. We attribute such oscillations of the speedup curves to communications delays between distributed processes and the number of levels of the preconditioner, which increases both the computation and the communications of distributed operations. Figure 5: L-shaped cantilever experiment: (**a**) geometric configuration, (**b**) boundary conditions, and (**c**) mesh parameterization and partitioning into four subdomains. We use an optimized implementation of AMG preconditioner with aggressive coarsening using block matrices (grouping into blocks all unknowns of a grid point) to solve the 3D arch-dam experiment. This instance uses incomplete LU factorization with zero fill-ins (ILU0) for the smoothing iterations. Figure 7a shows the wall-clock time for solving the single-arch dam experiment with a different number of unknowns. We address problems of about 32 M of DoF using the serial implementation, taking about one hour to solve.