# Accelerating Electromagnetic Field Simulations Based on Memory-Optimized CPML-FDTD with OpenACC
## 3 Algorithm and Flux Diagram  
CPML formulation plays a significant role in code implementation; in fact, any PML formulation is crucial in codification. A typical ABC setup over a computational domain is shown in Figure 2 and can be observed the PML region where the boundary conditions are applied. As previously mentioned, the CPML is implementable with a recursive sum in time, only in the absorbing region [31]; for this reason, we have two implementation options. Consider the variables implicated in the CPML as local in the absorbing region or the overall domain, simplifying the number of loops and variables but increasing the amount of memory used. The used CPML formulation reduces the number of memory arrays within the algorithm [33]. It can be therefore easily implemented in GPU architectures, due to their limited memory capacity. Coding the CPML in the FDTD method requires saving the spatial derivatives in a convolutional variable. It can be implemented, as mentioned, in two different ways: reserving memory space for them in all the domain or only in the absorption region. Figure 2: CPML Absorbing Boundary Condition implementation options. The advantage of allocating the convolutional variables in all the domain is that it only requires four computational cycles for the space derivatives and can be applied to vectorization techniques to gain performance. (see diagram depicted in Figure 3). Just allocating the convolutional variables in the CPML region saves memory but increases the implementation complexity and the number of computational cycles (see diagram depicted in Figure 4) [18; 34]. To avoid getting into low-level programming, OpenACC is a directive-based to generate parallel code [3], it is similar to the well-established OpenMP [35], a set of compiler directives, routines, libraries, and environment variables that can be used to specify high-level parallelism. Similarly, OpenACC allows you to tell the compiler how to generate parallel code based on CUDA C instead of writing it directly at a low level. Figure 3: Flux diagram showing the four cycles necessary if the convolutional variables are allocated in all the computational domain. The cycles are labeled as \(V_{1}\), \(V_{2}\), \(V_{3}\) and \(V_{4}\). Figure 4: Flux diagram showing the fourteen cycles necessary when the convolutional variables are allocated only in the absorption region. Saving Memory but increasing the number of cycles. The cycles are numbered as \(M_{1}\cdots M_{14}\). For code migration to OpenACC, the first action is to identify where the most significant computational load is located to reduce the execution time, so it is necessary to discern the degree of parallelization of the algorithm. In this case, is used a numerical method based on finite differences scheme based on a staggered grid. The characteristics of this method make the algorithm highly parallelizable with OpenACC as previously studied by [36; 37]. The execution time used for reading the initial conditions, memory allocation, or variables initialization variables does not cause overhead since they are only executed once during the entire program. The metric used to check if the loops are computationally enough is the intensity. The intensity of a loop refers to the relationship between floating-point operations and memory accesses. A relatively easy way to get this parameter is through the compiler (NVIDIA HPC SDK) using the flag -Minfo=intensity. The intensity for parallelization should at least be \(\geq\)1. Still, if any loop has an intensity less than 1, it is possible to include it in the parallelization if it is part of a larger context. The intensity is defined as \(I=\frac{f}{m}\), where \(f\) is the quantity of floating-point operations performed and \(m\) is the number of data moves. The intensities reported by the NVIDIA HPC compiler for both implementation options are shown in Tables 1 and 2 respectively.