# Accelerating Electromagnetic Field Simulations Based on Memory-Optimized CPML-FDTD with OpenACC
## 4 Simulation Settings and Results
### Free Space Propagation  
In this experiment it is setup an empty environment without absorbing obstacles nor reflecting surfaces and different frequencies were tested: 20.00 GHz, 5.00 GHz, and 2.45 GHz, using computational domains of 20 m \(\times\) 20 m, 10 m \(\times\) 10 m, and 5 m \(\times\) 5 m, respectively, with the source located in the center. Table 3 shows the configuration of the scenarios and Table 4 the quantity of the memory required for each case. It is necessary to mention that the time used to write to the disk is negligible because once transferred to the CPU memory, it is writing in parallel. This means that the kernel executions continue while disk writing is in process. Of course, the time used to transfer the data from the GPU to the CPU is considered part of the GPU total time, but the transfer is done asynchronously, taking on average 13\(\sim\)14 us. The location of the numerical control viewers for the free space propagation is shown in Figure 6.  
Computing times measured for the simulations using one CPU core with the Hyper-threading disabled are shown in Table 5. It is important to mention that the serial version is highly optimized with vectorization and the version implemented in OpenMP does not improve the performance, because the computational intensity is low and the problem is memory bound. Due to the benefits vectorization provides for speeding up codes, it is necessary to apply this technique to the serial version of the code for execution on the CPU since, for fair performance comparison, the best compiled version for the code should be used. Table 6 shows the execution times obtained using the GPUs an the Table 7 the corresponding speed-ups. The vector length is 1024 (block size), giving the best performance. Figure 7 depicts a bar graph comparing the times for the 2.45 GHz case; similar computing time behavior was found for 5.00 GHz (see Figure 8) and 20.00 GHz (see Figure 9). Speed-up factors were calculated taking as a baseline the CPU-MD times. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Version** & **2.45 Ghz** & **5.00 Ghz** & **20.00 Ghz** \\ \hline With Memory Saving (double) & 1155 MB & 2585 MB & 2923 MB \\ Without Memory Saving (double) & 1483 MB & 3353 MB & 3795 MB \\ With Memory Saving (single) & 635 MB & 1337 MB & 1519 MB \\ Without Memory Saving (single) & 803 MB & 1721 MB & 1959 MB \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Used memory for each scenario in the free space propagation, with and without memory saving and using single and double precision. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & **Mesh Size** & **CPML Thickness** & \(\Delta\pi,\Delta y\) & \(\Delta t\) & **Iterations** & **Disk** & **Source** \\ \hline
2.45 Ghz case & 3269 \(\times\) 3269 & 20 & 6.1182 \(\times\) 10\({}^{-3}\) & 1.4286 \(\times\) 10\({}^{-11}\) & 1160 & 50 & 580 \\
5.00 Ghz case & 5004 \(\times\) 5004 & 20 & 2.9979 \(\times\) 10\({}^{-3}\) & 7.0004 \(\times\) 10\({}^{-12}\) & 3480 & 100 & 1160 \\
20.00 Ghz case & 5338 \(\times\) 5388 & 20 & 7.4948 \(\times\) 10\({}^{-4}\) & 1.7501 \(\times\) 10\({}^{-12}\) & 3480 & 150 & 380 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Experiments setup for the free space scenery. Iterations refer to the number of time steps performed. Disk, to the number of iterations when the information has to be transferred to the CPU and saved to the disk. Source, refers to the number of operations when the source is turned off. \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Version** & **2.45 Ghz** & **5.00 Ghz** & **20.00 Ghz** \\ \hline GPU-MD (double) RTX 3060 & 00:49 & 05:22 & 06:05 \\ GPU-VD (double) RTX 3060 & 01:56 & 16:17 & 18:57 \\ GPU-MS (single) RTX 3060 & 00:22 & 02:45 & 02:46 \\ GPU-VS (single) RTX 3060 & 01:10 & 09:28 & 10:41 \\ GPU-MD (double) Titan RTX & 00:46 & 05:16 & 05:53 \\ GPU-VD (double) Titan RTX & 02:10 & 14:33 & 15:50 \\ GPU-MS (single) Titan RTX & 00:21 & 04:07 & 04:39 \\ GPU-VS (single) Titan RTX & 01:00 & 10:24 & 11:36 \\ \hline \hline \end{tabular}
\end{table}