# Heterogeneous Parallel Implementation of Large-Scale Numerical Simulation of Saint-Venant Equations
## 4 Performance Optimization and Testing
### Overlap Computation and Communication  
Scaling MPI applications on large high performance computing systems requires efficient communication patterns. Whenever possible, the application needs to overlap communication and computation to hide the communication latency. For the current parallel model, we consider implementing an efficient ghost cell exchange mechanism using non-blocking, peer-to-peer communication (mainly the MPI_Isend() and MPI_Irecv() functions), and domain decomposition of the grid.  
Figure 4: Schematic of data exchange.  
Figure 5: Simulation of water surface elevation at different time steps.  
First, we must confirm how to decompose the domain, that is, to confirm which part of the value to be calculated does not involve MPI communication. The grid we use is composed of real values inside and a dummy value of an outer circle of ghost cells, where real values are computed and dummy values are communicated. It can be seen from Figure 1 that the value of the ghost cell is only used when calculating the value of the outermost circle of the actual value. Therefore, as shown in Figure 6, we can come up with a solution for the domain decomposition. We call the outermost circle of the actual value the \"halo cell\", and the part of the actual value that removes the halo is called the \"inner filed\".  
The general iterative scheme is summarized as follows:  
1. Copy data of the ghost cells to send buffers;
2. Ghost cell exchange with Isend/Irecv calls;
3. Compute (part 1): Update the inner field of the domain;
4. Call MPI_Waitall();
5. Copy data from receive buffers to the ghost cells;
6. Compute (part 2): Update the halo cells;
7. Repeat.  
After completing the calculation and communication overlap, we performed a comparison test with the previous version, as shown in Table 2. The calculation amount of a single node is 200 iterations, and the grid size in each node is 2564 \(\times\) 4100.  
In this test, we performed a total of 200 iterations, and the time obtained after the average of five times was tested. The total time simply refers to the time to execute 200 iterations, excluding the previous initialization and subsequent output time. The sync time is the time to wait with MPI_Barrier() before the end of each iteration. It can be seen that by overlapping communication and calculation, the communication process is hidden in the calculation process, which can greatly reduce the time consumption, and even when the calculation amount is large enough, the communication time can be completely ignored.  
Afterwards, we conducted extended tests on this overlapped version. We tested the parallel performance of the program by continuously increasing the number of processes while keeping the amount of computation allocated by each process basically the  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Version** & **Calculating Time (s)** & **Sync Time (s)** & **Total Time (s)** \\ \hline Before & 240.61 & 6 & 246.61 \\ After & 189 & 5 & 194 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison test before and after overlap.  
Figure 6: Schematic of domain decomposition.  
same. The test results are shown in the Figure 7. This test uses four nodes (processes) as the benchmark, and these four nodes are arranged in a 2 \(\times\) 2 manner. The number of iterations and grid size are the same as before. It can be seen that the overlapping of communication computing can not only shorten the computing time but also maintain a good parallel efficiency.