# Heterogeneous Parallel Implementation of Large-Scale Numerical Simulation of Saint-Venant Equations
## 3 Heterogeneous Implementation  
First of all, our heterogeneous implementation works together through the CPU and accelerator. The CPU also participates in the calculation process while being responsible for communication, while the accelerator is only responsible for calculation. After that, we used MPI, OpenMP, Pthread, and OpenCL runtime libraries. Among them, MPI is a parallel program interface based on multiple processes with good performance, which is used in this paper for point-to-point communication between nodes. Both OpenMP and Pthread are thread-parallel interfaces. OpenMP is a portable API that is very convenient to use because it does not bind the program to a pre-set thread. But because of this, we cannot use OpenMP to manage specific threads individually. So, we introduced Pthread. The Pthread API handles most of the behavior required by threads. These behaviors include creating and terminating threads, waiting for threads to complete, and managing interactions between threads. Combining the advantages and disadvantages between the two, we use OpenMP when performing simple thread parallelism, and use Pthread when more complex thread operations are required. OpenCL is the first open, free standard for general-purpose parallel programming for heterogeneous systems. It is widely used in parallel processors such as multi-core CPUs, GPUs [24; 25; 26], accelerators, etc. We are here to process the accelerator and take care of some of the computing tasks, through OpenCL. A heterogeneous parallel framework can be used across multiple platforms, but different clusters are equipped with different equipment, and the program will be changed accordingly. This experiment is performed on the domestic advanced computing system \"SunRising-1\". The specific experimental environment is shown in Table 1. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Hard/Software** & **Name** & **Details/Version** \\ Environment & & \\ \hline Hard environment & CPU & 32-core domestic \(\times\) 86 processor \(\times\) 1 \\ (single node) & RAM & 16 GB DDR4 \(\times\) 8 \\  & & Domestic GPGPU accelerator \(\times\) 4, \\ Acceleration device & 16 GB HBM2 VRAM, & \\  & & bandwidth 1 TB/s \\ Network & & InfiniBand HDR network, \\  & & Fat-tree topology, 200 Gbps \\ \hline \hline \end{tabular}
\end{table} 
Table 1: SunRising-1 experimental environment. Figure 2: Schematic of a small-scale initial state. Figure 3 shows the schematic of our heterogeneous parallel framework. We first initialize MPI and use MPI_Type_contiguous() to create an MPI datatype by replicating an existing datatype (for example, MPI_INT, MPI_DOUBLE, etc.). These replications are created into contiguous locations, resulting in a contiguous datatype created. The created datatype is then committed using MPI_Type_commit(), before it can be used for communication. After entering the process parallelism, routinely initialize OpenCL, including obtaining the platform, device, creating context, etc. The next step is to implement the initial value conditions described in Section 2, and we call this process \"initialization\". The cluster used in our experiment is equipped with 32 CPU cores and four accelerators for a single node, so for the initialization process, we enable four threads to simultaneously call OpenCL to start the kernel function. This process does not involve complicated operations, so OpenMP is used. Thread parallelism ends after the kernel function finishes running. After that the calculation process starts. In this process, both the CPU and the accelerator participate in the calculation, so we divided two thread groups, one of which contains four threads to perform operations similar to the previous process to start the kernel function, and the other is to enable the remaining 28 threads participate in the computation. The grouping and waiting of threads are involved here, so Phread are used. After an iterative calculation is over, because of the existence of ghost cells, communication is required next.