# Acceleration of a Production-Level Unstructured Grid Finite Volume CFD Code on GPU
## 1 Introduction  
Computational fluid dynamics (CFD) play an important role in modern industry and science, which can be used to simulate and predict the physical and chemical properties of fluid motion, helping to improve the design and manufacturing of products such as aircraft [1] and buildings [2]. The development of CFD software has always been driven by high-performance computing (HPC) technology, and the improvement of HPC computing capabilities will bring revolutionary breakthroughs in CFD applications. The computational demands of CFD are increasing at an ever-faster pace as engineers attempt to model increasingly complex flow phenomena like chemically reactive flows using more high resolution method such as Large Eddy Simulation (LES) or Direct Numerical Simulation (DNS). The LES method requires a grid size of \(Re^{1.8}\), and DNS even reaches \(Re^{9/4}\). For general aircraft, the Reynolds number (\(Re\)) is usually at the million level, which requires a huge amount of grid. This places higher demands on high-performance computers. In recent years, the computing power of HPC has been advancing towards the exascale level [3]. However, manufacturing constraints and power requirements have forced computer vendors to seek continued improvements through vastly higher levels of parallelism and developing more complicated memory hierarchies [4]. In particular, with the rapid development of the General Purpose Graphics Processing Unit (GPGPU), GPUs with formidable computingpower and low power consumption have played an increasingly important role in scientific computing. [PERSON] et al. (2018) proposed a method to solve the one-dimensional magnetohydrodynamics (MHD) problem by building a CFD simulator within the Hadoop Ecosystem in heterogeneous environments, mainly in GPU, and achieved sufficient performance. Nowadays, many of the fastest supercomputers in the world's Top 500 are designed based on GPU architecture. ORNL's Summit supercomputer used a CPU/GPU heterogeneous architecture with 27,648 NVIDIA Volta V100 GPUs, and its next-generation system, Frontier, was based on AMD CDNA GPUs (2018). The Leonardo supercomputer of EuroHPC/CINECA uses 14,000 Nvidia GPUs (2018). Achieving the performance of exascale computing is necessitating a paradigm shift from traditional computing advances. There are generally two types of grids used in CFD simulations: structured grids and unstructured grids. The structured grid was constructed by elements within the domain that have the same adjacent elements (hexahedron). Unstructured grids are composed of elements of different shapes (tetrahedral, hexahedral, polyhedral, etc.). Currently, the mainstream industrial CFD software uses unstructured grids, mainly because unstructured grids are more suitable for handling complex geometric shapes, such as impeller blade (2018), automobile (2018), stirred tank reactor (2018), urban buildings (2018), and so on, than structured grids, and can adaptively adjust the size and shape of the grids to better adapt to the characteristics of fluid flow. In addition, unstructured grids can better handle complex problems such as moving bodies and multiphase fluids and have better scalability and flexibility (2018). However, unstructured grid CFD solvers encounter significant challenges when porting to GPU. The traditional parallelization strategy uses the idea of domain decomposition to divide the computational grid and uses Message Passing Interface (MPI) for communication. However, as the number of partitions increases, the overhead of copying data into (and from) communication buffers will cause parallel inefficiency. Algorithm inefficiency will also incur when the subdomains are small (2018). Under the new architecture, implementation of heterogeneous parallelism is required between CPU and GPU, i.e., utilizing multiple nodes to simultaneously process a grid system with communication by using MPI, and efficient and scalable shared memory parallel algorithms are required within GPU.