# Acceleration of a Production-Level Unstructured Grid Finite Volume CFD Code on GPU
## 5 Performance and Discussion
### Performance of Loop Parallelization  
The effectiveness of different reordering methods was compared to determine the best grid index manner. By simulating the M6 wing with the same calculation condition, the residual convergence history of different reordering methods was obtained. Figure 9 illustrates the results of Grid 1. The results show that the Cuthill-McKee method has the best performance. Compared to the original, it takes about 10% less time to reduce the residual by four orders of magnitude. The result is in line with expectations. According to Figure 5, the Cuthill-McKee method has the best data locality and the minimum matrix bandwidth after reordering. In the subsequent testing, all implementations are based on this method.  
The biggest performance obstacle of parallelization on GPU is the subroutines with write data conflicts. The performance of these subroutines can best reflect the effect of different parallelization methods. Figure 10 shows the acceleration achieved by the _UpdateRHS_ function in FlowStar after GPU parallelization compared to CPU serial computation. It can see that the acceleration effect of the reduction and face colouring strategy is almost the same, and both are better than atomic operations. It can also find from the results that the larger the grid size, the more significant the acceleration effect. It is because as the problem size increases, the proportion of data movement overhead between CPU and GPU decreases, and the benefits of utilizing GPU multi-core parallel acceleration will be more prominent.  
\begin{table}
\begin{tabular}{c c c} \hline
**Size Level** & **Total Cells** & **Total Faces** \\ \hline Grid 1 & 123,158 & 55,109 \\ Grid 2 & 804,367 & 352,639 \\ Grid 3 & 3,718,474 & 1,321,496 \\ Grid 4 & 10,834,928 & 4,519,009 \\ \hline \end{tabular}
\end{table}
Table 1: Grid information of M6 wing.  
\begin{table}
\begin{tabular}{c c c} \hline
**Size Level** & **Total Cells** & **Total Faces** \\ \hline Grid 1 & 6,518,485 & 2,318,948 \\ Grid 2 & 17,376,357 & 5,952,147 \\ \hline \end{tabular}
\end{table}
Table 2: Grid information of CHN-T1.  
Figure 8: The computational grid used for performance test. (**a**) The ONERA M6 wing. (**b**) The CHN-T1 airplane.  
Section 4.3 discusses that explicit management of shared memory can further improve memory access efficiency by caching data that need to be read and written repeatedly. Figure 11 gives the results of the speed-up ratio of the _UpdateRHS_ function before and after shared memory optimization. The effect of shared memory optimization based on the face colouring method is limited (red curve in the figure). This conclusion is consistent with the Reference [32]. The group colouring method (black curve in the figure) refers to the two-layered colouring or hierarchical colouring method of Reference [15]. The performance is still poor. The reason for achieving a good acceleration ratio in the original literature is that its results considered changing variable storage from the Array of Structure (AoS) to the Structure of Array (SoA). However, the data structure of FlowStar is already SoA, so the effect is not significant. In contrast, the optimization of shared memory proposed by this work based on the reduction strategy (blue curve) is impressive. After optimization, the performance can be further improved by three times, verifying the effectiveness of the method proposed in this paper.  
Furthermore, based on the above reduction strategy and shared memory optimization, this work investigated the acceleration effect of loop fusion described in Section 4.2. Table 3 gives the time consumption (s) before and after loop fusion for convective flux and  
Figure 10: Speed up of UpdateRHS subroutine for different grid level of M6 wing.  
Figure 9: Comparison of convergence history using different reordering methods.  
viscous flux. It is obvious that the time consumed after fusion is less, especially for the calculation of convective flux, which can be reduced by nearly half.