## 4 Experiments
### Results
Because the sparsity pattern of the matrix generated by the five-point finite different stencil had a strong data dependence using the level-scheduling algorithm for triangular solves, it became the bottleneck of the whole preconditioned GMRES on GPUs. On Tesla K20 GPUs, \(T_{p}\) using cuSPARSE library's block triangular solved accounts for over 80% of the average cost of 30 GMRES steps (i.e., \(30\times\frac{T_{gmres}}{T_{gmres}}\)). Due to the poor performance of the preconditioning step on GPUs, we did not observe obvious performance improvement when comparing \(T_{gmres}\) to that on the CPU platform which is shown in Figure 9. On Tesla V100 GPUs, the performance of the triangular solves using cuSPARSE 10.0 was much improved compared to Tesla K20 GPU. The sparsity pattern of the matrix had very strong data dependency that made the concurrency of the triangular solve low using the level-scheduling algorithm, so we attributed the performance improvement for cuSPARSE to the faster and more efficient memory on Tesla V100 GPUs. However, \(T_{p}\) on four GPUs was nearly the same as that on CPUs, which means the speedups was obtained only by the block sparse matrix-vector multiplication (block-SpMV) and vector operations in GMRES. Figure 6: driven cavity case: comparisons of \(T_{p}\) between Tesla K20 GPUs and Tesla V100 GPUs. \(T_{p}\) denotes the wall clock times for performing 30 preconditioning steps. Figure 4:0 Driven cavity case: comparisons of \(T_{gmres}\) between Tesla K20 GPUs and Tesla V100 GPUs. \(T_{gmres}\) denotes the wall clock times for running GMRES(30) to converge in each Newton step. Figure 4:1 Driven cavity case: comparisons of \(I_{gmres}\) between Tesla K20 GPUs and Tesla V100 GPUs. \(I_{gmres}\) denotes the total number of iterations used for GMRES(30) in each Newton step. Figure 4:2 Driven cavity case: \(T_{gmres}\) on host-1 and host-2 for all three linear systems. By using BJPB-ISAI, we see from Figures 7 and 8 that both \(T_{p}\) and \(T_{games}\) was much reduced. Note that \(T_{games}\) from BJPB-ISAI with \(S(|X_{i}|)\) is not reported in Figure 7 because the preconditioner constructed by \(S(|X_{i}|)\) was not accurate enough for GMRES(30) in the third Newton step to converge within 10,000 iterations. For BJPB-ISAI(\(|X_{i}|^{2}\)) and BJPB-ISAI(\(|X_{i}|^{3}\)), the preconditioners were effective enough to make GMRES converge even though they resulted in more GMRES iterations than the exact preconditioner from cuSPARSE. With increasing the density of the sparsity pattern, \(I_{games}\) gradually reduced, and BJPB-ISAI(\(|X_{i}|^{3}\)) was close to the preconditioning using exact block triangular solves in terms of \(I_{games}\). Before calculating the speedup on Tesla V100 over Tesla K20, we show a comparison of \(T_{BISAI}\) in Figure 4:3 Since there were three different linear systems that needed to be solved in this non-linear process, Block-ISAI for inverse approximations was executed three times, and Figure 10 reports the average values from these observations. Compared to \(T_{games}\) shown in Figure 7, \(T_{BISAI}\) is only a tiny part of \(T_{total}\) in (10). Compared to Block-ISAI(\(|X_{i}|^{2}\)), Block-ISAI(\(|X_{i}|^{3}\)) results in smaller values of \(T_{total}\) for all three linear systems, and an average of 3.3x and 5.1x speedups can be obtained for the three linear systems on Tesla K20 and Tesla V100 over the corresponding block triangular solves using cuSPARSE. Using BJPB-ISAI(\(|X_{i}|^{3}\)) and cuSPARSE's triangular solves for preconditioning on four GPUs, in Figure 11, we report the comparisons of the overall time for solving the non-linear problem with increasing mesh size from \(400\times 400\) to \(600  
Figure 4:4 Driven cavity case: overhead breakdown of gaining \(L_{i}^{inv}\) and \(U_{i}^{inv}\) using Block–ISAI with \(S(|X_{i}|^{2})\) and \(S(|X_{i}|^{3})\). **(top)**\(T_{ESP}\): time used for estimating the sparsity pattern. **(middle)**\(T_{SSTS}\): time used for solving small block triangular systems.