## 4 Experiments
### Results  
We used the matrix _atmosmod_ from SuiteSparse Matrix Collection [32] as the first test case. The matrix in CSR format had 1,270,432 rows (columns) and 8,814,880 non-zeros. The procedure in List 1 transformed the matrix to BCSR format using a block size of 4, and the resulting matrix consisting of 317,608 block rows (columns) and 2,190,844 non-zero blocks was solved by GMRES(30) with BJPB-ISAI and a relative tolerance of \(10^{-8}\). Figure 3 shows the performance results on up to four Tesla K20 and V100 GPUs respectively in terms of \(T_{p}\), \(T_{gmres}\), and \(I_{gmres}\), where \(T_{p}\) is the wall clock time for computing 30 preconditioning steps (i.e., (7)), \(T_{gmres}\) represents the total wall clock time for GMRES(30) to converge, and \(I_{gmres}\) is the total number of iterations for GMRES(30). To make a performance comparison between CPU and GPU results, we show \(T_{gmres}\) on CPUs in Figure 4. It is reported in [7] that the level-scheduling based triangular solve successfully gained speedups on the atmosmod case. By comparing \(T_{gmres}\) in Figure 4 to \(T_{gmres}\) obtained by block triangular solves from cuSPARSE library in Figure 3, we also observed an advantage of GPU block triangular solves over the CPU counterpart. When four GPUs were used, preconditioned GMRES using cuSPARSE's block triangular solves \begin{table}
\begin{tabular}{c c c} \hline \hline
**Tesla Product** & **Tesla K20** & **Tesla V100** \\ \hline GPU & GK110 & GV100 \\ Core Clock & 706 MHz & 1530 MHz \\ Stream Processors & 2496 & 5120 \\ Memory Interface & 320-bit DDR5 & 4096-bit HBM2 \\ Memory size & 5 GB & 16 GB \\ Peak FP32 TFLOPS & 3.52 & 15.7 \\ Peak FP64 TFLOPS & 1.17 & 7.8 \\ Shared Memory size & 16/32/48 KB & configurable up to 96 KB \\ Register File Size/SM & 256 KB & 256 KB \\ \hline \hline \end{tabular}
\end{table} 
Table 2: A comparison of the main parameters of Tesla K20 and Tesla V100 GPUs. could achieve 4.6\(\times\) and 1.44\(\times\) speedups on Tesla V100 and Tesla K20 GPU platforms over the pure CPU preconditioned GMRES on 16 cores. Then we looked at the results from BJPB-ISAI(\(|X_{i}|^{k}\)) (\(k\leq 3\)) and observed that \(I_{gmres}\) decreased with increasing the density of the sparsity pattern for \(L_{i}^{inv}\) and \(U_{i}^{inv}\). This is expected because the denser sparsity for \(L_{i}^{inv}\) and \(U_{i}^{inv}\) made the approximate inverses close to the exact ones. The penalty of increasing the sparsity densities of \(L_{i}^{inv}\) and \(U_{i}^{inv}\) was that it not only made the overhead of doing preconditioning increase but also made the wall clock times of solving \(L_{i}^{inv}\) and \(U_{i}^{inv}\) increase. To measure the real wall clock times for  
Figure 4: atmosmodd case: \(T_{gmres}\) on host-1 and host-2. Figure 3: Atmosmodd case: comparisons between cuSPARSE triangular solve-based preconditioning and BJPB-ISAI with three sparsity levels for \(L\)’s and \(U\)’s inverses. (**a**) \(T_{p}\): wall clock times for computing 30 preconditioning steps. (**b**) \(T_{gmres}\): total wall clock time for computing GMRES(30) to converge. (**c**) \(I_{gmres}\): the total number of iterations used for GMRES(30) to converge. GMRES with BJPB-ISAI algorithm, we included both \(T_{g nurses}\) and the overhead of gaining \(L_{i}^{inv}\) and \(U_{i}^{inv}\) as  
\[T_{total}=T_{g nurses}+T_{BISAI} \tag{10}\]  
where \(T_{BISAI}\) is the time used to gain \(L_{i}^{inv}\) and \(U_{i}^{inv}\). Figure 5 shows a comparison of \(T_{BISAI}\) on Tesla K20 and V100. Note that \(T_{BISAI}\) was measured as the maximum value among the used GPUs because there might be imbalanced workload when multiple GPUs were employed. We divided \(T_{BISAI}\) into \(T_{ESP}\), \(T_{SBTS}\) and \(T_{cs2 sc}\) where \(T_{ESP}\) is the time used for estimating the sparsity pattern (ESP) of \(|L_{i}|^{k}\) and \(|U_{i}|^{k}\), \(T_{SBTS}\) is the time used for solving the SBTS, and \(T_{cs2 sc}\) is the time used for format transformation in Algorithm 1.