## 5 Results and Analysis
### WPK1 and WPK2
#### 5.3.3 Effective Memory Bandwidth  
Effective memory bandwidth utilization indicates the processor usage efficiency by the kernels [88]. Specifically, effective memory bandwidth denotes the rate at which the bytes are read and written for a given kernel. The general formula to calculate the effective memory bandwidth was given in Equation (4). We did not find the specific formulas in the literature for calculating the effective memory bandwidth for each of the schemes that we have considered in this paper. Therefore, we looked at each of the schemes, its memory requirements, and devised equations to calculate the number of bytes read and written, \(brw\) (see Equation (4)), based on their storage requirements. Equation (4) was then used to calculate the effective memory bandwidth, \(e\_bandwidth\), in GB/s. Each scheme was executed 1000 times for each matrix in our dataset to get the average execution time for each of the 26 benchmark matrices. For further details on calculating effective memory bandwidth, see Nvidia documentation at [[https://devblogs.nvidia.com/how-implement-performance-metrics-cuda-cc/](https://devblogs.nvidia.com/how-implement-performance-metrics-cuda-cc/)]([https://devblogs.nvidia.com/how-implement-performance-metrics-cuda-cc/](https://devblogs.nvidia.com/how-implement-performance-metrics-cuda-cc/)). The formulas for the CSR and HYB schemes were reported in [58]) as given below:  
\[e\_brw\_HYB=(12m\times anpr)+8(m+n)+16(n-anpr), \tag{6}\]  
\[e\_brw\_CSR=(m+1+nnz)\times 4+(nnz+m+n)\times 8. \tag{7}\]  
The effective memory bandwidth for the SELL-P scheme was reported in [75], but no formula was given. After consulting [74; 75], we devised the formula to calculate \(brw\) for SELL-P as below:  
\[e\_brw\_SELLP=(12\times seg\_size\times\Sigma_{i=1}^{num\_seq}max\_npr\_seg_{i} )+4(2m+n)+\frac{m}{8}+4, \tag{8}\]  
Figure 8: SURAA: comparison of aggregate GFLOP/s of all the eight schemes. where \(num\_seg\) is the number of segments in the matrix, as determined by the SELL-P scheme, \(seg\_size\) is the size of the segments that is fixed, and \(max\_npr\_seg_{i}\) is the maximum number of nonzero elements in any one row of segment \(i\). It was difficult for us to calculate these numbers and therefore we approximated \(e\_brw\_SELLP\) as below:  
\[e\_brw\_SELLP=(12m\times anpr)+4(2m+n)+\frac{m}{8}+4. \tag{9}\]  
The effective memory bandwidth for the CSR5 scheme has not been reported before. After consulting [79] where this scheme was first introduced, we devised a formula to calculate \(brw\) for CSR5 as below:  
\[ \begin{split} e\_brw\_CSR5&=4\times(nnz+m+1+(1+ \frac{nnz}{\sigma\times 32})+\\ &(32\times 2)\times(1+\sigma))+8\times(nnz+m+n),\end{split}  \tag{10}\]  
where \(\sigma\) is a parameter that determines the column width of the blocks in their data structure. It can assume three different values depending on the value of \(anpr\) of a matrix. These values are 4, 32, or \(anpr\). The effective memory bandwidth for WPK1 and WPK2 was reported by [PERSON] et al. [78], but it was on a different GPU device (Nvidia GeForce GTX 480) and for different matrices. They did not give formulas to calculate the effective memory bandwidth. We looked at their paper and have derived the following formula to calculate the \(brw\):  
\[e\_brw\_WPKx=(12\times warp\_size\times\Sigma_{i=1}^{num\_warps}max\_npr\_warp_{ i}))+4(5m+3n)+\frac{m}{8}, \tag{11}\]  
where \(num\_warp\) is the number of warps in the matrix, as determined by the WPK1 and WPK2 schemes, \(warp\_size\) is the size of the warps which is fixed, and \(max\_npr\_warp_{i}\) is the maximum number of nonzero elements in any one row of warp \(i\).