# A Neural Network-Inspired Matrix Formulation of Chemical Kinetics for Acceleration on GPUs
## 3 GPU Performance Analysis
### Compute Times and Throughput
On the other hand, throughput performance is the ratio of kernel FLOPs to execution time in seconds. In summary, arithmetic intensity has units FLOPs-per-byte and throughput performance has units of FLOPs-per-second (typically, the performance is given in units of GigaFLOPs-per-second). Theoretical peak performance of the GPU (the horizontal black line in Figure 5a) is available only when kernels have sufficiently high arithmetic intensity; otherwise, the kernels are bandwidth-limited. The cutoff point at which a kernel goes from bandwidth to compute-limited is the Roofline elbow--for arithmetic intensities below the elbow, the maximum achievable performance decreases at a linear rate defined by the hardware (typically measured in Gigabytes-per-second). The effective GPU utilization for a kernel can then be visualized directly by plotting the two attributes of the kernel (throughput performance versus arithmetic intensity) and comparing with the theoretical device limits. Ideally, all kernels should approach the throughput limits of the hardware regardless of arithmetic intensity. Lastly, it should be noted that the Roofline model concerns rates  
Figure 4: Absolute compute time (time-to-solution) as a function of cell number \(N_{C}\) for the GPU evaluation of the species source terms for the mechanisms listed in Table 1. Saturation points for each mechanism group are indicated by the black arrows. and not absolute quantities (i.e., throughput performance and execution time are not interchangeable). In other words, kernels that are compute-limited do not necessarily run faster in physical time than those that are bandwidth-limited. Proceeding with the analysis, Figure 5a shows a Roofline plot _for all GPU kernels_ encountered during the source term evaluation using different \(N_{C}\) values for a single chemical mechanism (B3) for brevity, as global Roofline trends with respect to \(N_{C}\) for all mechanisms are similar. Each point represents a kernel (i.e., matrix multiplication operation, Gibbs matrix computation, exponentiation, etc. ), and the color of the point in Figure 5a denotes the \(N_{C}\) for which the kernel was evaluated. It is apparent that: (a) most of the kernels are bandwidth-limited; and (b) the kernels approach near-theoretical limits in all cases as \(N_{C}\) reaches the saturation point. In general, increasing the value \(N_{C}\) moves a kernel defined for a particular arithmetic intensity vertically on the Roofline plot, eventually saturating the GPU and achieving theoretical efficiency. Thus, for a large-scale application, one should allocate the MPI resources to accommodate a high enough \(N_{C}\) per GPU in order to ensure that node resources are utilized to their fullest extent. Examples of Rooflines extracted for _individual kernels_ are shown in Figure 5b,c. In these figures, the points are colored by reaction mechanism instead of \(N_{C}\) to assess direct effects of mechanism complexity. The spread of points for a single mechanism (a single color) represents the various \(N_{C}\) values at which the kernel was computed. Figure 5b shows the progression of the double-precision general matrix multiplication (DGEMM) operation provided by the cuBLAS backend, which is the main driver for the matrix-based formulation implemented in this work. Immediately apparent is an increase in arithmetic intensity with \(N_{R}\) (i.e., the points are cleanly clustered by reaction mechanism). The Class C mechanisms in particular allow for the DGEMM operation to access peak performance for most \(N_{C}\) values. The smaller mechanisms are near the tail-end of the bandwidth-limited region, though they still achieve theoretical peaks for sufficiently high \(N_{C}\).