# Benchmarking GPU Tensor Cores on General Matrix Multiplication Kernels through CUTLASS
## 2 Materials and Methods
### GPU Architecture and Programming Model
The standard deviations of all kernel executions are less than 5% and the results are stable. The theoretical peak performance for various kinds of data formats on different hardware units is listed in Table 2.

Theoretically, we can conclude that exploiting ASIC hardware units in the GPU could dramatically boost the performance for low-precision data formats. However, the substantial speedup may be restricted by some other factors, like limitations of CTA slots per SM, shared memory usage and memory access latency. Thus, benchmarks of GEMM on off-the-shelf commercial GPUs are necessary for engineers to assess the Tensor Core benefits. Before we continue to dig out the performance insight brought by the Tensor Core benchmarks, it should be pointed out that the CUTLASS naming conventions indicate the implementation details of kernels. Here is a GEMM kernel example:

\[\texttt{cutlass\_tensorop\_s1688 gemm\_128x128\_16x4\_tt\_align4}\]

Fields in the kernel name are separated by underlines. tensorop means the underlying hardware component used for the primary MMA operations is Tensor Core (as for CUDA Core, the corresponding name is simt). The s in the next field means the Tensor Core MMA instructions are targets for _single_ floating point numbers (d for double floating point numbers and h for half types). Then, 1688 indicates that the warp-level MMA tile shape is \(16\times 8\times 8\)--i.e., \(m=16,n=8,k=8\)--within each CTA tile. The 128x128_16 represents the CTA tile shape (\(M\times N\times K\)) is \(128\ \times\ 128\ \times\ 16\), and the trailing 4 is the number of \begin{table}
\begin{tabular}{c c} \hline \hline
**Data Format** & **Peak Performance (TFLOPS)** \\ \hline fp64 & 9.7 \\ \hline fp64 Tensor Core & 19.5 \\ \hline fp32 & 19.5 \\ \hline tf32 & 156 \\ \hline fp16 Tensor Core & 312 \\ \hline int8 Tensor Core & 624 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Theoretical peak performance of NVIDIA Tesla A100 PCIe version for different data formats and the underlying hardware units. pipeline stages for asynchronous global memory to shared memory copies (LDG.STS). The last field, align4, indicates that the maximum alignment between operands **A** and **B** is 4. ### Tensor Core vs. CUDA Core

We first compare GEMM kernels in the cases of with and without Tensor Core involved. The runtime comparisons are demonstrated in Figure 6. As we can see, the GFLOPS of GEMM kernels with Tensor Core are much higher than those without Tensor Core. The average speedup between kernels with and without Tensor Core is 1.98. It should be pointed out that some implementation details vary between these two kernels, including the CTA tile shape and the asynchronous shared memory fetching pipeline stages. Putting together Table 2 and Figure 6, we can figure out that the achieved fp32 GEMM performance via CUTLASS is about 95% of the peak fp32 performance, which is often regarded as a satisfying implementation. ### Data Formats

In this section, we mainly focus on the varied performance when applying CUTLASS GEMM kernels to different data types. As Figure 7 illustrates, there is a obvious trend that as the bitwidth of element type decreases, the achieved GEMM performance increases correspondingly. This is reasonable since the high-precision element type requires more bandwidth as well as cycles to execute the wider MMA instructions on Tensor Core. Another observation is that the achieved performance of extreme low-precision data types like u8 (unsigned 8-bit integer), and even s4 and u4, is much higher than that of conventional data types, which leads to the wide use of such types in deep learning model inferences. Figure 6: Comparison among kernels with and without Tensor Core, where both input and output types are fp32. Performance is measured by GFLOPS, which is a division of the total number of floating operations and the kernel duration. ### Tile Shapes and Split_k

In this section, we explore the relationship between the achieved performance, the CTA tile shape and orchestration.