# Benchmarking GPU Tensor Cores on General Matrix Multiplication Kernels through CUTLASS
## 2 Materials and Methods
### GPU Architecture and Programming Model
Works [15; 16; 17; 18] conducted micro-benchmarks to demonstrate the underlying mechanisms of Tensor Core, including the SASS instruction of mma_sync, the cost cycles for each MMA instruction as well as the tile orchestrations for each thread within one warp. [PERSON] et al. [19] pointed out the phenomenon that the utilization of CUDA Core and Tensor Core could not be high simultaneously; therefore, they proposed a source-to-source compiler that transforms a common kernel into a persistent SM-centric kernel, allowing workloads using Tensor Core or CUDA Core to be scheduled into the same SM. SIMD2 [20] used off-the-shelf commercial GPUs to emulate general purpose computation on the dedicated Tensor Core, revealing the potential of extending the calculation capacity for a wider range of workloads. Likewise, PET [21] proposed a transformation that allows partial equivalent operations and an automatic correction mechanism to ensure that the results are acceptable for tensor programs. Another direction is to fuse isolated kernels utilizing the CUDA Core or Tensor Core separately into monolithic macro-kernels and integrating architecture supports in hardware for this change [22]. There are also some workload-specific optimizations exploiting Tensor Core to accelerate the computation on GNN or sparse matrix operations [23; 24]. **GEMM** is a well-studied workload accelerated by GPUs, which has many adoptions in deep learning and scientific computing applications. There are some works focusing on exploiting Tensor Core to accelerate GEMM while maintaining the precision requirements [25]. Also, Mix-GEMM [26] explores the usage of GEMM on deep learning model \begin{table}
\begin{tabular}{c c c c} \hline \hline \(M,N,K\) & **w/o Async. Copy** & **w/ Async. Copy** & **Speedup** \\ \hline
4096 & 103,483 & 127,160 & 1.228 \\ \hline
8192 & 110,038 & 136,912 & 1.244 \\ \hline
16,384 & 122,674 & 139,075 & 1.113 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Speedups obtained via asynchronous global-to-shared memory copies. \begin{table}
\begin{tabular}{c c} \hline \hline
**Split\_k** & **Achieved Performance (TFLOPS)** \\ \hline
1 & 5641.83 \\ \hline
2 & 10,641.9 \\ \hline
4 & 17,577.3 \\ \hline
8 & 20,913.4 \\ \hline
10 & 19,182.4 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Achieved performance for fp32 GEMM kernel with the same compile-time configurations, except for the runtime-determined _split_k_ parameter. inference running on edge devices. The design of workload distribution on GEMM is also studied to better utilize the GPU on-chip resources [27]. Besides, other specialized linear algebra operations like SYMM and TRMM [28] can be implemented and optimized with CUTLASS. **GPU operator tuning** is the process of finding the most suitable tiling size and other kernel configurations like the number of pre-fetching stages for asynchronous copy. Nowadays, machine learning compilers like TVM [29] have enabled us to seek out the optimal solution with automatic search, but the search space is still to large for an exhaustive traverse. To bridge the gap between performant but less flexible vendor-provided libraries (like cuDNN) and highly customizable but less optimal libraries like CUTLASS, Bolt [30] was proposed to integrate CUTLASS into TVM as the backend of the search process for high-quality code generation. Besides, Graphene [31] was proposed to serve as an IR to minimize the effort for kernel experts to express tensor operations at a low level without losing general expression ability. ## 5 Conclusions

In this paper, we first illustrated the recent trend of emerging quantized calculation workload and architectural ASIC as the corresponding response from the GPU hardware vendors. Then, we briefly introduced the CUTLASS open-source templated library and its design principles for GEMM kernels. With CUTLASS, we built and conducted a series of benchmarks, revealing some insights and factors that will greatly impact the achieved kernel performance.