# Benchmarking GPU Tensor Cores on General Matrix Multiplication Kernels through CUTLASS
## 2 Materials and Methods
### GPU Architecture and Programming Model
We first compare two GEMM kernels under problem size \(M=N=K=8192\) and fp32 inputs. Configurations of the inspected kernels remain the same, except for the CTA shape: one is 128x128_16 and the other is 256x128_16. The performance metrics, measured in GFLOPS, obtained for both kernels are 36,956.7 and 115,962, respectively. There is a significant performance gap between these two kernels, even though almost all kernel parameters are identical except the CTA tile shape. As the thread block size (i.e., the number of threads in one CTA) and the orchestration of warps along each tile dimension are the same as well, we can conclude that configuring the CTA tile shape correctly has a huge impact on overall kernel performance. Another performance-impacting factor that deserves inspection is the _split_k_ setting, which controls the number of CTAs working on the same tile of the output matrix. _split_ \(k\) is introduced to tackle the problem that for some small GEMM workloads, the one-CTA-one-tile strategy will not saturate the entire GPU SM resources, extremely for cases with small \(M\) and \(N\) but large \(K\) problem sizes. For example, for a \(1024\times 1024\) output matrix, the abovementioned CTA orchestration with \(256\times 128\) tile shape strategy ends up with only \((1024/256)\times(1024/128)\) CTAs for this kernel, while there are 108 SMs in the A100 GPU. The number of fetch-calculate-store iterations required for each CTA to loop over alongside the \(K\) dimension is \(K/k\), where \(K\) is the reduced dimension of input matrices and \(k\) is the corresponding CTA tile dimension. To fully utilize the GPU resource and reduce the workload assigned to each CTA, one can parallelize the execution at the dimension \(K\) by orchestrating more than one CTA on the same tile of the output matrix. Thus, loop iterations required for that tile could be evenly distributed among multiple CTAs and, for each CTA, the number of iterations would then be reduced by _split_k_ times. Table 3 shows the achieved performance for fp32 GEMM kernels with a varying _split_k_ parameter and with \(M=N=512\) and \(K=8192\) problem size. As we can see, as the value of _split_k_ factor increases, the performance of the obtained kernel boosts as well. However, there is a kneepoint where increasing the _split_k_ value will harm the overall performance. It should be emphasized that enabling parallel _split_k_ mode with its value larger than one needs one extra reduction kernel to aggregate temporary results from different CTAs and the corresponding predefined global buffers into the final output tile destination. Hence, the larger _split_k_ is, the longer the reduction latency will be. Figure 7: CUTlass GEMM kernel performance comparisons on different input element types. The GEMM problem size is fixed as \(M=N=K=8192\). ### Asynchronous Copy

To implement a performant GEMM kernel, software pipelining is required to hide the global-to-shared memory access latency as much as possible. Asynchronous copy is a new hardware feature proposed in Ampere that allows programmers to explicitly invoke non-blocking global-to-shared memory access with no extra registers required for storing temporarily accessed data. Table 4 lists the relative speedup for kernel s1688 gemm_f16_256x128_32x2 gained via applying asynchronous global-to-shared memory copies. Observation shows that enabling this new hardware feature could bring averaged 1.195 speedups over different matrix sizes with fp16 input and fp32 output. Enabling async. copy brings more flexibility for the SM scheduler since Tensor Cores only accelerate the computation while the off-chip bandwidth remains the same. It offers the opportunities to pipeline and overlap the computation and memory access. ## 4 Related Works

**Tensor Core** has been a hot topic since the release of its first generation.