# Benchmarking GPU Tensor Cores on General Matrix Multiplication Kernels through CUTLASS
## 2 Materials and Methods  
### GPU Architecture and Programming Model  
The concept of a general purpose GPU was introduced in 2006. Figure 3 illustrates the SM architecture in NVIDIA's _Ampere_ GPU. As we can see, each SM contains four sub-processors, with integer and floating point units (CUDA Cores) inside. Besides, as aforementioned, NVIDIA has added Tensor Cores into the SM since the _Volta_ generation. CTAs are scheduled to sub-processors for execution based on the granularity of warp (32 consecutive threads in CUDA). Instructions from the warp may be issued and executed on CUDA Cores, Tensor Cores or the LD/ST units to fetch data from shared memory or global memory. Warps are swapped out and switched to the stall state while they are waiting for the necessary and dependent data from lower cache levels or off-chip memory. The warp scheduler is responsible for switching between the ready and stalled warps to hide memory access latency of LD/ST instructions, making the GPU a throughput-oriented processor compared to the CPU, which is latency-oriented. Figure 3: The architecture of SM in NVIDIA’s _Ampere_ GPU from NVIDIA’s official whitepaper [12]. Figure 2: CUTLASS pipeline phases and design abstraction. CUDA provides three warp-level collaborative APIs and specific C++ templated types for programmers to exploit the WMMA function. Listing 1 is the brief C++ statement sequence using WMMA APIs. ```
//definetheregisterfragment
``` ```
//loadatilecofmatrixatoregiterfragment
``` ```
//loadatilecofmatrixatoregiterfragment
``` //warpmatrixmultiplyoperation
```
//storethetilofoutputmatrixC
```



### Accelerating GEMM on GPU

GEMM is a widely used operator in data intensive applications. The computing pattern of the fully connected layer in deep neural networks is GEMM in fact, and countless engineering problems depend on GEMM in their computation. GEMM is defined as follows:

\[\mathbf{D}=\alpha\mathbf{A}\mathbf{B}+\beta\mathbf{C}, \tag{1}\]

where \(\alpha\) and \(\beta\) are given scalars. In reality, the output matrix \(\mathbf{D}\) is usually the same as the input matrix \(\mathbf{C}\) (like the bias tensor in the fully connected layer). There are multiple GEMM implementations designed to accelerate it in many-core architectures. One of the most common implementations in GPUs is tiled GEMM. Figure 5 demonstrates the process of tiled GEMM implementation. As discussed in Section 1, the output matrix is firstly partitioned into multiple tiles, whose shape is empirically determined by the on-chip cache size (\(Mtile\times Ntile\) in the green-colored tile in Figure 5). To obtain the results of this target tile, one has to use the corresponding tiles in both input matrices, colored as light blue and yellow in Figure 5. However, in most cases, the required memory space is far more than that of the on-chip cache capacity if we load all elements located in both input tiles onto the chip. Therefore, a common approach is to iterate both tiles at \(Ktile\) step to obtain partially temporary results at each step and, finally, accumulate the overall results. Figure 4: Memory and thread layouts of matrix tiles of mBnBk16 (i.e., \(8\times 16\) for tile from matrix \(\mathbf{A}\), \(16\times 8\) for tile from matrix \(\mathbf{B}\), and \(8\times 8\) for tile from the row-major output matrix \(\mathbf{C}\)) for one warp. The element type of input tiles is int8, and the accumulated output type is int32. The notation \(T_{x}:\{\cdots\}\) in the table cells represents that thread \(x\) in the warp is responsible for loading the matrix elements indicated between the brackets. Inside each tile step, the calculation as well as the matrix tile are partitioned into several iterations. For each iteration, the threads in the CTA will load the required data elements in the current iteration into shared memory to minimize off-chip memory accesses later, since there will be certain data reuse across warps within a CTA.