# Cross-Feature Transfer Learning for Efficient Tensor Program Generation
## 3 Design and Implementation
### Hardware-Aware Kernel Sampling  
We conducted an extensive and experimental analysis of the TenSet dataset to gain insights into the neural network and hardware features influencing metrics such as flops, throughput, and latency. The dataset comprises over 13,000 tasks derived from 120 networks, measured on six distinct hardware platforms. These measurements include throughput and latency, considering various neural network and hardware parameters, resulting in a dataset encompassing over 52 million measurements. In our analysis (as detailed in Table 1), we focused on the first four platforms to understand tasks, the applied schedules, associated performance, and hardware parameters. The latter half of the hardware serves for evaluating and establishing the viability of cross-device or transfer learning. In addition to neural network details such as tensor operations and input/output shapes, hardware parameters (as presented in Table 2) are also considered. While we specifically present hardware features for CPU and GPU, the approach can be extended to other heterogeneous devices. Through dataset analysis, we observed that high-performing kernels often correlate with specific hardware parameters. Employing probabilistic sampling helps mitigate biased kernel selection, preventing the inclusion of kernels that might lead to lower FLOPs or result in invalid computation graphs. We rigorously filtered out measurements and kernels deemed invalid or low performing. Notably, our observations underscore the impact of large search spaces, selected via random sampling of kernels, causing performance regression in existing cost modeling practices. \begin{table}
\begin{tabular}{l c c} \hline \hline
**Hardware Parameter** & **Definition** & **Hardware Class** & **Value (Bytes)** \\ \hline cache\_line\_bytes & chunks of memory handled by the cache & CPU; GPU & 64 \\ max\_local\_memory\_per\_block & maximum local memory per block in bytes & GPU & 2,147,483,647 \\ max\_shared\_memory\_per\_block & maximum shared memory per block in bytes & GPU & 49,152 \\ max\_threads\_per\_block & maximum number of threads per block & GPU & 1024 \\ max\_vthread\_extent & maximum extent of virtual threading & GPU & 8 \\ num\_cores & number of cores in the compute hardware & CPU & 24 \\ vector\_unit\_bytes & width of vector units in bytes & CPU; GPU & 64, 16 \\ warp\_size & thread numbers of a warp & GPU & 32 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Hardware Parameters Considered While Training. \begin{table}
\begin{tabular}{l c c} \hline \hline
**Hardware Platform** & **Processor** & **Remarks** \\ \hline Intel Platinum 8272 CL @ 2.60 GHz & CPU & 16 cores, AVX-512 \\ AMD EPYC 7452 @ 2.35 GHz & CPU & 4 cores, AVX-2 \\ ARM Graviton2 & CPU & 16 cores, Neon \\ NVIDIA Tesla T4 & GPU & Turing Architecture \\ NVIDIA GeForce RTX 2080 & GPU & Turing Architecture \\ NVIDIA A100 & GPU & Ampere Architecture \\ NVIDIA A40 & GPU & Ampere Architecture \\ NVIDIA H100 & GPU & Hopper Architecture \\ Intel Gold 5115 @ 2.40 GHz & CPU & 40 cores, Xeon \\ ARM A64 FK & CPU & 48 cores, aarch64 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Compute Hardware Description. In our efforts to streamline the tensor program generation, we strategically pruned the expansive search space. Relying on a search task driven by measurements from randomly sampled kernels proves unreliable, particularly when the search space lacks richness. Our observations underscore that certain hardware parameters, such as the number of cores, exhibit less significance as features compared to flop count, especially in relation to latency. This arises from the necessity for diverse kernel combinations and hardware features within the dataset under consideration. Consequently, we conducted an evaluation focusing on the interplay of FLOPs' count, kernel shape, and execution time for various tensor operations on a specific hardware platform. The outcomes, detailed in Table 3, reveal similar behavior between CPUs and GPUs in selecting optimal kernel shapes.