# Cross-Feature Transfer Learning for Efficient Tensor Program Generation
## 3 Design and Implementation
### Autotuner Architecture  
We have employed an attention-based [47] tuner to experiment with a joint neural network and hardware features as part of the task. In addressing the generation of tensor programs with reduced latency, it is crucial to recognize the convex nature of the problem, rendering search-based methodologies particularly useful. Leveraging schedule information from the comprehensive TenSet dataset, we focused on feature extraction, with the length of the schedule primitive serving as the designated sequence length. The internally processed features are subsequently encoded using the one-hot encoding technique, facilitating their integration into the embedding layer and determining the overall embedding size. As highlighted earlier in the discourse, our analysis targeted the identification of high-performing tensor kernels. Notably, our findings reveal a prevalence of these high-performing kernel tensor program schedules across both CPU and GPU architectures, underscoring their cross-platform efficacy. In outlining our model architecture, we employ an attention mechanism (Equation (3)) as the core component to grasp contextual features. This mechanism operates simultaneously, allowing for efficient parallel execution. Our model undergoes training using the existing dataset, and in the event of encountering new hardware, we fine-tune the learned autotuner to adapt to the updated conditions by learning new weights. To facilitate transfer learning and accommodate updates through fine-tuning, we incorporate multiple heads (Equation (4)), each tailored to the specific hardware parameters obtained from the system under test. This approach enhances the adaptability and performance of our model across diverse hardware configurations. \[\text{Attention}(Q,K,V)=\text{softmax}\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}} \bigg{)}V \tag{3}\]  
where  
\(Q\), \(K\), and \(V\) are the query, key, and value matrices, respectively;  
\(\text{d}_{k}\) is the dimension of the key vectors. \[\text{MultiHead}(Q,K,V)=\text{Concat}(\text{head}_{1},\dots,\text{head}_{h})W _{O} \tag{4}\]  
where  
\(\text{head}_{i}=\text{Attention}(\text{QW}_{Qi},\text{KW}_{Ki},\text{VW}_{Vi})\)  
\(\text{W}_{O}\) is the output transformation matrix. We designed our autotuner for efficient attention-based processing with applications in multiple tasks. The architecture incorporates a sequential encoder, composed of four linear \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline \multirow{2}{*}{**Sampled Kernels**} & \multicolumn{2}{c}{**\#Kernel\_Shapes**} & \multicolumn{2}{c}{**Max GFLOPs**} & \multicolumn{4}{c}{**Mean Execution Time (ms)**} \\ \cline{2-9}  & **CPU** & **GPU** & **CPU** & **GPU** & & **EPYC-7452** & **Graviton2** & **Platinum-8272** & **T4** \\ \hline T\_add & 229 & 388 & 8.59 & 8.59 & [4, 256, 1024] & 180.97 & 81.25 & 92.86 & 4.31 \\ Conv2 dOutput & 60 & 27 & 1.20 & 1.07 & [4, 64, 64, 32] & 40.94 & 14.21 & 19.11 & 2.07 \\ T\_divide & 24 & 69 & 0.003 & 0.003 & [8, 1, 1, 960] & 0.07 & 0.05 & 0.11 & 0.10 \\ T\_fast\_tanh & 9 & 9 & 0.008 & 0.008 & [4, 1024] & 0.43 & 0.43 & 0.53 & 0.97 \\ T\_multiply & 105 & 150 & 8.92 & 8.92 & [4, 256, 4096] & 320.74 & 48.08 & 95.65 & 0.55 \\ T\_relu & 300 & 1257 & 73.46 & 73.46 & [4, 14, 72, 8, 64] & 0.52 & 5.70 & 0.72 & 0.23 \\ T\_softmax\_norm & 27 & 27 & 0.016 & 0.016 & [4, 16, 256, 256] & 1.01 & 2.78 & 4.08 & 0.19 \\ T\_tanh & 9 & 9 & 0.905 & 0.629 & [8, 96, 96, 3] & 5.55 & 33.48 & 50.55 & 0.16 \\ conv2d\_winograd & 0 & 33 & NA & 0.868 & NA & NA & NA & NA & 0.93 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Neural Network And Hardware Features’ Characterization. layers with ReLU activation functions, to capture and transform input features. An attention mechanism is employed, allowing for parallel execution and facilitating the extraction of contextual information. The attention output is enhanced via residual connections in a series of residual blocks, each containing two linear layers followed by ReLU activation and batch normalization.