# Cross-Feature Transfer Learning for Efficient Tensor Program Generation
## 2 Background and Related Work
### Machine Learning-Based Autotuners  
Machine learning approaches for optimizing tensor programs [9; 21; 22; 23; 24; 25; 26; 27] are heavily researched for tensor compilers and deep learning workloads [5; 28]. The work from [PERSON] et al. [29] generates tensor programs for DL workloads by exploring optimization combinations through a hierarchical search space and optimizing subgraphs with a task scheduler. The use of LSTM to sequence optimization choices [30] has been explored. Moreover, [PERSON] et al. [31] automate generating and optimizing numerical software (version 3.6, [[https://www.netlib.org/atlas/](https://www.netlib.org/atlas/)]([https://www.netlib.org/atlas/](https://www.netlib.org/atlas/)), accessed date on 4 January 2024) for processors with deep memory hierarchies and pipelined functional units, making it adaptable across server and mobile platforms. Additionally, authors have employed reinforcement learning (RL) in various works. [PERSON] et al. [32] investigate transforming the integer linear programming solver into a graph neural network-based policy for auto-generating vectorization schemes. Also, domain-specific compilers such as COMET [33], JAX [34], and NWChem [35] are under active research for optimizing program execution. Lately, [PERSON] et al. [10] have proposed a one-shot tuner for the tensor compilers. Its limitation is that it considers only task-specific information, which prevents it from being applied for transfer learning. On the other hand, we employ neural network and hardware platform information so that the autotuner can learn better. [PERSON] et al. [36] introduce a novel approach to tensor program optimization, explicitly focusing on biased-diversity-based active learning. The primary objective is to streamline the training process while maintaining a consistent program performance optimization level comparable to the baseline Tenset. They implement a unique biased-diversity-based diversity scheme within their framework to achieve this. This scheme addresses a critical issue in active learning: the problem of overestimation when utilizing brute-force sampling techniques. The framework comprises two components: AL-based model pre-training and pre-trained-model-based program optimization. However, it is essential to note that this framework employs random program sampling without considering hardware characterization. Subsequently, these sampled programs are executed on the hardware to collect performance measurements. To generate unlabeled program datasets, the framework randomly samples transformed tensor programs from predefined optimization tasks. The authors employ a diversity-based selection approach to balance the distribution of these samples and, in particular, increase the proportion of high-performance programs. This strategy is designed to enhance the diversity of selected programs' performance, ultimately contributing to the effectiveness of the active learning process. In another research, [PERSON] et al. [37] address the challenge of latency estimation using evolving relational databases. To achieve this, they introduce two integral components: the Neural Network Latency Query System (NNLQ) and the Neural Network Latency Prediction System (NNLP). The central innovation of this system lies in its capacity to automatically convert deep neural networks (DNNs) into executable formats based on prior knowledge derived from offline latency measurements. The Latency Querying System employs a Graph Neural Network (GNN)-based foundation to extract cohesive graph embeddings for various DNN models. It also incorporates a multi-headed approach to facilitate concurrent latency prediction across multiple hardware platforms. This framework further empowers users to discern the supported tensor operators on a given hardware setup while considering achievable latency and the tradeoff between latency and accuracy.