BALTO: fast tensor program optimization with diversity-based active learning
Authors: Jun Bi
Xiaqing Li
Qi Guo
Rui Zhang
Yuanbo Wen
Xing Hu
Zidong Du
Xinkai Song
Yifan Hao
Yunji Chen
Venue: ICLR 2023 poster (2023)

Abstract: Tensor program optimization (TPO) based on pre-trained models can effectively reduce the computing time of deep neural networks. However, training of such models is prohibitively expensive, which highly depends on a large-scale dataset and thus requires tremendous time-consuming performance measurements (more than 1 million) on target platforms. In this paper, we propose BALTO, a fast TPO approach with biased-diversity-based active learning, aiming at reducing much lower training costs under similar optimization accuracy.The key insight is that random sampling of existing approaches suffers from a heavy redundancy of low-performance programs, which incurs tremendous duplicated time-consuming measurements. Inspired by this, BALTO removes such redundancy by introducing active learning (AL) to TPO for a much lower training cost. However, applying AL with a brute-force way in BALTO can lead to an overestimation problem. To address this, we further propose a biased-diversity-based diversity scheme specially designed for BALTO. We compare BALTO against TenSet on $6$ typical hardware platforms over $2$ learning models. Experimental results show that, on average, BALTO only requires 5% of the total performance measurements of TenSet to achieve the same or higher model accuracy. Moreover, the optimized tensor programs even outperform that of TenSet by 1.06% due to higher model accuracy.

Published as a conference paper at ICLR 2023
BALTO: FAST TENSOR PROGRAM OPTIMIZATION WITH BIASED-DIVERSITY-BAAEI ACTIVE LEARNING
Jun Bi Xiaqing Li", Qi Guo', Rui Zhang?, Yuanbo Wen', Xing Hu², Zidong Du²,
Xinkai Song', Yifan Hao", Yunji Chen
I 'University of Science and Technology of China
SKL of Processors, ICT, CAS Cambricon Technologies, Beijing, China
ABSTRACT
Tensor program optimization (TPO) based on pre-trained models can effectively reduce the computing time of deep neural networks. However, training of such models is prohibitively expensive, which highly depends on large-scale dataset and thus requires tremendous time-consuming performance measurements (more than million) on target platforms. In this paper, we propose BALTO, fast TPO approach with siieeddddveestty-buueeee active learning, aiming at significantly reducing training costs under similar program performance optimization ability. The key insight is that random sampling of existing approaches suffers from a heavy redundancy of wew-peforrmanc programs, which incurs tremendous time- consuming measurements. Inspired by this, BALTO removes such redundancy by introducing active learning (AL) to TPO for much lower training cost. However, applying AL with a brute-force way in BALTO can lead to an overestimation problem. To address this, we further propose a biased-diverss sorddddieesity-baaee scheme specially designed for BALTO. We compare BALTO against TenSet on 6 typical hardware platforms over 2 learning models. Experimental results show that, average, BALTO only requires 5% of the total measurements of TenSet to achieve the same or higher model accuracy. Moreover, the optimized tensor programs even outperform that of TenSet by 1.07% due to higher model accuracy.
diversity
INTRODUCTION
Tensor program optimization (TPO) can effectively reduce the computing time of neural networks by searching for high-performance programs in designed search space (Chen et al.) 2018 Zheng (et al., 22200a b). In TPO, neural networks are first represented as tensor programs that describe the computation of muuli--dimensionll data arrays. Then performances of these tensor programs measured on a target hardware platform. Such measurements are cons-mnnn and thus optimiz- ing a given network can cost several days
weeks, which greatly hinders the wide application of TPO (Zhu et al.) 2022). To accelerate TPO, pre-trained machine learning models (Adams et al.) 2019 Haj-Ali et al, 2020 Anderson al., 2020 ZZheng al.) 202T) are introduced to replace substantial part of the hardware measurements with performance predictions of a pre-trained model. For example, the state-of-the-art approach, TenSet (Zheng al. 202 can significantly reduce the optimization time by up to 10x through training on a large-scale and well-established dataset.
However, training the models prohibitively expensive. The main reason is that these models highly depend on large-scale training dataset. Unfortunately. collecting such a dataset involves massive performance measurements on hardware platforms, suffering from excessively long execution time. For example, for each hardware platform, around 8 million different tensor programs are required to be measured (Adams et al., 2019 /Zheng et al.) 2021). Even on a high-end GPU like V100, such measurements still can consume about 4000 GPU hours. This burden could be much worse in those capability- limitedd systems, e.g., mobile devices such as NVIDIA Jetson TX2, most of which often require more than 5000 GPU hours to conduct the measurements. More importantly, much larger dataset is required in real-world optimization tasks, so as to achieve better model generalization
"Corresponding author.

---

Published as a conference paper at ICLR 2023
on different tensor programs. Consequently, the size of the dataset increases, the number of the measurements can be significantly increased correspondingly. This can lead to great consumption of time and energy, and thus hinders wide deployment of ML-based TPO in industrial practice.
To look more closely at the inside of the large-
scale datasets sampled randomly by existing approaches, we conduct a deep exploration
Number of programs
10000
30000
[a]
Number of programs
5000
6000
(b)
25000
4000
of the distribution of these datasets. We ob-
20000
3000
19000
serve that random sampling adopted in exist- ing approaches can result in imbalanced train-
10000
5000
ing datasets, where the high-perte regh-performacce pro- grams are excessively less than that of the www-performance ones. We randomly sample 90,000 tensor programs of TenSet (as shown in Figure D and find that high-performance pro- grams only account for 19% and 8% the to- tal dataset on Platinum 8272CL CPU and T4 GPU respectively. In contrast, the low owwpeeformaace programs take the bulk of the total dataset (81% and 92% respectively). Such imbalance can fur- ther lead heavy redundancy of the low-performance programs that incurs tremendous time- consuming measurements. The main reason behind the redundancy is that the importance of gener- ated tensor programs highly different. fact, program optimization pays more attention those high-perffomance programs. Excessively more pew-performance programs cannot offer additional benefit of predicting high-performance programs, thus being a heavy redundancy.
0.02
0.32
0.62
0.92
0.02
0.32
0.62
0.92
Relative Performance
Relative Performance
The ratio of high-performance programs is much smaller than low-performance programs. (a) NVIDIA T4. (b) Platinum 8272CI..
Figure 1:
To this end, we propose BALTO, a fast TPO approach with biased-diversity-bass diversity active learning, aiming at significantly reducing training costs under similar optimization ability. The key insight is that random sampling of existing approaches suffers from a heavy redundancy of low- performance programs, which incurs tremendous time-consuming measurements. Inspired by this, BALTO removes such redundancy by introducing active learning (AL) to TPO for a much lower training cost. In this way, the measurements can be significantly lowered, thus greatly reducing the training cost of building pre-trained model. However, applying AL in brute-force way in BALTO can lead to an overestimation problem, where the relative performance accuracy is much better than that of the ground truth. To address this problem, we further propose a dineeedddveesity--bee based scheme specifically designed for BALTO, which can efficiently reduce the distribution imbalance of the sampled programs caused by overestimation.
the estimated
Finally, we integrate BALTO into TenSet, and compare BALTO with state-of-the-ar baseline TenSet on typical hardware platforms (i.e., two GPU platforms and four CPU platforms) two learn- ing models (i.c., XGBoost, MLP). The experimental results show that BALTO achieves same or higher model accuracy while only requiring 5% of TenSet's hardware measurements. Moreover, the optimized tensor programs even outperform that of TenSet by 1.07% due to higher model accuracy.
To the best of our knowledge, BALTO is the first work reduce the training cost of pre-trained- model-basee TPO by introducing AL. Summarily, our key contributions are three-fold:
We conduct a deep exploration on the distribution of large-scale datasets sampled randomly by existing approaches, and observe that the random sampling results in an imbalanced dataset, and thus suffers from heavy redundancy in the training dataset.
We propose BALTO, a fast TPO approach with active learning, aiming at significantly re- ducing training cost under similar optimization ability. To address the overestimation prob- lem, we further propose biased-divveriit-bbad based scheme specially deigned for BALTO.
We conduct comprehensive performance evaluation on six typical hardware platforms, indicating that BALTO achieves same or higher model accuracy with 20x reduction in performance measurements. Moreover, the optimized programs even outperform that of TenSet by 1.07% on NVIDIA T4.
2 BACKGROUND
2.1 TENSOR PROGRAM OPTIMIZATION.
2

---

Published as a conference paper at ICLR 2023
TPO is a process of transforming an input tensor program into another tensor program with optimal or near-optimal performance. Figure demonstrates Degunedeeoosstttttt cal TPO. As shown in F it mainly consists three modules, including program performance estimation, op- timization space exploration, and program performance evaluation. The program performance estimation module is used for predicting the performance of input programs, which usually composed of machine learning models or expert-designed models. The optimization space ex- ploration module searches for high-ppeffrmmacc program transformations in pre-defined search space and outputs a batch of programs with high scores on predicted per- formance. The program performance evaluation module compiles the batch of programs and measures their execu- tion times the hardware platform. The measured performance be used for further fine-tuning the models. Finally, the program with the highest measured performance is output as the optimiza- tion result.
Program Performance
Program Performance
Estimation
Evaluation
workflow of typi-
Update
Program
Optimization Space Exploration
Candidate Tensor Programs
Optimized Program
Figure 2: Workflow of TPO.
2.2 PRE-TRAINING OF PERFORMANCE MODEL.
Pre-training of the performance model has key stages typically: the sampling stage and the training stage. In the sampling stage, model pre-training usually depends on a large-scale dataset based program performance, which is generated by performance measurements ware platforms. Concretely, the sampling process mainly consists of three steps. First, a set of program optimization tasks are generated by the compiler, such as conv2d operator with different shapes. Second, for each generated task, random sampling is conducted in the optimization search space to produce set of tensor programs without performance measurement. Third, these tensor programs are delivered to the module of performance evaluation for performance measurement, and finally, the measured performance and the features tensor programs are kept the dataset.
target hard-
In the training stage, generally, the performance model aims at fitting the relative performance of the T_, programs, which can be denoted as = Tmaa Concretely, T denotes the throughput of the program, and Maa denotes the maximum throughput that can be reached by the corresponding optimization task of that program. Since the maximum throughput Tmaz of the task is unknown, T Tmax usually can be estimated by the maximum throughput of the sampled programs. Importantly, such estimation training models
can be quite accurate if given large number of samples. The commonly-use include XGBoost, MLP, and LSTM, etc, and the goal of model training is to minimize rmss or lambdarankloss. The evaluation criteria of the model is greatly critical. Compared with rmse or R²2 the Top-k performance predicted by the model on each optimization task is more related to the optimization quality of the compiler (Zheng et al. (02I]. For given set of tensor programs, the Top-k performance can be calculated as the division of the predicted Top-k programs' average performance and the optimal programs' performance.
2.3 DIVERSITY-BASED ACTIVE LEARNING
The training of machine learning, especially deep learning, highly depends on a large num- ber of labeled samples whose generation can be prohibitively expensive. Active learning (AL) popular training framework, aiming fectively reducing the requirement of expensive labels. In each iteration of an AL process, un- labeled samples are firstly selected from a pool with a fixed size, then the selected samples are delivered to be labeled, and finally these labeled samples are used for model training. The core of AL is the selection method which can effectively reduce the labeling redundancy while achieving similar model accuracy. A commonly used ap-
Algorithm Core-set Greedy Selection
Require: Dr, Du. B,
Output: S; is set of samples to be labeled
I: S. +
2: while [Se| B, de
3:
arg max, mine, Dijs d(a,, 4: S U [u}
5: return S.
3

---

Published as a conference paper at ICLR 2023
proach for selection is diversity- based selection, which selects representative samples for labeling to increase diversity.
The Core-set selection (Sener & Savarese 2018). one of the state-of-the-art diversity-basee approaches, formulates sample selection as solving the k-center optimization problem of min
min d();; s.t. |St| B,, where S is the samples selected at time step S.SDu 2,ES, ajED;US,
d(E;, #r) is measure of the distance between sample and Tj,, and B, is the total budget of samples at step l. Since the problem is NP-Hard, a common approach leverages the greedy strategy shown in Algorithm to obtain 2-OPT solution.
3 ACTIVE LEARNING BASED MODEL TRAINING
3.1 OVERVIEW OF BALTO
BALTO is a pre-trained-mee modeee---ddel-baeedd TPO approach that mainly consists of two parts, including AL-based model pre-training and pre-trained-model
program optimization, shown in Fig- The model pre-training part is an active learning process that consists of four steps including program sampling, model training, program selection, and performance evaluation. Regarding step- 1, BALTO samples programs randomly to form a large unlabeled dataset D, and a small labeled the hardware to obtain the labels. Then BALTO iterates step-2, step-3, and step-4 to train the model based on active learning. Regarding step-2, BALTO trains the model based Dj. Regarding step-3, BALTO leverages the trained model and the labeled data set to select batch of programs from D. for evaluation. Regarding step-4, the performance of programs are measured for updating Dj. Finally, the model training part outputs trained model for later program optimization tasks once the total measurement budget is exhausted.
ure
dataset Dj. Programs in labeled data set measured
The program optimization part takes the commonly used workflow described in Section 2.1
AL-based Model Pre-Training
Performance Evaluation
Program
Labelled Dataset
Upcate dataset
Sampling
M
Training dota
Unlabelled Dataset
Model
Cost model
Program Selection
Training
Exhousted
Program Optimization
Model
Program Optimizer
Input Tensor Program
Optimized Program
Figure 3: Overview BALTO.
3.2 PROGRAM SAMPLING
BALTO generates unlabeled program data sets by randomly sampling transformed tensor programs from pre-defined optimization tasks. optimization task corresponds to a machine learning op- erator (e.g.. MatMul) with specific shape, c.g., (1024, 1024, 1024). For given task, programs can be randomly generated either by performing random transformations on program sketches(c.g. TenSet (Zheng et al. 2021) and Ansor (Zheng al. 2020aa) by sampling from probabilistic- language-defined stochastic search spaces (e.g., MetaSchedule (Shao et al. 2022
Random sampling result in unbalanced distribution of the program performance. In such an unbalanced distribution, the number of low wwwperffomance programs accounts for a large part of the total samples, which makes the model easily fit these low-performance programs instead of the
4

---

Published as a conference paper at ICLR 2023
peghhperformacce ones given relatively small number of total samples. As result, the prediction of the model for those high-performance programs falls into a very low accuracy, and thus cannot effectively guide the compiler to identify sigh-peforrmnce programs. Therefore, it is essential to balance the distribution of the samples by an appropriate program selection strategy that can increase the proportion of perforerfor programs reasonably.
3.3 PROGRAM SELECTION
To balance the performance distribution of selected programs, BALTO leverages iiversity-basel selection to increase the diversity of selected programs' performance. We first propose an output- diversity-basee selection scheme that maximizes the diversity of the predicted performance of se- lected programs based on core-set selection. However, the selection scheme suffers from a perfor- mance overestimation problem which make the distribution unbalanced even under the proposed selection scheme. Thus we further propose a biased-diversity-basee selection to solve the problem.
nutput-diversity-basee selection. This selection scheme is based the greedy selection scheme shown in Algorithm[ I with doue (T; = - where / (2;) is the predicted performance of program I₁. The main advantage of the distance function is that can ensure much more balanced distribution of the sampling. This is because the diversity of the predicted performance makes the predicted performance of the sampled programs more decentralized into the range [0.1], rather than centralized around O, thus greatly reducing the proportion of low- now-performance programs.
Performance overestimation. For given task, the maximal program throughput Tmaz can be underestimated to a large extent under small number of performance measurements. Correspond- ingly, the underestimated Taaz further incurs overestimated relative performances of most programs at the early stage of AL process. As result, even with the diversity- selection, such overes- timation can still make the sampled programs imbalanced and mainly gathered at cow-performance regions.
Biased sioedddiveritt---ee based selection. To solve the overestimation problem, we propose biased selec- tion process that can effectively select the high- performance programs in of that the sampled programs are gathered at low-performance regions. Concretely, we assign a weight (i.e., f(x)) for program 2; encourage the algorithm to select programs with high scores predicted per- formance, so as to accelerate the exploration of programs in high-performance regions. Thus the distance function becomes diiased xT)) ) J (Ir.))ous (x,,),)))) Compared to the core-set selection that selects samples by solving the k-center problem, we formulate the biased selection as solving following weighted k-center problem:
min max fx;;) min S.D F:E8,
22,) ayED,
s.t. |S $ B,
Since the problem is NP-hard, we solve it via greedy strategy as in Algorithm
2
For each task, BALTO performs biased-diversity-baseee selection when the measured performances are gath- ered in the cow-performance regions and performs ouput--iiverstty-bani based selection when the distributions are well balanced. Although the biased selection scheme can alleviate the imbalance of performance distribution caused by overestimation, it scarifies
Algorithm 2 Biased-diversity-basse Selection
Require: Di, D., B,
Output: S, set of samples to be labeled at iteration
I: St
while [Sa| < B, do
u = arg max max E,.. (xx) minsseenue [((x))
S. - S. (u)
5: return S.
the diversity of high-performance programs to a certain extent. To increase the diversity, BALTO leverages the KL.-divergence D formance of measured programs of the task, as a criterion for choosing the selection scheme. P(z) with p(1) = 1 is a binomial distribution that describes a distribution of selected programs where all programs are pegeh-erformacce Q(x;) with g((x; is high--prrformance)) binomial distribution where a program with a relative performance of y; has a possibility of yı to be performanee DKL
-E[log(y:)] where yi represents the relative per-
yı is
effectively estimate how close the distribution of the currently sampled programs to P. As such, the lower DKL is, the closer the relative performance of the sampled programs is to 1 and the lower the diversity is. If DrL. z. is greater than constant threshold
5

---

Published as a conference paper at ICLR 2023
(i.e., C), BALTO enables the -iiseddiiversity--ee selection scheme select programs since the measured programs are gathered in the low-performance region. Otherwise, BALTO enables the output-diversity--oe bassd selection scheme since the overestimation problem is alleviated.
Selection complexity. BALTO trains the model for T' iterations. each iteration, BALTO iterates over all of the N tasks to select programs for measurement. For each task, the algorithm can select at most B, samples from D,, that consists of M programs of the given task. For a single selection of programs, the algorithm performs either the output-diversity-basee selection or the biased-diversity- based selection according to Dx Both of the two selection schemes have a complexity of O thus the total selection complexity O B, M or O(BM2) where B is the total measurement budget. Take TenSet for example, the B is 429,8 810, the M is 4,000, and the total selection time is 10.6 minutes. Thus, the adopted selection strategy in BALTO does not bring additional overhead to the training process.
PERFORMANCE EVALUATION
We integrate BALTO into TenSet to evaluate the performance comprehensively. The experimental methodology is five-fold. First, show that BALTO achieves comparable accuracy while with much less hardware measurements on 6 platforms. Second, we demonstrate that the models trained with much less performance measurements have comparable optimization ability hardware platform. Third, we demonstrate that BALTO is a generic optimization approach by inte- grating it to MetaScheduler. Fourth, we perform ablation study verify the effectiveness of the core components of BALTO. Fifth, we visualize the performance distribution of our proposed sampling strategy illustrate that the proposed strategy helps balancing the distribution of programs.
real-world
Baselines. To demonstrate the effectiveness of BALTO, we compare BALTO with five baselines including: 1) TenSet trains the model by measuring all the randomly sampled programs; 2) GSx (Yu
& Kim 2010) selects programs by maximizing the feature diversity greedily: 3) GSy (Wu et al.
2019) selects programs by maximizing the label diversity greedily: 4) iGS (Wu al. 2019) selects
programs by maximizing the label and feature diversity greedily; 5) ALT (Zeng et al. 2020) selects
programs by the uncertainty of the model prediction. The proposed output-iiversitybbas based selection represented Oursl and the soased-ddversity-baeeee selection is presented as Ours2.
4.1 COMPARISON OF BALTO AND TENSET FOR MODEL PRECISION
Dataset. We evaluate BALTO's effectiveness on the dataset provided by TenSet. The dataset consists of program performance measurement records from 6 different hardware platforms. Each platform includes total number of 8, 8,596,208 tensor program measurement records that are sampled from 2307 different types of tasks. We use 10% of the records as the test dataset and the remaining 90% records as the train dataset for the baselines. We further select at most 5% of the training dataset for training BALTO and other active learning approaches (i.e., GSx, GSy, iGS, and ALT). We report the Top-1 score on the test dataset based on 5 independent experiments.
Results. We compare BALTO with baselines on two ML models including XGBoost and MLP. All the models adopt the same hyperparamenters with TenSet. As shown in Table I BALTO out- performs all other active learning baselines and achieves the same even higher accuracy com- pared with TenSet. Compared to TenSet, BALTO reduces the hardware measurements by 20x. Take T4 for example, BALTO achieves accuracy improvement of 4.1 1% on XGBoost model and achieves the same accuracy on MLP model respectively. The reason that TenSet achieves a relatively lower accuracy that TenSet trains on a randomly sampled program dataset where the ratios of low-performance programs and high-performance programs are extremely imbalanced. Differently. such imbalance can be greatly alleviated by the proposed biased-diversity-bbe BALTO, thus delivering much lower training cost but similar or even higher accuracy.
active learning of
4.2 OPTIMIZATION PERFORMANCE ON WEAL-WORLD PLATFORM
To verify that cost models trained much fewer programs still effectively guide the TPO process, we evaluate BALTO on NVIDIA T4 and report the optimized execution time 5 commonly used neural networks of BALTO and the baselines.

---

Published as a conference paper at ICLR 2023
Table 1: Comparison between BALTO and TenSet 6 hardware platforms with 2 learning models. GPU- and GPU-2 represents NVIDIA T4 and NVIDIA K80, respectively. CPU-1, CPU-2, CPU-3, and CPU-4 represents Intel Platinum 8272CI, Intel E5-2673 v4, AMD EPYC 7452, and ARM Graviton2, respectively.
XGBoost
GPU-1
GPU-2
CPU-1
CPU-2
CPU-3
CPU-4
TenSet
84.7 + 0.4
84.9 4 0.1
82.7 + 0
83.0 =
83.9 $3.9
79.2 +
GSx
80.9
: 78.1
78
79.6
7
GSy
8
79.8
7
80
81440
7
iGS
0
80.9
79:20
8.2
0.4
76.8
ALT
8
85.2
81.5
83.7
84.6
78.8
Ours1
87.3 0.5
87.6
86.2
85.6
8.13
80.6
Ours2
88.8 0.2
0.1
+
86.4 + 0.5
8.2
81.6
MLP
GPU-1
GPU-2
CPU-1
CPU-2
CPU-3
CPU-4
TenSet
90.5 1 0.3
0.5
86.5 1 0.8
86.1 0.4
88.1 0.5
80.8 1 0.8
GSx
13.51..
0.9
76.2 0.2
74.8 0.3
77.6
69.5 0.7
GSy
1.9
0.3
7.11
1.6
7..7
68.9
iGS
0.4
0.2
7.21
2
0.9
69.2
ALT
8.4
0.2
1.2
1.2
0.9
2.
Ours1
89.6
0.7
8.4
8..4
8.6
191
Ours2
90.4 0.6
90.3
16.9+100
87.2 1.1
89.1 1.3
81.1 =
Benchmarks. We evaluate the optimization results on 5 commonly used neural networks including ResNet-50 (He al. 2016). MobileNet-v2 (Sandler al.) 2018), ResNext-50 (Xie al. (017). BERT-tiny and BERT-base (Devlin et al. 2018). For the three CNN models, we set the batch size and the input shape to 224 224. For BERT models, we the sequence length be equal
to 128. We assign at most 1,000 trials of hardware measurements for each network and report the average execution time based on 5 independent experiments of program optimization.
Table Evaluations of BALTO and TenSet on 5 commonly used neural networks.
XGBoost
ResNet-50
MobileNet-v22
ResNext-50
BERT-tiny
BERT-base
TenSet
3.89 +: 0.4 ms
0.60 + 0.04 ms
3.41 + 0.31 ms
3.06 + 0.30
10.8 + 0.2 ms
GSx
3.92 + 0.1
0.62 0.08
3.62 + 0.09
3.11 + 0.05
12.2 + 0.7
GSy
4.39 + 0.4 ms
0.63 0.05 ms
3.65 0.56 ms
3.03 +
11.1 1.1 ms
iGS
3.81 + 0.1 ms
0.65 : 0.03 ms
3.71 0.73 ms
3.26 + 0.21 ms
11.0 1.0 ms
ALT
3.62 + 0.1 ms
0.59 + 0.02 ms
3.30 + 0.04 ms
2.81 0.13 ms
11.1 + 1.6 ms
Oursl
3.68 + 0.5 ms
0.59 0.06 ms
3.27 + 0.1
2.84 + 0.09
10.9 + 0.3 ms
Ours2
3.30 + 0.2 ms
(1.58 0.04 ms
3.23 0.17 ms
2.76 + 0.05
10.8 + 0.4 ms
MLP
ResNet-50
MobileNet-v2
ResNext-50
BERT-tnn
BERT-base
TenSet
3.28 4: 0.15 ms
0.57 +: 0.03 ms
3.26 + 0.32 ms
2.80 1: 0.09ms
10.0 1 0.4 ms
GSx
4.13 1.13 0.17
0.64 1 0.08
3.50 + 0.32
3.16 1 0.13
11.5 1 0.8
GSy
3.79 0.58 ms
0.62 0.07 ms
3.40 0.24 ms
1 0.15 ms
11.5 0.5 ms
iGS
3.58 + 0.09 ms
0.62 16066.00 ms
3.43 0.13 ms
2.91 + 0.01 ms
11.3 1.1 ms
ALT
4.04 404::0 0.25 ms
0.75 1 0.19 ms
3.30 - 0.16 ms
3.06 + 0.16 ms
11.6 0.9 ms
Ourst
3.43 + 0.42 ms
0.60 0.02 ms
3.30 = 0.33
4.097
10.8 + 0.5 ms
Ours2
3.33 + 0.17 ms
0.58 0.00 ms
3.14 0.17
2.71 0.13
9.9 4 0.4
Results. As shown in Table BALTO achieves the same better optimization performance com- pared the baselines. Specially, when using the XGBoost cost model, BALTO achieves an average of 1.07x performance improvement over TenSet. The improvement comes from more accurate performance model that helps the compiler to identify the high-performance programs efficiently. When using the MI.P cost model, the performance of BALTO is the same that of TenSet since the two approaches' models have the same accuracy on T4. The experiments' results show that models trained on much fewer measured programs can still guide the optimization efficiently. Thus, do not need to measure as much of programs as existing optimization frameworks do.
4.3 COMPATIBILITY OF BALTO

---

Published as a conference paper at ICLR 2023
BALTO is generic pre-trained-mort wowerddmmodl--bebeeeeee tensor program optimization approach that is compatible with other TPO frameworks. To demonstrate this, we integrate BALTO into MetaScheduler (Shao et al. 2022) perform
1.12
250
Pretrained
MetaSchoduler
Relative Performance
Total CPU CPU hours
200
Ours
1.04
150
100
auto-tensorization using TensorIR (Fnng et al.
2022). Take the GEMM operator for exam- ple, we randomly sample 800,000 unmeasured programs at first. To train the model, we then use BALTO to measure at most 5% of the to- tal programs on an Intel Xeon Gold 6226R CPU. To verify the optimization ability of the trained model, we generate 80 tasks with dif- ferent problem sizes (i.e., N 512 + 64
0.96
50
0.92
2000
4000
6000
Problem Size
Number of tasks
Figure 4: Comparison of optimized performance and tuning time. (a) Optimized performances of GEMM op- erators with shape (N, N, N). (b) Total tuning time.
0, 0, 79). We experiment on two baselines including MetaScheduler and tne-trrined-mmmelleee MetaScheduler (i.e., the Pretrained). The MetaScheduler guides the search with model trained from scratch and the Pretrained trains the model by measuring all the 800,000 programs. For given task, we limit the total trials of MetaScheduler 1000 and the Pretrained as well as BALTO to 100. Regarding the optimization ability, we compare the relative performance which is normalized MetaScheduler. As shown in Figure a)), the relative performance of BALTO to MetaScheduler is between 0.93 and 1.08x with 0.992 x on average, which demonstrates that BALTO achieves similar optimization ability. Regarding the tuning time, BALTO reduces the pre-training overhead by x and the single task optimization time by 10x. As shown in Fig the total tuning time of MetaScheduler grows rapidly with the number of optimization tasks while our tuning time grows much slower.
4.4 ABLATION STUDY
Effectiveness of biased selection. As shown in Figure 5 biased selection with a proper C (e.g., C' 1) can outperform the output-diversity-basdd approach and the random selection approach. At the early stage of the training, the accuracy score of the biased selection approach grows much faster than that of the output-diversitybaaseee effectively reduces the distribution imbalance caused by the overestimation problem. Meanwhile, after the early stage of the training, the estimation of the relative performance becomes more accurate as well as the distribution becomes more balanced. Therefore, the biased selection switches to the selection and finally achieves a higher precision score under the same budget.
approach, since the bias for nigh-performance programs
output-diversity-baar
Optimization ability sub-graphs. As shown in Figure Figure 7 BALTO achieves compa- rable optimization ability with TenSet on two representative sub-graphs including batched matmul and 2D convolution. Both BALTO and TenSet converge faster than Ansor since the pre-trained mod- els are accurate enough to guide the search without massive hardware measurements. For sub-graphs (i.e., batched matmul) with large search spaces, Ansor requires to converge since it searches with an inaccurate model that is trained from scratch. Meanwhile, the traiined-model-baseee approaches search such spaces more quickly and efficiently under the guidance of an accurate model.
trials (i.e., greater than 1000)
Per Performance (TFLOPS)
Performa Performand ann.orrmnnee (TLOSS)
Top-1 accuracy(55
855
ce
80
butout siverstty
Ansor
Ar sor
25
random
Tenset
Tenset
dissed-ielecioon C-1
200
400
600
800
1000
200
400
660
800
1000
Percentage total programs(*6)
Measurement trials
Measurement trials
Figure Comparison of differ- ent selection schemes.
Figure 6: Batched matmul,
Figure 7: 2D convolution.
8

---

Published as a conference paper at ICLR 2023
4.5 VISUALIZATION
Visualization of the performance distributions. Figure is a histogram of program performance distributions at two different time steps (e.g., the first time step and the last time step). The y-axis represents the percentage of programs and the x-axis represents the relative performance. As shown in Figure [ the program distribution of the first time step is ex- tremely unbalanced since the programs are sampled randomly. After 15 steps of program sampling based on active learning, the program distribution of the last time step becomes more balanced, which clearly demonstrates the effectiivneess of our seasedddiversity-bile based AL approach.
35
Ratio of programs
First
30
25
Last
20
15
10
0.02
0.32
0.62
0.92
Relative Performance
Figure 8: Performance distributions.
5 RELATED WORK
Tensor optimization frameworks. Commonly used TPO frameworks include online-trained-
model-guided program optimization (Chen et al. 2018 Zheng al. 2020a; 2022 Ahn et al. 20220]
Zeng et al. 2020), and pedeffnnd-moder guddi- program optimization (Zhu et al. 2022 Dave et al,
2019 Adams et al. 2019 Haj-Ali et al. 2020; Anderson et al., 2020 Zheng et al. | 10221). Predefined-
model-guided program optimization can optimize the given tensor programs in a short time and its predefined model can be either designed manually by domain experts (Zhu et al. [202) Dave al. 2019) or pre-trained using randomly generated programs (Adams et al. 2019) Haj-Ali et al.) 2020 Anderson al., 2020;; Zheng et al. 202I). The manually designed model are usually designed for specific hardware platforms and thus hard to adapt to new platforms. The pre-trained-model based approaches can effectively adapt the new platforms but require large number of hardware measurements of randomly sampled programs. The online-trained wooeeeddd-oe----eeee -ggidee approaches relies training the model with data collected online. Thus this kind of optimization approaches time con- suming for each input optimization task. Although BALTO is a fast tensor optimization approach approaches, the optimization schemes in BALTO can also
designed for the hne-trainnddmoddel-eee be applied to the mollne-trainnee mooeeeddddddllgieded approaches, which is our future work.
Active learning. AL is widely used for reducing the expensive annotation cost for model training (Aghdam et al. 2019: Joshi et al. 2009 Culotta & McCallum 2005). The acquisition function can reduce the redundancy in sampling by only anotating informative or representative samples and thus is the core to AL. Typically, the most frequently used acquisition strategies can be divided into uncertainty-baet based approaches (Yoo & Kweon 2019] |Roth & Small (006) and diversity- -aatet ap- proaches (Sener & Savarese 2018 & Koy-Chowdhury
(2155) Uncertainty-basee -based approaches tend to query samples that are most uncertain (Lewis & Catlett 1994). MC Dropout (Gal & Ghahra- mani 2016)adopts random dropout for obtaining CNN's ' prediction uncertainty. (Beluch et al.j 2018 uses an ensemble of classifiers for obtaining the uncertainty. Diversity-based acquisition em- phasizes on the diversity of selected samples. GSx (Yu Kim 2010) maximizes the diversity in the feature space, while iGS (Wu et al., 2019) maximizes the diversity in both feature space and label space. Our active learning approach is based on the Core-Set framework and explores the diversity of predicted programs' performance.
6 CONCLUSION
In this paper, we propose BALTO, a fast TPO approach with btased-diversity reduce the training cost of the pre-trained model while with the same or higher prediction accuracy. The key insight is that there exists a heavy redundancy of low- low-performance programs when sampling randomly with existing approaches. To remove such redundancy and for a much lower training cost, active learning is introduced in BALTO. Moreover, we further propose scheme specially designed for BALTO to address the overestimation problem. The empirical results demonstrate that BALTO consistently shows superior training performance (i.e., 20 reduction in required hardware measurements.) on a wide range of environment configurations (i.c., 6 hardware platforms and 2 learning models) and even better optimization performance on T4.
active learning to
hiased-diversity-

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We would like to thank the reviewers for their valuable suggestions. This work is partially supported by the National Key R&D Program of China (under Grant 2021ZD01 the NSF of China (under Grants U22A2028, 61925208, 62102399, 62002338, 62222214, U19B2019), CAS Project for Young Scientists in Basic Research (YSBR-029), Youth Innovation Promotion Association CAS and Xplore Prize.
REFERENCES
Andrew Adams, Karima Ma, Luke Anderson, Riyadh Baghdadi, Tzu-Mao Li, Michaël Gharbi, Benoit Steiner, Steven Johnson, Kayvon Fatahalian, Frédo Durand, et al. Learning to optimize halide with tree search and random programs. ACM Transactions Graphics (TOG), 38(4):
1-12, 2019.
Hamed Habibi Aghdam, Abel Gonzalez- Gonzalez-Garcia, Joost van de Weijer, and Antonio M. López. Ac- tive learning for deep detection neural networks. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3671 3671-3679, 2019.
Byung Hoon Ahn, Prannoy Pilligundla, Amir Yazdanbakhsh, and Hadi Esmaeilzadeh. Chameleon: Adaptive code optimization for expedited deep neural network compilation. arXiv preprint arXiv:2001.08743, 2020.
Luke Anderson, Andrew Adams, Karima Ma, Tzu-Mao Li, and Jonathan Ragan-Kelley. Learning to schedule halide pipelines for the gpu. arXiv preprint arXiv:20/2.07145 2020.
William H. Beluch, Tim Genewein, A. Nümberger, and Jan M. Köhler. The power of ensembles for active learning in image classification. 2018 IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 9368-9377 2018.
Tianqi Chen, Lianmin Zheng, Eddie Yan, Ziheng Jiang, Thierry Moreau, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. Learning to optimize tensor programs. Advances in Neural Informa- fion Processing Systems, 31, 2018.
Aron Culotta and Andrew McCallum. Reducing labeling effort for structured prediction tasks. In AAAI, 2005.
Shail Dave, Youngbin Kim, Sasikanth Avancha, Kyoungwoo Lee, and Aviral Shrivastava. Dmazerunner: Executing perfectly nested loops on dataflow accelerators. ACM Transactions on Embedded Computing Systems (TECS), 18(5s):1-27, 2019.
Jacob Devlin, Ming-Wos Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800. 2018.
Siyuan Feng, Bohan Hou, Hongyi Jin, Wuwei Lin, Junru Shao, Ruihang Lai, Zihao Ye, Lianmin Zheng, Cody Hao Yu, Yong Yu, and Tianqi Chen. Tensorir: An abstraction for automatic ten- sorized program optimization. ArXiv, abs/2207.0ns 004296, 2022.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. ArXiv, abs/1 2016.
Ameer Haj-Ali, Hasan Gene, Qijing Huang, William Moses, John Wawrzynek, Krste Asanovié, and lon Stoica. Protuner: tuning programs with monte carlo tree search. arXiv preprint arXiv:2005.13685 2020.
Mahmudul Hasan and Amit K. Roy- nhy-Chhwd, Context aware active learning of activity recogni- tion models. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 4543-4551, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Published as a conference paper at ICLR 2023
Ajay J. Joshi, Fatih Murat Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classification. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp- 2372-2379, 2009.
David D. Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning. In ICML, 1994.
Dan Roth and Kevin Small, Margin-based active learning for structured output spaces. In ECML, 2006.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Ching Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 45 4510-4520, 2018.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference Learning Representations, 2018. URL https://
Junru Shao, Xiyou Zhou, Siyuan Feng. Bohan Hou, Ruihang Lai, Hongyi Jin, Wuwei Lin, Masahiro Masuda, Cody Hao Yu, and Tianqi Chen. Tensor program optimization with probabilistic pro- grams. ArXiv, abs/2205.1366 2022.
Dongrui Wu, Chin-Teng Lin, and Jian Huang. Active learning for regression using greedy sampling. Inf. Sci., 474:90-105, 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 1492-1500, 2017.
Donggeun Yoo and In So Kweon. Learning loss for active learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 93 102, 2019.
Hwanjo Yu and Sungchul Kim. Passive sampling for regression. 2010 IEEE International Confer- ence on Data Mining, pp. 1151-1156, 2010.
Xi Zeng, Tian Zhi, Zidong Du, Qi Guo, Ninghui Sun, and Yunji Chen. Alt: Optimizing tensor compilation in deep learning compilers with active learning. In 2020 IEEE 38th International Conference on Computer Design (ICCD), pp. 623-630. IEEE, 2020.
Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, et al. Ansor: Generating Performerfrm tensor programs for deep learning. In 14th USENIX symposium on operating systems design and imple- mentation (OSDI 20), pp. 863-879, 2020a.
Lianmin Zheng, Ruochen Liu, Junru Shao, Tianqi Chen, Joseph E Gonzalez, Ion Stoica, and Ameer Haj Ali. Tenset: A large-scale program performance dataset for learned tensor compilers. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Size Zheng, Yun Liang, Shuo Wang, Renze Chen, and Kaiwen Sheng. Flextensor: An automatic schedule exploration and optimization framework for tensor computation on heterogeneous sys- tem. Proceedings of the Swenty-Fifth International Conference on Architectural Support Pro- gramming Languages and Operating Systems, 2020b.
Size Zheng, Renze Chen, Anjiang Wei, Yicheng Jin, Qin Han, Liqiang Lu, Bingyang Wu, Xiuhong Li, Shengen Yan, and Yun Liang. Amos: enabling automatic mapping for tensor computations on spatial accelerators with hardware abstraction. In ISCA, pp. 874 887, 2022.
Hongyu Zhu, Ruofan Wu, Yijia Diao, Shanbin Ke, Haoyu Li, Chen Zhang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Wei Cui, al. (ROLLER): Fast and efficient tensor compilation for deep learning. In 16th USENIX Symposium Operating Systems Design and Implementation (OSDI 22), pp. 233-248, 2022.
11

---

Published as a conference paper at ICLR 2023
A APPENDIX
You may include other additional sections here.
12


Reviews:
[{"review_id": "wn2CJLFjaxS", "reviewer": "Reviewer_3pkd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces an active learning solution to sample configuration points to learn a data-driven cost model. They observe that most configurations are low performant and use a biased program selection strategy to find programs with higher performance. The results show that BALTO can achieve similar or better performance by requiring on a fraction of hardware measurements when the cost model is trained using their sampled dataset.", "summary_of_the_review": "I enjoyed reading the paper. They clearly articulate a valid observation that program auto-tuning practitioners face in training cost models, which is the training distribution is skewed towards non-performant programs. Therefore, cost models tend to be erroneous in estimating cost of high performant regions, minimizing their efficacy.\n\nUsing active learning to come up with the biased sampling strategy IMO is a novel technique to tackle this problem. However, the sampling process that solves eq. 2 and 3 is not clear. The paper mentions about Core-set, however adapting it to programs in not explained properly. I think this is an important detail which should be elaborate more in the paper. I am not sure whether the programs are still generated using a probabilistic grammar or using some other methodology. \n\nThe results seem convincing, insofar as the auto-tuning time is reduced drastically. This is because of the efficacy of the cost model. \nOverall, I think the paper presents a nice solution to the data imbalance problem, but details were missing to fully appreciate it. I will change my score once the authors provide more clarifying details about their program selection / sampling process.", "strength_and_weaknesses": "Strengths:\n\n* Intuitive observation and solution to a real world auto-tuning problem.\n* Strong results on reduction of auto-tuning time.\n\nWeaknesses\n\n* It is unclear how the sampling process happens. Core-set algorithm is mentioned, but how this is adapted to program sampling is not presented.\n* Implementation details of the core algorithm is unclear in the program setting", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "S23u6YV6rWE", "reviewer": "Reviewer_UWiq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes BALTO, a biased-diversity-based activation learning approach for fast tensor program optimization. BALTO combines active learning and a biased-diversity-based diversity scheme.\nBALTO can achieve the same or higher model accuracy with only 5% of the data samples.", "summary_of_the_review": "In summary, this paper proposes an effective approach to accelerate tensor program optimization. However, the novelty is limited. I think this is a borderline paper.", "strength_and_weaknesses": "Strength:\n- The application of core-set with biased selection is novel. The method is well-motivated and addresses the right problems of TPO.\n- The results are impressive. It reduces the required number of data samples by 20x.\n\nWeaknesses:\n- Table 1 and Table 2 should include results from more baselines. Currently, they only list BALTO and a simple baseline without any active learning. Although Sec 4.3 does some ablation study, there is a concern about cherry-picking with a single data point in Sec 4.3.\n- More analysis of algorithm 1 should be included such as complexity and running time. In the final evaluation, the search time should also be included.\n- The real technical contribution of this paper is sec 3.3. But I think the contribution is too small for an ICLR paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MtackPRD_q", "reviewer": "Reviewer_7kGb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper identifies that with Tensor Programming Optimisation in general the dataset collected is significantly imbalanced. Using this they develop an active learning approach which favours the selection of new samples as those which are hoped to be in areas with less prior knowledge. The authors show that this allows them to gain the same performance with far fewer samples.", "summary_of_the_review": "The paper seems a marginal improvement in the area - though without comparison to the real state-of-the-art work it is difficult to judge if there is a benefit to this approach or not. The work could benefit from more analytical analysis of the results. Many of the equations are presented without any justification or motivation as to why they may be beneficial.", "strength_and_weaknesses": "The introduction to the paper fails to set the context for the work making it difficult to follow for all but those who know the area well. By contrast the second section gives a very low-brow overview of the area. A shortened version of this should be merged into the introduction thus increasing space for other material. \n\nThe use of active learning for TPO is not a new idea. A simple google search reveals \"ALT: Optimizing Tensor Compilation in Deep Learning Compilers with Active Learning\". It is very surprising that the authors do not compare themselves with this work. It does not appear in the related work nor is it used to compare results with. To clam TenSet is state-of-the-art is rather misleading. A full comparison with other Active Learning approaches should be performed in order to justify if this work is better than these other works. ATL was not the only AL paper out there.\n\nFor the results there was no indication of whether the results were from one run or averaged over many. There has been a lot of criticism in AI recently for authors cherry-picking results and the authors here should show that their results are true irrespective of runs or random seeds.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "GjuDVuUVZL6", "reviewer": "Reviewer_nyuX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel active learning-based approach that accurately addresses an long-existing pain point in deep learning compilation, namely tuning speed. Its contribution could be summarized as:\n* Presented a careful investigation into the root cause of the unnecessarily long tuning speed, which is lack of diversity in sampling that leads to failure of comprehensive exploration of the entire search space, while large chunk of exploratory trials are wasted in apparently low-quality candidates;\n* A bias-diversity-based active learning-based approach is proposed to mitigate the problem revealed by the investigation. First, it applies the existing methodology of diversity-based approach by carefully controlling the sample generation to be aware of the similarity between samples; Then, a bias selection scheme is used to alleviate the imbalanced distribution that diversity-based approach could lead to (in terms of performance);\n* Experiments on popular models are shown to demonstrate the superior performance of the proposed solution.\n", "summary_of_the_review": "In summary, the reviewer believes that the paper makes significant contribution in terms of insightful investigation of the weakness of existing sampling process, namely over-sampling of weak candidates (imbalanceness) and underestimation of the peak performance (estimated maximal throughput). The concrete active learning-based algorithm, as proposed in this paper, properly solves the issue being found. This work is particularly valuable in terms of its insights and future direction it leads to.\n\nThe reviewer is willing to adjust the score accordingly if more diverse setting of experiments and more detailed comparison between this work and previous lines of research is present.\n", "strength_and_weaknesses": "Strength\n* As a practitioner of deep learning compilation, I am enthusiastic about this paper because of the accurate analysis presented in this paper. While active learning does not necessarily have to be “the only” approach, as the solution proposed the paper, it apparently does solve the problem revealed by the analysis, and most importantly, the insights presented in the paper provides a promising direction for follow-up works to further explore.\n* The output-diversity-based program selection, plus biased-diversity-based active learning, are quite novel and invented with deep understanding of the characteristics of tensor program optimization. For example, algorithm in Section 3.3 comes from very careful analysis that the estimated maximal program throughput could be underestimated, and thus it’s used to avoid overestimation of relative performance.\n* Experiments are designed carefully to align with numbers in the baseline, i.e. TenSet, and much better performance is showcased in those very well aligned experiments.\n\nWeakness\n* While experimented in several standard representative models, given the proposed solution is algorithmcally generic, it would be desirable to demonstrate that it could be further extended to more complicated scenarios, for example, automatic tensorization in TensorIR [1], auto tuning with sparse computation [2]. The reviewer doesn’t believe it’s any major weakness, but is curious if it could be generic enough to generalize to broader usecases.\n* More ablation study is desirable to present with more in-depth the effectiveness of such exploration algorithm in different subgraphs, for example, the difference between batched matmul in BERT and winograd convolution in ResNet; On the other hand, the characteristics of models selected in Table 2 might overlap, for example, between BERT-base and BERT-tiny - is there any reason to have both in the table?\n* The relationship of some lines of work is worth more discussion. For example, is this algorithm compatible with the programming model proposed in MetaSchedule [3]? What samples does the resulting models favor, compared with human prior-based approaches like Roller [4]?\n\n[1] Feng, Siyuan, Bohan Hou, Hongyi Jin, Wuwei Lin, Junru Shao, Ruihang Lai, Zihao Ye et al. \"TensorIR: An Abstraction for Automatic Tensorized Program Optimization.\" arXiv preprint arXiv:2207.04296 (2022).\n\n[2] Ye, Zihao, Ruihang Lai, Junru Shao, Tianqi Chen, and Luis Ceze. \"SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learning.\" arXiv preprint arXiv:2207.04606 (2022).\n\n[3] Shao, Junru, Xiyou Zhou, Siyuan Feng, Bohan Hou, Ruihang Lai, Hongyi Jin, Wuwei Lin, Masahiro Masuda, Cody Hao Yu, and Tianqi Chen. \"Tensor Program Optimization with Probabilistic Programs.\" arXiv preprint arXiv:2205.13603 (2022). \n\n[4] Zhu, Hongyu, Ruofan Wu, Yijia Diao, Shanbin Ke, Haoyu Li, Chen Zhang, Jilong Xue et al. \"{ROLLER}: Fast and Efficient Tensor Compilation for Deep Learning.\" In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 233-248. 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster