Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes BALTO, a biased-diversity-based activation learning approach for fast tensor program optimization. BALTO combines active learning and a biased-diversity-based diversity scheme.
BALTO can achieve the same or higher model accuracy with only 5% of the data samples.
strength_and_weaknesses: Strength:
- The application of core-set with biased selection is novel. The method is well-motivated and addresses the right problems of TPO.
- The results are impressive. It reduces the required number of data samples by 20x.

Weaknesses:
- Table 1 and Table 2 should include results from more baselines. Currently, they only list BALTO and a simple baseline without any active learning. Although Sec 4.3 does some ablation study, there is a concern about cherry-picking with a single data point in Sec 4.3.
- More analysis of algorithm 1 should be included such as complexity and running time. In the final evaluation, the search time should also be included.
- The real technical contribution of this paper is sec 3.3. But I think the contribution is too small for an ICLR paper.
clarity,_quality,_novelty_and_reproducibility: The paper is well written with enough background information.
I believe most of the experiments can be reproduced.
summary_of_the_review: In summary, this paper proposes an effective approach to accelerate tensor program optimization. However, the novelty is limited. I think this is a borderline paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces an active learning solution to sample configuration points to learn a data-driven cost model. They observe that most configurations are low performant and use a biased program selection strategy to find programs with higher performance. The results show that BALTO can achieve similar or better performance by requiring on a fraction of hardware measurements when the cost model is trained using their sampled dataset.
strength_and_weaknesses: Strengths:

* Intuitive observation and solution to a real world auto-tuning problem.
* Strong results on reduction of auto-tuning time.

Weaknesses

* It is unclear how the sampling process happens. Core-set algorithm is mentioned, but how this is adapted to program sampling is not presented.
* Implementation details of the core algorithm is unclear in the program setting
clarity,_quality,_novelty_and_reproducibility: Novelty: The paper explores a novel avenue for the data diversity problem with a clear observation driving their decisions.

Clarity: It its unclear how exactly the algorithms are translated to sample programs.
summary_of_the_review: I enjoyed reading the paper. They clearly articulate a valid observation that program auto-tuning practitioners face in training cost models, which is the training distribution is skewed towards non-performant programs. Therefore, cost models tend to be erroneous in estimating cost of high performant regions, minimizing their efficacy.

Using active learning to come up with the biased sampling strategy IMO is a novel technique to tackle this problem. However, the sampling process that solves eq. 2 and 3 is not clear. The paper mentions about Core-set, however adapting it to programs in not explained properly. I think this is an important detail which should be elaborate more in the paper. I am not sure whether the programs are still generated using a probabilistic grammar or using some other methodology. 

The results seem convincing, insofar as the auto-tuning time is reduced drastically. This is because of the efficacy of the cost model. 
Overall, I think the paper presents a nice solution to the data imbalance problem, but details were missing to fully appreciate it. I will change my score once the authors provide more clarifying details about their program selection / sampling process.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper identifies that with Tensor Programming Optimisation in general the dataset collected is significantly imbalanced. Using this they develop an active learning approach which favours the selection of new samples as those which are hoped to be in areas with less prior knowledge. The authors show that this allows them to gain the same performance with far fewer samples.
strength_and_weaknesses: The introduction to the paper fails to set the context for the work making it difficult to follow for all but those who know the area well. By contrast the second section gives a very low-brow overview of the area. A shortened version of this should be merged into the introduction thus increasing space for other material. 

The use of active learning for TPO is not a new idea. A simple google search reveals "ALT: Optimizing Tensor Compilation in Deep Learning Compilers with Active Learning". It is very surprising that the authors do not compare themselves with this work. It does not appear in the related work nor is it used to compare results with. To clam TenSet is state-of-the-art is rather misleading. A full comparison with other Active Learning approaches should be performed in order to justify if this work is better than these other works. ATL was not the only AL paper out there.

For the results there was no indication of whether the results were from one run or averaged over many. There has been a lot of criticism in AI recently for authors cherry-picking results and the authors here should show that their results are true irrespective of runs or random seeds.
clarity,_quality,_novelty_and_reproducibility: The writing in the paper is not always clear. There are many places where words need to be added in to make the sentences clear.

The quality of analysis as outlined above needs to be improved.

The novelty of the work does not seem that great given the fact that works exist which use Active Learning and are not discussed here. Nor is the work compared to other optimisation techniques used.
summary_of_the_review: The paper seems a marginal improvement in the area - though without comparison to the real state-of-the-art work it is difficult to judge if there is a benefit to this approach or not. The work could benefit from more analytical analysis of the results. Many of the equations are presented without any justification or motivation as to why they may be beneficial.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel active learning-based approach that accurately addresses an long-existing pain point in deep learning compilation, namely tuning speed. Its contribution could be summarized as:
* Presented a careful investigation into the root cause of the unnecessarily long tuning speed, which is lack of diversity in sampling that leads to failure of comprehensive exploration of the entire search space, while large chunk of exploratory trials are wasted in apparently low-quality candidates;
* A bias-diversity-based active learning-based approach is proposed to mitigate the problem revealed by the investigation. First, it applies the existing methodology of diversity-based approach by carefully controlling the sample generation to be aware of the similarity between samples; Then, a bias selection scheme is used to alleviate the imbalanced distribution that diversity-based approach could lead to (in terms of performance);
* Experiments on popular models are shown to demonstrate the superior performance of the proposed solution.

strength_and_weaknesses: Strength
* As a practitioner of deep learning compilation, I am enthusiastic about this paper because of the accurate analysis presented in this paper. While active learning does not necessarily have to be “the only” approach, as the solution proposed the paper, it apparently does solve the problem revealed by the analysis, and most importantly, the insights presented in the paper provides a promising direction for follow-up works to further explore.
* The output-diversity-based program selection, plus biased-diversity-based active learning, are quite novel and invented with deep understanding of the characteristics of tensor program optimization. For example, algorithm in Section 3.3 comes from very careful analysis that the estimated maximal program throughput could be underestimated, and thus it’s used to avoid overestimation of relative performance.
* Experiments are designed carefully to align with numbers in the baseline, i.e. TenSet, and much better performance is showcased in those very well aligned experiments.

Weakness
* While experimented in several standard representative models, given the proposed solution is algorithmcally generic, it would be desirable to demonstrate that it could be further extended to more complicated scenarios, for example, automatic tensorization in TensorIR [1], auto tuning with sparse computation [2]. The reviewer doesn’t believe it’s any major weakness, but is curious if it could be generic enough to generalize to broader usecases.
* More ablation study is desirable to present with more in-depth the effectiveness of such exploration algorithm in different subgraphs, for example, the difference between batched matmul in BERT and winograd convolution in ResNet; On the other hand, the characteristics of models selected in Table 2 might overlap, for example, between BERT-base and BERT-tiny - is there any reason to have both in the table?
* The relationship of some lines of work is worth more discussion. For example, is this algorithm compatible with the programming model proposed in MetaSchedule [3]? What samples does the resulting models favor, compared with human prior-based approaches like Roller [4]?

[1] Feng, Siyuan, Bohan Hou, Hongyi Jin, Wuwei Lin, Junru Shao, Ruihang Lai, Zihao Ye et al. "TensorIR: An Abstraction for Automatic Tensorized Program Optimization." arXiv preprint arXiv:2207.04296 (2022).

[2] Ye, Zihao, Ruihang Lai, Junru Shao, Tianqi Chen, and Luis Ceze. "SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learning." arXiv preprint arXiv:2207.04606 (2022).

[3] Shao, Junru, Xiyou Zhou, Siyuan Feng, Bohan Hou, Ruihang Lai, Hongyi Jin, Wuwei Lin, Masahiro Masuda, Cody Hao Yu, and Tianqi Chen. "Tensor Program Optimization with Probabilistic Programs." arXiv preprint arXiv:2205.13603 (2022). 

[4] Zhu, Hongyu, Ruofan Wu, Yijia Diao, Shanbin Ke, Haoyu Li, Chen Zhang, Jilong Xue et al. "{ROLLER}: Fast and Efficient Tensor Compilation for Deep Learning." In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 233-248. 2022.
clarity,_quality,_novelty_and_reproducibility: - Clarity & qualty. The paper is clearly written and easy to follow. With rich insights presented, the reviewer believes the paper provides a promising direction for future exploration. A quick side note: given it’s written as an insightful analysis paper, it would be even more clear to avoid unnecessary formula, for example, Equation (1)(2), and present the insights in a plain text that readers don’t have to think twice.
- Novelty. The paper tackles a long-existing problem in a novel way, by creatively applying and improving active learning according to the insights and observation in tensor program optimization.
- Reproducibility. Given the limited time budget, the reviewer does not have the bandwidth to evaluate the artifact or code.

summary_of_the_review: In summary, the reviewer believes that the paper makes significant contribution in terms of insightful investigation of the weakness of existing sampling process, namely over-sampling of weak candidates (imbalanceness) and underestimation of the peak performance (estimated maximal throughput). The concrete active learning-based algorithm, as proposed in this paper, properly solves the issue being found. This work is particularly valuable in terms of its insights and future direction it leads to.

The reviewer is willing to adjust the score accordingly if more diverse setting of experiments and more detailed comparison between this work and previous lines of research is present.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an approach called biased active-learning for tensor program optimization, or BALTO. The high level idea is to use diversity-based active learning for selecting program samples (which increases the proportion of high-performance programs in the training set). This mitigates a known failure mode in TPO, which rely on random sampling and therefore are heavily imbalanced towards low-performance programs. Experiments on six different hardware platforms show the benefits of BALTO with up to 20X reduction in measurement costs.

Original reviews on this paper were split. Some reviewers were positive about the paper, highlighting that the authors had addressed a real-world application with measurable improvements over current state-of-the-art. Other reviewers pointed out issues with the clarity of writing, as well as the (relative) lack of novelty, given that other approaches for TPO based on active learning have already been proposed. Additional baseline comparisons, and more details about the experimental setup, may have helped here.

After a discussion among the reviewers and the AC, the overall majority sentiment seemed to point favorably in the direction of the paper. The authors are recommended to take into account the comments made by the reviewers while preparing revisions.

Recommendation: accept. 

justification_for_why_not_higher_score: While the paper has many positive aspects, questions related to novelty and clarity of writing still remain, preventing me from recommending a higher score.
justification_for_why_not_lower_score: Two of the reviewers were domain experts working in this exact application area (program optimization), and they were the most positive about the paper. I feel that even though the paper does not break novel ground from the ML techniques standpoint, the fact that the approach seems to be genuinely useful in applications makes it a worthy candidate for acceptance.
Decision: Accept: poster