Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes an optimizing compiler  for DNN's based on adaptive sampling and reinforcement learning, to drive the search of optimal code in order to reduce compilation time as well as potentially improve the efficiency of the code produced. In particular, the paper proposes to use PPO to optimize a code optimization "search" policy, and then use K-mean clustering over a set of different proposed compilation proposals, from which to perform adaptive sampling to reduce compilation time while still keeping a high diversity of the proposed solution pools during exploration. At the same time the authors claim that using RL will learn a better search strategy compared to random search - such as simulated annealing which is used by competing methods - thus producing faster and better solutions.
The paper show results of up to 4x speedup in compilation time (autotuning time) while obtaining a slightly better or similar efficiency of the generated code (in term of execution time). This is a well written extensive research with good results. The authors mention it is (will be) integrated in the open source code of TVM. However I could find no mention in the paper of whether the code will be released with this publication, and I would like to solicit the authors to clarify their code release strategy and timing.
My other question pertains to whether or not  compilation time is an key metric to target. It is important to some extent, but I would say that aside from exponential / super-polynomial behaviour of auto-tuning algorithms, a multiple hours / days process to create the best optimized code for a certain network / hardware platform might not be such a big hurdle for a community already used to multiple days / weeks / months to train the same models. I believe that focusing on the efficiency of the optimized code produced would probably be a better metric of success.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The authors proposed a method for code optimization for deploying neural networks. The main idea is to formulate it as a search task over tuning knobs in the code template, and to apply reinforcement learning to optimize the configurations for the tuning knobs with respect to a cost model. The cost model is trained based on a subset of representative samples from the RL controller and their corresponding hardware cost measurements.

The paper is very well written and the proposed method seems technically sound. The authors did a good job combining existing techniques in a complementary manner. For example, the usage of RL for efficient search space exploration and the usage of clustering for selecting representative samples for on-device measurements.

Some concerns:
* The authors are referring to the usage of reinforcement learning as Adaptive Exploration and the usage of clustering as Adaptive Sampling. While combining them to tackle the task of neural network compilation is interesting, these techniques themselves are very standard and hence come with limited technical novelty.
* The proposed workflow seems to involve a nontrivial amount of additional hyperparameters, e.g., those in the RL controller as well as those for clustering. It might be useful to discuss about the overhead caused by hyperparameter tuning, as otherwise numbers reported in Table 2 (based on a single trial) could be misleading.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper proposes a new solution called CHAMELEON for deep learning code optimization, which accelerates the process of compiling codes and achieves faster training and inference of deep networks. The proposed method can be used to compile various deep network architectures. Experimental results show that the proposed method outperforms the previous method with large margin.

The paper exploits reinforcement learning to address code compilation, which is novel for me. 

The experimental results are convincing, the paper evaluates the proposed evaluation in multiple aspects.

I am totally new in the area. I will refer to the other reviews' review and the authors' rebuttal to have my final decision.


Post-rebuttal
Thank the other two reviewers and the authors to help me better understand the paper. I think I have no concern on the paper so I still give 6.
review_title: Official Blind Review #3

Meta-review: This paper proposes to optimize the code optimal code in DNN compilers using adaptive sampling and reinforcement learning. This method achieves  significant speedup in compilation time and execution time. The authors made strong efforts in addressing the problems raised by the reviewers, and promised to make the code publicly available, which is of particular importance for works of this nature.   

Decision: Accept (Poster)