Chameleon: Adaptive Code Optimization for Expedited Deep Neural Network Compilation
Authors: Byung Hoon Ahn
Prannoy Pilligundla
Amir Yazdanbakhsh
Hadi Esmaeilzadeh
Venue:  (2020)
Keywords: Reinforcement Learning
Learning to Optimize
Combinatorial Optimization
Compilers
Code Optimization
Neural Networks
ML for Systems
Learning for Systems

Abstract: Achieving faster execution with shorter compilation time can foster further diversity and innovation in neural networks. However, the current paradigm of executing neural networks either relies on hand-optimized libraries, traditional compilation heuristics, or very recently genetic algorithms and other stochastic methods. These methods suffer from frequent costly hardware measurements rendering them not only too time consuming but also suboptimal. As such, we devise a solution that can learn to quickly adapt to a previously unseen design space for code optimization, both accelerating the search and improving the output performance. This solution dubbed Chameleon leverages reinforcement learning whose solution takes fewer steps to converge, and develops an adaptive sampling algorithm that not only focuses on the costly samples (real hardware measurements) on representative points but also uses a domain-knowledge inspired logic to improve the samples itself. Experimentation with real hardware shows that Chameleon provides 4.45x speed up in optimization time over AutoTVM, while also improving inference time of the modern deep networks by 5.6%.

Published as a conference paper at ICLR 2020
CHAMELEON: ADAPTIVE CODE OPTIMIZATION FOR EXPEDITED DEEP NEURAL NETWORK COMPILATION
Byung Hoon Ahn', Prannoy Pilligundla', Amir Yazdanbakhsh", Hadi Esmaeilzadeh'
University of California, San Diego
2 Google Research
bhahndeng. wws.. edu ppilligueeng ucsd. edu ayazdan@googiee.com
hadi uusengucss.edu
ABSTRACT
Achieving faster execution with shorter compilation time foster further diver- sity and innovation in neural networks. However, the current paradigm of exe- cuting neural networks either relies on hand-optimized librarics, traditional com- pilation heuristics, or very recently genetic algorithms and other stochastic meth- ods. These methods suffer from frequent costly hardware measurements rendering them not only too time consuming but also suboptimal. As such, we devise so- lution that can learn to quickly adapt to a previously unseen design space for code optimization, both accelerating the search and improving the output performance. This solution dubbed CHAMELEON leverages reinforcement learning whose solu- tion takes fewer steps to converge, and develops adaptive sampling algorithm that not only focuses on the costly samples (real hardware measurements) on rep- resentative points but also uses nomain-knowledge inspired logic improve the samples itself. Experimentation with real hardware shows that CHAMELEON pro- vides x.45xxpect speed up in optimization time over AutOTVM, while also improving inference time of the modern deep networks by 5.6%.
INTRODUCTION
The enormous computational intensity of Deep Neural Networks (DNNs) have resulted in develop- ing either hand-optimized kernels, such as NVIDIA cuDNN or Intel MKL that serve as backend for variety programming environment such as TensorFlow (Abadi et al. 2016) and PyTorch (Paszke) let al. 2019). y. However, the complexity of the tensor operations in DNNs and the volatility of algo- rithms, which has led to unprecedented rate of innovation (LeCun 2019). calls for developing auto- mated compilation frameworks. To imitate or even surpass the success of hand-optimized libraries, recent research has developed stochastic optimization passes: for general code, STOKE (Schkufza et al. 2013). and neural network code, TVM (Chen et al. 201 and TensorCooprehennsions (Vasi- lache et al. 2018). TVM and TensorComprrhenniions are based on random or genetic algorithms to search the space of optimized code for neural networks. AutoTTVM (Chen al., 2018b) builds on top of T'M and leverage boosted trees (Chen & Guestrin 2016) as part of the search cost model to avoid measuring the fitness of each solution (optimized candidate neural network code), and instead predict its fitness. However, even with these innovations the optimizing compilation time can be around 10 hours for R8 (He et al. 2016), and even more for deeper or wider networks.
Since the general objective is to unleash new possibilities by developing automatic optimization passes, long compilation time hinders innovation and could put the current solutions in a position of questionable utility. To solve this problem, we first question the very statistical guarantees which the aforementioned optimization passes rely on. The current approaches are oblivious to the patterns in the design space of schedules that are available for exploitation, and causes inefficient search or even converges to solutions that may even be suboptimal. Also, we notice that current approaches rely on greedy sampling that neglects the distribution of the candidate solutions (configurations). While greedy sampling that passively filter samples based on the fitness estimations from the cost models work, many of their hardware measurements (required for optimization) tend to be redundant and wasteful. Moreover, we found that current solutions that rely on greedy sampling lead to significant fractions of the candidate configurations being redundant over iterations, and that any optimizing

---

Published as a conference paper at ICLR 2020
compiler are prone to invalid configurations which significantly prolongs the optimization time. As such, this work sets out to present an Adaptive approach dubbed CHAMELEON to significantly reduce the compilation time and offer automation while avoiding dependence to hand-optmiization enabling far more diverse tensor operations in the next generation DNNs. We tackle this challenge from two fronts with the following contributions:
(1) Devising an Adaptive Exploration module that utilizes reinforcement learning to adapt to unseen design space of new networks to reduce search time yet achieve better performance.
(2) Proposing an Adaptive Sampling algorithm that utilizes clustering to adaptively reduce the num- ber of costly hardware measurements, and devising a domain -nowleekn inspired Sample Syn- thesis to find configurations that would potentially yield better performance.
Real hardware experimentation with modern DNNs (AlexNet, VGG-16, and ResNet-18) on a high- end GPU (Titan Xp), shows that the combination of these two innovations, dubbed CHAMELEON, yields x.45 speedup over the leading framework, AuM CHAMELEON is publicly available in the project page: hnttps://bitber
cxg/acct-ab/chameeee
2 CHALLENGES IN DEEP NEURAL NETWORK COMPILATION
The general lyfe- of deep learning models from its birth to deployment comprises of two major stages. First stage is the designing and the training of a deep learning model by a research scientist, with the primary goal of achieving the highest feasible accuracy. Then, with general demand to enable the intelligence on a wide range of devices (from mobile CPUs in the edge to cloud-scale GPUs), the second stage has emerged for the deployment of the pre-trained deep learning model to target hardware by a deployment engineer. These stages are each iterative processes: research scientists iterate until reaches the target performance in terms of accuracy whereas the deploy- ment engineers iterate until the performance in terms of inference speed with a given hardware satisfies the given constraints. Importantly, these two stages are most often separate processes, and this paper mainly focuses on the second stage (deployment) of the cycle with an overarching goal of accelerating the overall deployment cycle by reducing the optimizing compilation time without compromising the performance of the output code.
2.1 COMPILATION WORKFLOW FOR DEEP NEURAL NETWORKS
DNN Model
Target- Independent
Target- Dependent Passes
Code Templica
Optimizing Compiler
Measurement
Harzhoee
Passes
Design
Optimized -(0*)
Code
Programmer
Frantend Compiler
Hardware
Figure 1: Overview of our model compilation workflow, and highlighted is the scope of this work.
Figure I Iillustrates how compiler for DNNs takes input model M and emits optimized code r(O*) that runs the model efficiently on given hardware. This flow is commensurate with Tensor- Comprehensions (Vasilache et al.) 2018) and TVM (Chen al.) 2018a), using which we implement CHAMELEON that is available as a separate package for adoption in even other frameworks. The first phase of the workflow is the frontend compiler which performs the translation from the compiler and applies target-independent and white-box target-dependent optimizations that do not incorpo- rate a measure of runtime. Target-independent passes transform the input DNN model without specificity to the target hardware. Operator fusion and data layout transformation in TVM are some examples of these passes, which lie in the same category as dead-code elimination or loop-invariant code motion in GCC (Stallman & DeveloperCommunityy (2099) or LLVM (Lattner & Adve 2004)). Target-dependent passes, on the other hand, the compiler takes the hardware architecture (larget) into account while optimizing the program; however, this also does not actively leverage runtime measures. The last stage is a black-box optimization pass, called optimizing compiler, that given a measure of performance at runtime from the hardware can further optimize the code. CHAMELEON falls in this class by offering an optimizing compiler that adapts to different design space to be more swift in optimizing deep neural networks compared to conventional approaches.
2

---

Published as a conference paper at ICLR 2020
KNOBS
DEFINITION
ResNet 18 Optimization Time
tile.f. tile.y. tile.x
Factors for tiling and binding # of filters height, and width of feature maps. Factors for tiling reduction axis such as #
Oggan Dodentstonn Thre (Hours
2,50
Physical ement
Search Algortthm
2.0H
tilerrc, tlle.ry, tile.xx
1,5H
of channels, height, and width of filters. Threshold of number of steps in the loop be automatically unrolled. Explicitly unroll loop, this may let code generator to generate pragma unroll hint.
step unrollexxlicci
2,0
0.5H+
0.011
L5
L7
LS
LID LOLLE
Controlution Layer
Figure 2: AVMoTVM optimization
Table 1: Knobs in the design space optimize convolution.
time for ResNet-18 Titan Xp.
2.2 OPTIMIZING COMPILER FOR DEEP NEURAL NETWORKS
Optimizing compilers (Kennedy& & Allen 2001) usually take a black-box approach and use hardware measurements to configure the optimization based on measure of fitness f of each solution. In order make the problem tractable, the optimizing compilers for deep neural networks reduce the problem down to tuning the knobs 0 for the output code template т, and can be formulated as:
θt argmax f(T())),
for θ E De.
(I)
A combination of assignment to the knobs is said to be configuration + = (01, 02.,0...... 0p) while the dimensions the design space De is defined by the knobs. As such, in Equation| 1 an optimizing compiler starts from a code template for each layer, and makes use of a search algorithm and real hardware measurements to efficiently find the best configuration O E 'De In this context, there are three variables that determine the effectiveness of the optimizing compiler: (1) large and diverse enough design space that covers a variety of transformations, (2) an effective search algorithm to adequately navigale this space, and (3) a mechanism cul down the number of costly hardware measurements that check the fitness of a solution. Table[]]ists. the knobs for performing convolution on GPU, where it is crucial that the code (1) maximizes data reuse, (2) uses the shared memory wisely, and (3) minimizes bank conflicts. The knobs optimize various aspects of the execution, including tiling (e.g., tile x, ti unrolling (e.g., auto unroll max step and unroll unroll. explicit), and these knobs define a design space with 10a0 possibilities. Given the vastness of the design space, the remaining challenges are designing an effective search algorithm and designing a mechanism that reduces the cost of each step in the search (i.e. reducing the need to measure the hardware).
2.3 CHALLENGES DEEP NEURAL NETWORK COMPILATION
As shown in Figure 2 optimizing compilation for DNNs may still take an eon even with the advances
from prior works (Chen et al. 20120088 8a bj Vasilache et al., 2018) With active research (You et al. 2017
Goyal et al. 2017 Codreanu et al. 201 Akiba al. 2017
You et al. 2018 Mattson et al. 2019
that has been able to cut down the training time to only few hours YYou et al. 2017; Goyal et al.
2017) and even minutes YYou et al.) 2018 Akiba al. 2017 big models (e.g., ResNet-50 (He et al. 2016)) for ImageNet, it renders the optimizing compilation time of the current solutions seem even more prominent. Especially, since the above-mentioned compilers have been integrated to the deep learning pipelines of major players in the industry (Liu et al. 2019: Rotem et al. 2018 Vasilache et al., 2018), many users of these pipelines including the deployment engineers must go through the compilation workflow depicted in Figure I numerous times. Therefore, current long compilation time can be a hindrance deploying DNN in various hardware, hence a major bottleneck in enabling intelligence on wider range of target platforms.
Furthermore, as we explore various neural topologies (Xie et al. 2019: Wortsman et al. 2019) for better performance as illustrated in Ahn et al. (2020), even deeper or wider networks (Szegedy) ctt al. 2015 Zagoruyko & Komodakis 2016), and new operations (Howard et al. 2017) to achieve higher performance (LeCun) 2019). we are forced to optimize the networks more frequently. The long optimization times are multiplied with such trend, leaving the practical utility of the current compiler solutions to question. As such, the primary goal of this work is reducing the optimizing compilation time to meet the immediate needs of the industry for expedited DNN compilation to foster further diversity and innovation in designing DNNs.
3

---

Published as a conference paper at ICLR 2020
Funtimme Measurcmon
Cost Model
Ho use os tampltte code generstion))
Template
Query
Extimaned
knith mn
Adaptive Exploration
Adaptive Sampling
Code Generator
knab
Contggs
Sumpled Carligs
Generated
Fodee
Heckow
Spacee Th
Generated ((8)
Conpgg
Code
Figure 3: Overall design and compilation overview of the CHAMELEON.
Such long optimization time results from the inefficiency of simulated annealing which (while it stochastically guarantees a reasonable solution after huge number of iterations) faily to capture the patterss in the design space that can be exploited during the search. On the other hand, we can see in the figure that majority of the optimization time is spent on reaching for measurements on real hardware that is used as a feedback for the aforementioned search. Also, current approach even suf- fers from numerous invalid configurations that not only wastes the limited hardware measurement budget that the compiler starts with, but also incurs serious overhead to reset the target hardware for subsequent hardware measurements. As such, it is important that a sampling mechanism that selects potential configurations for hardware measurements to be smarter to ensure that each measurement maximizing the chances of achieving good solution and that it evades the invalid configurations. However, the current approaches rely on greedy sampling that passively sample based on the esti- mations from the cost models. This not only has a tendency overfit but also neglect that solutions are distributed con-uniformly and that there are numerous invalid configurations.
3 CHAMELEON: ADAPTIVE CODE OPTIMIZATION FOR EXPEDITED DEEP NEURAL NETWORK COMPILATION
As discussed Section 2 current solutions fall short of providing a swift optimization framework for optimizing emergent deep neural networks, because of the futility of the search in adapting to the design space from a random walk based search algorithm and the inefficiency of the physical hardware measurements from the greedy sampling. Therefore, developing overcome current challenges to unfetter neural network innovation from a prolonged optimization times be boiled down two problems: improving the the search algorithm to better adapt to the design space, and improving the sampling algorithm to both better adapt to the distribution of the solutions and decrease the possibility of running into invalid configurations. As such we make two innovations in the optimizing compiler for deep neural networks to develop CHAMELEON by applying reinforcement learning the search that can adapt to new design spaces (Adaptive Exploration) and devising Adaptive Sampling that replaces the current greedy sampling.
framework that can
3.1 OVERALL DESIGN OF CHIAEELEN
Figure outlines the overall design of our optimizing compiler, dubbed CHAMELEON and gives an overview the optimizing compilation process. CHAMELEON takes code template 7 for each layer in the network and the corresponding design space De as its input, and iteratively optimizes the code for configuration O to finally output T r(@*). The proposed Adaptive Exploration maneuvers the design space while using cost model proxy for hardware measurements the output set of candidate configurations Se. These configurations are then sampled with Adaptive Sampling so that the sampled configurations S% subsume the initial candidate configurations while reducing its number significantly. The sampled configurations S6B are then passed to the code generator which combines the input template and the configurations S% create a of ((0) that sent real hardware for runtime measurements, Runtimes from the hardware are used as the measure of fitness
'Chameleon is an animal that is capable of Adapting to their environments which helps them survive. In work, CHAMELEON is entity that Adapts to the variations in the design space and the distribution of the candidate configurations, enabling expedited deep neural network compilation.
4

---

Published as a conference paper at ICLR 2020
and update the cost model to enhance the exploration of the subsequent iterations. After multiple iterations, T(@") with the best fitness f (shortest runtime) is selected as an output for the layer.
3.2 ADAPTIVE EXPLORATION: LEARNING ABOUT THE UNSEEN DESIGN SPACE TO EXPEDITE CONVERGENCE OF OPTIMIZATION
As stated Section 2 the current state-of-the-ard approach (Chen et al. 2018b) that leverages sim- ulated annealing relies on the stochastic guarantees of its random walks. Therefore, the current approach requires numerous iterations of exploration to converge to a reasonable solution causing long compilation hours, thus insufficient to enable disruptive innovations neural networks. We take an inspiring approach that avoids naive dependence on the stochastic guarantee of simulated annealing and leverage a technique that can learn adapt to unseen design space to not only ac- celerate convergence but also bring some performance gains. As such, we develop Adaptive Explo- ration by leveraging Reinforcement Learning (RI.), which is concerned with learning maximize reward given an environment by making good exploration and exploitation tradeoffs, in case maximizing fitness f of the explored configurations Se.
Reinforcement learning formulation. Our RL-based Adaptive Exploration module uses an actor-critic style RL, where policy network learns emit a set of directions (vector of incre- men//decrement/sayy for each knob in the design space that will increase f of the next configuration and the value network learns the design space De to estimate the value of the action. The first layer of these networks that takes the current configuration θ input is shared to foster information shar- ing among the two networks, and its output fed into the subsequent layers the networks. These networks not only learn the dependencies among the different knobs of the design space (which are interrelated) that helps our module navigate through the design space but also lean the potential gains of the modifications to the configurations.
Palicy Network
Config Updater
Policy Network
Config Updater
Palicy Network
Config Updater
A Li
Cunfg
e Curfi
Curfin
Contigs Su
Soarch
Slop
Figure Adaptive Exploration Module of CHAMELEON in action.
Learning procedure. Having formulated the RL-based Adaptive Exploration Module, an itera- tion of our optimization begins with a set initial configurations and takes multiple search steps (episode) for each of the configurations. As shown in Figure the agent makes action and ap- plies it to the configuration using configuration updater to get another configuration that potentially has better f. After finishing multiple search steps the episode, all configurations Se evaluated using a cost model, which its return values are used as a surrogate reward to update our agent, to reduce the number of costly hardware measurements. By taking this approach, f of Se improves as our module progresses through the episodes. In other words, by repeating multiple episodes and iterations, our Adaptive Exploration Module gradually learns to locate good configurations.
3.3 ADAPTIVE SAMPLING: ADAPTING TO THE DISTRIBUTION TO REDUCE COSTLY HARDWARE MEASUREMENTS
Reducing number of costly hardware measurements. After the exploration step (regardless of the exploration method), we observe that the candidate configurations are clustered in subregions of the design space and these clusters are non-uni non-uniformly distributed (Figure We also find that, while the design space's is surface is discrete and un-smooth, large fraction of configurations within each cluster achieve similar runtime (Figure 6). Utilizing these characteristics of the design space, we devise Adaptive Sampling that can sample
of candidates, by adapting the shape of the
5

---

Published as a conference paper at ICLR 2020
80
302
82-
cluster
60-
cluster
40
Majorit contoourations disolaus simiar gefformance to the centrodd
20
23
20
40
50
80
100
0
20
40
60
80
100
Defference runtime from the centroud (S)
Figure Cumulative Distribution Function (CDF) of the difference in runtime among the configurations the cluster.
(a) VGG- 16 4th layer
(b) ResNet-11 11th layer
Figure 5: Clusters of candidate configurations.
design space and the non-uniformity of the distribution while leaving the performance of optimiza- tion intact. We first leverage clustering algorithm to find configurations that are representative of each cluster; the sampling module uses centroids the representative configurations. Our Adaptive Sampling iterates over a different number of clusters for their respective centroids and the L2 loss.
In the context of optimizing compiler, selecting the number of centroids for clustering entails making the important tradcoff between selecting more centroids for better performance or fewer centroids for a reduced number of hardware measurements. As such, we must devise a method that would automatically make the tradeoff in a reasonable manner. We take advantage of the decreasing trend in the aforementioned 1.2 loss Swift Meta-Search to determine the number of clusters. By setting the threshold (hyperparameter) allows the compiler determine the point of diminishing return (knee of the curve), inflection point beyond which fewer centroids may lead to performance degradation and more clusters would prolong the optimization substantially. Overall, our sampling curtails the number of hardware mea- surements so that it is just enough to subsume the entire subspace of the candidate configurations.
increase the number of centroids, and devise T'hreshold-based
Improving candidate configurations using sample synthesis. While the above sampling algo- rithm significantly reduces the number of hardware measurements compared to the conventional greedy sampling, without impacting the performance of the output code, critical issue of redundancy among the candidate configurations. We find that the exploration al- gorithm (regardless of the type) combined with the greedy sampling frequently leads redundancy among the candidate configurations over different iterations of optimization due to the overfitting of the cost model from the greediness of the sampling. Even though the exploration algorithm tries to explore unvisited regions of the design space, these explored (not exploited) configurations are discarded due to the greedy sampling which entirely depends on the cost model for its selections of the configurations. Therefore, the current greedy sampling algorithm has its limitation in focusing the hardware measurements to the same region over and over.
still left with
On the other hand, find that from code optimization point of view, know that many of the automated approaches for black-box optimization are prone to invalid configurations, which results from too large a tile that goes over the input feature map boundary errors during memory accesses (cannot be solved analytically). These invalid configurations not only blow the chances for better exploration but also leads the subsequent hardware measurement. We try to overcome both of these limitations by devising Sample Synthesis. When our compiler runs into redundant samples, the proposed synthesis method analyzes the candidate samples to determine the most probable (most frequent = mode function) non-invalid choice for each knob to come up with new configuration. This statistical combina- tion of the most frequent knob settings yield configurations that combine the strengths of different knobs to converge
extra optimization time overhead reset the physical hardware for
better overall solution. In spirit, the recombination (crossover) operator in genetic algorithms also tries to combine the best features of the solutions with high fitness values. Algorithm presents the integration of our Adaptive Sampling and the Sample Synthesis.
3.4 IMPLEMENTATION DETAILS
Architecture exploration for the adaptive exploration. We Proximal Policy Optimization (PPO) [Schulman et al.) 2017), policy gradient that has been shown to adapt to various problems and have good sample complexity, our reinforcement learning algorithm. Since reinforcement

---

Published as a conference paper at ICLR 2020
Algorithm Adaptive Sampling and Sample Synthesis
procedure ADAPTVEES ve)
sg: candidate configs, to: visited configs
2: new www_ccndddatesss.co +- 0. previous + 00
3: for in range(8. 64) do
4: new new_candidates, clusters, L2_loss t- K-means.run(sa, k)
5: if Threshold L. loss 2 previous loss then break Exit loop at knee loss curve
6: previous loss + L2 loss
7: end for
8: for candidate in new_candidates do
Replace visited config with new config
candidate in ve then new rendidatts.replaccccanddii mode(se)) 10: end for
9:
11: return new _w__candidates
> Feed to Code Generator to make measurements on hardware
end procedure
learning could incur computational overhead that could prolong the optimization time, optimize the actor-critic networks through architecture exploration to find good tradeoff for size of these networks (that determines the computational overhead) and the optimization performance.
Design choices for the adaptive sampling. We a K-means Clustering to determine centroids of the configurations, because X-means has been shown effective in practice and it only requires KC, radius in other algorithms which much more challenging tune. For example, DBSCAN (Ester et al. 1996) or mean-shift clustering (Comaniciu & Meer 2002) are very sensitive the above hyperparameters. On the other hand, can be framed as a lever to balance the perfor- mance and speed of optimizing compilation which abstracts away the aforementioned challenges, enabling the threshold-basee Swift Meta-Search that identifies the optimal number of clusters.
Hyperparameter tuning. Hyperparameter tuming is a very important task machine learning- based tools and models. As such, we present the hyperparameters we used for the evaluation in Tin appendix), which its tuning took several days. For the hyperparameters in Table 8 (in ap- pendix), ), we used the same set of values that were used in the AutoTVV paper (Chen et al.) (0188) in order to conduct fair comparison or CHAMELEON. Additionally, for parameters used in the Adap- tive Exploration module, which is not present in AutoTVM, we have tuned the hyperparameters using the set of layers presented in Table 5 (in appendix). We emphasize, however, that the hyper- parameters have been tuned offline before the deployment of CHAMELEON, and the hyperparameters are not changed during the use of the framework or the experimentation. So the tuning overhead is not part of the compilation after the Adaptive Exploration module is tuned once before releasing the compiler to the deployment practitioners.
4 EVALUATION
We integrate CHAMELEON into TVM CChn al. 2018a) perform component evaluation and compare with AutoTVM (Chen et al., 2018b 8b). We first evaluate components of CHAMELEON in Section 4.1 and Section 4.2 set of convolution layers sampled from AlexNet (Krizhevsky al.)
2012). VGG-16 (Simonyan & Zisserman 2015), and 18 (He et al.) 2016). Then we pro- vide end-to-end evaluation of CHAMELEON on both set of layers and end-to-end deep models, in Section 4.3 Due to space limitations, we present only the representative plots in the paper, and the complete set of results and the details of the parameters are provided in the appendix.
4.1 ADAPTIVE EXPLORATION: IMPROVING EFFICACY OF SEARCH ALGORITHM
In the previous approach (Chen et al. 20 20)),), authors have built a cost model to estimate fitness instead of performing costly measurements real hardware, then used simulated annealing to find potentially optimal configurations. Figure 7(a compares the number of search steps taken per iter- ation to reach converge to the solution simulated annealing and Adaptive Exploration. respec- tively. Overall, observation is that CHAMELEON's Adaptive Exploration requires 2.88 x less search steps compared simulated annealing find good solution. This comes from the ability of the

---

Published as a conference paper at ICLR 2020
Adaptive Exploration
Adaptive Sampling
AutoTVM
Simuleted Aemesling
3
gved AUTOTVW
Explonatior
AutOTVM
Exploration Time
Hardware Misstrammntt
llaaations
Iogrowereent oxer Aer.eaaaooo
2.10k
1.50
/Ataptiie Sumpling
2.00
2.80
(Fpllcationn /
1.50
0.6%
CHAMELEON
xttrattions
1.00
tlerstion
CHAMELEON
L4
CHAMELEON significantly reduces octimization
(a) Reduction in number (b) Reduction in number of hard- (c) Illustration of how the each component of steps for convergence.
ware measurements.
CHAMELEON reduces the optimization time.
Figure 7: Component evaluation of CHAMELEON.
inforcement learning algorithm in Adaptive Exploration Module (1) learn the correlation between different dimensions, and (2) reuse information across different iterations, instead of starting from scratch while naively relying on the stochastic guarantees of simulated annealing process.
4.2 ADAPTIVE SAMPLING: REDUCING NUMBER OF COSTLY HARDWARE MEASUREMENTS
Figure 7(b) summarizes the effect of applying CHAMELEON's Adaptive Sampling module sim- ulated annealing and reinforcement learning based search. First, the results show that using Adaptive Sampling helps the framework make less hardware measurements regardless of the search algorithm used. The Adaptive Sampling algorithm reduces the number of measurements by when used with simulated annealing and 2.33 with reinforcement learning One observation is that the Adaptive Sampling is more effective with reinforcement learning search. This comes from the reinforcement learning agent's s capacity to better localize the search to meaningful sam- ples (exploitation) while still aiming to find good solution by making diverse search (exploration).
Diversity exploration of AutoTVM aims to spread out the candidate con- figurations with a regularizing effect that fosters uniform sampling. In
1.4g
Simileted Arneeing
1.4g
Reindieremeen Learning
(eversity Exploacion
Introvmment over SunnTe
Sampling
22
1.11x
2.J4
Imd Indrosemann ere Duvee
111x
contrast, our Adaptive Sampling uses
1.50
100x
1.0
1::5
01991
a clustering algorithm to perform
1.0
3:55
3.0c
0.214
more measurements the regions
0.x0x
with higher likelihood of achieving better output performance, leading a non-uniform sampling. While Au-
0.8c
88
12
9m
0.cc
gm
(a) Simulated Annealing.
(b) Reinforcement Learning.
toTVM states that diversity-aware lection had no meaningful impact on most of the evaluated workloads, our Adaptive Sampling brings significant improvement as depicted in Figure 8 As shown, Adaptive Sampling brings an average of 13.5% and 19.0% improvement on simulated annealing and reinforcement learning, respectively.
Figure 8: Comparison to AutoTTMM's diversity exploration.
4.3 INTEGRATION: REDUCING OPTIMIZATION TIME AND OUTPUT INFERENCE TIME
CHAMELEON integrates components into the workflow: RL-based Adaptive Exploration (AE) and Adaptive Sampling (AS). This section compares the performance of CHAMELEON with Au- toTVM (Chen al. 2018b) that leverages Simulated Annealing (SA) for its exploration.
Layer evaluation. over number hardware measurements during optimization. The figure illustrates that our Adaptive Exploration finds better configurations than simulated annealing which results in better output code performance, and the Adaptive Sampling reduces number of hardware measurements significantly during optimization. Also, CHAMELEON's 's Adaptive Exploration and Adaptive Sampling working in tandem emits better code with shorter optimization time than others. As such, Figure (U(a))comparss optimization time and the performance of the output code in CHAMELEON and Auto'I VVM confirm
Figure the trend of output code performance of ResNet- 18's 11th layer
8

---

Published as a conference paper at ICLR 2020
significantly number measurements
while even improving the output code performance (from 4.71 5.26)
5.22
Simubited Annealing (V20
4.71
Adaptive Exploration
Simuusted Annealing Adoative Sampling
Adeeptive Expleration Adspr Sampling [CHaMPLsCN)
200
460
650
Hardware messtrements
Figure 9: Layer evaluation of output performance for ResNet-18's 11th layer.
the observation. CHAMELEON achieved 1.17xbetter performance with 4.82 shorter optimization time compared to AVM. Overall, the results suggest that Adaptive Exploration effectively maneuvers the design space, and Adaptive Sampling reduces hardware measurements and the overall optimization time while even improving output performance.
End-to-end evaluation. Up until we continue our discussion to the applicability of CHAMELEON to optimization of end-to-end deep neural networks. TOCh shows that CHAMELEON spends 3.59x, 5.73 and 4.28 less time than Au to optimize Ale VGG-16, and BesNet-18, respectively. On average, our work shows 4.45 x45xomtiii time speedup while achieving up 6.4% improvement in terms of performance of output code. Inference time in Figure To(b) illustrates the speedup for optimized code. Raw numbers are available Table and Table 3 Adaptive Exploration and the reduced number of hardware measurements from Adaptive Sampling.
have focused evaluation with subset of layers. Now
All all, such improvements result from efficient
Optimization Time
Output Perfoumance
Optimization
Output Performance
Impro wwwoooooeeenn ent over Automo
1.38X
122x
121x
1,200
AutoTV
Coreeea Comeeni ower AuTSTUM
633G
6.28x
7.5k
56. 23
1ix
3.600
8.82x
5.07x
1.00
50%
Spocdip
1.6x
Mediee 18
Restet
18
grmmon
Restett8 geanean
(a) Layer evaluation.
(b) End-to-end evaluation.
Figure 10: Layer and end-to-end evaluation. Dashed lines denote Au's'' performance.
NETWORK
SA
AE
SA AS
AE AS (CHAMELEON)
(AutoTVM)
AlexNet
4.31 Hours
4.06 Hours
1.25 Hours
1.20 Hours
VGG-16
11.18 Hours
8.82 Hours
2.57 Hours
1.95 Hours
ResNet-18
9.13 Hours
7.39 Hours
2.14 Hours
2.13 Hours
Table 2: End-to-end evaluation of the optimization time for deep networks.
NETWORK
SA
AE
SA AS
AE AS (CHAMELEON)
(AutoTVM)
AlexNet
1.0277
1.0207
0.9762 ms
0.9673
VGG-16
3.9829 ms
3.9710 ms
3.8733 ms
3.8458 ms
R8sNet-18
1.0258 ms
0.9897 ms
0.9897 ms
0.9831 ms
Table 3: End-to-end evaluation of the output performance for deep networks.
5 RELATED WORKS
CHAMELEON uniquely offers solution that exclusively enables (i) Reinforcement Learning and (ii) Sampling in the context of (iii) Optimizing Compilers for neural networks, As such, we discuss the related work from each of the three independent research directions.

---

Published as a conference paper at ICLR 2020
Optimizing compilers. TensorComprrhemsiions (Vasilache et al. 2018 and TVM (Chen et al. 2018a) use genetic algorithm and simulated annealing to choose parameters of polyhedral optimiza- tion for neural networks. In a more general context, some computing libraries (Whaley & Dongarra 1998 Frigo & Johnson, 1998) make use of black box optimization and also profiling-based compi- lation passes (Chang al. 1991 Novillo 2014) utilize runtime information generate optimized code. Later, VuM (Chen et al. 2018b) (8b) incorporates learning with boosted trees within the cost model for TVM reduce the number of real hardware measurements. While CHAMELEON is inspired and builds on these prior works, unlike them, it is based on reinforcement learning for Adaptive Ex- ploration, and Adaptive Sampling that leverages clustering to reduce the number of measurements.
Reinforcement learning for aapee-parramter optimization. There are a growing body of stud- ies on using reinforcement learning to perform various optimizations (Gao et al., 2018 Mirhoseini
al. 2017: Nareyek 2003 Mao et al,, 2016 Xu al. 2018; Mao al. 2019 for a variety of objectives including hyper--aramann parameter optimization for neural networks. For instance, DeepArchi- tect (Negrinho & Gordon 2017) and NAS (Zoph & Le 2017) use reinforcement learning to automate
the process of designing deep neural network models and their associated parameters. HAQ (Wang) jet al. 2019)) and Rel.eQ (Elthakeb et al.] 2018) use reinforcement learning to chose levels of quan- tization for the layers of given deep neural network. AMC (He al., 2018) formulates neural network compression as a RL problem. A most recent effort (Paliwal et al. -hich will be RL with graph neural networks and genetic
published concurrent to ours in ICLR algorithms to optimize DNN execution. Our work exclusively explores a different problem, that is optimizing compilers using reinforcement learning.
Sampling algorithms for learning. Active learning is a broad field (Settles) 2009 Cohn et al. 1996 SSugiyaaa 2006 Cai et al. 2013) Goetz et al. 2018: Wu et al, 2019) that uses measure of the change in the model to decide which training data elements should be used to update the model. Passive learning (Yu & Kim 2010 O'Neill et al. (017) is an alternative view that independent of the model, analyze the distribution of the training data set and selects subset. The Adaptive Sampling algorithm for CHAMELEON shares similarities with Passive learning but it differs in its context. The sampling is designed to reduce the number of samples (configuration) for hardware measurement from the exploration of the design space whilst performing an optimization to accelerate the process.
6 CONCLUSION
We present CHAMELEON allow optimizing compilers to adapt to unseen design spaces of code schedules to reduce the optimization time. This paper is also an initial effort to bring reinforcemens learning the realm of optimizing compilers for neural networks, and also develop an Adap- five Sampling with domain-knowledde inspired Sample Synthesis to not only reduce the number of samples required to navigate the design space but also augment its quality in terms of fitness. Experimentation with real-world deep models shows that CHAMELEON not only reduces the time for compilation significantly, but also improves the quality of the code. This encouraging result suggests significant potential for various learning techniques to optimizing deep learning models.
ACKNOWLEDGEMENT
We thank the anonymous reviewers for their insightful comments. We also thank Jinwon Lee and Jangho Kim for the fruitful discussions and feedbacks on the manuscript. This work was in part supported by generous gifts from Qualcomm, Google, Microsoft, and Xilinx as well as the Semiconductor Research Corporation (SRC) contract #2019-SD-2888 National Science Founda- ECCS# 1609823, CCF#1553199 Air Force Office of Scientific Research (AFOSR) Young Investigator Program (YIP) award 1PA9550-17-1-02777 National Insti- tute of Health (NIH) award BR02550, and Air Force Research Laboratory (AFRL) and De- fense Advanced Research Project Agency (DARPA) under agreement number #FA8860-2002270009 The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed implied, of AFRL, DARPA the U.S. Government.
tion (NSF) awards
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. TensorFlow: A system for large-scale machine learning. In OSDI, 2016.
Byung Hoon Ahn, Jinwon Lee, Jamie Menjay Lin, Hsin-Pai Cheng, Jilei Hou, and Hadi Esmaeilzadeh. Or- dering chaos: Memory-awart scheduling of irregularly wired neural networks for edge devices. In MLSys, 2020.
Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda. Extremely large minibatch SGD: training ResNet-50 on ImageNet in 15 minutes. arXiv,, 2017. |tLass::///axx http://gggpppppaee 04325. pdt
Wenbin Cai, Ya Zhang, and Jun Zhou. Maximizing expected model change for active learning in regression. In ICDM, 2013.
Pohua P Chang. Scott A Mahlke, and Wen-Mei W Hwu. Using profile information assist classic code optimizations. Software: Practice and Experience, 1991.
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In KDD, 2016.
Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang. Yuwei Hu, Luis Ceze, TVM: An automated end-to-end optimizing compiler for deep learning. In OSDI, 2018a.
Tianqi Chen, Lianmin Zheng, Eddie Yan, Ziheng Jiang, Thierry Moreau, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy, Learning to optimize tensor programs. NeurIPS, 2018b.
Valeriu Codreanu, Damian Podareanu, and Vikram Saletore. Achieving deep learning training less than 40 minutes on ImageNet-IK K & best accuracy and training time on ImaeeNet-22K & P15 with scale-out Intel Xeon (Xeon Phi TM architectures. 2017. URL htttttssts///ooooo surt ml/en//
int-iineellle imagenet- -nrranning-- -eon--eeo---p------ less than- 40-minutem//mn///////
David A Cohn, Zoubin Ghahramani, and Michael Jordan. Active learning with statistical models. JAIR, 1996.
Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis. TPAMI, 2002.
Ahmed Elthakeb, Prannoy Pilligundla, Amir Yazdanbakhsh, Sean Kinzer, and Hadi Esmacilzadeh. ReLeQ:
A reinforcement learning approach for deep quantization of neural networks. arXiv, 2018. URL https:
001.001700 01704.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. density-based algorithm for discovering clusters density-based algorithm for discovering clusters in large spatial databases with noise. In KDD, 1996.
Matteo Frigo and Steven G Johnson. FFTW: An adaptive software architecture for the FFT. In ICASSP, 1998.
Yuanxiang Gao, Li Chen, and Baochun Li. Post: Device placement with cross-entropy minimization and proximal policy optimization. NeurlPS, 2018.
Jack Goetz, Ambuj Tewari, and Paul Zimmerman. Active learning for non-parametric regression using purely random trees. NeurlPS, 2018
Priya Goyal, Piotr Dollár, Ross Girshick, Picter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in hour. arXiv, 2017. httstttps:
32677.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang. Li-Jia Li, and Song Han. AMC: AutoML for model compression and acceleration on mobile devices. In ECCV, 2018.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv, 2017. |ttttt:::///axii wwtp http//dp///1/ 200E/1704.0. 000.0.00886..p..
Ken Kennedy and John R Allen. Optimizing compilers for modern architectures: dependence-hasee ap- proach. Morgan Kaufmann Publishers Inc.,, 2001.
11

---

Published as a conference paper at ICLR 2020
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, 2012.
Chris Lattner and Vikram Adve. LLVM: compilation framework for lifelong program analysis transfor- mation, CGO, 2004.
Yann LeCun. Deep learning hardware: Past, present, and future. In ISSCC, 2019.
Yizhi Liu, Wang, Ruofei Yu, Mu Li, Sharma, and Yida Wang. Optimizing CNN model inference on CPUs. In USENIX ATC, 2019.
Hongzi Mao, Mohammad Alizadeh, Ishai Menache, and Srikanth Kandula. Resource management with deep reinforcement learning. In HotNets, 2016.
Hongzi Mao, Malte Schwarzkopf. Shaileshh Bojja Venkatakrishnan, Zili Meng. and Mohammad Alizadeh. Learning scheduling algorithms for data processing clusters. In SIGCOMM, 2019.
Peter Mattson, Christine Cheng, Cody Coleman, Greg Diamos, Paulius Micikevicius, David Patterson, Hanlin Tang, Yoon Wei, Peter Bailis, Victor Bittorf, al. MLPerf training benchmark. arXiv, 2019, URL
https://aaxxww.or/ppdrrrr 015000000001.....000000000000000 pdf
Azalia Mirhoseini, Hieu Pham, Quoe Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Naveen Kumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean. Device placement optimization with reinforcement learning. In ICMI,, 2017.
Alexander Nareyek. Choosing search heuristics by non-stationary reinforcement learning. In Metaheuristics: Computer Decision-Makiing Springer, 2003.
Renato Negrinho and Geoff Gordon. DeepArchitect: Automatically designing and training deep architectures. arXiv, 2017. URL
Diego Novillo. SamplePGO the power of profile guided optimizations without the usability burden. In LLVM Compiler Infrastructure in HPC, 2014.
Jack O'Neill, Sarah Jane Delany, and Brian MacNamee. Model-free and model- active learning for regression. In Advances in Computational Intelligence Systems. Springer, 2017.
Aditya Paliwal, Felix Gimeno, Vinod Nair, Yujia Li, Miles Lubin, Pushmeet Kohli, and Oriol Vinyals. Re- inforced genetic algorithm learning for optimizing computation graphs. In ICLR, 2020. URL https:
//openreview.no net. http:r/www.e...eeefoooddddddddde
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
Nadav Rotem, Jordan Fix, Saleem Abdulrasool, Garret Catron, Summer Deng, Roman Dzhabarov, Nick Gib- son, James Hegeman, Meghan Lele, Roman Levenstein, et al. Glow: Graph lowering compiler techniques for neural networks, arXiv, 2018. URL/htcs:://arke http:/axxxv....g.poo/// wwtpp//p//gg///// 00907. 7.p
Eric Schkufza, Rahul Sharma, and Alex Aiken. Stochastic superopttimization. In ASPLOS, 2013.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alcc Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv, 2017. URL httpsstts:///// http:///wwwwvvvoaaa///oo//
06347.
Burr Settles. Active learning literature survey. Technical report, University of Wisconsin-Madison Department of Computer Sciences, 2009.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Richard M Stallman and GCC DeveloperCommunity. Using the GNU compiter collection: a GNU manual for GCC version 4.3.3. CreateSpace, 2009.
Masashi Sugiyama. Active learning in approximately lincar regression based on conditional expectation of generalization error. JMLR, 2006.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions, In CVPR, 2015.
12

---

Published as a conference paper at ICLR 2020
Nicolas Vasilache, Oleksandr Zinenko, Theodoros Theodoridis, Priya Goyal, Zachary DeVito, William Moses, Sven Verdoolaege, Andrew Adams, and Albert Cohen. Tensor Comprehensions: Framework agnostic high-performance machine learning abstractions. arXiv, 2018. URL httppp:///acxi.oogg
pct/1802. 04730, pdf
Kuan Wang, Zhijian Liu, Yujun Lin, Lin, and Song Han. HAQ: Hardware-aware automated quantization with mixed precision. In CVPR, 2019.
Clinton Whaley and Jack Dongarra. Automatically tuned linear algebra software. SC, 1998.
Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. Discovering neural wirings. NeurIPS, 2019.
Dongrui Wu, Chin-Teng Lin, and Jian Huang. Active learning for regression using greedy sampling. Informa- tion Sciences, 2019.
Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He. Exploring randomly wired neural networks image recognition. ICCV, 2019.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. In NeurlPS, 2018.
Yang You. Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv, 2017. 0tt.///100.0388..00. .df
Yang You, Zhao Zhang, Cho-Jui Hsich, James Demmel, and Kurt Keutzer. ImageNet training in minutes. In ICPP, 2018.
Hwanjo Yu and Sungchul Kim. Passive sampling for regression. In ICDM, 2010.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. BMVC, 2016.
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. ICLR, 2017. URL
13

---

Published as a conference paper at ICLR 2020
APPENDIX
A EXPERIMENTAL SETUP
A.I DNN MODELS AND LAYERS
Table 4: Details of the DNN models used in evaluating CHAMELEON.
NETWORK
DATASET
NUMBER OF TASKS
AlexNet VGG-16
ImageNet ImageNet
ResNet-18
ImageNet
Table 5: Details of the layers used in evaluating CHAMELEON.
NAME
MODEL
LAYER TYPE
TASK INDEX
LI
AlexNet
convolution
AlexNet
convolution
L3
VGG-16
convolution
L4
VGG-16
convolution
2
L5
VGG-16
convolution
4
ResNet-18
convolution
ResNet-18
convolution
L8
ResNet-18
convolution
11
A.2 HARDWARE SPECIHICATION
Table 6: Details of the hardware used for evaluation of CHAMELEON.
SPECIHCATIONS
DETAILS
GPU
Titan Xp
Host CPU
3.4G Hz Intel Core i7
Main Memory
32GB 2400 MHz DDR3
14

---

Published as a conference paper at ICLR 2020
A.3 PAPEA-PARAMETES
Table 7: Hyper-parameters uses in CHAMELEON.
HYPERPARAMETER
VALUE
DESCRIPTION
iterationopt
16
number of iterations for optimization process (cquivalent to 1000 hardware measurements) type of loss used for cost model
modeGBT
xgb-reg 64
bGBT
maximum batch size of planning in GBT (Chen & Guestrin 2016
cost model per iteration of optimization process
episode,t
128
number of episodes for reinforcement learning
step
500
maximum steps of one reinforcement learning episode threshold used for beta--searc in sampling
thresholdmeta
2.5
Table 8: Syyer-parameeee uses in AutoTVM (Chen et al. 2018b
HYPERPARAMETER
VALUE
DESCRIPTION
E(baBT)
1000
total number of hardware measurements type of loss used for model
modeGBT
xgb-reg
bGBT
64
batch size planning GBT (Chen Guestrin 2016
Tisa
128
number of Markov chains in parallel simulated annealing maximum steps of one simulated annealing run
slepsa
500
Table 9: Hyper-parameters used in CHAMELEON's PPO
(Schulman et al. 2017) search agent.
HYPERPARAMETER
VALUE
Adam Step Size
x 10
Discount Factor
0.9
GAE Parameter
0.99
Number Epochs
3
Clipping Parameter
0.3
Value Coefficient
1.0
Entropy Coefficient
0.1
15

---

Published as a conference paper at ICLR 2020
B ADDITIONAL EXPERIMENTAL RESULTS
B.1 OPTIMIZATION TIME BREAKDOWN FOR DNN MODELS
Algorithm
2.57
2.11
3.08
2.00
2.01
Outmieat Thre
1.54
2.00
71 60.
0.54
25.12%
13.00%
STS
aachets Corvolation Layer
VGG-1E Comclttin Layer
Raslen-1Ps Comeclatin Layer
(a) AlexNet.
(b) VGG-16.
(c) ResNet-18 18.
Figure 11: MuMoTVM optimization time for AlexNet Krizhevsky al. 2012) and VGG-16 (Si-)
monyan & Zisserman 2015), and ResNet-18 (He et al. 2016) on Titan Xp. Numbers bars denote fraction of time for measurements.
B.2
PERFORMANCE VS. NUMBER OF MEASUREMENTS FOR DNN MODELS
Simulated
Simulated Arnealing Adaptive Sampling
Adaptive Exploration
Acaptive Exploration Adaptive Sampling (CHAMELOO))
834
F14
XX
433
423
20
Phetcs HHAEEeA
64
69
Figure 12: Layer evaluations for AlexNet (Krizhevsky et al. 2012)
16

---

Published as a conference paper at ICLR 2020
Simulated Annealing (AutoTVM) Adaptive Exploration
Simulated Annealing Acaptive Sampling
Adaptive Exploration Adaptive Sampling (CHAMELSON)
360
320
120
205
150
180
222
Prysat
40
Pradiaa
Figure 13: Layer evaluations for VGG-16 Simonyyn & Zisserman 2015)
Simulated Anneaing (AutoTVM) Adaptive Exploration
Simulated Annealing Adaptive Sampling
Exploration Adaptive Sampling (CHaNELSON)
4.2
3.5
4.0
F20
304
30%
120
119
432
Peotical Wenoreeer
Figure 14: Layer evaluations for ResNet-18 (He et al. 2016
17


Reviews:
[{"review_id": "SkeyGWrHKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new solution called CHAMELEON for deep learning code optimization, which accelerates the process of compiling codes and achieves faster training and inference of deep networks. The proposed method can be used to compile various deep network architectures. Experimental results show that the proposed method outperforms the previous method with large margin.\n\nThe paper exploits reinforcement learning to address code compilation, which is novel for me. \n\nThe experimental results are convincing, the paper evaluates the proposed evaluation in multiple aspects.\n\nI am totally new in the area. I will refer to the other reviews' review and the authors' rebuttal to have my final decision.\n\n\nPost-rebuttal\nThank the other two reviewers and the authors to help me better understand the paper. I think I have no concern on the paper so I still give 6.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkxnE68RYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an optimizing compiler  for DNN's based on adaptive sampling and reinforcement learning, to drive the search of optimal code in order to reduce compilation time as well as potentially improve the efficiency of the code produced. In particular, the paper proposes to use PPO to optimize a code optimization \"search\" policy, and then use K-mean clustering over a set of different proposed compilation proposals, from which to perform adaptive sampling to reduce compilation time while still keeping a high diversity of the proposed solution pools during exploration. At the same time the authors claim that using RL will learn a better search strategy compared to random search - such as simulated annealing which is used by competing methods - thus producing faster and better solutions.\nThe paper show results of up to 4x speedup in compilation time (autotuning time) while obtaining a slightly better or similar efficiency of the generated code (in term of execution time). This is a well written extensive research with good results. The authors mention it is (will be) integrated in the open source code of TVM. However I could find no mention in the paper of whether the code will be released with this publication, and I would like to solicit the authors to clarify their code release strategy and timing.\nMy other question pertains to whether or not  compilation time is an key metric to target. It is important to some extent, but I would say that aside from exponential / super-polynomial behaviour of auto-tuning algorithms, a multiple hours / days process to create the best optimized code for a certain network / hardware platform might not be such a big hurdle for a community already used to multiple days / weeks / months to train the same models. I believe that focusing on the efficiency of the optimized code produced would probably be a better metric of success.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylKi-56tH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a method for code optimization for deploying neural networks. The main idea is to formulate it as a search task over tuning knobs in the code template, and to apply reinforcement learning to optimize the configurations for the tuning knobs with respect to a cost model. The cost model is trained based on a subset of representative samples from the RL controller and their corresponding hardware cost measurements.\n\nThe paper is very well written and the proposed method seems technically sound. The authors did a good job combining existing techniques in a complementary manner. For example, the usage of RL for efficient search space exploration and the usage of clustering for selecting representative samples for on-device measurements.\n\nSome concerns:\n* The authors are referring to the usage of reinforcement learning as Adaptive Exploration and the usage of clustering as Adaptive Sampling. While combining them to tackle the task of neural network compilation is interesting, these techniques themselves are very standard and hence come with limited technical novelty.\n* The proposed workflow seems to involve a nontrivial amount of additional hyperparameters, e.g., those in the RL controller as well as those for clustering. It might be useful to discuss about the overhead caused by hyperparameter tuning, as otherwise numbers reported in Table 2 (based on a single trial) could be misleading.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to optimize the code optimal code in DNN compilers using adaptive sampling and reinforcement learning. This method achieves  significant speedup in compilation time and execution time. The authors made strong efforts in addressing the problems raised by the reviewers, and promised to make the code publicly available, which is of particular importance for works of this nature.   

Decision: Accept (Poster)