Chameleon: Adaptive Code Optimization for Expedited Deep Neural Network Compilation

1 INTRODUCTION . The enormous computational intensity of Deep Neural Networks ( DNNs ) have resulted in developing either hand-optimized kernels , such as NVIDIA cuDNN or Intel MKL that serve as backend for a variety of programming environment such as TensorFlow ( Abadi et al. , 2016 ) and PyTorch ( Paszke et al. , 2019 ) . However , the complexity of the tensor operations in DNNs and the volatility of algorithms , which has led to unprecedented rate of innovation ( LeCun , 2019 ) , calls for developing automated compilation frameworks . To imitate or even surpass the success of hand-optimized libraries , recent research has developed stochastic optimization passes : for general code , STOKE ( Schkufza et al. , 2013 ) , and neural network code , TVM ( Chen et al. , 2018a ) and TensorComprehensions ( Vasilache et al. , 2018 ) . TVM and TensorComprehensions are based on random or genetic algorithms to search the space of optimized code for neural networks . AutoTVM ( Chen et al. , 2018b ) builds on top of TVM and leverage boosted trees ( Chen & Guestrin , 2016 ) as part of the search cost model to avoid measuring the fitness of each solution ( optimized candidate neural network code ) , and instead predict its fitness . However , even with these innovations the optimizing compilation time can be around 10 hours for ResNet-18 ( He et al. , 2016 ) , and even more for deeper or wider networks . Since the general objective is to unleash new possibilities by developing automatic optimization passes , long compilation time hinders innovation and could put the current solutions in a position of questionable utility . To solve this problem , we first question the very statistical guarantees which the aforementioned optimization passes rely on . The current approaches are oblivious to the patterns in the design space of schedules that are available for exploitation , and causes inefficient search or even converges to solutions that may even be suboptimal . Also , we notice that current approaches rely on greedy sampling that neglects the distribution of the candidate solutions ( configurations ) . While greedy sampling that passively filter samples based on the fitness estimations from the cost models work , many of their hardware measurements ( required for optimization ) tend to be redundant and wasteful . Moreover , we found that current solutions that rely on greedy sampling lead to significant fractions of the candidate configurations being redundant over iterations , and that any optimizing compiler are prone to invalid configurations which significantly prolongs the optimization time . As such , this work sets out to present an Adaptive approach dubbed CHAMELEON to significantly reduce the compilation time and offer automation while avoiding dependence to hand-optimization , enabling far more diverse tensor operations in the next generation DNNs . We tackle this challenge from two fronts with the following contributions : ( 1 ) Devising an Adaptive Exploration module that utilizes reinforcement learning to adapt to unseen design space of new networks to reduce search time yet achieve better performance . ( 2 ) Proposing an Adaptive Sampling algorithm that utilizes clustering to adaptively reduce the number of costly hardware measurements , and devising a domain-knowledge inspired Sample Synthesis to find configurations that would potentially yield better performance . Real hardware experimentation with modern DNNs ( AlexNet , VGG-16 , and ResNet-18 ) on a highend GPU ( Titan Xp ) , shows that the combination of these two innovations , dubbed CHAMELEON , yields 4.45×speedup over the leading framework , AutoTVM . CHAMELEON is publicly available in the project page : https : //bitbucket.org/act-lab/chameleon . 2 CHALLENGES IN DEEP NEURAL NETWORK COMPILATION . The general life-cycle of deep learning models from its birth to deployment comprises of two major stages . First stage is the designing and the training of a deep learning model by a research scientist , with the primary goal of achieving the highest feasible accuracy . Then , with a general demand to enable the intelligence on a wide range of devices ( from mobile CPUs in the edge to cloud-scale GPUs ) , the second stage has emerged for the deployment of the pre-trained deep learning model to a target hardware by a deployment engineer . These stages are each iterative processes : research scientists iterate until it reaches the target performance in terms of accuracy whereas the deployment engineers iterate until the performance in terms of inference speed with a given hardware satisfies the given constraints . Importantly , these two stages are most often separate processes , and this paper mainly focuses on the second stage ( deployment ) of the cycle with an overarching goal of accelerating the overall deployment cycle by reducing the optimizing compilation time without compromising the performance of the output code . 2.1 COMPILATION WORKFLOW FOR DEEP NEURAL NETWORKS . Figure 1 illustrates how a compiler for DNNs takes an input modelM and emits an optimized code τ ( Θ∗ ) that runs the model efficiently on a given hardware . This flow is commensurate with TensorComprehensions ( Vasilache et al. , 2018 ) and TVM ( Chen et al. , 2018a ) , using which we implement CHAMELEON that is available as a separate package for adoption in even other frameworks . The first phase of the workflow is the frontend compiler which performs the translation from the compiler and applies target-independent and white-box target-dependent optimizations that do not incorporate a measure of runtime . Target-independent passes transform the input DNN model without specificity to the target hardware . Operator fusion and data layout transformation in TVM are some examples of these passes , which lie in the same category as dead-code elimination or loop-invariant code motion in GCC ( Stallman & DeveloperCommunity , 2009 ) or LLVM ( Lattner & Adve , 2004 ) . Target-dependent passes , on the other hand , the compiler takes the hardware architecture ( target ) into account while optimizing the program ; however , this also does not actively leverage runtime measures . The last stage is a black-box optimization pass , called optimizing compiler , that given a measure of performance at runtime from the hardware can further optimize the code . CHAMELEON falls in this class by offering an optimizing compiler that adapts to different design space to be more swift in optimizing deep neural networks compared to conventional approaches . KNOBS DEFINITION tile f , tile y , tile x Factors for tiling and binding # of filters height , and width of feature maps . tile rc , tile ry , tile rx Factors for tiling reduction axis such as # of channels , height , and width of filters . auto unroll max step Threshold of number of steps in the loop to be automatically unrolled . unroll explicit Explicitly unroll loop , this may let code generator to generate pragma unroll hint . Table 1 : Knobs in the design space to optimize convolution . 2.2 OPTIMIZING COMPILER FOR DEEP NEURAL NETWORKS . Optimizing compilers ( Kennedy & Allen , 2001 ) usually take a black-box approach and use hardware measurements to configure the optimization based on a measure of fitness f of each solution . In order to make the problem tractable , the optimizing compilers for deep neural networks reduce the problem down to tuning the knobs θ for the output code template τ , and can be formulated as : Θ∗ = argmax Θ f ( τ ( Θ ) ) , for Θ ∈ DΘ . ( 1 ) A combination of assignment to the knobs is said to be a configuration Θ = ( θ1 , θ2 , ... , θn ) while the dimensions of the design space DΘ is defined by the knobs . As such , in Equation 1 , an optimizing compiler starts from a code template τ for each layer , and makes use of a search algorithm and real hardware measurements to efficiently find the best configuration Θ∗ ∈ DΘ . In this context , there are three variables that determine the effectiveness of the optimizing compiler : ( 1 ) a large and diverse enough design space that covers a variety of transformations , ( 2 ) an effective search algorithm to adequately navigate this space , and ( 3 ) a mechanism to cut down the number of costly hardware measurements that check the fitness of a solution . Table 1 lists the knobs for performing convolution on a GPU , where it is crucial that the code ( 1 ) maximizes data reuse , ( 2 ) uses the shared memory wisely , and ( 3 ) minimizes bank conflicts . The knobs optimize various aspects of the execution , including tiling ( e.g. , tile x , tile y , . . . ) , unrolling ( e.g. , auto unroll max step and unroll explicit ) , and these knobs define a design space with 1010 possibilities . Given the vastness of the design space , the remaining challenges are designing an effective search algorithm and designing a mechanism that reduces the cost of each step in the search ( i.e . reducing the need to measure the hardware ) . 2.3 CHALLENGES IN DEEP NEURAL NETWORK COMPILATION . As shown in Figure 2 , optimizing compilation for DNNs may still take an eon even with the advances from prior works ( Chen et al. , 2018a ; b ; Vasilache et al. , 2018 ) With active research ( You et al. , 2017 ; Goyal et al. , 2017 ; Codreanu et al. , 2017 ; Akiba et al. , 2017 ; You et al. , 2018 ; Mattson et al. , 2019 ) that has been able to cut down the training time to only few hours ( You et al. , 2017 ; Goyal et al. , 2017 ) and even minutes ( You et al. , 2018 ; Akiba et al. , 2017 ) on big models ( e.g. , ResNet-50 ( He et al. , 2016 ) ) for ImageNet , it renders the optimizing compilation time of the current solutions seem even more prominent . Especially , since the above-mentioned compilers have been integrated to the deep learning pipelines of major players in the industry ( Liu et al. , 2019 ; Rotem et al. , 2018 ; Vasilache et al. , 2018 ) , many users of these pipelines including the deployment engineers must go through the compilation workflow depicted in Figure 1 numerous times . Therefore , current long compilation time can be a hindrance to deploying DNN in various hardware , hence a major bottleneck in enabling intelligence on wider range of target platforms . Furthermore , as we explore various neural topologies ( Xie et al. , 2019 ; Wortsman et al. , 2019 ) for better performance as illustrated in Ahn et al . ( 2020 ) , even deeper or wider networks ( Szegedy et al. , 2015 ; Zagoruyko & Komodakis , 2016 ) , and new operations ( Howard et al. , 2017 ) to achieve higher performance ( LeCun , 2019 ) , we are forced to optimize the networks more frequently . The long optimization times are multiplied with such trend , leaving the practical utility of the current compiler solutions to question . As such , the primary goal of this work is reducing the optimizing compilation time to meet the immediate needs of the industry for expedited DNN compilation to foster further diversity and innovation in designing DNNs . Adaptive Sampling . Such long optimization time results from the inefficiency of simulated annealing which ( while it stochastically guarantees a reasonable solution after huge number of iterations ) fails to capture the patterns in the design space that can be exploited during the search . On the other hand , we can see in the figure that majority of the optimization time is spent on reaching for measurements on real hardware that is used as a feedback for the aforementioned search . Also , current approach even suffers from numerous invalid configurations that not only wastes the limited hardware measurement budget that the compiler starts with , but also incurs serious overhead to reset the target hardware for subsequent hardware measurements . As such , it is important that a sampling mechanism that selects potential configurations for hardware measurements to be smarter to ensure that each measurement is maximizing the chances of achieving a good solution and that it evades the invalid configurations . However , the current approaches rely on greedy sampling that passively sample based on the estimations from the cost models . This not only has a tendency to overfit but also neglect that solutions are distributed non-uniformly and that there are numerous invalid configurations .

Summary: The authors proposed a method for code optimization for deploying neural networks. The main idea is to formulate it as a search task over tuning knobs in the code template, and to apply reinforcement learning to optimize the configurations for the tuning knobs with respect to a cost model. The cost model is trained based on a subset of representative samples from the RL controller and their corresponding hardware cost measurements.