Auto-tuning Matrix Multiplication and Convolution for Deep Learning on CPUs

Deep learning ( DL ) compilers have emerged aiming to reduce the gap between1 abundant , fast-growing DL models and the lag of high performance implemen-2 tations of these models on diverse hardware devices . In this work , we introduce3 several optimization strategies , combining analytic ideal cache models with ma-4 chine learning models trained with real hardware measures , and integrate them into5 a unified auto-tuning framework , called AutoMCL , to improve the performance6 of DL compilers on both the operation level and the end-to-end model inference.7 We evaluate AutoMCL and compare it with state-of-the-art on multiple CPUs.8 End-to-end evaluations show that AutoMCL outperforms TensforFlow on fully9 connected and convolutional neural networks with respectively a geometric mean10 of 9.29× and 1.54× speedup . Over the baseline AutoTVM , on average , AutoMCL11 achieves respectively 1.37× and 2.16× speedup in inference and optimization time12 for fully connected neural networks and gains 2.55 % performance improvement in13 inference for convolutional neural networks with 1.91 % more optimization cost.14 1 Introduction15 Deep learning models have found wide applications in image and sound recognition , natural language16 translation , game playing , etc . The success of deep learning benefits greatly from the accessibility17 of DL frameworks , such as TensforFlow [ 4 ] , PyTorch [ 19 ] and MXNet [ 8 ] , which not only ease the18 burden of coding but also provide high performance supports through efficient low-level libraries,19 such as Intel oneMKL [ 2 ] or NVIDIA cuDNN [ 3 ] . However , it is difficult to make the library20 development , which requires tremendous manual engineering effort entangled with hardwares and21 often takes months or even years to finish , keep pace with the rapid innovation of DL models . As22 a result , many newly introduced neural networks or operators may lack optimal implementation23 support on the target hardwares , thus hindering the further innovation of DL models . To address24 this challenge , DL compilers ( e.g . TVM [ 9 ] and TensorComprehensions [ 25 ] ) emerged [ 16 ] , whose25 goal is to automatically compile high-level declarations of DL operators into efficient low-level code26 across various hardware devices , including CPUs , GPUs , FPGAs , and ASICs.27 To make the DL compilers appealing , it is essential to keep their performance competitive or even28 superior to that of DL frameworks or hand-optimized libraries . To achieve this , state-of-the-art DL29 compilers , such as TVM and its successor AutoTVM [ 10 ] , extend the decoupled compute/schedule30 principle of Halide [ 20 ] to separate target hardware intrinsics from computation description and31 optimization sequence specification composed of transform primitives to ease the process of high-32 level optimization , and leverage machine learning to automate low-level optimizations . The success33 of DL compilers relies on high-quality schedules as well as effective searching and learning strategies34 to find optimal parameters . Recently , new progress have been made on automating the design of35 schedule primitives , enlarging the parameter space to expose more tuning opportunities and utilizing36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . heuristic and learning approaches , in particular reinforcement learning , to explore the parameter37 space more effectively to find optimal candidates . Among these work , AdaTune [ 15 ] , Ansor [ 29 ] ,38 CHAMELEON [ 5 ] , FlexTensor [ 30 ] and Cortex [ 11 ] are built on top of TVM while the value function39 method [ 23 ] and TIRAMISU [ 6 ] are respectively based on Halide and the polyhedral model.40 Most of these optimizations have been focusing on the loop level optimizations , such as loop tiling,41 loop split and fuse , loop unroll , loop reordering , vectorization , etc . The algorithm level optimization,42 on the other hand , is hard to automate and still requires human ’ s expertise . Moreover , while enlarging43 the tuning space may potentially include better candidates , it also calls more effort to find the optimal44 solution and often leads to getting suboptimal solution in limited budget . Thus , it remains a great45 challenge to prune the parameter space efficiently to avoid unnecessary exploration , which may also46 help increase the chance of optimal solutions to be picked earlier . A purely analytical modeling47 approach for optimizing convolutions [ 17 ] was recently proposed towards this direction.48 In this work , we propose several new strategies aiming to leverage both analytic model and machine49 learning to generate more efficient code in shorter compilation time targeting on the CPU platforms,50 the ubiquity of which implies that a great number of users can benefit from such improvement . Our51 main contributions are three-fold:52 • We introduce new strategies for initializing and filtering the tiling size space for matrix53 multiplication and convolution based on analytic models.54 • We introduce several new competitive schedules for matrix multiplication and convolution55 in both algorithm and loop level to enlarge the schedule space.56 • We integrate the proposed strategies into a new auto-tuning framework called AutoMCL,57 which leverages TVM ’ s frontend computational graph optimization and backend code58 generation functionalities . We conduct operator level and end-to-end evaluations showing59 that the overall performance of AutoMCL is superior to AutoTVM in both inference and60 optimization time on typical fully connected or convolutional neural networks.61 2 Background62 The operations matrix multiplication and convolution appear widely in many deep neural networks63 and improving their performance is critical to speed up the the training and inference . Matrix multi-64 plication has been implemented on CPU in many basic linear algebra libraries , such as ATLAS [ 27 ] ,65 GotoBLAS [ 13 ] and Intel oneMKL [ 2 ] . The convolution operation was also implemented on CPU in66 several standalone libraries , such as Intel oneDNN [ 1 ] . In the context of deep learning , there is a a67 strong demand to deploy a well-trained model to a great variety and amount of devices such that the68 model can infer in real time on the target hardwares . This offers new challenges and opportunities for69 auto-tuning the performance of these two operations for fixed size input tensors [ 28 , 18 ] .70 Matrix multiplication and 2D-convolution operators . Mathematically , the matrix multiplica-71 tion operator matmul takes two matrices AM×K and BK×N as input and computes their prod-72 uct matrix CM×N . In this paper , we would assume that the operator takes two matrices DM×K73 and WN×K and computes a new matrix CM×N by Cij : = ∑K−1 k=0 AikBjk . The 2D-convolution74 operator conv2d , in its simplest form , takes a tensor D of dimensions B × IC × DH × DW ,75 a tensor W of dimensions OC × IC × KH × KW , two stride sizes s1 , s2 , and produces76 a tensor C of dimensions B × OC × OH × OW , where OH = ( DH − KH ) /s1 + 177 and OW = ( DW − KW ) /s2 + 1 . Each element of C is computed according to the rule78 Cb , o , y , x : = ∑IC−1 i=0 ∑KH−1 ky=0 ∑KW−1 kx=0 Db , i , s1y+ky , s2x+kxWo , i , ky , kx . In general , it may also takes79 two padding sizes PH , PW and two dilation sizes d1 , d2 and produces a tensor of dimen-80 sions B × OC × OH × OW , where OH = ( DH + 2PH − ( KH − 1 ) d1 − 1 ) /s1 + 1 and81 OW = ( DW + 2PW − ( KW − 1 ) d2 − 1 ) /s2 + 1.82 Ideal cache model . The ideal cache model was introduced in [ 12 ] for studying the cache complexity83 of algorithms . It assumes that the computer has a two-level memory hierarchy consisting of an ideal84 cache of Z words with cache line size C , where Z C , and an arbitrarily large main memory . To85 access a word in main memory , it first searches it in cache . If the word does not reside in the cache,86 a cache miss occurs and a cache line containing the word is loaded into the cache from the main87 memory . It assumes that the cache is fully associative and the line with furthest access in the future88 will be replaced if new data is loaded into a full cache . The cache complexity counts the number of89 cache misses . For instance , the ( worst ) cache complexity for scanning n words continuously stored in90 an array is dn/Ce+ 1 . In general , the cache complexity of an algorithm operating on a tensor largely91 depends on the layout of the tensor and the ordering for visiting the dimensions of the tensor.92 3 Components of AutoMCL93 We design a few optimization passes and evaluate the effectiveness of each optimization strategy94 individually and only append experimentally proven working optimizations to our framework . Fig . 195 provides an overview of the framework , named AutoMCL . 96 Enlarging the space of schedules . The DL compiler TVM provides two default computes for the97 matrix multiplication operator , namely DNMM and RPMM and one default compute CONV for gen-98 eral 2D-convolution . We introduce another four alternative computes TMM , TTMM , DPMM , LPMM99 for matrix multiplication and two alternative computes Im2colDNMM332 and Im2colRPMMV for100 convolution by converting convolution to matrix multiplication in an im2col manner . Table 1 summa-101 rizes the specification of each compute for matrix multiplication . The computes for convolution can102 be found in the supplemental material . We manually write schedule template for each new compute103 and improve the default schedule templates for DNMM , RPMM , CONV respectively as DNMM332104 ( single-level tiling to double-level tiling ) , RPMMV ( adding missing vectorization for some loop ) and105 CONVOpt ( loop reordering according to the cache complexity analysis in Theorem 2 and its remark ) .106 We analyze the cache complexity with the ideal cache model for each schedule template , stated as107 Theorem 1 and Theorem 2 , whose detailed proof can be found in the supplemental material . Note108 that all the nested loops will be tiled in the schedules . This would lead to a better cache complexity if109 the data required for computing a tile all fit in cache . This assumption depends both on the tile and110 cache size but should not depend on the input tensor size ( with the kernel sizes as an exception since111 they are usually small ) . Table 2 and Table 3 summarize the assumptions.112 Let V ` be the length of vectorization , C ` be the cache line size , Z be the cache size , and D ` be the113 size of tensor data type in bytes . Let Vw : = V ` /D ` , Cw : = C ` /D ` , Zw : = Z/D ` .114 Theorem 1 . Assume that Tm ( Mt , Kt , Nt ) < ZwCw and Mt|M , Kt|K , Nt|N , the cache complexity115 Cm ( M , K , N , Mt , Kt , Nt ) for each schedule is listed as below:116 TMM : MMt N Nt ( Mt ( dKtCw e+ 1 ) K Kt +Nt ( dKtCw e+ 1 ) K Kt +Mt ( d NtCw e+ 1 ) ) TTMM : KKt N Nt ( Kt ( d NtCw e+ 1 ) +Nt ( dKtCw e+ 1 ) ) + MMt N Nt ( Mt ( dKtCw e+ 1 ) K Kt +Kt ( d NtCw e+ 1 ) K Kt +Mt ( d NtCw e+ 1 ) ) DNMM : MMt N Nt ( d KKt e ( Mt +Nt ) ( dKtCw e+ 1 ) +Mt ( dNtKtCw e+ 1 ) +Mt ( d NtCw e+ 1 ) ) LPMM : MMt ( d KMt Cw e+ 1 ) +M ( d KCw e+ 1 ) + d MtNt Cw e+ 1 + MMt N Nt ( Mt ( d NtCw e+ 1 ) + KKt ( Nt ( dKtCw e+ 1 ) + dKtMtCw e+ 1 ) ) RPMM : NNt ( d KNt Cw e+ 1 ) +N ( d KCw e+ 1 ) ) + d MtNt Cw e+ 1 + MMt N Nt ( Mt ( d NtCw e+ 1 ) + KKt ( Mt ( dKtCw e+ 1 ) + dKtNtCw e+ 1 ) ) DPMM : MMt ( dKMtCw e+ 1 ) +M ( d KCw e+ 1 ) + NNt ( dKNtCw e+ 1 ) +N ( d KCw e+ 1 ) +dMtNtCw e+ 1 + M Mt N Nt ( Mt ( d NtCw e+ 1 ) + d KKt e ( dKtMtCw e+ 1 + d KtNt Cw e+ 1 ) ) . Theorem 2 . Assume that Tc ( Mt , Kt , Nt ) < ZwCw and OWt|OW , ICt|IC , OCt|OC , the cache com-117 plexity for CONVOpt is:118 dB ∗ ( DH + 2PH ) ∗ IC ∗ ( DW + 2PW ) Cw e+ 1 + ( B ∗DH ∗ IC ∗ ( dDW Cw e+ 1 ) ) +OC ∗ IC ICt ∗ ( dKH ∗KW ∗ ICt Cw e+ 1 ) ( 1 ) +IC ∗KH ∗KW ∗ OC OCt ∗ ( dOCt Cw e+ 1 ) + ( B ∗ OC OCt ∗OH ∗ OW OWt ∗ ( dOWt ∗OCt Cw e+ 1 ) ) +B ∗ OC OCt ∗OH ∗ OW OWt ∗ IC ∗KH ∗KW ∗ ( dOCt Cw e+ 1 ) +B ∗ OC OCt ∗ IC ∗OH ∗ OW OWt ∗KH ∗ ( d ( s2 ∗ ( OWt − 1 ) + ( KW − 1 ) ∗ d2 + 1 ) Cw e+ 1 ) +B ∗ OC OCt ∗OH ∗OW ∗ ( dOCt Cw e+ 1 ) + ( B ∗OH ∗OW ∗ OC OCt ∗ ( dOCt Cw e+ 1 ) ) +B ∗OC ∗OH ∗ OW OWt ∗ ( dOWt Cw e+ 1 ) , and the cache complexity of Im2col-CONV is:119 dB ∗ IC ∗ ( DH + 2PH ) ∗ ( DW + 2PW ) Cw e+ 1 + ( dB ∗ IC ∗DH ∗DW Cw e+ 1 ) + B ∗OH ∗OW ∗ IC ICt ∗ ( dICt ∗KH ∗KW Cw e+ 1 ) + ( 2 ∗OC ∗ IC ICt ∗ ( dKH ∗KW ∗ ICt Cw e+ 1 ) ) +B ∗OH ∗OW ∗ IC ∗KH ∗ ( d ( OWt − 1 ) ∗ s2 + ( KW − 1 ) ∗ d2 + 1 Cw e+ 1 ) +Cm ( B ∗OH ∗OW , IC ∗KH ∗KW , OC , OWt , ICt ∗KH ∗KW , OCt ) +B ∗OC ∗OH ∗ d OW OWt e ∗ ( dOWt Cw e+ 1 ) + ( B ∗OH ∗OW ∗ d OC OCt e ∗ ( dOCt Cw e+ 1 ) ) . Remark 1 . For the cache complexity of CONV in TVM , we only need to replace the bold part in120 Table 3 with OCt ∗ ICt ∗ ( dKWCw e+ 1 ) +KW ∗ ICt ∗ ( dOCt/Cwe+ 1 ) and ( 1 ) in Theorem 2 by121 OC ∗ IC ∗KH ∗ ( dKWCw e+ 1 ) . It is usually larger than that of CONVOpt for the same tiling size.122 Learning to choose schedules . We first evaluate the performance of each schedule on a dataset123 consisting of matrices with sizes ranging from small to large . The experiments , reported in Section 4,124 show that each one can be exclusively the best for certain types of sizes . We then choose the top125 four best performed schedules as candidates and train a boosted tree model by Xgboost [ 7 ] , with the126 matrix size as input feature , to automatically select the best one for a particular size.127 Initializing the tiling size space . Suppose that there are m dimensions to be tiled and the size of128 each dimension is Xi , i = 1 , . . . , m. Then the number of valid one-level tilings is ∏m i=1 Xi , which129 is one billion for Xi = 1000 . Thus one has to set up a reasonable initial tiling size space . For130 instance , in TVM , there are two basic strategies depending on the tiling size being a factor of Xi or a131 power of 2 . Suppose that there are m dimensions X1 , . . . , Xm to be tiled , and each dimension has a132 nested tiling of levels di , i = 1 , . . . , m. Then the initial configure space for the factor strategy is a133 direct product of the sets Gi : = { ( X ( 0 ) i , . . . , X ( di ) i ) | ∏di j=0 X ( j ) i = Xi } , i = 1 , . . . , m. We adopt134 this factor strategy for 2D-convolution . For matrix multiplication , we propose a more sophisticated135 strategy , motivated by both the factor strategy of TVM and the analytic model of [ 21 ] to balance136 cache locality and load balancing among parallel threads . This strategy is described by Algorithm 1.137 Filtering the tiling size space . Let G ( Zt , Yt , Xt ) be the initial tiling size space for the com-138 pute/schedule pair ( O , S ) , where Zt , Yt , Xt denote the innermost tiling sizes for the tiled dimensions139 Z , Y , X . Let T ( Xt , Yt , Xt ) be the cache fit formula Tc or Tm . Let X be the dimension for vectoriza-140 tion and Xt be the tiling size for this dimension . We would only consider the tiling size satisfying141 both X ≥ min ( Xt , Vw ) and T < Zw/Cw and filter out the rest ones from G.142 Learning to choose optimal configurations . Except for the default schedule CONV of TVM , the143 configuration space for all the schedules considered in this work is solely formed by different tiling144 sizes . The schedule CONV has another knob unroll_kw to decide whether to unroll the for loop145 involving the kernel dimension KW . The size of the configuration space in our experiments is146 usually less than 10 , 000 thanks to the initialization and filter strategies . For this moderate size , we147 find that the rather direct tuning strategy described by Algorithm 2 works quite well in practice.148 4 Evaluation149 We developed AutoMCL on top of TVM ( 0.6.0 ) and it will be released in open source . Three Intel150 CPUs ( Intel i7-G9700F , Intel i7-9750H , Intel i9-9900 ) and one AMD CPU ( AMD-Ryzen9-3900X ) are151 used for evaluation . More detailed hardware information can be found in the supplemental material.152 We first evaluate each optimization strategy individually based on TVM on randomly generated153 datasets consisting of tensors of various sizes , in order to see if a particular optimization can speed up154 either optimization time or inference time . Then we evaluate the whole integrated framework on both155 the operation and the end-to-end level for typical fully connected and convolutional neural networks.156 The maximum number of trials for the whole tuning and the early stopping are set respectively as157 10 , 000 and 400 for most of the experiments . The only exception is the end-to-end evaluation of158 CNNs , where we set the two numbers respectively as 500 and 300.159 Algorithm 1 : InitConfigSpace ( O , S ) Input : A compute/schedule pair ( O , S ) for matmul , the number of parallel threads p. Output : The initial configure space G for tiling . 1 begin 2 if S has 1-level tiling then 3 initialize G′ , Gx , Gy , Gk , Gyx respectively as ∅ ; 4 for all factors py of p do 5 px : = p/py ; let Gy and Gx be respectively all the factors of dM/pye and dN/pxe ; 6 Gyx : = { ( Mt , Nt ) |Mt ∈ Gy , Nt ∈ Gx } 7 let Gk be all the factors of K ; G : = { ( 1 , Mt , 1 , Nt , Kt ) | ( Mt , Nt ) ∈ Gyx , Kt ∈ Gk } ; 8 else if S has 2-level tiling then 9 initialize G′ , Gx , Gy , Gk , Gyx respectively as ∅ ; 10 for all factors py of p do 11 px : = p/py ; 12 let Gy : = { ( Mo , Mt ) : MoMt|dM/pye } ; Gx : = { ( No , Nt ) : NoNt|dN/pxe ; 13 Gyx : = { ( Mo , Mt , No , Nt ) | ( Mo , Mt ) ∈ Gy , ( No , Nt ) ∈ Gx } 14 let Gk be all the factors of K ; 15 G : = { ( Mo , Mt , No , Nt , Kt ) | ( Mo , Mt , No , Nt ) ∈ Gyx , Kt ∈ Gk } ; / * Due to limitation of TVM , it is additionally rquired that Mt|M for LPMM , Nt|N for RPMM and Mt|M , Nt|N for DPMM . * / 16 return G Algorithm 2 : AutoConfig ( O , S , G , m , n , b ) Input : The compute/schedule pair ( O , S ) , the configuration space G for ( O , S ) , the maximum number of trials m , the batch size n for restarting training , the batch size b for a parallel run . Output : The optimal configuration . 1 begin 2 D : = ∅ ; t : = 0 ; randomly pop n configurations from G and put in N ; 3 while true do 4 while N 6= ∅ do 5 choose b configurations B from N ; N : = N \B ; 6 in parallel , run the code compiled from the tuple ( O , S , c ) , c ∈ B , on hardware ; 7 add B examples labelled with ( averaged ) running timings to D ; t : = t+ |N | ; 8 if G 6= ∅ and t < m then 9 train a ML model with D and predict the running timings of ( O , S , c ) , c ∈ G ; 10 pop the best ( shortest predicted timing ) n configurations N from G ; 11 else 12 break ; 13 return the configurations in D with the shortest running time Comparison of different schedules . To make a fair comparison , we create two testing datasets160 consisting of examples of various dimension sizes for matrix multiplication and convolution . For161 matrix multiplication , a dimension size is chosen in three different scales , with small size in { 1 , 8 , 16 } ,162 medium size in { 64 , 256 } and large size in { 1024 , 4096 } , which creates 73 different combinations.163 We remove 5 extreme size cases and add additional 120 examples with each dimension randomly164 taking values in 1 .. 4096 . For convolution , we create a dataset of the same size ( 458 ) as matrix mul-165 tiplication . The dimensions of each convolution example ( DB×IC×DH×DW , WOC×IC×KH×KW ) 166 with stride s and padding size p randomly take values by the following rule : B ∈ { 1 , 32 , 128 } ,167 IC ∈ { 20 · · · 214 } , OC ∈ { 20 .. 214 } , DH = DW ∈ { 1 · · · 256 } , KH = KW ∈ { 1 , 3 , 5 , 7 } ,168 s ∈ { 1 , 2 } , p = b ( KH − 1 ) /2c . In addition , we only keep examples with each dimension size less169 than 4096 in their im2col representations.170 Fig . 2 reports the proportions of examples with the shortest running time or the lowest cache misses171 ( measured by the ideal cache model ) for different schedules implementing matrix multiplication or172 convolution . The experiments show that each schedule can be exclusively the best for certain types of173 tensor sizes . Here we allow a 0.02 tolerance for being the best . Our manually improved schedule174 DNMM332 , RPMMV and CONVOpt indeed work better than their counterparts . Moreover , the real175 and the theoretical measure correlate quite well for the “ top performed ” schedules , except for the two176 based on DNMM , which however have a different vectorization dimension from the others . 177 Evaluation of automatic schedule chosen . With performing exclusively the best on at least 5 % of178 the dataset as a criterion , four “ top performed ” schedules DNMM332 , RPMMV , LPMM , TMM are179 selected for matmul and three are selected for conv2d . For matmul , we adopt Xgboost to automatically180 choose the best schedule among the four for a given problem size . The dataset is the same as the181 one in last subsection , from which 40 randomly chosen examples are reserved for the testing dataset182 and the rest for the training dataset . Fig . 3 reports the performance on the testing dataset , where183 AutoSchedule denotes the learned schedule and OptSchedule stands for choosing schedules in a static184 manner as TVM but with DNMM and RPMM replaced respectively by DNMM332 and RPMMV.185 For conv2d , the learning approach does not work quite well and we instead use CONVOpt as the 186 default implementation since it performs better than CONV while having the same advantage as187 CONV on leveraging NCHWc layout optimization [ 18 ] in the end-to-end inference.188 Evaluation of tiling size space initialization and filter . Fig . 4 illustrates how the two default189 schedules for matmul ( DNMM when M ≤ 16 and RPMM when M > 16 ) perform when being190 combined with different strategies for initializing the tiling size space . The left image shows the191 speedup over the base ( factor ) . The middle and right images show the space swell ratio over the192 base ( factor ) . Our strategy pfactor shrinks the tiling size space more than 40 % for matrices with193 powers of 2 sizes without an obvious performance loss . For the dataset consisting of matrices of194 prime number sizes , pfactor brings 1.2 speedup on average.195 Fig . 5 show that the filter strategy further reduces tiling space size while not loosing performance.196 Comparison of different configuration space exploiting strategies . Fig . 6 compares AutoTVM ’ s197 exploration module ( SA+RANK ) and AutoMCL ’ s performance model ( REG ) on tuning GEMMs of198 different sizes . The left and right image show the average performance of tuned matrix multiplications199 and the average tuning time . 200 Evaluation of AutoMCL on the operation and the end-to-end level . Now we evaluate the per-201 formance of AutoMCL , which integrates all the optimization strategies introduced in Section 3 , on202 optimizing matmul and conv2d for both fully connected neural networks ( FCNNs ) [ 26 ] and typical203 convolutional neural networks ( CNNs ) ResNet-50 [ 14 ] , Inception-v3 [ 24 ] , and VGG16 [ 22 ] .204 Ablation analysis . We analyze the effects of adding different optimizations on the performance,205 where OS and AS stand for using respectively the optimized and the automatically chosen schedules.206 5 Conclusion207 In this paper , we have introduced a framework AutoMCL to auto-tune the matrix multiplication and208 the 2D-convolution operations in fully connected and convolutional neural networks by leveraging209 both analytic and machine learning models . Experiments show that it outperforms AutoTVM on both210 inference speed and optimization cost for FCNNs and is competitive to AutoTVM for CNNs . In the211 future , we plan to further improve its performance by designing better strategies on automatically212 choosing the optimal schedule.213 References214 [ 1 ] Intel oneDNN . https : //01.org/oneDNN.215 [ 2 ] Intel oneMKL . https : //software.intel.com/content/www/us/en/develop/tools/216 oneapi/components/onemkl.html.217 [ 3 ] NVIDIA cuDNN . https : //developer.nvidia.com/cudnn.218 [ 4 ] Martín Abadi , Paul Barham , Jianmin Chen , Zhifeng Chen , Andy Davis , Jeffrey Dean , Matthieu219 Devin , Sanjay Ghemawat , Geoffrey Irving , Michael Isard , et al . Tensorflow : A system for220 large-scale machine learning . In 12th { USENIX } symposium on operating systems design and221 implementation ( { OSDI } 16 ) , pages 265–283 , 2016.222 [ 5 ] Byung Hoon Ahn , Prannoy Pilligundla , Amir Yazdanbakhsh , and Hadi Esmaeilzadeh.223 Chameleon : Adaptive code optimization for expedited deep neural network compilation . arXiv224 preprint arXiv:2001.08743 , 2020.225 [ 6 ] Riyadh Baghdadi , Abdelkader Nadir Debbagh , Kamel Abdous , Fatima Zohra Benhamida,226 Alex Renda , Jonathan Elliott Frankle , Michael Carbin , and Saman Amarasinghe . Tiramisu : A227 polyhedral compiler for dense and sparse deep learning , 2020.228 [ 7 ] Tianqi Chen and Carlos Guestrin . Xgboost : A scalable tree boosting system . In Proceedings of229 the 22nd acm sigkdd international conference on knowledge discovery and data mining , pages230 785–794 , 2016.231 [ 8 ] Tianqi Chen , Mu Li , Yutian Li , Min Lin , Naiyan Wang , Minjie Wang , Tianjun Xiao , Bing Xu,232 Chiyuan Zhang , and Zheng Zhang . Mxnet : A flexible and efficient machine learning library for233 heterogeneous distributed systems . arXiv preprint arXiv:1512.01274 , 2015.234 [ 9 ] Tianqi Chen , Thierry Moreau , Ziheng Jiang , Lianmin Zheng , Eddie Yan , Haichen Shen , Meghan235 Cowan , Leyuan Wang , Yuwei Hu , Luis Ceze , et al . TVM : An automated end-to-end optimizing236 compiler for deep learning . In 13th USENIX Symposium on Operating Systems Design and237 Implementation ( OSDI 18 ) , pages 578–594 , 2018.238 [ 10 ] Tianqi Chen , Lianmin Zheng , Eddie Yan , Ziheng Jiang , Thierry Moreau , Luis Ceze , Carlos239 Guestrin , and Arvind Krishnamurthy . Learning to optimize tensor programs . Advances in240 Neural Information Processing Systems , 31:3389–3400 , 2018.241 [ 11 ] Pratik Fegade , Tianqi Chen , Phil Gibbons , and Todd Mowry . Cortex : A compiler for recursive242 deep learning models . CoRR , abs/2011.01383 , 2020.243 [ 12 ] Matteo Frigo , Charles E. Leiserson , Harald Prokop , and Sridhar Ramachandran . Cache-oblivious244 algorithms . ACM Trans . Algorithms , 8 ( 1 ) , January 2012.245 [ 13 ] Kazushige Goto and Robert A van de Geijn . Anatomy of high-performance matrix multiplication.246 ACM Transactions on Mathematical Software ( TOMS ) , 34 ( 3 ) :1–25 , 2008.247 [ 14 ] Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep residual learning for image248 recognition . In 2016 IEEE Conference on Computer Vision and Pattern Recognition ( CVPR ) ,249 pages 770–778 , 2016.250 [ 15 ] Menghao Li , Minjia Zhang , Chi Wang , and Mingqin Li . Adatune : Adaptive tensor program251 compilation made efficient . Advances in Neural Information Processing Systems , 33 , 2020.252 [ 16 ] Mingzhen Li , Yi Liu , Xiaoyan Liu , Qingxiao Sun , Xin You , Hailong Yang , Zhongzhi Luan , Lin253 Gan , Guangwen Yang , and Depei Qian . The deep learning compiler : A comprehensive survey.254 IEEE Transactions on Parallel and Distributed Systems , 32 ( 3 ) :708–727 , 2020.255 [ 17 ] Rui Li , Yufan Xu , Aravind Sukumaran-Rajam , Atanas Rountev , and P. Sadayappan . Analytical256 characterization and design space exploration for optimization of cnns . In Proceedings of the257 26th ACM International Conference on Architectural Support for Programming Languages and258 Operating Systems , ASPLOS 2021 , page 928–942 , New York , NY , USA , 2021 . Association for259 Computing Machinery.260 [ 18 ] Yizhi Liu , Yao Wang , Ruofei Yu , Mu Li , Vin Sharma , and Yida Wang . Optimizing CNN model261 inference on cpus . In 2019 USENIX Annual Technical Conference ( USENIX ATC 19 ) , pages262 1025–1040 , Renton , WA , July 2019 . USENIX Association.263 [ 19 ] Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan,264 Trevor Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , et al . Pytorch : An imperative265 style , high-performance deep learning library . arXiv preprint arXiv:1912.01703 , 2019.266 [ 20 ] Jonathan Ragan-Kelley , Connelly Barnes , Andrew Adams , Sylvain Paris , Frédo Durand , and267 Saman Amarasinghe . Halide : A language and compiler for optimizing parallelism , locality , and268 recomputation in image processing pipelines . SIGPLAN Not. , 48 ( 6 ) :519–530 , June 2013.269 [ 21 ] Yukinori Sato , Tomoya Yuki , and Toshio Endo . An autotuning framework for scalable execution270 of tiled code via iterative polyhedral compilation . ACM Transactions on Architecture and Code271 Optimization ( TACO ) , 15 ( 4 ) :1–23 , 2019.272 [ 22 ] Karen Simonyan and Andrew Zisserman . Very deep convolutional networks for large-scale273 image recognition . In Yoshua Bengio and Yann LeCun , editors , 3rd International Conference274 on Learning Representations , ICLR 2015 , San Diego , CA , USA , May 7-9 , 2015 , Conference275 Track Proceedings , 2015.276 [ 23 ] Benoit Steiner , Chris Cummins , Horace He , and Hugh Leather . Value learning for throughput277 optimization of deep learning workloads . In Proceedings of the 4th MLSys Conference , San278 Jose , CA , USA , 2021.279 [ 24 ] Christian Szegedy , Vincent Vanhoucke , Sergey Ioffe , Jon Shlens , and Zbigniew Wojna . Re-280 thinking the inception architecture for computer vision . In 2016 IEEE Conference on Computer281 Vision and Pattern Recognition ( CVPR ) , pages 2818–2826 , 2016.282 [ 25 ] Nicolas Vasilache , Oleksandr Zinenko , Theodoros Theodoridis , Priya Goyal , Zachary DeVito,283 William S Moses , Sven Verdoolaege , Andrew Adams , and Albert Cohen . Tensor comprehen-284 sions : Framework-agnostic high-performance machine learning abstractions . arXiv preprint285 arXiv:1802.04730 , 2018.286 [ 26 ] Yu Emma Wang , Gu-Yeon Wei , and David Brooks . Benchmarking tpu , gpu , and cpu platforms287 for deep learning . arXiv preprint arXiv:1907.10701 , 2019.288 [ 27 ] R. Clinton Whaley and Jack J Dongarra . Automatically tuned linear algebra software . In SC ’ 98:289 Proceedings of the 1998 ACM/IEEE conference on Supercomputing , pages 38–38 . IEEE , 1998.290 [ 28 ] Huaqing Zhang , Xiaolin Cheng , Hui Zang , and Dae Hoon Park . Compiler-level matrix multipli-291 cation optimization for deep learning . arXiv preprint arXiv:1909.10616 , 2019.292 [ 29 ] Lianmin Zheng , Chengfan Jia , Minmin Sun , Zhao Wu , Cody Hao Yu , Ameer Haj-Ali , Yida293 Wang , Jun Yang , Danyang Zhuo , Koushik Sen , et al . Ansor : Generating high-performance294 tensor programs for deep learning . In 14th { USENIX } Symposium on Operating Systems Design295 and Implementation ( { OSDI } 20 ) , pages 863–879 , 2020.296 [ 30 ] Size Zheng , Yun Liang , Shuo Wang , Renze Chen , and Kaiwen Sheng . Flextensor : An automatic297 schedule exploration and optimization framework for tensor computation on heterogeneous298 system . In Proceedings of the Twenty-Fifth International Conference on Architectural Support299 for Programming Languages and Operating Systems , pages 859–873 , 2020.300 Checklist301 The checklist follows the references . Please read the checklist guidelines carefully for information on302 how to answer these questions . For each question , change the default [ TODO ] to [ Yes ] , [ No ] , or303 [ N/A ] . You are strongly encouraged to include a justification to your answer , either by referencing304 the appropriate section of your paper or providing a brief inline description . For example:305 • Did you include the license to the code and datasets ? [ Yes ] See Section ? ? .306 • Did you include the license to the code and datasets ? [ No ] The code and the data are307 proprietary.308 • Did you include the license to the code and datasets ? [ N/A ] 309 Please do not modify the questions and only use the provided macros for your answers . Note that the310 Checklist section does not count towards the page limit . In your paper , please delete this instructions311 block and only keep the Checklist section heading above along with the questions/answers below.312 1 . For all authors ... 313 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s314 contributions and scope ? [ Yes ] 315 ( b ) Did you describe the limitations of your work ? [ Yes ] See Section 4 on “ Evaluation of316 automatic schedule chosen ” .317 ( c ) Did you discuss any potential negative societal impacts of your work ? [ N/A ] 318 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to319 them ? [ Yes ] 320 2 . If you are including theoretical results ... 321 ( a ) Did you state the full set of assumptions of all theoretical results ? [ Yes ] 322 ( b ) Did you include complete proofs of all theoretical results ? [ Yes ] , but only in the323 supplemental material due to space limit.324 3 . If you ran experiments ... 325 ( a ) Did you include the code , data , and instructions needed to reproduce the main ex-326 perimental results ( either in the supplemental material or as a URL ) ? [ Yes ] , in the327 supplemental material.328 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they329 were chosen ) ? [ Yes ] , in the supplemental material.330 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-331 ments multiple times ) ? [ Yes ] , see Section 4 on “ Ablation analysis ” 332 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type333 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] , in the supplemental material.334 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 335 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] 336 ( b ) Did you mention the license of the assets ? [ Yes ] , in the supplemental material.337 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 338 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re339 using/curating ? [ N/A ] 340 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable341 information or offensive content ? [ N/A ] 342 5 . If you used crowdsourcing or conducted research with human subjects ... 343 ( a ) Did you include the full text of instructions given to participants and screenshots , if344 applicable ? [ N/A ] 345 ( b ) Did you describe any potential participant risks , with links to Institutional Review346 Board ( IRB ) approvals , if applicable ? [ N/A ] 347 ( c ) Did you include the estimated hourly wage paid to participants and the total amount348 spent on participant compensation ? [ N/A ] 349

Summary: This work introduces AutoMCL - a new auto-tuning framework aimed at making compiled output higher performing, while improving the compile-time as well though better pruning of the search space. Authors focus primarily on lower-level library primitives, specifically: fully connected and convolution layers. Their primary contribution is a mix of analytical and ML-driven models. More specifically authors introduce some analytical models, suited for assessing effectiveness of various tiling strategies on CPUs, with the help of ideal cache models and parameters learning/modeling cache complexity. These help both cutting down the search space with respect to prior work based on ML-based learning, and find in some cases better schedules as well.  Net gain on optimization/compile time is more significant (more than 2x over AutoTVM).  Gain in inference performance through better schedule generation at kernel level is less - about 37% for full-connected networks. CNNs show just about 2% improvement in throughput and optimization time.