Ternary MobileNets via Per-Layer Hybrid Filter Banks

1 INTRODUCTION . Deeper and wider convolutional neural networks ( CNNs ) has led to outstanding predictive performance in many machine learning tasks , such as image classification ( He et al . ( 2016 ) ; Krizhevsky et al . ( 2012 ) ) , object detection ( Redmon et al . ( 2016 ) ; Ren et al . ( 2015 ) ) , and semantic segmentation ( Chen et al . ( 2018 ) ; Long et al . ( 2015 ) ) . However , the large model size and corresponding computational inefficiency of these networks often make it infeasible to run many real-time machine learning applications on resource-constrained mobile and embedded hardware , such as smartphones , AR/VR devices etc . To enable this computation and size compression of CNN models , one particularly effective approach has been the use of resource-efficient MobileNets architecture . MobileNets introduces depthwise-separable ( DS ) convolution as an efficient alternative to the standard 3-D convolution operation.While MobileNets architecture has been transformative , even further compression of MobileNets is valuable in order to make a wider range of applications available on constrained platforms ( Gope et al . ( 2019 ) ) . Model quantization has been a popular technique to facilitate that . Quantizing the weights of MobileNets to binary ( -1,1 ) or ternary ( -1,0,1 ) values in particular has the potential to achieve significant improvement in energy savings and possibly overall throughput especially on custom hardware , such as ASICs and FPGAs while reducing the resultant model size considerably . This is attributed to the replacement of multiplications by additions in binary- and ternary-weight networks . Multipliers occupy considerably more area on chip than adders ( Li & Liu ( 2016 ) ) , and consume significantly more energy than addition operations ( Horowitz ( 2014 ) ; Andri et al . ( 2018 ) ) . A specialized hardware can therefore trade off multiplications against additions and potentially accommodate considerably more adders than multipliers to achieve a high throughput and significant savings in energy for binary- and ternary-weight networks . However , prior approaches to binary and ternary quantization ( Rastegari et al . ( 2016 ) ; Alemdar et al . ( 2016 ) ; Li & Liu ( 2016 ) ; Tschannen et al . ( 2018 ) ) incur significant drop in prediction accuracy for MobileNets . Recent work on StrassenNets ( Tschannen et al . ( 2018 ) ) presents a more mathematically profound way to approximate matrix multiplication computation ( and , in turn , convolutions ) using mostly ternary weights and a few full-precision weights . It essentially exploits Strassen ’ s algorithm to approximate a matrix multiplication of a weight matrix with feature maps , where the elements of the product matrix are generated by different combination of few intermediate terms through additions . Computation of each of the intermediate terms requires a multiplication along with combination of different elements of weights and feature maps through additions . The number of intermediate terms ( also called hidden layer width ) in StrassenNets therefore determines the addition and multiplication budget of a convolutional layer and in turn decides the approximation error of the corresponding convolution operation . While the results in ( Tschannen et al . ( 2018 ) ) using StrassenNets demonstrates no loss in predictive performance when compared to full-precision models for few networks , the effectiveness of StrassenNets is quite variable , however , depending on the neural network architecture . We observe , for example , that while strassenifying is effective in reducing the model size of DS convolutional layers , this might come with a prohibitive increase in the number of addition operations , reducing the energy efficiency of neural network inference . The exorbitant increase in additions primarily stems from the use of wide hidden layers for closely approximating each convolutional filter in a network layer . While this might be required for some of the convolutional filters in a layer , our observations indicate that all filters may not require wide strassenified hidden layers . As different filters in a network layer tend to capture different features , they may respond differently to ternary quantization , and , in turn , to strassenified convolution with a specific hidden layer units . Some filters can be harder to approximate using ternary bits than others , and have larger impact on the model accuracy loss . Furthermore , given a constrained hidden layer budget for StrassenNets , a group of filters extracting fairly similar features at a layer may respond favorably to ternary quantization , while other filters of the layer extracting significantly different features from those may not . Guided by these insights , we propose a layer-wise hybrid filter banks for the MobileNets architecture capable of giving start-of-the-art accuracy levels , while requiring a fraction of the model size and considerably fewer MAC and multiplication operations per inference . The end-to-end learning of hybrid filter banks makes this possible by keeping precision critical convolutional filters in fullprecision values and strassenifying quantization tolerant filters only to ternary values . The filters that are most sensitive to quantization errors perform traditional convolutions with input feature maps , whereas ternary quantization tolerant filters can perform strassenified convolutions using narrow hidden layers . We apply this proposed quantization scheme to the state-of-the-art MobileNets-V1 architecture . The hybrid filter banks for MobileNets achieves a 46.4 % reduction in multiplications , and a 51.07 % reduction in model size while incurring modest increase in additions . This translates into a 27.98 % savings in energy required per inference while ensuring no degradation in throughput on a DNN hardware accelerator consisting of both MAC and adders when compared to the execution of baseline MobileNets on a MAC-only hardware accelerator . The hybrid filter banks accomplishes this with a very minimal loss in accuracy of 0.51 % . To the best of our knowledge , the hybrid filter banks proposed in this work is a first step towards quantizing the already compute-efficient MobileNets architecture to ternary values with a negligible loss in accuracy on a large-scale dataset , such as ImageNet . The remainder of the paper is organized as follows . Section 2 elaborates on the incentives behind the use of per-layer hybrid filter banks for the MobileNets architecture and provides a brief overview of current quantization algorithms along with our observations of applying them to the MobileNets architecture . Failing to find a good balance between accuracy and computation costs shifts our focus towards designing layer-wise hybrid filter banks for MobileNets . Section 3 describes our hybrid filter banks . Section 4 presents results . Section 5 compares hybrid filter banks against prior works and Section 6 concludes the paper . 2 MODEL QUANTIZATION LIMITATIONS FOR MOBILENETS . Quantization is an extremely popular approach to make DNNs , in particular convolutional neural networks ( CNNs ) , less resource demanding . This section briefly reviews the important existing works on ternary quantization , which we focus on in this paper , and illustrates their limitations to motivate the development of per-layer hybrid filter banks for quantizing MobileNets to ternary values . 2.1 TERNARY QUANTIZATION OF WEIGHTS . In order to observe the impact of ternary quantization ( Courbariaux et al . ( 2015 ) ; Rastegari et al . ( 2016 ) ; Lin et al . ( 2017 ) ; Cai et al . ( 2017 ) ; Li & Liu ( 2016 ) ; Zhu et al . ( 2016 ) ; Zhou et al . ( 2016 ) ) , we apply the ternary weight quantization method from ( Li & Liu ( 2016 ) ) over the baseline MobileNetsV1 architecture . It approximates a full-precision weight W fp by a ternary-valued W t and a scaling factor such that W fp ≈ scaling factor ∗ W t. Ternary quantization of the weights of MobileNets achieves substantial reduction in model size but at the cost of significant drop ( by 9.66 % , see Table 1 ) in predictive performance when compared to the full-precision model . Any increase in the size of the MobileNets architecture to recover the accuracy loss while using ternary quantization will lead to a significant increase in the number of addition operations . Recent work on StrassenNets ( Tschannen et al . ( 2018 ) ) , which we describe next , has shown the potential to achieve near state-of-the-art accuracy for a number of deep CNNs while maintaining acceptable increase in addition operations . 2.2 STRASSENNETS . Given two 2 × 2 matrices , Strassen ’ s matrix multiplication algorithm computes their product using 7 multiplications instead of the 8 required with a naı̈ve implementation of matrix multiplication . It essentially converts the matrix multiplication operation to a 2-layer sum-product network ( SPN ) computation as shown below : vec ( C ) =Wc [ ( Wbvec ( B ) ) ( Wavec ( A ) ) ] ( 1 ) Wa , Wb ∈Kr×n 2 and Wc ∈Kn 2×r are ternary matrices with K ∈ { −1 , 0 , 1 } , vec ( A ) and vec ( B ) are the vectorization of the two input square matrices A , B ∈ Rn×n ; and vec ( C ) represents the vectorized form of the product A × B. denotes the element-wise product . The ( Wbvec ( B ) ) and ( Wavec ( A ) ) of the SPN compute r intermediate factors each from additions , and/or subtractions of elements of A and B realized by the two associated ternary matrices Wa and Wb respectively . The two generated r-length intermediate factors are then element-wise multiplied to produce the rlength ( Wbvec ( B ) ) ( Wavec ( A ) ) . The outmost ternary matrix Wc later combines the r elements of the product ( Wbvec ( B ) ) ( Wavec ( A ) ) in different ways to generate the vectorized form of product matrix C. Therefore , the width of the hidden layer of the SPN , r , decides the number of multiplications required for the Strassen ’ s matrix multiplication algorithm . For example , given two 2 × 2 matrices , ternary matrices Wa and Wb with sizes of 7 × 4 can multiply them using 7 multiplications and 36 additions . It is important to note that Strasssen ’ s algorithm requires a hidden layer with 7 units here to compute the exact product matrix that a naı̈ve matrix multiplication algorithm can obtain using 8 multiplications . Building on top of Strassen ’ s matrix multiplication algorithm , the StrassenNets work ( Tschannen et al . ( 2018 ) ) instead realizes approximate matrix multiplications in DNN layers1 using fewer hidden layer units compared to the standard Strassen ’ s algorithm required to achieve the exact product matrix . StrassenNets makes this possible by training a SPN-based DNN framework end-to-end to learn the ternary weight matrices from the training data . The learned ternary weight matrices can then approximate the otherwise exact matrix multiplications of the DNN layers with significantly fewer multiplications than Strassen ’ s algorithm . The approximate transforms realized by the SPNs , adapted to the DNN architecture and application data under consideration , can enable precise control over the number of multiplications and additions required per inference , creating an opportunity to tune DNN models to strike an optimal balance between accuracy and computational complexity . 1A convolutional operation in DNN layers can be reduced to a general matrix multiplication ( GEMM ) . In the context of strassenified matrix multiplications of a network layer , A is associated with the weights or filters of the layer and B is associated with the corresponding activations or feature maps . As a result , after training , Wa and vec ( A ) can be collapsed into a vector â = Wavec ( A ) , as they are both fixed during inference . The success of StrassenNets in achieving significant compression for 3 × 3 convolutions ( Tschannen et al . ( 2018 ) ) and increasing visibility of DS convolutions in resource-constrained networks inspired us to apply StrassenNets over the already compute-efficient MobileNets architecture to reduce its computational costs and model size even further . Further compression of DS layers will not only enable more energy-efficient networks leading to longer lasting batteries , but also will open up the opportunities for more complex use-cases to fit in the limited memory budget of emergent DNN hardware accelerators . Among the various MobileNets architectures ( Howard et al . ( 2017 ) ; Sandler et al . ( 2018 ) ; Howard et al . ( 2019 ) ) , in this work we extensively study the quantization of MobileNets-V1 ( Howard et al . ( 2017 ) ) . MobileNets-V1 stacks one 3x3 and 13 DS convolutional layers . A DS convolution first convolves each channel in the input feature map with a separate 2-D filter ( depthwise convolution ) and then uses 1x1 pointwise convolutions to combine the outputs in the depth dimension .

Summary: This paper proposes a novel quantization method towards the MobileNets architecture, with the consideration of balance between accuracy and computation costs. Specifically, the paper proposes a layer-wise hybrid filter banks which only quantizes a fraction of convolutional filters to ternary values while remaining the rest as full-precision filters. The method is tested empirically on ImageNet dataset.