Smart Ternary Quantization

1 INTRODUCTION . Deep Neural Networks ( DNN ) models have achieved tremendous attraction because of their success on a wide variety of tasks including computer vision , automatic speech recognition , natural language processing , and reinforcement learning ( Goodfellow et al. , 2016 ) . More specifically , in computer vision DNN have led to a series of breakthrough for image classification ( Krizhevsky et al. , 2017 ) , ( Simonyan & Zisserman , 2014 ) , ( Szegedy et al. , 2015 ) , and object detection ( Redmon et al. , 2015 ) , ( Liu et al. , 2015 ) , ( Ren et al. , 2015 ) . DNN models are computationally intensive and require large memory to store the model parameters . Computation and storage resource requirement becomes an impediment to deploy such models in many edge devices due to lack of memory , computation power , battery , etc . This motivated the researchers to develop compression techniques to reduce the cost for such models . Recently , several techniques have been introduced in the literature to solve the storage and computational limitations of edge devices . Among them , quantization methods focus on representing the weights of a neural network in lower precision than the usual 32-bits float representation , saving on the memory footprint of the model . Binary quantization ( Courbariaux et al. , 2015 ) , ( Hubara et al. , 2016 ) , ( Rastegari et al. , 2016 ) , ( Zhou et al. , 2016 ) , ( Lin et al. , 2017 ) represent weights with 1 bit precision and ternary quantization ( Lin et al. , 2015 ) , ( Li & Liu , 2016 ) , ( Zhu et al. , 2016 ) with 2 bits precision . While the latter frameworks lead to significant memory reduction compared to their full precision counterpart , they are constrained to quantize the model with 1 bit or 2 bits , on demand . We relax this constraint , and present Smart Ternary Quantization ( STQ ) that allows mixing 1 bit and 2 bits layers while training the network . Consequently , this approach automatically quantizes weights into binary or ternary depending upon a trainable control parameter . We show that this approach leads to mixed bit precision models that beats ternary networks both in terms of accuracy and memory consumption . Here we only focus on quantizing layers because it is easier to implement layer-wise quantization at inference time after training . However , this method can be adapted for mixed precision training of sub-network , block , filter , or weight easily . To the best of our knowledge this is the first attempt to design a single training algorithm for low-bit mixed precision training . 2 RELATED WORK . There are two main components in DNN ’ s , namely , weight and activation . These two components are usually computed in full precision , i.e . floating point 32-bits . This work focuses on quantizing the weights of the network , i.e . generalizing BinaryConnect ( BC ) of Courbariaux et al . ( 2015 ) and Ternary Weight Network ( TWN ) of Li & Liu ( 2016 ) towards automatic 1 or 2 bits mixed-precision using a single training algorithm . In BC the real value weights w are binarized to wb ∈ { −1 , +1 } during the forward pass . To map a full precision weight to a binary weight , the deterministic sign function is used , wb = sign ( w ) = { +1 w ≥ 0 , −1 w < 0 . ( 1 ) The derivative of the sign function is zero on R \ { 0 } . During back propagation , this cancels out the gradient of the loss with respect to the weights after the sign function . Therefore , those weights can not get updated . To bypass this problem , Courbariaux et al . ( 2015 ) use a clipped straight-through estimator ∂L ∂w = ∂L ∂wb 1|w|≤1 ( w ) ( 2 ) where L is the loss function and 1A ( . ) is the indicator function on the set A . In other words ( 2 ) approximates the sign function by the linear function f ( x ) = x within [ −1 , +1 ] and by a constant elsewhere . During back propagation , the weights are updated only within [ −1 , +1 ] . The binarized weights are updated with their corresponding full precision gradients . Rastegari et al . ( 2016 ) add a scaling factor to reduce the gap between binary and full-precision model ’ s accuracy , defining Binary Weight Network ( BWN ) . The real value weights W in each layer are quantized as µ × { −1 , +1 } where µ = E [ |W| ] ∈ R. Zhou et al . ( 2016 ) , generalize the latter work and approximates the full precision weights with more than one bit while Lin et al . ( 2017 ) approximate weights with a linear combination of multiple binary weight bases . 2.1 TERNARY WEIGHT NETWORKS . Ternary Weight Network ( TWN ) ( Li & Liu , 2016 ) is a neural network with weights constrained to { −1 , 0 , +1 } . The weight resolution is reduced from 32 bits to 2 bits , replacing full precision weights with ternary weights . TWN aims to fill the gap between full precision and binary precision weight . Compared to binary weight networks , ternary weight networks have stronger expressive capability . As pointed out in ( Li & Liu , 2016 ) , for a 3×3 weight filter in a convolutional neural network , there is 23×3 = 512 possible variation with binary precision and 33×3 = 19683 with ternary precision . Li & Liu ( 2016 ) find the closest ternary weights matrix Wt to its corresponding real value weight matrix W using { µ̂ , Ŵt = arg min α , Wt ‖W− µWt‖22 , s.t . µ ≥ 0 , wtij ∈ { −1 , 0 , 1 } , i , j = 1 , 2 , ... , n. ( 3 ) The ternary weight Wt is achieved by applying a symmetric threshold ∆ Wt = +1 wij > ∆ , 0 |wij | ≤ ∆ , −1 wij < −∆ . ( 4 ) Li & Liu ( 2016 ) define a weight dependant threshold ∆ and a scaling factor µ that approximately solves ( 3 ) . TWN is trained using stochastic gradient descent . Similar to BC and BWN schemes ; ternary-value weights are only used for the forward pass and back propagation , but not for the parameter updates . At inference , the scaling factor can be folded with the input X X W ≈ X ( µWt ) = ( µX ) Wt , ( 5 ) where denotes the convolution . Zhu et al . ( 2016 ) proposed a more general ternary method which reduces the precision of weights in neural network to ternary values . However , they quantize the weights to asymmetric values { −µ1 , 0 , +µ2 } using two full-precision scaling coefficients µ1 and µ2 for each layer of neural network . While the method achieve better accuracy as opposed to TWN , its hardware implementation becomes a challenge , because there are two unequal full precision scaling factors to deal with . Our method provides a compromise between BC and TWN and trains weights with a single trainable scaling factor µ . Weights jumps between ternary { −µ , 0 , +µ } and binary { −µ , +µ } . This provides a single algorithm for 1 or 2 bits mixed precision . 2.2 REGULARIZATION . Regularization term is the key to prevent over-fitting problem and to obtain robust generalization for unseen data . Standard regularization functions , such as L2 or L1 encourage weights to be concentrated about the origin . However , in case of binary network with binary valued weights , it is more appropriate to have a regularization function to encourage the weights about µ × { −1 , +1 } , with a scaling factor µ > 0 such as R1 ( w , µ ) = ∣∣|w| − µ∣∣ , ( 6 ) proposed in Belbahri et al . ( 2019 ) . A straightforward generalization for ternary quantization is R2 ( w , µ ) = ∣∣∣∣∣∣|w| − µ2 ∣∣− µ2 ∣∣∣∣ . ( 7 ) Regularizer ( 6 ) encourages weights about { −µ , +µ } , and ( 7 ) about { −µ , 0 , +µ } . The two functions are depicted in Figure 1 . These regularization functions are only useful when the quantization depth is set before training start . We propose a more flexible version to smoothly move between these two functions using a shape parameter . 3 SMART TERNARY QUANTIZATION . Here we propose an adaptive regularization function that switches between binary regularization of ( 6 ) and ternary regularization of ( 7 ) R ( w , µ , β ) = min ( ∣∣|w| − µ∣∣ , tan ( β ) |w| ) , ( 8 ) in which µ is a trainable scaling factor , and β ∈ ( π4 , π 2 ) controls the transition between ( 6 ) and ( 7 ) . As a special case β → π2 converges to the binary regularizer ( 6 ) and β → π 4 coincide with the ternary regularizer ( 7 ) , depicted in Figure 2 . A large value of tan ( β ) repels estimated weights away from zero thus yielding binary quantization , and small tan ( β ) values encourage zero weights . The shape parameter β controls the quantization depth . Quantization is done per layer , therefore we let β very per layer . We recommend to regularize β about π2 i.e . preferring binary quantization apriori R ( w , µ , β ) = min ( ∣∣|w| − µ∣∣ , tan ( β ) |w| ) + γ| cot ( β ) | , ( 9 ) in which γ controls the proportion of binary to ternary layers . For a single filter W the regularization function is a sum over its elements R ( W , µ , β ) = I∑ i=1 J∑ j=1 min ( ∣∣|wij | − µ∣∣ , tan ( β ) |wij | ) + γ| cot ( β ) | . ( 10 ) Large values of γ encourage binary layers . In each layer , weights are pushed to binary or ternary values , depending on the trained value of the corresponding β . A generalization of ( 9 ) towards Lp norms of Belbahri et al . ( 2019 ) is also possible . However , here we only focus on regularizer constructed using the L1 norm as the accuracy did not change significantly by using Lp norm with different values of p. The introduced regularization function is added to the empirical loss function L ( . ) . The objective function defined on weights W , scaling factors µ , and quantization depths β is optimized using back propagation L ( W , µ , β ) = L ( W ) + L∑ l=1 λl Kl∑ k=1 R ( Wkl , µkl , βl ) , ( 11 ) where k indexes the channel , and l indexes the layer . One may use a different regularization constant λl for each layer to keep the impact of the regularization term balanced across layers , indeed different layers may involve different number of parameters . We set λl = λ # Wl where λ is a constant , and # Wl is the number of weights in layer l. We propose to use the same threshold-based function of Li & Liu ( 2016 ) ( 3 ) , but with a fixed threshold ∆l per layer l. Note that Li & Liu ( 2016 ) propose a weight-dependant threshold . We let the possibility for the weights to only accumulate about { −µ , +µ } and not about 0 , depending on β . One may set ∆l to have the same balanced weights in { −µ , 0 , +µ } at initialization for all layers and let the weights evolve during training . Formally , if σl is the standard deviation of the initial Gaussian weights in layer l , we propose ∆l = 0.2× σl . The probability that a single weight lies in the range [ −∆l , ∆l ] is ≈ 0.16 . All the weights falling in this range will be quantized as zeros after applying the threshold function . Weights are naturally pushed to binary or ternary values depending on βl during training . Eventually , a threshold δ close to π2 ≈ 1.57 defines the final quantization depth for each layer . Final quantization depth of layer l : { Binary βl ≥ δ , Ternary βl < δ

Summary: The paper discusses a generalization to low bit quantization and combines the approaches of binary and ternary quantization methods. Past methods such as Binary Connect and Binary Weights Network have shown that you can train a network efficiently with 1-bit quantization, and methods such as Ternary Weights Network demonstrate 2-bit quantization with weights taking one of {-1, 0, 1} * mu, with mu being a scale computed per weight tensor. The authors generalize these two methods so that the choice of binary vs ternary weights can be made per layer automatically during training. The primary contribution to make that work is by incorporating a generic regularizer with addition hyper-parameters to trade-off between the binary weight regularizer and ternary weight regularizer. In addition to that, the regularization also includes a prior to make the layers prefer binary weights by default. This is done by adding a cost that penalizes the choice of ternary weights for each layer.