Gradient $\ell_1$ Regularization for Quantization Robustness

1 INTRODUCTION . Deep neural networks excel across a variety of tasks , but their size and computational requirements often hinder their real-world deployment . The problem is more challenging for mobile phones , embedded systems , and IoT devices , where there are stringent requirements in terms of memory , compute , latency , and energy consumption . Quantization of parameters and activations is often used to reduce the energy and computational requirements of neural networks . Quantized neural networks allow for more speed and energy efficiency compared to floating-point models by using fixed-point arithmetic . However , naive quantization of pre-trained models often results in severe accuracy degradation , especially when targeting bit-widths below eight ( Krishnamoorthi , 2018 ) . Performant quantized models can be obtained via quantization-aware training or fine-tuning , i.e. , learning full-precision shadow weights for each weight matrix with backpropagation using the straight-through estimator ( STE ) ( Bengio et al. , 2013 ) , or using other approximations ( Louizos et al. , 2018 ) . Alternatively , there have been successful attempts to recover the lost model accuracy without requiring a training pipeline ( Banner et al. , 2018 ; Meller et al. , 2019 ; Choukroun et al. , 2019 ; Zhao et al. , 2019 ) or representative data ( Nagel et al. , 2019 ) . But these methods are not without drawbacks . The shadow weights learned through quantizationaware fine-tuning often do not show robustness when quantized to bit-widths other than the one they were trained for ( see Table 1 ) . In practice , the training procedure has to be repeated for each quantization target . Furthermore , post-training recovery methods require intimate knowledge of the relevant architectures . While this may not be an issue for the developers training the model in the first ∗Work done during internship at Qualcomm AI Research †Qualcom AI Research is an initiative of Qualcomm Technologies , Inc. place , it is a difficult step for middle parties that are interested in picking up models and deploying them to users down the line , e.g. , as part of a mobile app . In such cases , one might be interested in automatically constraining the computational complexity of the network such that it conforms to specific battery consumption requirements , e.g . employ a 4-bit variant of the model when the battery is less than 20 % but the full precision one when the battery is over 80 % . Therefore , a model that can be quantized to a specific bit-width “ on the fly ” without worrying about quantization aware fine-tuning is highly desirable . In this paper , we explore a novel route , substantially different from the methods described above . We start by investigating the theoretical properties of noise introduced by quantization and analyze it as a ` ∞-bounded perturbation . Using this analysis , we derive a straightforward regularization scheme to control the maximum first-order induced loss and learn networks that are inherently more robust against post-training quantization . We show that applying this regularization at the final stages of training , or as a fine-tuning step after training , improves post-training quantization across different bit-widths at the same time for commonly used neural network architectures . 2 FIRST-ORDER QUANTIZATION-ROBUST MODELS . In this section , we propose a regularization technique for robustness to quantization noise . We first propose an appropriate model for quantization noise . Then , we show how we can effectively control the first-order , i.e. , the linear part of the output perturbation caused by quantization . When the linear approximation is adequate , our approach guarantees the robustness towards various quantization bit-widths simultaneously . We use the following notation throughout the paper . The ` p-norm of a vector x in Rn is denoted by ‖x‖p and defined as ‖x‖p : = ( ∑n i=1 |xi|p ) 1/p for p ∈ [ 1 , ∞ ) . At its limit we obtain the ` ∞-norm defined by ‖x‖∞ : = maxi |xi| . The inner product of two vectors x and y is denoted by 〈x , y〉 . 2.1 ROBUSTNESS ANALYSIS UNDER ` p-BOUNDED ADDITIVE NOISE The error introduced by rounding in the quantization operation can be modeled as a generic additive perturbation . Regardless of which bit-width is used , the quantization perturbation that is added to each value has bounded support , which is determined by the width of the quantization bins . In other words , the quantization noise vector of weights and activations in neural networks has entries that are bounded . Denote the quantization noise vector by ∆ . If δ is the width of the quantization bin , the vector ∆ satisfies ‖∆‖∞ ≤ δ/2 . Therefore we model the quantization noise as a perturbation bounded in the ` ∞-norm . A model robust to ` ∞-type perturbations would also be robust to quantization noise . To characterize the effect of perturbations on the output of a function , we look at its tractable approximations . To start , consider the first-order Taylor-expansion of a real valued-function f ( w+∆ ) around w : f ( w + ∆ ) = f ( w ) + 〈∆ , ∇f ( w ) 〉+R2 , ( 1 ) where R2 refers to the higher-order residual error of the expansion . We set R2 aside for the moment and consider the output perturbation appearing in the first-order term 〈∆ , ∇f ( w ) 〉 . The maximum of the first-order term among all ` ∞-bounded perturbations ∆ is given by : max ‖∆‖∞≤δ 〈∆ , ∇f ( w ) 〉 = δ ‖∇f ( w ) ‖1 . ( 2 ) To prove this , consider the inner product of ∆ and an arbitrary vector x given by ∑n i=1 nixi . Since |ni| is assumed to be bounded by δ , each nixi is bounded by δ|xi| , which yields the result . The maximum in Equation 2 is obtained indeed by choosing ∆ = δ sign ( ∇f ( w ) ) . Equation 2 comes with a clear hint . We can guarantee that the first-order perturbation term is small if the ` 1-norm of the gradient is small . In this way , the first-order perturbation can be controlled efficiently for various values of δ , i.e . for various quantization bit-widths . In other words , an effective way for controlling the quantization robustness , up to first-order perturbations , is to control the ` 1-norm of the gradient . As we will shortly argue , this approach yields models with the best robustness . This conclusion is based on worst-case analysis since it minimizes the upper bound of the first-order term , which is realized by the worst-case perturbation . Its advantage , however , lies in simultaneous control of the output perturbation for all δs and all input perturbations . In the context of quantization , this implies that the first-order robustness obtained in this way would hold regardless of the adopted quantization bit-width or quantization scheme . The robustness obtained in this way would persist even if the perturbation is bounded in other ` pnorms . This is because the set of ` ∞-bounded perturbations includes all other bounded perturbations , as for all p ∈ [ 1 , ∞ ) , ‖x‖p ≤ δ implies ‖x‖∞ ≤ δ ( see Figure 8 ) . The robustness to ` ∞-norm perturbations is , therefore , the most stringent one among other ` p-norms , because a model should be robust to a broader set of perturbations . Controlling the ` 1-norm of the gradient guarantees robustness to ` ∞-perturbations and thereby to all other ` p-bounded perturbations . In what follows , we propose regularizing the ` 1-norm of the gradient to promote robustness to bounded norm perturbations and in particular bounded ` ∞-norm perturbations . These perturbations arise from quantization of weights and activations of neural networks . 2.2 ROBUSTNESS THROUGH REGULARIZATION OF THE ` 1-NORM OF THE GRADIENT We focused on weight quantization in our discussions so far , but we can equally apply the same arguments for activation quantization . Although the activations are not directly learnable , their quantization acts as an additive ` ∞-bounded perturbation on their outputs . The gradient of these outputs is available . It therefore suffices to accumulate all gradients along the way to form a large vector for regularization . Suppose that the loss function for a deep neural network is given byLCE ( W , Y ; x ) where W denotes the set of all weights , Y denotes the set of outputs of each activation and x the input . We control the ` 1-norm of the gradient by adding the regularization term∑ Wl∈W ‖∇WlLCE ( W , Y ; x ) ‖1 + ∑ yl∈Y ‖∇ylLCE ( W , Y ; x ) ‖1 to the loss , yielding an optimization target L ( W ; x ) = LCE ( W , Y ; x ) + λw ∑ Wl , ∈W ‖∇WlLCE ( W , Y ; x ) ‖1 + λy ∑ yl∈Y ‖∇ylLCE ( W , Y ; x ) ‖1 , ( 3 ) where λw and λy are weighing hyper-parameters . 2.3 ALTERNATIVES TO THE ` 1-REGULARIZATION The equivalence of norms in finite-dimensional normed spaces implies that all norms are within a constant factor of one another . Therefore , one might suggest regularizing any norm to control other norms . Indeed some works attempted to promote robustness to quantization noise by controlling the ` 2-norm of the gradient ( Hoffman et al. , 2019 ) . However , an argument related to the curse of dimensionality can show why this approach will not work . The equivalence of norms for ` 1 and ` 2 in n-dimensional space is stated by the inequality : ‖x‖2 ≤ ‖x‖1 ≤ √ n ‖x‖2 . Although the ` 2-norm bounds the ` 1-norm from above , it is vacuous if it does not scale with 1/ √ n. Imposing such a scaling is demanding when n , which is the number of trainable parameters , is large . Figure 1 shows that there is a large discrepancy between these norms in a conventionally trained network , and therefore small ` 2-norm does not adequately control the ` 1-norm . A very similar argument can be provided from a theoretical perspective ( see the supplementary materials ) . To guarantee robustness , the ` 2-norm of the gradient , therefore , should be pushed as small as Θ ( 1/ √ n ) . We experimentally show in Section 4 that this is a difficult task . We therefore directly control the ` 1-norm in this paper . Note that small ` 1-norm is guaranteed to control the first orderperturbation for all types of quantization noise with bounded support . This includes symmetric and asymmetric quantization schemes . Another concern is related to the consistency of the first-order analysis . We neglected the residual term R2 in the expansion . Figure 3 compares the induced loss after perturbation with its first-order approximation . The approximation shows a strong correlation with the induced loss . We will see in the experiments that the quantization robustness can be boosted by merely controlling the first-order term . Nonetheless , a higher-order perturbation analysis can probably provide better approximations . Consider the second-order perturbation analysis : f ( w + ∆ ) = f ( w ) + 〈∆ , ∇f ( w ) 〉+ 1 2 ∆T∇2f ( w ) ∆ +R3 . Computing the worst-case second-order term for ` ∞-bounded perturbations is hard . Even for convex functions where ∇2f ( w ) is positive semi-definite , the problem of computing worst-case secondorder perturbation is related to the mixed matrix-norm computation , which is known to be NPhard . There is no polynomial-time algorithm that approximates this norm to some fixed relative precision ( Hendrickx & Olshevsky , 2010 ) . For more discussions , see the supplementary materials . It is unclear how this norm should be controlled via regularization .

Summary: the authors propose a regularization scheme that is applied during regular training to ease the pose-training quantization of a network. Modeling the quantization noise as an additive perturbation bounded in \ell_\inf norm, they bound from above the first-order term of the perturbations applied to the network by the \ell_1 norm of the gradients. Their claims are also supported by experiments and qualitative illustrations.