Gradient $\ell_1$ Regularization for Quantization Robustness
Authors: Milad Alizadeh
Arash Behboodi
Mart van Baalen
Christos Louizos
Tijmen Blankevoort
Max Welling
Venue:  (2020)
Keywords: quantization
regularization
robustness
gradient regularization

Abstract: We analyze the effect of quantizing weights and activations of neural networks on their loss and derive a simple regularization scheme that improves robustness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized on-demand to different bit-widths as energy and memory requirements of the application change. Unlike quantization-aware training using the straight-through estimator that only targets a specific bit-width and requires access to training data and pipeline, our regularization-based method paves the way for ``on the fly'' post-training quantization to various bit-widths. We show that by modeling quantization as a $\ell_\infty$-bounded perturbation, the first-order term in the loss expansion can be regularized using the $\ell_1$-norm of gradients. We experimentally validate our method on different vision architectures on CIFAR-10 and ImageNet datasets and show that the regularization of a neural network using our method improves robustness against quantization noise.

Published as a conference paper at ICLR 2020
GRADIENT l REGULARIZATION FOR QUANTIZATION ROBUSTNESS
Milad Alizadeh* 2.1 Arash Behboodi', Mart van Baalen', Christos Louizos', Tijmen Blankevoort', and Max Welling'
Qualcomm AI Research
Qualcomm Technologies Netherlands B.V.
{behboodi, mart, clouizos, t jmen, mwell 1ellinng inniinggeeen /agt qualccuu. com
"University of Oxford
milad, alizadeh@cs oo. ax.ac.uk
ABSTRACT
We analyze the effect of quantizing weights and activations of neural networks on their loss and derive a simple regularization scheme that improves robust- ness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized on- demand to different bit-widths as energy and memory requirements of the ap- plication change. Unlike quantization-ave estimator that only targets a specific bit-width and requires access training data and pipeline, our regularization-base training quantization to various bit-widths. We show that by modeling quantiza- tion as -bounded perturbation, the first-order term in the loss expansion can regularized using the -norr gradients. We experimentally validate ef- fectiveness of our regularization scheme on different architectures CIFAR-10 and ImageNet datasets.
training using the
method paves the way for "on the fly" post-
INTRODUCTION
Deep neural networks excel across variety of tasks, but their size and computational requirements often hinder their real-world deployment. The problem is more challenging for mobile phones, embedded systems, and IoT devices, where there stringent requirements in terms of memory, compute, latency, and energy consumption. Quantization of parameters and activations is often used to reduce the energy and computational requirements of neural networks. Quantized neural networks allow for more speed and energy efficiency compared to floating-pon point models by using ixxe-pooint arithmetic.
However, naive quantization of pre-trained models often results in severe accuracy degradation, especially when targeting bit-widths below eight (Krishnamoorthi 2018). Performant quantized models can be obtained via quantization-awar shadow weights for each weight matrix with backpropagation using the straight-through estimator (STE) (Bengio et al.) 2013), or using other approximations (Louizos et al. 2018). Alternatively, there have been successful attempts recover the lost model accuracy without requiring a training
training or fine-tuning, i.e., learning full-precision
pipeline (Banner et al. 2018 Meller et al. 2019: Choukroun et al. 2019 Zhao et al.) 2019) or representative data (Nagel et al. 2019).
But these methods are not without drawbacks. The shadow weights learned through quantization- aware fine-tuning often do not show robustness when quantized bit-widths other than the one they were trained for (see Table [] In practice, the training procedure has to be repeated for each quantization target. Furthermore, post-training recovery methods require intimate knowledge of the relevant architectures. While this may not be an issue for the developers training the model in the first
Work done during internship at Qualcomm Al Research
Qualcom AI Research is initiative of Qualcomm Technologies, Inc.

---

Published as a conference paper at ICLR 2020
place, it is difficult step for middle parties that are interested in picking up models and deploying them to users down the line, c.g., as part of a mobile app. In such cases, one might be interested in automatically constraining the computational complexity of the network such that it conforms to specific battery consumption requirements, e.g. employ 4-bit variant of the model when the battery is less than 20% but the full precision when the battery is over 80%. Therefore, model that can be quantized to specific bit-width "on the fly" without worrying about quantization aware fine-tuning highly desirable.
In this paper, we explore a novel route, substantially different from the methods described above. We start by investigating the theoretical properties of noise introduced by quantization and analyze it as a lo -ounded perturbation. Using this analysis. we derive a straightforward regularization scheme to control the maximum first-order induced loss and learn networks that are inherently more robust against post-training quantization. We show that applying this regularization at the final stages of training, or as a fine-tuning step after training, improves post-training quantization across different bit-widths at the same time for commonly used neural network architectures.
FIRST-ORDER QUANTIZATION-
MODELS
In this section, we propose a regularization technique for robustness quantization noise. We first propose an appropriate model for quantization noise. Then, we show how we can effectively control the first-order, i.e., the linear part of the output perturbation caused by quantization. When the linear approximation is adequate, approach guarantees the robustness towards various quantization bit-widths simultancously.
We use the following notation throughout the paper. The fy-norm of a vector x in R" is denoted by | and defined ||>e||p = (E [ar/|P) PP)//P for p (o). At its limit we obtain the -nomm defined by ||0||>> == max; [c:] The inner product of two vectors x and is denoted by (æz, y)-
2.1 ROBUSTNESS ANALYSIS UNDER l,- BOUBDEDD ADDITIVE NOISE
The егтоr introduced by rounding in the quantization operation can be modeled as generic additive perturbation. Regardless of which bit-width is used, the quantization perturbation that is added to each value has bounded support, which is determined by the width of the quantization bins. In other words, the quantization noise vector of weights and activations in neural networks has entries that are bounded. Denote the quantization noise vector by A. If 5 is the width of the quantization bin, the vector satisfies
|| S 8/2. Therefore we model the quantization noise as perturbation bounded in the -nom. A model robust to -type perturbations would also be robust to quantization noise.
To characterize the effect of perturbations the output of a function, we look at its tractable ap- proximations, To start, consider the first-order Taylor- expansioonnnnii a real valued- cullued-fnnciion f(w A) around w:
f(w + A) f(w) (A, (((w)) + R2,
(1)
where Ra refers to the higher-order residual error of the expansion. We set Ro aside for the moment and consider the output perturbation appearing in the first-order term (A, Vf(w)). The maximum of the first-order term among all lox bbounded perturbations A is given by:
A V.(w)) = /|l f(w))l
max || Ill
To prove this, consider the inner product of A and an arbitrary vector given by Σn nya, Since |ng| assumed to be bounded by 5, each nyæ; is bounded by d|m,) which yields the result. The maximum in Equation 2 is obtained indeed by choosing A sign (fw)))
Equation comes with a clear hint. We can guarantee that the first-order perturbation term small if the é -norm of the gradient is small. In this way, the first-order perturbation can be controlled efficiently for various values of 8, i.e. for various quantization bit-widths. In other words, an ef- fective way for controlling the quantization robustness, up to first-order perturbations, is to control the é -norm of the gradient. As we will shortly argue, this approach yields models with the best robustness.
2

---

Published as a conference paper at ICLR 2020
Quantization config (6.
Quantization config 5)
10a
Quantization config 4)
2500
fy-norms of gradients
10a
KL Regularized
2000
10-1
1500
10-2
500
Regularized Network
10
10
10-2
10-1
10"
10a
1z-norms of gradients
Baseline
Figure 1: f1- and fz-norms of the gradients for CIFAR-10 test-set mini-batches. Note the differ- ence between the scales on the horizontal and tical axis. We observe that our regularization term decreases the f--norm significantly, compared to its unregularized counterpart.
Figure 2: KL-divergence of the floating point predictive distribution to the predictive distribu- tion the quantized model for CIFAR-10 test- set mini-batches. We observe that the regulariza- tion leads to smaller gap, especially for smaller bit- widths.
This conclusion based on worst-case analysis since it minimizes the upper bound of the first-order term, which is realized by the worst-case perturbation. Its advantage, however, lies in simultancous control of the output perturbation for all 5s and all input perturbations. In the context of quantization, this implies that the first-order robustness obtained in this way would hold regardless of the adopted quantization bit-width or quantization scheme.
The robustness obtained in this way would persist if the perturbation is bounded in other lp- norms. This is because the set of fo -bounded perturbations includes all other bounded perturba- 8 (see Figure g The robustness to
tions, as for all p [1, oc), || ||p < 6 implies ||az|| loo -nom perturbations is, therefore, the most stringent one among other Ep norms, because a model should be robust a broader of perturbations. Controlling the L -noromm of the gradient guarantees robustness to fe perturrbaati and thereby to all other ep-bounded perturbations.
In what follows, we propose regularizing the 4n-mr of the gradient promote robustness to bounded norm perturbations and in particular bounded -nom perturbations. These perturbations arise from quantization weights and activations of neural networks.
2.2 ROBUSTNESS THROUGH REGULARIZATION THE e -NORM OF THE GRADIENT
We focused on weight quantization in our discussions so far, but we can equally apply the same arguments for activation quantization. Although the activations are not directly learnable, their quantization acts as additive lxo -bounded perturbation their outputs. The gradient of these outputs is available. therefore suffices to accumulate gradients along the way to form a large vector for regularization.
Suppose that the loss function for a deep neural network is given by Log (W, Y; x) where W denotes the set of all weights, Y denotes the set of outputs of each activation and the input. We control the fi-norm of the gradient by adding the regularization term
Σ
WVw,LoE(WW) (W, x1)), +
Σ WEY
IV LCE (W))
W
to the loss, yielding optimization target
L(W: = LOE (c Xu Σ
Lww,
Σ W
Y
W.EW
33)
where Au and Ny weighing hyper-parameters.

---

Published as a conference paper at ICLR 2020
0.25
Irrddaeed loee Pistarder predicion
0.20
0.15
0.05
0.00
0.10
Number
Figure 3: Predicting induced using first-order terms. We added foo -oundedd noise with 6 correspond- ing to 4-bit quantization to all weights of ResNet-18 18 and compared the induced loss on the CIFAR-10 test-set with the predictions using gradients. While not perfect, the first-order term is not insignificant.
2.3 ALTERNATIVES TO THE L <E-REGULARIZATION
The equivalence of norms in finite-dimensional normed spaces implies that all norms are within constant factor of one another. Therefore, one might suggest regularizing any norm control other norms. Indeed some works attempted to promote robustness to quantization noise by controlling the fo-norm the gradient (Hoffman et al. 2019). However, an argument related the curse of dimensionality can show why this approach will not work. The equivalence of norms for & and & n-dimensional space stated by the inequality:
|||||| S |<=||| S v ||2|||-
Although the f2-ormm bounds the -nomm from above, it is vacuous if does not scale with 1/vT. Imposing such scaling is demanding when which is the number of trainable parameters, is large. Figure shows that there large discrepancy between these norms trained network, and therefore small & norm does not adequately control the fornm. A very similar argument can be provided from a theoretical perspective (see the supplementary materials).
conventionally
To guarantee robustness, the fo-norm of the gradient, therefore, should be pushed small as (l//vn). We experimentally show in Section that this is a difficult task. We therefore directly control the f--nrrm in this paper. Note that small f-norm is guaranteed control the first order- perturbation for all types of quantization noise with bounded support. This includes symmetric and asymmetric quantization schemes.
Another concern is related to the consistency of the first-order analysis. We neglected the residual term R2 in the expansion. Figure 3 compares the induced loss after perturbation with its first-order approximation. The approximation shows a strong correlation with the induced loss. We will see in the experiments that the quantization robustness can boosted by merely controlling the first-order term. Nonetheless, a higher-order perturbation analysis can probably provide better approximations. Consider the second-order perturbation analysis:
f(w + A) = f(w) + A f(ww)) + " A +
Computing the worst-case second-order term for -buunded perturbations is hard. Even for convex functions where f²f(w) ) is positive semi-definite, the problem of computing worst-case second- order perturbation is related to the mixed matrix-norm computation, which is known be NP- hard. There is no polynomial-time algorithm that approximates this norm to some fixed relative precision (Hendrickx & Olshevsky 2010). For more discussions, see the supplementary materials. It is unclear how this norm should be controlled via regularization.
3 RELATED WORK
closely related line of work to ours is the analysis the robustness of the predictions made by neural networks subject adversarial perturbation in their input. Quantization be similar scenario where non-adversarial perturbations are applied to weights and activations instead. Cisse al. (2017) proposed method for reducing the network's sensitivity small perturbations

---

Published as a conference paper at ICLR 2020
by carefully controlling its global Lipschitz. The Lipschitz constant of a linear layer is equal to the spectral norm of its weight matrix, i.e., its largest singular value. The authors proposed regularizing weight matrices in each layer to be close to orthogonal: Ew,ew [W?':; W -1|22. All singular values of orthogonal matrices are one; therefore, the operator does not amplify perturbation (and input) in any direction. [Lin et al. (2019) studied the effect of this regularization in the context of quantized networks. The authors demonstrate the extra vulnerability of quantized models to adversarial attacks and show how this regularization, dubbed "Defensive Quantization", improves the robustness of quantized networks. While the focus of Lin et al.] (2019) is on improving the adversarial robustness, the authors report limited results showing accuracy improvements of post- training quantization.
The idea of regularizing the norm of the gradients has been proposed before (Gulrajani et al. 2017) in the context of GANs, as another way to enforce Lipschitz continuity. A differentiable function is I-Lipschitz if and only if it has gradients with 42 æ2-norm of most everywhere. hence the authors penalize the f--nrrm of the gradient of the critic with respect to its input. This approach has major advantage over the methods mentioned above. Using weight regularization is only well-defined for 2D weight matrices such as in fully-connected layers. The penalty term is often approximated for convolutional layers by reshaping the weight kernels into 2D matrices. Sedghi et al. (2018) showed that the singular values found in this weight could be very different from the actual operator norm of the convolution. Some operators, such as nonlinearities, are also ignored. Regularizing Lipschitz constant through gradients does not suffer from these shortcomings, and the operator- norm is reg- ularized directly. Guo et al. (2018) demonstrated that there exists an intrinsic relationship between sparsity in DNNs and their robustness against for and f2 attacks. For binary linear classifier, the authors showed that they could control the Ex robustness, and its relationship with sparsity, by reg- ularizing the L norm of the weight tensors. In the case of a linear classifier, this objective is, in fact, equivalent to our proposed regularization penalty.
Finally, another line of work related to ours revolves around quantization-aware training. This can, in general, be realized in two ways: 1) regularization and 2) mimicking the quantization procedure during the forward pass of the model. In the first case, we have methods (Yin al. 2018 Achter- hold et al. 2018) where there are auxiliary terms introduced in the objective function such that the optimized weights are encouraged to near, under some metric, to the quantization grid points, thus alleviating quantization noise. In the second case, we have methods that rely on either the STE (Courbariaux et al. 2015; Rastegari et al. 2016 Jacob et al. 2018), stochastic rounding (Gupta) surrogate objectives and gradients (Louizos al. 2018 Shayer al. 2017). While all of the methods above have been effective, they still suffer from a major limitation; they target one-specific bit- width. In this way, they not appropriate for use-cases where want to be able to choose the bit-width "on the fly".
cet al. 2015]Gyysll
4 EXPERIMENTS
In this section we experimentally validate the effectiveness of our regularization method on im- proving post-training quantization. We use the well-known classification tasks of CIFAR-10 with ResNet-18 18 (He et al. (016) and VGG-like (Simonyan & Zisserman 2014) and of ImageNet with
ResNet- We compare our results for various bit-widths against (1) unregularized baseline
networks (2) Lipschitz regularization methods (Lin et al.] 2019; Gulrajani et al.] 2017) and (3) quantization-awan -ware fine-tuned models. Note that Gulrajani et al. (2017) control the Lipschitz con- stant under an E2 metric by explicitly regularizing the f2-norm of the gradient, while[Lin et al. (2019) essentially control an upper bound on the f--norm of the gradient. Comparing against these base- lines thus gives insight into how our method of regularizing the fn-norm of the gradient compares against regularization of the l fg-norm of the gradient.
4.1 EXPERIMENTAL SETUP
Implementation and complexity Adding the regularization penalty from Equation 3 to the train- ing objective requires higher-order gradients. This feature is available in the latest versions of frame- works such Tensorflow and PyTorch (of which we have used the latter for all our experiments). Computing V V V Vll1 using automatic differentiation requires O x C' x E) extra computations, where F. is the number of elementary operations in the original forward computation graph, and C
5

---

Published as a conference paper at ICLR 2020
1.0
0.9
0.8
0.8
Accuracy
0.7
Accuracy
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
Baseline Regularized
Baseline
0.2
0.2
Regularized
0.1
0.1
(8,FP)
(6.FP)
(S.FP)
(4.FP)
(8.4)
(7,4)
(6,4)
(5,4)
(4,4)
Quantization Config
Quantization Config
(a)
(b)
Figure 4: Accuracy regularized VGG-like after post-training quantization. We trained models with different initializations and show the mean accuracy for each quantization configuration. The error hars indicate min/max observed accuracies. (a) Weight-only quantization (b) Activation quantization fixed to 4-bits
is fixed constant (Baydin et al. 2018 This can be seen from the fact that C| is a function R/wl - R. where w denotes the number of weights and the computation of the gradient w.r.t. the loss contains elementary operations, as many as the forward pass. In practice, enabling regular- ization increased time-per-epoch time CIFARIO from 14 seconds to 1:19 minutes for VGG, and from 24 seconds to 3:29 minutes for ResNet-18. On ImageNet epoch-time time increased from 33:20 minutes to 4:45 hours for ResNet-18. 18. The training performed on single NVIDIA RTX 2080 Ti GPU.
However, in experiments observed that it is necessary enable regularization from the beginning, as the fı-norm of the gradients decreases naturally up to a certain point as the training progresses (See Appendíx [or more details). We therefore only enable regularization in the last 15 epochs of training or as an additional fine-tuning phase. We experimented with tuning A and Xy in Equation separately but found no benefit. We therefore set Av X for the remainder of this section.
We use grid-search to find the best setting for A. Our search criteria ensuring that the perfor- mance of the unquantized model is not degraded. In order to choose a sensible range of values we first track the regularization and cross-entropy loss terms and then choose range of that ensures their ratios are in the same order of magnitude. We do not perform any quantization for validation purposes during the training.
Quantization details We uniform symmetric quantization (Jacob et al. 2018 Krishnamoorthi 2018) in all our experiments unless explicitly specified otherwise. For the CIFAR 10 experiments fix the activation bit-widths 4 bits and then vary the weight bits from 8 to 4. For the Imagenet experiments we use the same bit-width for both weights and activations. For the quantization-awarc fine-tuning experiments we employ the STE experiments we perform a hyperparameter search over learning rates for each of the quantization bit-widths and use fixed weight decay of 1e 4. For our experiments with defensive quantization (Lin et al. 2019) we perform hyperparameter search over the scaling parameters of the regularizer and the learning rate. We limit the search over the scaling parameters to those mentioned (Lin) al.) 2019) and do not weight decay. When applying post-training quantization activation ranges using the batch normalization parameters as described in (Nagel al. 2019).
fixed (symmetric) quantization grid. In all these
set the
When model is fine-tuned target bit-width and evaluated on a higher bit-width, can trivially represent the original quantized weights and activations by ignoring the higher-order bits, or quantize using the higher bit-wi.th. As using the higher bit-width quantize shadow weights and activations introduces noise to the model and might yield lower results, we try both approaches and only report result if quantization using the higher bit-width gives better results.

---

Published as a conference paper at ICLR 2020
Reguneene 20 anreect
Figure 5: Random cross sections of decision boundaries in the input space. To generate these cross- sections, we draw a random example from the CIFAR-10 test set (represented by the black dot in the center) and pass random two-dimensional hyper-plane C R through it. We then evaluate the network's output for each point
hyper-ppaae planee. Various colors indicate different classes. Softmax's 's maximum values determine the contours. The top row illustrates the difference between the baseline and the regularized VGG-like networks (and their quantized variants) when they all classify example correctly. The bottom row depicts case where the quantized baseline misclassifies example while the regularized network predicts the correct class. We see that regularization pushes the decision boundaries outwards and enlarges the decision cells.
4.2 EFFECTS OF REGULARIZATION
In order get better understanding of proposed regularizer, first adopt the visualization method from Hoffman et al. (2019) and illustrate the effects that the quantization in general, and our method in particular, have on the trained classifier's decision boundaries. The result can be seen in Figure 5 where we empirically observe that the regularized networks "expands" its decision cells.
Secondly, we investigate in Figure [ the f and lz- fz-norms of the gradients for CIFAR-10 test batches on the VGG-like model. We can observe that while the fz-norms of the gradient are small the unregularized model, the -norms are orders magnitude larger. Consequently, when fine- tuning the same model with our method, we see strong decrease of the f--orm.
Finally, we investigate how the predictive distribution the floating point model, p(y[x), changes when quantize either unregularized baseline model regularized with our method, thus obtaining {)y[x). x). We measure this discrepancy using the KL-divergence when using the predictive distribution of the quantized model, i.e. D (p(y)=>))l9(())) over each test batch. Since our method improves robustness of the loss gradient against small per- turbations, we would expect the per-class probabilities to be more robust to perturbations well, and thus more stable under quantization noise. The result can be seen in Figure 2 where we indeed observe that the gap is smaller when quantizing regularized model.
the original predictive
averaged
4.3 CIFAR-10 & IMAGENET RESULTS
The classification results from our CIFAR-10 10 experiments for the VGG-like and ResNet18 networks are presented in Table [ whereas the result from our Imagenet experiments for the ResNet18 net- work can be found in Table 2 Both tables include all results relevant to the experiment, including results on our method, Defensive Quantization regularization, L2 gradient regularization and fine- tuning using the STE.
Comparison to "Defensive Quantization" As explained in Section 3 Defensive Quantization (Lin et al. 2019) aims to regularize each layer's Lipschitz constant to be close to 1. Since the

---

Published as a conference paper at ICLR 2020
VGG-like
ResNet-18
FP
(8,4)
(6,4)
(4,4)
FP
(8,4)
(6,4)
(4,4)
No Regularization
92.49
79.10
78.84
11.47
93.54
85.51
85.35
83,98
DQ Regularization
91.51
86.30
84.29
30.86
92.46
83.31
83.34
82.47
L2 Regularization
91.88
86.64
86.14
63.93
93.31
84.50
84.99
83.82
L1 Regularization (Ours)
92.63
89.74
89.78
85.99
93.36
88.70
88.45
87.62
STE @ (8,4)
91.28
89.99
32.83
89.10
87.79
86.21
STE @ (6,4)
90.25
39.56
90.77
88.17
STE (4,4)
89.79
89.98
Table I: Test accuracy (%) for the VGG-like and ResNet-18 models on CIFAR-10. STE @ (X.X) indicates the weight-activation quantization configuration used with STE for fine-tuning. DQ denotes Defensive Quantization (Lin et al. 2019). For the No Regularization row of results we only report the mean of 5 runs. The full range of the runs is shown in Figure 4
Configuration
(8,8)
(6.6)
(4.4)
No Regularization
69.70
69.20
63.80
0.30
DQ Regularization
68.28
67.76
62.31
0.24
L2 Regularization
68.34
68.02
64.52
0.19
L1 Regularization (Ours)
70.07
69.92
66.39
0.22
L1 Regularization (Ours) (A 0.05)
64.02
63.76
61.19
55.32
STE @ (8,8)
70.06
60.18
0.13
STE @ (6,6)
69.63
11.34
STE (4,4)
$7.50
Table 2: Test accuracy for the ResNet-18 architecture on ImageNet. STE @ (X.X) indicates the weight-activation quantization configuration used with STE for fine-tuning. In addition to the À found through the grid-search which maintains FP accuracy, we also experimented with a stronger 0.05 show that (4,4) accuracy can be recovered the price of overall lower performance.
regularization approach taken by the authors is similar to our method, and the authors suggest that their method be applied regularization for quantization robustness, we compare their method to ours. As the experiments from the original paper differ methodologically from ours in that we quantize both weights and activations, all results on defensive quantization reported in this paper are produced by us. We were able to show improved quantization results using defensive quantization for CIFAR-10 on VGG-like, but not on any of the experiments on ResNet18. We attribute this behavior to too stringent regularization in their approach: the authors regularize all singular values of their (reshaped) convolutional weight tensors to be close to one, using a regularization term that is essentially fourth power regularization of the singular values of the weight tensors (see Appendix C). This regularization likely inhibits optimization.
Comparison to explicit ε--norm gradient regularization We consider the & regularization of gradient, as proposed by Guirajani et al. (2017). as a generalization of the DQ regularization. Such regularization has two key benefits over DQ: 1) we can regularize the singular values without reshaping the convolutional kernels and 2) we impose a less stringent constraint as we avoid enfore- ing all singular values to be close to one. By observing the results at Table andd] 2 we see that the l regularization indeed improves upon DQ. Nevertheless, it provides worse results compared to our l regularization, an effect
explain by the analysis of Section 2
Comparison to quantization-aware fine-tuning While in general we cannot expect our method to outperform models to which quantization-awan fine-tuning is applied on their target bit- bit-widths, as this case the model can adapt to that specific quantization noise, we do see that our model performs par better when comparing to bit-widths lower than the target bit-width. This is in line with our expectations: the quantization -ware fine-tuned models are only trained to be robust to specific noise distribution. However, method ensures first-order robustness regardless of bit-
8

---

Published as a conference paper at ICLR 2020
width quantization scheme, explained in Section 2 ImageNet. We hypothesize that this is caused by the fact that we tune the regularization strength À to the highest value that does not hurt full-precision results. While stronger regularization would harm full-pre full-precision performance, would also most likely boost bit results, due to imposing robustness larger magnitude, i.e. 6, of quantization noise. Table T includes results for higher value of that is in line with this analysis.
The only exception is the 4 bit results
CONCLUSION
In this work, analyzed the effects of the quantization noise the loss function of neural net- works. By modelling quantization as an lo -bounded perturbation, we showed how we can con- trol the first-order term of the Taylor expansion of the loss by straightforward regularizer that encourages the l norm of the gradients to be small. We empirically confirmed its effectiveness, demonstrating that standard post-training quantization to such regularized networks can maintain good performance under a variety of settings for the bit-width of the weights and activations. As result, our method paves the way towards quantizing floating-poin models "on the fly" according to bit-widths that are appropriate for the resources currently available.
ACKNOWLEDGMENTS
We would like thank Markus Nagel, Rana Ali Amjad, Matthias Reisser, and Jakub Tomezak for their helpful discussions and valuable feedback.
REFERENCES
Jan Achterhold, Jan Mathias Koehler, Anke Schmeink, and Tim Genewein. Variational network quantization. International Conference Learning Representations, 2018.
R Banner, Y Nahshan, E Hoffer, and D Soudry. Post training 4-bit quantization of convolution networks for rapid-deployment. CoRR, abs/1810.05723, 1:2, 2018.
Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of machine learning research, 18( (8(153), 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1r.Xiv1308.... 3432, 2013.
Vijay Bhattiprolu, Mrinalkanti Ghosh, Venkatesan Guruswami, Euiwoong Lee, and Madhur Tul- siani. Inapproximability of matrix norms. arXiv preprint arXiv:1802.07422 2018.
Yoni Choukroun, Eli Kravchik, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. arXiv preprint arXiv: 192200002222 2019.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. In Proceedings of the 34th International Conference on Machine Hourning-Volume 70, pp. 854-863. JMLR. org, 2017.
Matthieu Courbariaux, Yoshua Bengio, and lierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pp. 3123-3131, 2015.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, pp- 5767-5777, 2017.
Yiwen Guo, Chao Zhang, Changshui Zhang, and Yurong Chen. Sparse dnns with improved adver- sarial robustness. In Advances in neural information processing systems, pp. 242-251, 2018.

---

Published as a conference paper at ICLR 2020
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning, pp. 1737-1746, 2015.
Philipp Gysel. Ristretto: Hardware-oriented approximation of convolutional neural networks. arXiv preprint ar0iv: 06402, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Julien M. Hendrickx and Alex Olshevsky. Matrix p-Norms Are NP-Hard to Approximate If p + 1,2, oc. SIAM Journal Matrix Analysis and Applications, 3 31(5):2802-2122 2010.
Wassily Hoeffding. Probability Inequalities for Sums of Bounded Random Variables. Journal of the American Statistical Association, 58(301):13-30, March 1963.
Judy Hoffman, Daniel A. Roberts, and Sho Yaida. Robust Learning with Jacobian Regularization. /tt///988002222 [es, stat], August 2019. arXiv: 1908.02729
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalemichenko. Quantization and training of neural networks for effi- cient aateger-arithmeticoone inference. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.088 2018.
Lin, Chuang Gan, and Song Han. Defensive quantization: When efficiency meets robustness. arXiv preprint arXiv: 199040844444, 2019.
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Re- laxed quantization for discretized neural networks. arXiv preprint arXiv:8810001875. 2018.
Eldad Meller, Alexander Finkelstein, Uri Almog, and Mark Grobman. Same, same but different recovering neural network quantization error through weight factorization. In International Conference on Machine Learning. 2019.
(et.: 04721, 2019. Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. arXiv preprint arXiv:1906.04
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pp. 525-542. Springer, 2016.
Hanie Sedghi, Vineet Gupta, and Philip M Long. The singular values of convolutional layers. arXiv preprint arXiv: 2018.
Oran Shayer, Dan Levi, and Ethan Fetaya. Learning discrete weights using the local reparameteri- zation trick. arXiv preprint arXiv: 1710.0779,, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Penghang Yin, Shuai Zhang, Jiancheng Lyu, Stanley Osher, Yingyong Qi, and Jack Xin. Bina- ryrelax: A relaxation approach for training deep neural networks with quantized weights. SIAM Journal on Imaging Sciences, 11(4):2055-2225 2018.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. In International Conference on Machine Learning. pp. 7543-7552, 2019.
10

---

Published as a conference paper at ICLR 2020
Engli pagirccal cuandration parturaation diosbutton
18
02
Figure 6: Quantization noise uniformly distributed. In this plot show the quantization noise on each individual weight an ImageNet trained ResNet18 model. The noise is scaled by the width of the quantization bin for cach weight quantizer. This plot shows that quantization noise is uniformly distributed between -5/2 and 6/2.
A ROBUSTNESS ANALYSIS FOR QUANTIZATION PERTURBATIONS
In this section, we address two questions in more details, first regarding regularization of the 2n-norm of gradient and second regarding non-unin non-uniform quantization schemes.
We argued above that regularizing the f2-norm of gradient cannot achieve the same level of bustness as regularization of the & f--norm of gradient. We provide here another, more theoretical, argument. The following inequality shows how the fn-norm of gradient controls the first-order per- turbation:
(A, Vs(w)) S | (
This is a simple Cauchy-Shward inequality. Therefore, if the f2-norm of the gradient is inversely proportional the power of the perturbation, the first-order term is adequately controlled. However, using a theoretical argument, we show that the power of the f perturbation can blow up with the dimension as a vector A in E. with A /no. In other words, the length of the quantization noise behaves with high probability as @(vn), which implies that the the fo-m of the gradient should be as small as ( /n).
can reach an 42 f2-norm of approximately
We show that this can indeed occur with high probability for any random quantization noise with the bounded support. Note that for symmetric uniform quantization schemes, quantization noise can approximated well by uniform distribution over [-6/2, 6,/2,,6/ 6/2] where 8 is the width of the quantization bin. See Figures[oJfor the empirical distribution of quantization noise on the weights of a trained network. Our argument, however, works for any distribution supported over 15/2,6, 8/2), and, therefore, it includes asymmetric quantization schemes over uniform quantization bin.
Consider a vector æ (#I,..., FT,) T R." with entries D; randomly and independently drawn from a distribution supported on [-8/2, 8/2]. We would like to show that ||ж|| is well concentrated going write down the above norm as the sum of
around its expected values. To do that independent zero-mean random variables. See that:
nd
|| = E Σ
x
nl (x+)
12
Besides. note that x. E [0, 82 /4]. Therefore 2f 8 122 is zero-mean random variable that lies in the interval [-8 12, 82 /6). We can now use Hoeffding's inequality. To be self-contained, we include the theorem below.
Theorem A.1 (Hoeffding's inequality, (Hoefiding 1963)). Let XI...., Xn be a sequence of in- dependent zero-mean random variables such that X, almost surely supported on (ai, by) for
11

---

Published as a conference paper at ICLR 2020
i EE (I....,n.. Then, for all O, it holds that
2t
P
X;
26
S eexp
Li=l (4 a;)²
t²
P
X
28
S 2xx
(5)
E (b: a;)²
Applying Theorem A.1 to our setting, we obtain:
PP ( | no² n8² /12 2 - exx
212
n(00 n((2//423 /4)2
So with probability 1 1 we have:
no
1/2
||01| - no² n <
log (2/e)
32
Therefore, if the quantization noise A has entries randomly drawn from a distribution over [-6/2, 6/2]. then with probability e, the squared fg-norm of A, i... | 1 lies in the interval no2 nd* log(2/e), nd nd* log(2/e) In other words, the length of the vector behaves
with high probability (vr).. This result holds for any quantization noise with bounded support.
the quantization bins are non-uniformly chosen, and if the weights can take arbitrarily large val- ues, the quantization noise is no-longer bounded in general. As long the quantization noise has a Gaussian tail, i.e., it is a subgaussian random variable, one can use Hoeffding's inequality for subgaussian random variables to show similar concentration result above. The power of the perturbation will, therefore, behave with (v) and the f--nrrm of the gradient cannot effectively control the gradient. Note that nonuniform quantization schemes ware implementations, hence, our focus on uniform cases. Besides, the validity of this assumption about nonuniform quantization noise requires further investigation, which relegated our future works.
commonly used for hard-
B SECOND-ORDER PERTURBATION ANALYSIS
We start by writing the approximation of f(-) up to the second-order term: - ATv + R3-
f(w + A) f(w) (A, ff(w))
The worst-case second-order term under &go -bounded perturbations is given by
max (ww)
|| $6
The above value is difficult quantify for general case. We demonstrate this difficulty by consider- ing some special cases.
Let's start with convex functions, for which the Hessian (2 is positive semi-definite. In this case, the Hessian matrix admits a square root, and the second-order term can be written as:
2 f(w) = (")""
| ((w))
Therefore the worst-case analysis of the second-term amounts to
(Vw) A 2
max f ||
max ||]l 58
55
The last term is the mixed oc + 2-norm of (V2 As a reminder, the p + q- q-matrix norm defined as
Alll-= == max |
= max sell
(y. =: || || 1 -pp
11
12

---

Published as a conference paper at ICLR 2020
where q denote the dual norms of p and q, i.e. satisfying 1/p 1/p" 1/q 1/q" = =.
The worst case second-order perturbation is given by:
A f(m)) = 8² (V
max T t0
Unfortunately the 00 + 2-norm is known to be NP-hard (Hendrickx & Olshevsky 2010); see Bhattiprolu al. (2018) for more recent study). As a matter of fact, if f()) positive semi- defimite, and hence the function is convex, the problem above corresponds to maximization of convex functions, which is difficult as well.
For general Hessian, the problem is still difficult to solve. First note that:
2 f'w) = max Tr (V² ||m 26
max Fx >
We therefore replace A with positive semi-defintite matrix of rank denoted by N. The worst case second-order perturbation can be obtained by solving the following problem:
max Tr (V2f(w)N N NER"x xm
(6)
subject to N =
Nu S 8² for E (1,......)
Bank(N)=11
The last constraint, namely the rank constraint, is discrete constraint. The optimization problem above is therefore NP-hard to solve. To sum up, the worst case second-order perturbation cannot be efficiently computed, which poses difficulty for controlling the second-order robustness.
There are, however, approximations available in the literature. A common approximation, which is widely known for the Max-Cut and community detection problems, consists of dropping the rank- constraint from the above optimization problem to get the following semi-definite program:
Tr (V f(w)
max N NEReX subject to N 2 0
Nu 8 for i {I....,
Unfortunately this approximation, apart from being costly to solve for large n, does not provide regularization parameter that be included in the training of the model.
is not clear how we can control the second-order term through a tractable term.
C DEFENSIVE QUANTIZATION IMPOSES A 4TH POWER CONSTRAINT ON SINGULAR VALUES
From basic linear algebra we have that
|W Tr(WTW) = o
ie., the Frobenius norm is equal to sum of the squared singular values of W. From this we can conclude that the regularization term W w Ill introduced by Lin et 2019) thus equals
W W 7 Σ - Σ 03 (W) 1/
and therefore imposes 4th power regularization term on the singular values of W. softer regu- larization be introduced by regularizing Tr(W" W -T) instead.
13

---

Published as a conference paper at ICLR 2020
D GRADIENT-PENALIY PROGRESSION IN NON-REGULARIZED NETWORKS
Optimizing our regularization penalty requires computing gradients of the gradients. While this easily done by bouble-backpropagation modern software frameworks it introduces overhead (as discussed in Section 4.1] and makes training slower. However, as the training progresses, the gradients unregularized networks tend to become smaller as well, which is inline with our reg- ularization objective. It is therefore not necessary to apply the regularization from the beginning of training. Figure / we show examples of how the regularization objective naturally decreases during training. We also show how turning the regularization on in the final epochs where the regu- larization objective is oscillating can push the loss further down towards zero.
Reg Regudarization applled
Regallrtission applied step
Ondlent L1 Peraby
1600
T0X
20K
23K
30K
3AK
Towniing Slep
Triining Blep
(a) VGG-like
Reg appiled
Regularzston applied step
1200
1200
Drddent Penaty
1000
Punalyy
1000
000
800
600
800
400
400
19K
29K
30K
Training Stop
Training
(b) ResNet-18
Figure 7: The gradients in unregularized networks tend become smaller training progresses. This means large parts of the training there no need apply the regularization. The plots on the left show regularization penalty in unregularized networks. The plots on the right show how turning on the regularization in the last 15 epochs of the training can push the regularization loss even further down.
E loc BBUUDDDD PERTURBATIONS INCLUDE OTHER BOUNDED-NORM PERTURBATIONS
Figure 3 show that the bounded perturbations include all other bounded-nome perturbations.
14

---

Published as a conference paper at ICLR 2020
Loc
l2
l1
Figure 8: -bounded vectors include other bounded- norm vectors. In this plot we show that the pertur-
bations with bounded -nomm are subset of &o bouunded perturbations. For 1, 2, oc, we plot the vectors with ||l|| 1.
15


Reviews:
[{"review_id": "r1gvs0imcr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper shows that if we add L1 regularization on gradient in the training phase, the obtained model can achieve better post-training quantization performance since it is more robust to Linf perturbation on the weights. I like the intuition of the paper but there are several weaknesses: \n\n1. The main concern is that the proposed method cannot outperform quantization-aware fine-tuning. This probably limits the application of the method --- it will only be used when there's not enough time budget for quantization-aware fine tuning for each specific choice of #bits. It will be good if the authors can discuss in what practical scenario their algorithm can be applied. \n\n2. The method is only tested under uniform symmetric quantization. I believe to demonstrate that the L1 regularized models are indeed easier to be quantized, we need to test it on several different kinds of quantizations. \n\n3. I have concerns about the hyper-parameter selection for lambda. The authors mentioned that lambda is chosen by grid-search, but what's the grid search criteria? In other words, are the hyper-parameters trying to minimize the validation error of the \"unquantized model\", or they are minimizing the validation error of the \"post-quantized model\"? \n\n4. Some minor suggestions: \n\n- The current paper uses boldfaced n as perturbation which is quite confusing (since small n is the dimension). I would suggest to replace it by something else, e.g, \\Delta. \n\n- Section 2.3 seems redundant. It's clearly that L1 regularization is better given it's the dual norm of Linf, so clearly it's better than L2 norm. You have proved L2 is not good anyway in experiments. \n\n===========\n\nAfter seeing the rebuttal, my concerns about the parameters have been well addressed. Also, I agree with the authors that there are use cases for post quantization, and personally I think post quantization is much easier to do in practice than quantization-aware training. However, this is quite subjective so the fact that the proposed method doesn't outperform quantization-aware training is still a weakness of the paper. \n\nI would like to slightly raise the score to borderline/weak-accept. I hope the authors can have some experiments on non-uniform quantization if the paper is being accepted; I really think that will demonstrate the strength of the method. People will likely to use this method if it can consistently improve many different kinds of post quantization. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1eFtky0tr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper models the quantization errors of weights and activations as additive l_inf bounded perturbations and uses first-order approximation of loss function to derive a gradient norm penalty regularization that encourage the network's robustness to any bit-width quantization. The authors claim that this method is better than previous quantization-aware methods because those methods are dedicated to one specific quantization configuration.\n\nThe derivation of the proposed method is not complex but I like the idea that models quantization error as additive perturbation in this context and how it eventually connects with gradient penalty that's widely used in GAN training and adversarial robustness.\n\nQuestions:\n\n1. What is the capital N in the time complexity of gradient computation in Sec. 4.1? The authors should discuss in details the time complexity of the proposed regularization well because this is an essential problem of the regularization, which involves double back-propagation and should be computationally heavy. For the same reason, I'd like to see the training time comparison, and more results with deeper networks.\n\n2. Compared to STE, one of the quantization-aware methods, the proposed method is not very competitive even in the setting when a STE network, which is specially trained for 6,6 bits but quantized to 4,4 bits, can outperforms the proposed method. This contradicts with the claimed strength of the proposed method. Will it be better when we regularize more, if we want the model to perform well when quantized to 4,4 bits? It would be better if there is a set of experiments of different regularization hyperparameters.\n\n***********************\n\nUpdate: I'd like to keep my score after reading the authors' response to all reviewers. I think the authors do address some questions but the paper still has some weakness in terms of performance.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxmIYPEYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: the authors propose a regularization scheme that is applied during regular training to ease the pose-training quantization of a network. Modeling the quantization noise as an additive perturbation bounded in \\ell_\\inf norm, they bound from above the first-order term of the perturbations applied to the network by the \\ell_1 norm of the gradients. Their claims are also supported by experiments and qualitative illustrations.\n\nStrengths of the paper:\n- The paper is clearly written and easy to follow. In particular, section 2.1 clearly motivates the formulation of the regularization term from a theoretical point of view (reminiscent of the formulation of adversarial examples) and Figures 1 and 2 motivate the regularization term from a practical point of view. I found Figure 5 particularly enlightening (the regularization term \"expands\" the decision cells). \n- The method is clearly positioned with respect to previous work (in particular using \\ell_2 regularization of the gradients) \n- Experiments demonstrate the effectiveness fo the method. \n\nWeaknesses of the paper:\n- The link between the proposed objective and the sparsity could be made clearer: does this objective enforce sparsity of the gradients, the weights, and how does this affect training?\n\n\nJustification of rating:\nThe paper clearly presents a regularization method to improve post-training quantization. The approach is motivated both from a theoretical point of view and from a practical point of view. The latter aspect is of particular interest for the community. The claims are validated by a limited set of experiments that are seem nonetheless well executed.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers uniformly suggest acceptance. Please take their comments into account in the camera-ready. Congratulations!
Decision: Accept (Poster)