Differentiable Model Compression via Pseudo Quantization Noise
Authors: Alexandre Défossez
Yossi Adi
Gabriel Synnaeve
Venue: Accepted by TMLR ()

Abstract: We propose DiffQ a differentiable method for model compression for quantizing model parameters without gradient approximations (e.g., Straight Through Estimator). We suggest adding independent pseudo quantization noise to model parameters during training to approximate the effect of a quantization operator. DiffQ is differentiable both with respect to the unquantized weights and the number of bits used. Given a single hyper-parameter balancing between the quantized model size and accuracy, DiffQ optimizes the number of bits used per individual weight or groups of weights, in end-to-end training. We experimentally verify that our method is competitive with STE based quantization techniques on several benchmarks and architectures for image classification, language modeling, and audio source separation. For instance, on the ImageNet dataset, DiffQ compresses a 12 layers transformer-based model by more than a factor of 8, (lower than 4 bits precision per weight on average), with a loss of 0.3\% in model accuracy. Code is available at github.com/facebookresearch/diffq

Published in Transactions on Machine Learning Research (09/2022)
Differentiable Model Compression via Pseudo Quantization Noise
Alexandre Défosser* Meta AT, PATR Team, Paris, France
defossea@ft.com com
Yossi Adi
adiyoss@ib..cmm
Meta AI, FAIR Team, Tel-Aviu, Israel
Gabriel Synnaeve
gab gab@ff.co com
Meta AT, FATR Team, Poris, France
Reviewed on OpenReview: https: / openneveee............m.m net/ forum? id= D6 jnKai che
Abstract
We propose DIFEQ differentiable method for model compression for quantizing model parameters without gradient approximations (e.g., Straight Through Estimator). We suggest adding independent pseudo quantization noise to model parameters during training to approximate the effect of a quantization operator. DIFFQ is differentiable both with respect to the unquantized weights and the number of bits used. Given a single hyper-parameter balancing between the quantized model size and accuracy, DIFFQ optimizes the number bits used per individual weight or groups of weights, in end-to- tnd training. We experimentally verify that our method is competitive with STE based quantization techniques on several benchmarks and architectures for image classification, language modeling, and audio source separation. For instance, on the ImageNet dataset, DIFFQ compresses 12 layers transformer- based model by more than a factor of 8, (lower than 4 bits precision per weight on average), with a loss of 0.3% in model accuracy. Code is available at github.com/
ggttub.comm//ccebookeeeen www.eeebooooeeerrcccc/oo/oooaae /diffq
1 Introduction
An important factor in the adoption of a deep learning model for real-world applications how easily it can be pushed to remote devices. It has been observed that larger models usually lead better performance, for instance with larger ResNets (He et al., 2016) achieving higher accuracies than smaller ones. In response, the community has worked toward smaller, and more efficient models (Tan & Lo. 2019). Yet an EfficientNet-B3 is still almost. 50MB, considerable amount if the model is to be included in online applications, or updated with limited network capabilities. For other applications, such as language modeling (Vaswani et al. 2017 or source separation (Défossez et al., 2211)), the typical model size is closer to 1GB, ruling out any kind of mobile usage. Efficient model compression is thus important for on device adoption of deep learning models. Thus, focus in the present work reducing model size, rather than achieving computational gains.
The simplest method to reduce model size consists in decreasing the number of bits used to encode individual weights. For instance, using 16 bits floating point numbers halves the model size, while retaining a sufficient approximation of the set of real numbers, IR, to train with first-order optimization methods (Micikevicius ct al. 2018). When considcring lower procision, for instance, 8 or 4 bits, the set of possible values is no longer a good approximation of IR, hence preventing the use of first-order optimization methods. Specifically, uniform quantization requires using the round function, which has zero gradients wherever is differentiable.
Quantization can be done post-processing step to regular training. However, errors accumulate in multi- plicative fashion across layers, with a possibly uncontrolled decrease in the model accuracy. Courbariaux el al.
* EEqul contribution.

---

Published in Transactions on Machine Learning Research (09/2022)
(2016 and later Krishnamoorthi (2018) propose to use gradient Straight-Througg Eatrouggh-sttmatoor (STE) [Bengio
et al. 2013) in order to provide a non-zero gradient to the original weights. This allows the model to adapt
to quantization during training and reduces the final degradation of performance. However, Fan et al. 20021
noticed instability and bias in the learned weights, as STE is not the true gradiont to the function.
The nature of quantization noise has been extensively studied as part of Analog-to-Digital Converters (ADC). In particular, a useful assumption to facilitate the design of post-processing filters for ADC is the independence of the imput value and the "Pseudo Quantization Noise" (PQN), formalized by Widrow et al. (1966). In this work, we show that it also applies to deep learning model quantization, and provides a simple framework in which the output and the quantized model size are both diffcrentiable, without any use of STE. This allows to optimally set the number of bits used per individual weight (or group of weights) to achieve a trade-off between size and accuracy, in a single training and at almost no extra cost. Even when the number of bits to use is fixed, we show that unlike STE, using independent pseudo quantization noise does not introduce bias in the gradient and achieves higher performance. Although PQN has been proposed before for quantization (Baskin et. al., 2018a b), it has never been used on its own without any need for STE or other quantization methods, while achieving state-of-the-art performance.
Our Contribution: (i) With DIFFQ, we propose to use pseudo quantization noise only to approximate quantization at train time, as a diffcrentiable altornative to STE, both with respect to the unquantized weights and number of bits used.
(ii) We provide differentiable model size estimate, so that given a single penalty level X, DIFFQ optimizes the number of bits per weight group of weights to achieve a given trade-off between model size and accuracy. (iii) We provide extensive experimental validation using various models (ConvNets and Transformers) and domains (image classification, language modeling, audio source separation). We demonstrate the efficiency of DIFFQ both in providing small footprint models with comparable performance to the uncompressed ones, together with easy and stable optimization, using only one sensitive dyper-parameet
2 Related Work
Early network quantization methods focused on low-bitwidth networks such as BinaryNet Courbariaux et al. (2015 2016), XOR-Nets Rastegari et al.] (2016), or Ternary networks Li et al. (2016); Wu et al. (2018).
Although these methods produce highly quantized models, their performance not on par with uncompressed
ones. To improve accuracies, higher bitwidth quantization methods were studied Jung ct al. (2019 : Zhang
et al. (2018a); Mishra et al. (2017). These methods followed the STE approach Bengio et al. 20113) STE
allows the gradients to be backpropagated through the quantizers and, thus, the network weights can be adapted with gradient descent Courbariaux et al. (2016).
Variational approaches were used to make the categorical distribution over quantized weights differentiable. Louizos et al. (219) uses a Gumbel-softmax (Jang et al., 2017) but requires hyper-parameters and has no bitwidth tuning. DIFFQ has a single paper-prramterr and supports automatic bitwidth tuning. [Shayer et al. (2018) relies on a Central Limit Theorem (CLT) application, however this prevents weights from converging to deterministic value, which would break the assumptions of the CLT. With DIFFQ, weights are free to converge to any optimal value. Finally Ullrich et al. (2017) uses a gaussian mixture model trained on top of the weights, adding significant complexity both in terms of code, and computation. In contrast, DIFFQ adds only one penalty term to the loss, optimized along the rest of the model in an end-to-end fashion.
An alternative to use a smoothed version of the quantization operator, possibly with a trained wettok (Chen et al. 2019), however as the smoothed operator converges to the true one, gradients will eventually be zero almost everywhere. Gong et all (2019) use meta-network to provide gradients despite quantization. However, their implementation for training the meta-network still relies on STE.
Additive noise injection has been studied by Baskin el al. (0188)), although only during the first few epochs, after which STE based approximation is used. This work was extented to non uniform quantization (Baskin ct al. 20185). In contrast, DIFFQ uses only noise injection, and as demonstrated Results Scction, achieves a better accuracy for an equivalent compression level than both methods. Non uniform quantization was also studied by Pollno et al. (2018), but without differentiability with respect to the weights, with worse
2

---

Published in Transactions on Machine Learning Research (09/2022)
performance than DIFFQ. Additive noise was also studied in the context of image compression Balll et al. 2017: Choi et al. 2019) in order to provide a differentiable psoudo-quantizatinn operator. However, those work rely on an explicit estimation of the quantized values entropy, in particular with respect to a distribution of images. This formalism breaks down when having to quantize a single model, not a distribution, and DIFFQ uses a simpler approach where the bitwidth directly tuned. More recently, Park et al. (2022) extended our method for activation quantization.
An important contribution from DIFFQ is the automatic tuning of the bitwidth using mixed Other mixed-precision quantization methods are based Reinforcement Learning
[Wang et al.
2019 Elthakeb
ct al. 2020 Liu et al., 2021), second-order optimization [Dong et al.
2019
2020; Yao et al. 2021), and
differentiable quantization methods (Uhlich et al. 2020; Wang et al.) 2020). Comparing to DIFFQ, such
methods are more complex (e.g., require plenty of parameter tuning), more computationally heavy, and most importantly based on STE approximations. Wang et al.
(2019); Elthakeb et al. (2019) suggested learning a bitwidth assignment policy using reinforcement learning methods. In contrast, our method select bitwidth along training, using only first. order optimization. [Jain al. (2019): Esser et. al. (2020), and Bhalgat el al. (2020) proposed learning the quantizer step-size or Syyammic-range using STE, but do not allow to select the bitdwidth. Our experiments show that DIFFQ outperforms (Esser et al. 2020) (LSQ) both on most vision and natural language tasks. Uhlich et al. 2020 proposed a re-parametrization that allows to select the bitwidth for each layer through first order optimization, while also relying on STE. The re-parametrization is more complex than the additive noise used in DIFFQ, and suffers from the biased gradient of STE. Results suggest that DIFFQ achieves similar or better trade-offs between model size and accuracy. Besides, in the present work we explore setting bitwidth for individual groups of weights within each layer, rather than layer-wise.
The limitations of STE methods for quantization were first noticed by Liu & Mattina 2019). They recommend using a linear combination of the unquantized and quantized weight, with the gradient flowing only through the unquantized contribution. In a similar spirit, Fan et al. (2021) sample for each layer and iteration whether to use the quantized or unquantizod weight. Both methods reduce the bias from STE, but also remove some of the quantization noise during training. In contrast our method allows to keep a full pseudo quantization noise without the STE bias. Liu et al.] (2022) proposed the Generalized STE method to deal with gradient instabilities by calculating the expectation of the stochastic quantization during the backward phase. Finally, Nagel et al. 20222 extend the analysis we present in Section 3.3 the oscillations of weights when using STE and suggest tracking the weight oscillations in order to freeze them when needed, an ad-hoc solution.
A last line of related work is Product Quantization (PQ) Stock et al. (2019), where code words are being learned to quantize blocks of weights rather than single weights. This method achieves a higher compression level than per-weight. quantization but also requires carefully choosing the size of the codebooks for each layer. In contrast, our method requires only choosing a single lyper-parameter to balance between model size and accuracy. Besides, as noted by Fan et al. (2021). per-weight quantization and PQ can be combined. We compare with PQ vision and language tasks: while PQ can reach smaller model size than DIFFQ, it can also suffer from unacccptable accuracy loss, in particular for language modeling.
3 Background
Let us consider a weight vector w Rdd where d E N, typically the weights of convolution or linear layer. Each entry of the vector is lypically coded over 32 bits with floating-point precision. We aim to reduce the number of possible states to 28, where B < 32 is the number of bits of precision. First, we assume w, E [0, 1] for all 1 < i < d. In practice, one would first normalize w as
w min(w) max(w) min(w)
i
and provide the tuple (min(w), max( scparately as a 32 bits IEEE float. Given that for typical deep learning models d > 1, storing this range has a negligible cost. For readability, we describe the method for scalar values w E [0, 1), however, this can be easily extended to vectors w E Rd.
3

---

Published in Transactions on Machine Learning Research (09/2022)
3.1 Uniform quantization
The simplest quanlization methods consist of laking 2 points evenly spaced in the range [0, 1] and round each entry of w to the nearest point. One can then store the rounded value by its index, which requires only B bits. Formally, we quantize a number w E [0, 1] over B bits as
Vw e 10. E N.. Qww B) round (w (2 - 1))
(1)
28
While the intuitive definition of quantization is for an integer number of bits, we can extend the previous definitions to a real-valued number of bits B E RR.. Note that variants of this scheme exist, for instance, symmetric uniform quantization, which enforces that 0 is always exactly represented (Krishnamoorthi 2018
3.2 Optimization of the quantized weights
The weight vector w is typically obtained through the process of training predictor function parameterized by w, denoted as fw: to minimize a loss function L.
min L(fw), werd
(2)
where T((fxr) is the empirical risk over given dataset. The process of quantizing vector over B bits introduces a quantization noise N (ww, B) = Q(w, B) which is unaware of the training objective L. Even if w is close to the optimum, Q(w, B) might deteriorate arbitrarily the performance of the predictor.
Thus, given a fixed budget of bits B, one would ideally like to minimize the empirical risk when considering the quantization process,
min L(fq(e.m))
wERd
where fQ(w,B) is the predictor function using the quantized model parameters.
Unfortunately, the gradients of Q(w, B) are zero over its definition domain because of the rounding operation, and as result, it cannot be optimized using first-order optimization methods such as SGD Adam (Kingma & Ba. 2015). One possible solution is to replace the Jacobian of Q(-, B) with the identity matrix during the backward phase, as suggested in the STE method (Bengio et al. 2013). The STE method was popularized
for quantization as the Quantization Aware Training (QAT) technique by KKrishnamorrthi (018)).
3.3 The instability and bias in STE
As described by Fan et al. 2021), following the STE approach can cause instability during training and bias in the models' gradients and weights. As a result optimization will fail to converge to the optimal value even on simple cascs. To demonstrate that, consider the following 1D least-mean-square problem, where B E N., the optimal weight w. E [0, 1] such that Q(w,, B) A w., and Q(w., B) (0, 1). Given a random variable X R with o² E [x]] such that 0 < o² < oo, we would like to minimize the following using STE based QAT:
min L(w) = (XQ(w, w. B) - -Xw.)² wE[U,1]
(4)
We immediately have that the optimum is achieved for Q(w, B) Q(w,, B). Let us try to optimize equation 4 using SGD with STE starting from Wa w. with Wn the sequence of iterates. We call w_ and w4 the quantized values just under and above W., and we assume without loss of generality that Q(w., B) w4- The expected gradient with STE at iteration n is given by
G, = o2= B) - w.).
(5)
In particular, Go = 2 w.) > 0, and G, will stay positive until Q(w,, B) w_. At this point, we will have Gn 0, and will stay so until Q(w,, B) = w+. Thus, we observe that using STE, Q(u,, B) will oscillate between w- and wt, while the optimal value is w4. The pattern of oscillation will depend on the learning
4

---

Published in Transactions on Machine Learning Research (09/2022)
0.14
70
Weight Value
Accuracy (%)
50
40
30
0.08
20
QAT4
DifIQ
0.06
10
100
Iteration
Epochs
(a)
(b)
Figure 1: (a) Using STE and SGD to optimize the 1D least-reau-square problem given by equation | (with B and X a.s.). Q(w,, B) oscillates between the quantized value just above (w+) and just under (w-) the unquantized ground truth w., while w., oscillates around the boundary (w+ + w- w-) (b) Model accuracy vs. epochs for ImageNet using EfficientNet-b3, Results are presented for both QAT over bits and DIFFQ.
rate and relative position of w. within the segment [w_, w+]. Taking smaller step size will reduce the amplitude of the oscillations of wa, but not of Q(wn, B), which is what interests us. Indeed, Wn. oscillations are centered at the boundary (w4 + w-)/2. We provide one example of those oscillations on Figure ! with a.s. and a step size of 0.5.
w. = 0.11, B 4,
Extrapolating to model with millions of parameters, at any point in time, significant fraction of the weights could be quantized to suboptimal value due to the oscillations implied by the STE method. We conjecture that this behavior explains the oscillations of the accuracy observed when training an EfficientNet-b3 with QAT using 4 bits per weight on ImageNet (see Figure [(b).. In the following section, we introduce DIFFQ, a method based on independent additive pseudo quantization noise, that does not suffer from such a bias, while approximating well enough quantization noise to perform efficient quantization aware training.
4 Method
Pseudo quantization noise. A classical assumption in digital signal processing when working with quantized signals is that the quantization noise is approximated by independents uniform variables over /2 A/2] with A بيं the quantization step. This approximation was studied in depth by Widrow et al. 1996 as Pseudo Quantization Noise (PQN). Following this assumption, we define the pseudo quantization function Q for all E R and B E R4. as
A U 1,11, 2
Q(I, x
(6)
with U[-1, 1] an independent sample from the uniform distribution over [-1, 1]. This pseudo quantization function differentiable with respect to æ and B. Unlike QAT, this differentiability does not require an STE. It also provides meaningful gradient with respect to the number of bits used B (extended to be real-valued).
If we look back at the example from Figure using now equation instead of STE, the expected gradients for SGD become
Gn = F x. ((w,,
Win A 2 ul u1[1,,1] W.F
= We),
(7)
5

---

Published in Transactions on Machine Learning Research (09/2022)
which cancels out for Wn Wa, that at convergence we indeed have Q(wn. B) Q(w,, B), i.c. the gradient estimate is unbiased and converges to the right solution.
Mixed precision. We used a common precision B for all the entries of the weight vector w. One can instead use different values for different entries. Formally, the entries in w are grouped by considering wo E Rexd/s with 9 the group size and d/g the number of groups. We can then extend the definition Q(w, B)) given by equation and equation to use a number of bits b, for the group 8, with E R+44/9,
Training objective. Given w E Rexd/9 with 9 groups of d/g entries, and a number of bits E No, we define the model size, expressed in MegaByles (IMB 8 220 bits)
d/g
M(b) = 2
b.
(8)
A typical objective of quantization is to achieve the best possible performance within given model size budget or to achieve the smallest model size that reaches a given performance, i.c. we want to minimize with b E Nd/9 and w E Rexd/s either,
min (fQ((.)))) s.t. M(b)= S m.
min M(6), wib
Or
s.t. [(fQ(wo)) 1.
We can relax to be real valued, and replace Q by our differentiable pseudo quantization function Q. Then, following the ezact penalty method (Bertsckas) (1997). Section 4.2, Bertsekas (2014), Chapter 4), there is X(m) > 0 (or X(I) for the right hand side problem), such that the left hand size problem equivalent to
min (fa((.))) + A(m)M((), wib
(10)
which is fully differentiable with respect and and can be optimized with first order optimization.
Parametrization. In practice, the number of bits used for each group b TR4 is obtained from logit. parameter 1 E R2, that we have
b = bmin + (1)(mmmxx - bmin):
(11)
with a is the sigmoid function, and 6min and bmax the minimal and maximal number of bits to use. The trainable paramoter is initialized so that b b= binit- We set binit 8.
Evaluation and noise distribution. At evaluation time, we round the value b obtained from equation 10 as b= (b) and quantize w as Q(w, b). Thus, the amount of quantization noise at evaluation can larger than the amount of noise injected at train time. We observed that using noise distribution with larger support, such as Gaussian noise with unit variance (i.e. 3 times the variance of 1])), makes the model more robust to this operation. An empirical comparison between uniform and Gaussian noise can be found in Table B.7 in the Appendix. Thus in the rest of the paper, always use Gaussian noise at train time.
True model size. The mode size given by equation g is used at train time but does not account for part of the true model size. At evaluation time, we represent each weight by the integer obtained from the rounding operation in equation [ For each layer in the network, we also store two 32 bits float numbers for the minimum and maximum scale. Finally, the actual value of b must be coded, as it is no longer a fixed constant. For each layer, we compute the maximum value of C, log2 [1 + b, bmia) over all groups E {1, d/g). 8-bit integer, and for each group, encode b, buin max
We encode once the value max(C)) bits. The true size for one layer, expressed in MegaBytes, is thus given by
d/g b
M(b) = 223
32 + 8 + g
max C*) +
(12)
6

---

Published in Transactions on Machine Learning Research (09/2022)
Table 1: Comparison of DIFFQ against baselines presented in the Related Work section. Sizes marked with are reported after Huffman coding, following Polino et al. 2018 Accuracics marked with are the best rather than last one match previous practices.
MODEL
METHOD
Top-1 Acc. (%)
M.S. (MB)
CIFARI0
RESNET-18
UNCOMPRESSED
95.3
42.7
RESNET-18
UNIQ
BASKIN ET AL.
2018B
89.1
2.7
RESNET-18
NICE
BASKIN
2018A
92.7
2.7
RESNET-18
DIFEQ (OURS)
93.9
2.7
RESNET-20
UNCOMPRESSED
92.7*
1.48
RESNET-20
DQ UULICH ET AL.
2020
91.1*
0.07
RESNET-20
DIFFQ (OURS)
91.6*
0.06
CIFAR100
WIDE-RESNED
UNCOMPRESSED
76.2
139.4
WIDERRENNT
DIFEQUANT POLINO ET AL.
2018
49.3
7.9
WIDEEREESNE
DIFFQ (OURS)
75.6
4.7
IMAGENET
RESNET-18
UNCOMPRESSED
70.9*
44.6
RESNET-18
META-QUANT CHEN ET AL.
2019
60.3
1.3
RESNeT-18
DQ UhLICH ET AL. 2020
70.1
5.4
RESNET-18
LSQ 4 BITS ESSER AL.
2020
70.7*
5.6
RESNET-18
DIFFQ (OURS)
71.1* *
5.3
RESNET-50
UNCOMPRESSED
77.1* *
97.5
RESNET-50
LSQ 4 BrTS ESSER ET AL.
2020
76.2*
12.3
RESNET-50
LSQ 3 BrTS ESSER ET AL.
2020
75.6*
9.3
RESNET-50
DIFQQ (OuRS)
76.6*
10.5
REsNeT-50
DIFFQ (OuRS)
76.3*
8.8
5 Results
We present experimental results for language modeling, audio source separation, and image classification. We show that DIFFQ can often provide a model with comparable performance to the uncompressed one while producing a model with smaller footprint than the baseline methods (STE based). We provide finer analysis of different aspects of DIFFQ hyper-past parameerasssssss and their impact on quantized models in next Section. Finally, discuss limitations of DiffQ in the Limitation Section. Both experimental code, and generic framework usable with any architecture in just. few lines, is available on Github github.com, fawcebookreeerrcc/cooocoo /diffq All typer-parameter for optimization and model definition are detailed in the Appendix. In all tables, 1 (resp. 4) indicates that highest best (resp. lowest best). All results referred to as "QAT" are obtained using the formula given by equation [ with layer-wise min-max scaling of the weights. When using DIFFQ, we use the same per layer min-max scaling. When also doing activation quantization, we use per-channel min-max scaling of the activations. All DIFFQ experiments use Gaussian noise explained in Section $
5.1 Comparison to related work
On Table I. we compare DIFFQ to some of the related work presented Section P Compared with the NICE (Baskin al. 2018a) and UNIQ (Baskin al.) (2018)) methods, which also rely on additive noise, DIFFQ achieves significantly better accuracy for the same model size. We then compare to the differentiable

---

Published in Transactions on Machine Learning Research (09/2022)
Table 2: Language modeling results for 16 layer Transformer trained on Wikitext-103. We also test combining weight and activation quantization. We compared DIFFQ to QAT and Quant-Noise (QN) method proposed by Pan et al. (2021) (models with were trained with layer-drop of 0.2 Fan et al. (0019) Activations are quantized over 8 bits, with per-channel scaling.
WEIGHTS
ACTIVATION PPL 4
M. S. (MB) 4
UNCOMPRESSED
18.1
942
BITS
BITS
18.3
236
QAT SAITS
8 BITS
19.7
236
QAT 4AITS
8 BITS
29.9
118
LSQ 4 BITS ESSER ET AL.
2020
8 BITS
18.9
118
DIFEQ 1=
8 BITS
18.1
130
DIFEQ 1=
8 BrTS
18.6
113
UNCOMPRESSED
18.3
942
QN BITST FAN ET AL.
2021
QN 8 BITS
18.7
236
QN 4 BITST FAN ET AL.
202T
QN 8 BrTS
19.5
118
PQi FAN ET AL. [2021)
20.7
38
quantization method by (Polino al. 2018): which only optimizes the non uniform quantization points, not the pre-quantization weights. Following their practice, we report numbers after Huffman coding, We achieve model almost half as small, with gap of 25% in accuracy, proving that optimizing pre-quantization weights is more important than tuning non uniform quantization grid. Meta-Quant (Chen et al.) 2019) achieves smaller model size than DIFFQ, with 1 bit per weight, a regime where the PQN assumption breaks down, at the price of losing nearly 10% of accuracy. Finally, compared with two quantization methods: DQ by [Uhich] et al. (220)) and LSQ by Esser et al.] (2020). When considering DQ, DIFFQ achieves slightly smaller model size and better accuracy on ImageNet using ResNet-18, and a 15% smaller model with sightly better accuracy for Resnet-20 trained on CIFAR-10. Comparing LSQ [ DIFFQ achieves better accuracy with smaller model size on ImageNet using both ResNet-18 18 and ResNet-50. Additional comparison between DIFFQ and LSQ for higher compression rates can be on Table B.1 in the Appendix.
5.2 Language Modeling
Vaswann et al. 2017 based language model on the Wikitext-103 text. corpus (Merity et al. 2016), following Baevski & Auli 22019) and using the Fairseq framework (Ott al. 2019). Results are presented in Table 2 We compare to the Quant-Noise method by Fan et al. (2021), but use a reduced layer-drop (Tan et al. 2019 of 0.1 instead of 0.2. This both improves the baseline, as well as the performance DIFFQ models. For DIFFQ, we explicitly set the gradient for the number of bits parameters to zero for all layers that have been dropped. In order to test the compatibility of DIFFQ with efficient int8 kernels, we further quantize the activations to bits using PyTorch native support (Paszke al. 2019).
We trained 16 layers transformer
The transformer model has some tied parameters (e.g. word embedding in the first and pre-softmax layer). It is important to detect such tied parameters with DIFFQ. We use single shared bits parameter when a parameter tensor is rensed multiple times, and for each forward, we sample a single pseudo quantization noise per group of shared weights and reuse it appropriately. Failure to do so led to a significant worsening of the performance at validation time.
While QAT breaks down when trying get to 4 bits precision (perplexity of 29.9), using DIFFQ allows achieve lower model size (113MB vs. 118 MB for QAT 4 bits) with a perplexity closer to the uncompressed one (18.6, vs. 18.1 uncompressod). We also tried fine-tuning pro-trained model with LSQ (Esser et al. 2020 While this works better than QAT, LSQ reaches worst perplexity for slightly larger model size
We used own LSQ implementation, with only weight quantization, since no official code available. Comparison with the results reported in Eeser al. (2020) can he found on Table B.1

---

Published in Transactions on Machine Learning Research (09/2022)
Table 3: Music source separation results for the Demucs model Défossoz et al.
2019
We report Signal-to-
Distortion Ration (SDR) together with Model Size (M.S.).
SDR (dB) T
M. S. (MB)
UNCOMPRESSED
6.31
1014
QAT 4BITS
5.99
130
QAT 5BITS
6.27
162
DIFEQ (A-3e-4)
6.28
120
than DIFFQ (18.9 perplexity for 118 MB). Similarly, Quant-Noise (Fan et al. 2021) improves on QAT but performs worse than DIFFQ, even when using more than twice as many bits. With just 4.4 bits per weight on average, DIFFQ achieve the same perplexity as the baseline. We also compare to PQ (Stock et al. 2019); as reported by Fan et al.] (2021). While PQ achieves higher compression levels, with just 38MB, its perplexity is the worst of all methods.
5.3 Music Source Separation
We use the Demucs architecture by Défossez et al. (2019) with 64 initial hidden channels. The model is trained on the standard MusDB benchmark (Rafi et al. 2017), for 180 epochs, and evaluated with the Signal- To-Distortion Ratio (SDR) metric (Vincent al.) 2006). The unquantized model 1GB. We compare DIFFQ with QAT training with either 5 or 4 bits, with the results presented in Table With bits, QAT is able to replicate almost the same performance the uncompressed model. When trying further compress the model to 4 bits per weight, QAT leads to sharp decrease of the SDR, losing 0.3dB, for a 130MB model. DIFFQ achieves a model size of 120MB, with only drop of 0.03dB of SDR compared to the uncompressed baseline.
5.4 Image Classification
Next, we evaluated three image classification benchmarks: ImageNet Deng et al. (2009) CIFAR- and CIFAR-100 Krizhevsky et al. (2009). For CIFAR-10 and C1F results are reported for Mobile.Net- vl Howard et al. (2017), ResNet-18 He et al. 20066) and Wide-ResNet with 28x10, depth and width levels respectively Zagoruyko & Komodakis (2016). ImageNet results are reported using Efficient Tan & Le
(2019) and DeiT-B Touvron et al. (0020) models. More details regarding hyper-parameters and augmentations used can be found in the Appendix.
CIFAR10 & CIFAR-100. Results for CIFARIO and CIFAR100 are depicted in Figures 2(a) and 2(b)
We compare DIFFQ, QAT and LSQ (without activation quantization) using 2, 3, and bits quantization. Performance of the uncompressed model is additionally presented as bound- To better understand the effect of the penalty level X on both model size and accuracy, we train models with DIFFQ using different penalty levels. Exact results are presenled in Table B.2 in the Appendix, together with a detailed analysis.
Results suggest DIFFQ models reach comparable performance to the LSQ and outperforms QAT models while producing models with smaller footprint. When considoring 2 bits quantization, QAT is always worse than both LSQ and DIFEQ. While LSQ works well for Resnet18, suffers from large drops in accuracies for MobileNet and WedeResNet, failing entirely to train for MobileNet CIFAR10, despite initialization from pre-trained model.
ImageNet - DeiT. Results for ImageNet using DeÎT-B model are presented in Table 4 We compared DIFFQ to QAT when training with 4 and 8 bits. Both QAT with 8 bits and DIFFQ reach comparable performance to the uncompressed model, while DIFFQ yields model almost half of the size as QAT, however still bigger than QAT with 4 bits. When we increase X, we get a smaller model-size than QAT with 4 bits but with better accuracy levels.
9

---

Published in Transactions on Machine Learning Research (09/2022)
MobileNet
ResNet-18
WRN
MobileNet
ResNet-1
WRN
100
80
70
80
60
Accuracy (%)
Accuracy (%)
60
50
*
40
40
DiffQ
DiffQ
30
QAT
QAT
20
LSQ
20
LSQ
Uncomp.
Uncomp.
10
2
2
2
24
2
2
2
2
2
2
2
2
Model Size (MB)
Model Size (MB)
(a) CIFAR10
(b) CIFAR100
Figure 2: Model accuracy and size on CIFARIO (a) and CIFAR100 (b) using MobileNet, ResNet-18, and WideResNet (WRN) models for various penalty levels using DIFFQ, QAT, LSQ, and the baseline.
Table 4: Image classification results for the ImageNet benchmark. Results are presented for DIFFQ and QAT
using 4 and 8 bits using the DeiT model TTouvron et al. 2020 We report Top-1 Accuracy (Acc.) together with Model Size (M.S.).
ToP-1 Acc. (%) f
M.S. (MB) +
UNCOMPRESSED
81.8
371.4
QAT 4BTTS
79.2
41.7
QAT 8BITS
81.6
82.9
DIFFQ (A=1e-2)
82.0
45.7
DIFFQ (A=0.1)
81.5
33.02
ImageNet Efficient.Net. We evaluate the performance of DIFFQ on the memory-efficiet EfficientNet-B3 model. Results are depicted on Figure B.1 (c) as well as in Table B.5 both in the Appendix. Both QAT bits and DIFFQ achieves similar accuracy (QAT 81.3 %, DIFFQ 81.5%) but with a smaller model size for DIFFQ (8.5MB vs. 12MB for QAT). When considering QAT 4 bits, DIFFQ produces a smaller model with a significantly better accuracy level (80.8)). For QAT 4, we noticed considerable instability close to the end of the training, Figure B.1 (b) in the Appendix.
5.5 Analysis
Bits Histogram. Figure presents the weight bitwidth assignment over layer groups for the EfficientNet-Bs: Tan & Le (2019) and DeiT Touvron et al. (2020) models trained ImageNet. The capacity distribution over depth for ConvNets (EfficienNeet-B3) and Transformers (DeiT) different (fp32 shows uncompressed capacity). Notice, that the quantization trends are different too: for the ConvNet, smaller bitwidths are used for deeper layers of the model while large bitwidth is more common in the first layers (except for the last linear layer which seems to need some precision). For the Transformer, this effect of varying quantization by layer similar but less pronounced, due to the more symmetric nature the architecture.
Fixed bitwidth. On Table B.4 the Appendix, we compare QAT to DIFFQ using fixed number of bits, i.e. comparing strictly PQN to STE. On MobileNet, ResNet-18, and NedeResNet for both CIFAR1O and
10

---

Published in Transactions on Machine Learning Research (09/2022)
CIFAR100, DIFFQ outperforms QAT, with gap especially noticeable for 2 bits models, a regime where QAT becomes unstable, as we described in previous section.
Group size. We additionally evaluate the affect of the group-size, on model size and accuracy, by optimizing DIFFQ models using 9 E {1, 8, oo). When g=oc, we use a single group for the entire layer. Results for ResNet-18 using CIFAR-100 are depicted in Figure [(a] the Appendix. Interestingly, we observed that increasing 9, yiclds a smaller model size on the expense of minor docrease in performance. However, when setting -=00 model performance (model size and accuracy) is comparable to g=8 for this task.
Runtime overhead and loading time. Using DifiQ usually increase the training time by some amount. On the language modeling task, the time per batch went from 115ms to 125ms. When training a ResNet18 on CIFAR-10, it increased from 120ms to 150ms. For the Demucs model, it went from 0.9s to 1.1s. However, when training the EfficientNet-b3 model, we observed that the time per batch would nearly double. Thus it seems that for most architectures the training time overhead is limited, although the worst case can be up twice as slow. At evaluation time, decompressing the Demucs model from its variable bitwidth compact representation takes around 2.81 scconds on MacBook Pro with 2.4 GHz 8 coros Intel i9 processor.
5.6 Limitations
The model size given by equation 12 is obtained with a traditional encoding of the quantized model. However, more efficient coding techniques exist when the entropy of the data is low, such as Huffman coding (Huffman 1952 Using the ZLib library, obtain estimate of the Huffman compressed model size after quantization. For instance, for the language model described in Table 2 the QAT 8 model gets further compressed from 236MB to 150MB, showing that the entropy of its quantized weight is significantly lower than the maximal one for 8 bits integers. However, the DIFFQ model naive size is 113MB, and after compression by ZLib, gets to 122MB. This is a sign that the entropy is close to its maximal value, with ZLib adding only overhead for no gain. In equation 10 we only penalize the naive number of bits used, while asking for the best possible accuracy. In that case, the model maximally use the entropy capabilities for given number of bits. An interesting line of research would be to replace the model size equation S to account for the actual entropy of the data, for instance with differentiable kernel density estimation. We leave that for further rescarch.
Another limitation of DifiQ is that it can make training up to twice as slow, due to the extra parameters to optimize for and the more complex gradient calculation graph. Besides, in order to achieve a specific model size or accuracy, one has to lune the penalty parameter.
6 Discussion
We presented DIFFQ, a novel and simple differentiable method for model quantization via pseudo quantization noise addition to models' parameters. Given a single syper-parameter that quantifies the desired trade-off between model size and accuracy, DIFFQ can optimize the number of bits used for cach trainable parameter group of parameters during model training. We conduct expensive experimental evaluations on various domains using different model architectures. Results suggest that DIFFQ is superior to the baseline methods on several benchmarks from various domains. On ImageNet, Wikitext-103, and MusDB, we achiove a model size that is smaller than a 4 bits quantized model, while retaining the same performance as the unquantized baseline. For future work, we consider adapting the model size penalty to account for Huffman encoding, which could allow to further reduce the model size when it is gzipped. Another line of work would be using PQN improve activation quantization, enabling 4-bits kernels for larger number of tasks.
References
Alexci Baevski and Michael Auli. Adaptive input representations for neural language modeling. In Proc. of the International Conference on Learning Representations, 2019.
Johannes Ballé, Valero Laparra, and Eero Simoncelli. End-to-end optimized image compression. In ICLR, 2017.
11

---

Published in Transactions on Machine Learning Research (09/2022)
433
g32
2 bits
3 bits
23
3 bits
4 bits
bits
5 bits
bits
25
6 bits
20
bits
7 bits
8
8 bits
9 bits
of sie
15
10 bits
Loga 02 size
11 bitx
15
overeadd
12 bitts
10
overhead
5
3
4
6
7
10
0
3
4
6
7
10
Layez group
Layer group
(a) LffficientNet-BB
(b) DeilT
Figure 3:
We group layers of a given architecture into 11 groups (group 0 being closest to the input, and closest the output), and report for each group its contribution to the model size. We compare the baseline EfficientNNe--33 (above) and DeiT (below) models (floating point 32 bits) and the quantized models with DIFFQ (A=5e-3 for EfficientNet-B33 x=le-2 for DeiT). For quantized model, we also report the distribution over each bitwidth within each group of layers. Scale is logarithmic across layers, and linear inside each one. Finally, "overhead" shows the capacity needed to encode the bitwidth used for each group of weights.
Chaim Baskin, Natan Liss, Yoav Chai, Evgenii Zheltonozhskii, Eli Schwartz, Raja Giryes, Avi Mendelson, and Alexander M Bronstein. Nice: Noise injection and clamping estimation for neural network quantization. arXiv preprint arxiv: 010.80000100 2018a.
Chaim Baskin, Eli Schwartz, Evgenii Zheltonozhskii, Natan Liss, Raja Giryes, Alex M Bronstein, and Avi Mendelson. Uniq: Uniform noise injection for non-uniform quantization of neural networks, arXiv preprint arXiv:1804.10969, 2018b.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint wrXic:1308.3432. 2013.
Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3):334 334, 1997.
Dimitri Bertsekas. Constrained optimization and Lagrange multiplier methods. Academic press, 2014.
Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsqt: Improving low-bit quantization through learnable offsets and better initialization. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pottern Recognition Workshops, pp. 696-697, 2020.
Shangyu Chen et al. Melaquant: Learning to quantize by learning to penetrate non-differentiable quantization. In Advances in Neural Information Processing Systems, 2019.
Yoojin Choi, Mostala El-Khany, and Jungwon Lee. Variable rate deep image compression with a conditional autoencoder, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3146 3154, 2019.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. arXiv preprint arXiv:1511.00963 2015.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1, arXiv preprint Xrx:: 602. 02830, 2016.
12

---

Published in Transactions on Machine Learning Research (09/2022)
Alexandre Défossez, Nicolas Usunier, Léon Bottou, and Francis Bach. Music source separation in the waveform domain. arXiv preprinl or 1xxiv 13324,, 2019.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Databasc. In CVPR, 2009.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 293-302, 2019.
Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace- weightt quantization of neural networks. Advances in neural information processing syslems, 33:18518 18529, 2020.
Ahmed Elthakeb, Prannoy Pilligundla, FatemehSadat Mireshghallah, Amir Yazdanbakhsh, Sicuan Gao, and Hadi Esmaeilzadeh. Releq: An automatic reinforcement learning approach for deep quantization of neural networks. In NeurIPS ML for Systems workshop, 2018, 2019.
Ahmed T Elthakeb, Prannoy Pilligundla, Fatemehsadat Mireshghallah, Amir Yazdanbakhsh, and Hadi Esmaeilzadeh. Releq: A reinforcement learning approach for automatic deep quantization of neural networks. IEEE micro, 40(5):37-45, 2020.
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswany, and Dharmendra S Modha. Learned step size quantization. In Proc. of the International Conference on Learning Representations, 2020.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In Proc. of the International Conference on Learning Representations, 2019.
Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Remi Gribonval, Herve Jegou, and Armand Joulin. Training with quantization noise for extreme model compression. In ICLR 2021, 2021.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjic Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In Proceedings of the IEFE International Conference on Computer Vision, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. aiXiv preprint arXiu: 774.04061.... 2017.
D. A. Huffman. A method for the construction of minimum-redundanny codes. Proceedings of the IRE, 40(9): 1098-1101) 1952. doi:
Yerlan Idelbayev. Proper ResNet implementation for CIFARIO/CIFARI00 PyTorch. https:/ https:///ithubb.om akamaster /pytorcc sewpoorchh__eeeett_eeeeeeeeeeeee resnet [ 2018
Sambhav R Jain, Albert Gural, Michael Wu, and Chris H Dick. Trained quantization thresholds for accurate
and efficient fixed-point inference of deep neural networks. arXie preprint 1rXiv:1903. 080666 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In Proc. of the International Conference on Learning Representations, 2017.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4350-4359,
2019.
13

---

Published in Transactions on Machine Learning Research (09/2022)
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of the International Conference on Learning Regresentations, 2015.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whilepaper. arXiv preprint S.ie:1806.08344 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. arXiv preprint or 04711, 2016.
Jing Liu, Jianfei Cai, and Bohan Zhuang. Sharpness-aware quantization for deep neural networks. arXiv preprint Xrv:222111. 12277, 2021.
Zechun Liu, Kwang-Ting Cheng, Dong Huang, Eric P Xing, and Zhiqiang Shen. Nommnformn-oo-mifom quantization: Towards accurate quantization via generalized straight-through estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Zhi- MaigGang Liu and Matthew Mattina. Learning low- oow-precision neural networks without straight- straight-through through estimator (ste). arXiv preprint orXiv:1903.0100 01061, 2019.
Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proc. of the International Conference on Learning Representations, 2019.
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling, Relaxed quantization for discretized neural networks. In Proc. of the International Conference on Learning Representations, 2019.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In Proc. of the International Conference Learning Represendations, 2016.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Frich Flsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In Proc. of the International Conference on Learning Represenlations, 2018.
Asit Mishra, Eriko Nurvitadhi, Jeffrey J Cook, and Debbie Marr. Wrpn: Wide reduced--recissoon networks. In Proc. of the International Conference on Learning Representations, 2017.
Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, and Tijmen Blankevoort. Overcoming oscillations in quantization-aware training. Xi preprint orXiv:2203. 1 1086, 2022.
Myle Ott,, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Sein Park, Junhyuk So, Juncheol Shin, and Eunhyeok Park. Nipq: Noise injection pseudo quantization for automated dnn optimization. arXiv preprint arXiv:2206. 008000 2022.
Adam Paszke al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, 2019.
Antonio Polino et al. Model compression via distillation and quantization. In Proc. of the International Conference on Learning Representations, 2018
Zafar Rafii Antoine Liutkus, Fabian-Robert Stöter, Stylianos loannis Mimilakis, and Rachel Bittner. The musdb18 corpus for music scparation, 2017.
Mohammad Rastegari, Vicente Ordoncz, Joseph Redmon, and Ali Farhadi. Xnor-nct: Imagenct classification using binary convolutional neural networks. In Europeon conference on computer vision, pp. 525-542 Springer, 2016.
14

---

Published in Transactions on Machine Learning Research (09/2022)
Oran Shayer, Dan Levi, and Ethan Fetaya. Learning discrete weights using the local reparameterization trick. In Proc. of the International Conference on Learning Representations, 2018.
Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. And the bit goes down: Revisiting the quantization of neural networks. In Proc. of the International Conference Learning Representations, 2019.
Christian Szegedy, Vincent Vanhoucke, Sergey Loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pollern recognilion, pp. 2818 2826, 2016.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proc. of the International Conference Machine Learning, 2019.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude, 2012.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. ar preprint Xiv:2001 12877, 2020.
Stefan Uhlich, Lukas Mauch, Fabien Cardinanx, Kazuki Yoshiyama, Javier Alonso Garcia, Stephen Tiedemann, Thomas Kemp, and Akira Nakamura. Mixed precision dnns: All you need a good parametrization. Proc. of the International Conference on Learning Representations, 2020.
Karen Ullrich, Edward Mcods, and Max Welling. Soft weight-sharing for neural network compression. In Proc. of the International Conference on Learning Represendations, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Miia Polosukhin. Atlention is all you need. In Proc. of Newral Informalion Processing Systems, 2017.
Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. IEEE Transactions on Audio, Speech and Language Processing, 2006.
Kuan Wang et al. Haq: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8612 22--8620, 2019.
Ying Wang, Yadong Lu, and Tijmen Blankevoort. Differentiable joint pruning and quantization for hardware efficiency. In Europeaa Conference on Computer Vision, pp. 259 277. Springer, 2020.
Bernard Widrow, Istvan Kollar, and Ming-Chame Liu. Statistical theory of quantization. JEEE Transactions on instrumentation and measurement, 45(2):353 361, 1996.
Ross Wightman. Pytorch image models. https:/
2019.
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with inlegers in deep neural networks. In Proc. of the International Conference on Learning Representations, 2018.
Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang, Qijing Huang, Yida Wang, Michael Mahoney, et al. Hawq-v3: Dyadic neural network quantization. In International Conference on Machine Learning, pp. 11875-11886. PMLR, 2021.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. -032, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. ar preprint (tf:::6605071.40 2016.
Dongqing Zhang, Jiaolong Yang. Dongqiangzi Ye, and Gang Hua. Lq Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pp. 365-382 201 Sa.
15

---

Published in Transactions on Machine Learning Research (09/2022)
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Pazz mixup: Beyond empirical risk minimization. In Proc. of the International Conference on Learning Representations, 201 8b.
16


Reviews:
[{"review_id": "8JsHO614YK", "reviewer": "Reviewer_SyNn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Insightful example that differentiates DiffQ from QAT with STE\n- Formulation of model-size aware optimization with grouping and mixed-precision\n- Evaluation of various DNN applications including CNN and Transformer-based image classification, Transformer-based language model, and music source separation.\n\nWeakness:\n- There are several unsolved questions about the proposed techniques. \n1) There are three claims in the proposed method: 1) noise injection for QAT without STE, 2) mixed-precision quantization derived by differentiable bit-width parameter, and 3) grouping of weights for fine-grain (mixed-precision) quantization. It is not clear how much each of these sub-techniques helps improve the model accuracy. \n2) Table B.4 (Appendix) shows the accuracy comparison of the fixed-precision quantization. It is strange to see that DiffQ's accuracy is quite lower than LSQ for some cases (especially, ResNet18). Why does this happen?\n3) Figure B.1(a) (Appendix) shows the accuracy comparison for different group sizes, but only for one network with very limited group size options. Noting that changing bit-precision in small granularity (g=1, 4, 8) is not practical, the author should consider experimenting with larger group size options, such as channel-wise grouping. Also, it is strange that the accuracy difference between g=inf and g=1 is marginal in Figure B.1(a). Is this a commonly observed phenomenon? If so, would group size matter?\n4) Table B.7 (Appendix) shows that the quantization performance of different noise distributions. It is strange to see that the uniform distribution achieves much lower accuracy compared to the Gaussian since it contradicts the authors' assumption of quantization error modeling motivated by Widrwo et al., 1996. Does this experimental result imply that there is something wrong with the assumption of the error modeling? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a noise injection method for differentiable quantization-aware training without STE. The proposed method, DiffQ, replaced the quantization operation on weights with a simple quantization noise injection modeled as a uniform random noise. The benefit of this simple derivation is that the model is differentiable with respect to weight as well as the bit-precision. Therefore, the authors further proposed a mixed-precision quantization mechanism where weights are grouped and quantized with different bit-precision. The authors provided extensive performance comparison with prior quantization methods. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please provide a more extensive ablation study and in-depth analysis of the three key techniques included in the proposed method to clarify their contributions to the improvement in the quantization performance.", "broader_impact_concerns": "I don't have specific concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IeChMvI1x6", "reviewer": "Reviewer_eAuD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Relevance:** the topic of aggressive model compression at 8 bits and below discussed in this paper is of clear interest for the scientific community and industry alike.\n\n**General comments:** the paper is well structured and easy to follow. The discussion of the literature properly cites and compares with closely related work. The implementation of the DiffQ method via PQN is adequately described and could be reproduced by a practitioner. The provided code is clean and well organized.\n\n**Novelty:** noise injection at train time as a replacement for quantization has been previously reported (as acknowledged by the authors: see Baskin2018a,b), although only during part of the training. This work extends this concept by replacing quantization with PQN across the whole training and by using additive gaussian noise (instead of uniform or other forms). In addition, the authors leverage the differentiability w.r.t both inputs and bits to (1) allow for mixed precision (i.e., enable a flexible choice of number of bits for different layers / groups of parameters) and (2) explore a range of trade-offs between model size and accuracy. In my view, this clears the bar for novelty.\n\n**Results:** solid results on image classification on a range of relevant models and datasets. Results on language models and music source separation are good but more limited. At least they add some degree of confidence on the applicability of the technique across different tasks.\n\n**Weaknesses and concerns:** \n\n1. DiffQ is only applied to weight quantization. This is effective for model compression but limits the applicability of this method to workload acceleration, which, in practical terms, requires the activations to be set to low-precision as well. However, it appears that recent work that builds up on the preprint of this very manuscript has already extended the work to activations as well (already cited: Park 2022), so my concerns are partially mitigated.\n\n2. Due to hardware requirements, mixed precision strategies have limited applicability compared to uniform precision methods. DiffQ can also be applied with fixed quantization but results presented in the supplementary material appears not to show consistent improvement (see next point). This takes from the value of the proposed technique.\n\n3. My main concern regards the claim that improved performance are to be attributed to having access to an end-to-end differentiable method, which avoids \"STE instabilities\". Although this claim is repeated throughout the manuscript, I don't think the authors have provided adequate evidence to support it. For example, fig 1b shows improved performance of DiffQ compared to 4 bit QAT but DiffQ relies on mixed precision while this instance of QAT does not, so the improvement can't be conclusively attributed to better handling of the backpropagation and better representation of the optimal weights by DiffQ. A fair comparison would be between models at fixed precision. When the authors did these experiments (Table B.4 suppl. material), results are not conclusive, *with DiffQ performing sometimes better, sometimes comparable, sometimes worse than their counterpart with regular quantizers set at an equivalent precision*. I can see the argument set in Section 3.3 (on a toy model), that at any given time QAT weights may be sub-optimal due to oscillations. I just don't think the improvements reported in this paper can be conclusively linked to this potential instability and not in part or entirely to the use of mixed precision. It should also be noted that, in my view, the papers cited in support of such \"STE instabilities\" tend to hand-wave the subject entirely or, alternatively, they address very aggressive quantization (2, 3 bits), which do not apply to the examples DiffQ is compared to in this manuscript (4,5,8 bits).\n\n4. Use of gaussian noise instead of uniform noise (one of the main elements of novelty) appears to be really effective in improving performance but the authors provide very limited explanation of why this may be the case.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a model compression method (DiffQ) based on additive noise injection during training, as an alternative to the widely used parameters quantization. Such Pseudo Quantization Noise (PQN) mimics the level of noise associated with quantization, while providing a differentiable pathway (with respect to both inputs and number of bits) for gradient propagation. PQN differs from common quantization techniques which rely on the non-differentiable *round()* operator to discretize the parameters and thus, during training backprop, are typically forced to bypass the operator by means of a Straight Through Estimator.\n\nThe paper shows that DiffQ outperforms other quantization strategies on 3 different tasks (image classification, language modeling, music source separation), achieving the most favorable trade-offs in terms of accuracy versus model compression.\n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "As discussed in the previous section, my concerns at point 3 on supposed STE instabilities should be addressed, via a re-writing of the authors claims and/or with more conclusive experimental demonstrations and/or with better support from the literature.\n\nI have a favorable view of this paper for all the reasons outlined in the previous section but, as it stands, these claims are not properly supported and must be fixed for me to recommend acceptance.\n\n**Other minor changes**\n- Some more comments on the effectiveness of gaussian noise would be valuable\n\n- Quantization Aware Training refers to a class of techniques, while LSQ is a type of quantizer, which can be employed as part of QAT methods. In the various tables, what quantizers are used when results for QAT are reported? Are they symmetric / asymmetric, minmax quantizers, ...? It should be specified, at least as a table note / footnote.\n\n- in fig.3 the label \"bits bits\" is quite confusing and a better name is warranted. I believe this is in reference to the total bits extracted from eq.12?\n\n- I would recommend the authors to proofread the manuscript more carefully, as typos can be found, such as:\n    - pag 7: compering\n    - pag 10: symetric\n    - pag 10: archiecture\n    - pag 10: expanse", "broader_impact_concerns": "Broader Impact Concerns are not discussed in the manuscript. I don't have specific concerns or requests on my side either.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vRw6oU3JNT", "reviewer": "Reviewer_NAyT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The manuscript is easy to follow, with clear illustrations.\n2. The related work section of the manuscript is thorough.\n3. Useful information is also presented in the Appendix and the supplementary codes.\n4. Although DIFFQ is not the first in this research direction, I still think it is meaningful to explore more ways to replace the standard STE method.\n5. The experiments are conducted not only on computer vision tasks but also on NLP and audio source separation.\n6. Most previous quantization works are targeting a moderate accuracy range (mostly less than 80%) while models with very high accuracy can be harder to quantize. It's good to see that EfficientNetB3 and DeiT are also explored in the paper.\n\nWeaknesses:\n1. The DIFFQ method achieves marginal improvement over LSQ on ImageNet with different ResNets.\n2. The analysis of why STE is sub-optimal is from other papers. Given that previous works have already tried to leverage the pseudo quantization noise, the novelty of this manuscript is not particularly strong.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In the manuscript, the authors propose DIFFQ, which applies pseudo quantization noise during training to avoid using the standard STE method. \n\nApplications on various tasks, such as image classification, NLP, and audio source separation, are taken to validate the effectiveness of the DIFFQ method.\n\nThe experimental results show that DIFFQ can achieve comparable or better results compared to the previous works. Additionally, the authors use different and state-of-the-art neural architectures (EfficientNetB3, DeiT, etc) in their experiments, which strengthens the claim of better performance.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. In the manuscript the authors mention that DIFFQ is superior to STE. It would be also important to show/discuss/compare the computational cost of applying DIFFQ and standard QAT. For example, the latency per iteration, or the throughput.\n2. small issue, page 11, Figure 3, \"Layer groups wise size\" is not very clear.\n3. Overall I think the manuscript is decent, which can lead to an acceptance if the limitations mentioned in the reviews are well addressed.", "broader_impact_concerns": "I don't have concerns about the broader impact of this manuscript.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AipwmWNMY4", "reviewer": "Reviewer_d8XE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n+ The proposed method is easy to follow, inspired by a classical assumption in the signal processing area. \n\n+ The authors conducted experiments on multiple benchmarks, beyond image classification. \n\n\nCons:\n\n- Currently, the analysis is not convincing, the authors derive the method based on a historical assumption, but for neural networks, it is unclear how this assumption will affect the optimization. I expect to see some theoretical analysis on convergence. \n\n- Is this method applicable to activation quantization? Quantization on weights is not that hard compared to activation quantization. Therefore, it makes me question the real improvements over STE. An ablation study should be conducted. \n\n- The mixed precision on model size is not a hard problem since the former layers have very few parameters, making them easily learn high bits. However, these early layers do not bring less computation energy or latency saving than later layers. Hence, model size mixed precision is easier for high accuracy and less likely to really achieve hardware acceleration in practice. \n\n- The paper fails to compare recent works in QAT, they claimed the state of the arts QAT methods are proposed in the year 2020. The authors are strongly recommended to compare with works like [1, 2] (see ref below). \n\n[1] Liu Z, Cheng K T, Huang D, et al. Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 4942-4952.\n\n[2] Nagel M, Fournarakis M, Bondarenko Y, et al. Overcoming Oscillations in Quantization-Aware Training[J]. arXiv preprint arXiv:2203.11086, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors focus on quantization-aware training and propose introducing a Pseudo quantization noise formula. The noise injection method is supposed to emulate the effect of quantization. The noise form can also be combined into a framework to learn mixed precision networks.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The authors need to provide a theoretical justification of this method, and comprehensive comparison with existing SOTA. ", "broader_impact_concerns": "Nothing worth concern. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The response from the authors and new revision of the paper addressed the bulk of the reviewers' concerns about this paper. After reading the most recent revision, I have two recommended revisions that will help make the paper clearer.
1. Make it clearer that variable bit-width quantization of models has value beyond inference time computational savings. From reading some of the authors' comments, I think that one goal of this work is to train models that are smaller to communicate to edge devices, even if the models are subsequently unpacked to larger bit widths. The reviewers seem to have assumed that the only value to reduced bit widths (or, at least, the primary value) is lower computational costs during inference.
2. Say a bit more about the added Gaussian noise. Specifically, the current revision does not say how the variance of the noise is selected. Is it set to match the variance of the more theoretically justified uniform noise in Eq. 6? Also, the **main paper should make it clear that the noise in all experiments is Gaussian.** Do not bury this detail in the appendix.

Decision: Accept (poster)