Training with Quantization Noise for Extreme Model Compression
Authors: Pierre Stock
Angela Fan
Benjamin Graham
Edouard Grave
Rémi Gribonval
Herve Jegou
Armand Joulin
Venue: ICLR 2021 Poster (2021)
Keywords: Compression
Efficiency
Product Quantization

Abstract: We tackle the problem of producing compact models, maximizing their accuracy for a given model size. A standard solution is to train networks with Quantization Aware Training, where the weights are quantized during training and the gradients approximated with the Straight-Through Estimator. In this paper, we extend this approach to work with extreme compression methods where the approximations introduced by STE are severe. Our proposal is to only quantize a different random subset of weights during each forward, allowing for unbiased gradients to flow through the other weights. Controlling the amount of noise and its form allows for extreme compression rates while maintaining the performance of the original model. As a result we establish new state-of-the-art compromises between accuracy and model size both in natural language processing and image classification. For example, applying our method to state-of-the-art Transformer and ConvNet architectures, we can achieve 82.5% accuracy on MNLI by compressing RoBERTa to 14 MB and 80.0% top-1 accuracy on ImageNet by compressing an EfficientNet-B3 to 3.3 MB.

Published as a conference paper at ICLR 2021
TRAINING WITH QUANTIZATION NOISE FOR EXTREME MODEL COMPRESSION
Pierre Stock
Angela Fan"
Benjamin Graham
Facebook Al Research, Inria
Facebook Al Research, LORIA
Facebook Al Research
Edouard Grave
Rémi Gribonval Inria
Hervé Jégou Facebook Al Research
Armand Joulin
Facebook Al Research
Facebook AI Research
ABSTRACT
We tackle the problem of producing compact models, maximizing their accuracy for given model size. standard solution is train networks with Quantization Aware Training (Jacob et al. 2018). where the weights are quantized during training and the gradients approximated with the Straight-Througe Estimator (Bengio) ct al. 2013). In this paper, extend this approach to work beyond int fixed- point quantization with extreme compression methods where the approximations introduced by STE severe, such as Product Quantization. Our proposal is only quantize a different random subset of weights during each forward, allowing for unbiased gradients to flow through the other weights. Controlling the amount of noise and its form allows for extreme compression rates while maintaining the performance of the original model. As result we establish new state-of-the-art compromises between accuracy and model size both in natural language processing and image classification. For example, applying our method to state-of-the-us Transformer and ConvNet architectures, we can achieve 82.5% accuracy on MNLI by compressing RoBERTa to 14 MB and 80.0% top-1 accuracy on ImageNet by compressing EfficientNet-B3 to 3.3 MB.
INTRODUCTION
Many of the best performing neural network architectures in real-world applications have large number of parameters. For example, the current standard machine translation architecture, Trans- former (Vaswani et al., (007),, has layers that contain millions of parameters. Even models that are designed to jointly optimize the performance and the parameter efficiency, such as EfficientNets (Tan) Le. 2019), still require dozens to hundreds of megabytes, which limits their applications to domains like robotics virtual assistants.
Model compression schemes reduce the memory footprint of overparametrized models. Pruning (LeCun et al. (990) and distillation (Hinton et al. 2015) remove parameters by reducing the number of network weights. In contrast, quantization focuses on reducing the bits per weight. This makes quantization particularly interesting when compressing models that have already been carefully optimized in terms network architecture. Whereas deleting weights or whole hidden units will inevitably lead to drop in performance, demonstrate that quantizing the weights be performed with little to no loss in accuracy.
Popular postprocessing quantization methods, like scalar quantization, replace the floating-pons weights of a trained network by a lower-precision representation, like fixed-width integers (Vanhoucke 201T. These approaches achieve a good compression rate with the additional benefit of accelerating inference supporting hardware. However, the errors made by these approximations
"Equal contribution. Corresponding authors: patocke com, angel a fb. com I Univ Lyon, Inria, CNRS, ENS de Lyon, UCB Lyon 1, LIP UMR 5668, F-69342, Lyon, France !Code available at https: https//sss.........oooooo//////s quant_noise

---

Published as a conference paper at ICLR 2021
26
X wio Quank-Noise
X Training without Quant-Noise
Training with Quant-Noise
Training Time Weight Matrix
Quantization Weight Matrix
Training Time Weight Marix
Quanizzation Weight Matrix
Perplexity
24
22.
w/ Quant-Noise
Original Model
18
Size (MB)
10
Figure 1: Quant-Noise trains models to be resilient to inference- time quantization by mimicking the effect of the quantization method during training time. This allows for extreme compression rates without much loss in accuracy on a variety of tasks and benchmarks.
accumulate in the computations operated during the forward pass, inducing significant drop in performance (Stock et al. 2019).
A solution to address this drifting effect is to directly quantize the network during training. This raises two challenges. First, the discretization operators have a null gradient - the derivative with respect to the input is zero almost everywhere. This requires special workarounds to train network with these operators. The second challenge that often comes with these workarounds is the discrepancy that appears between the train and test functions implemented by the network. Quantization Aware Training (QAT) (Jacob et al.) 2018) resolves these issues by quantizing all the weights during the forward and using straight through estimator (STE) (Bengio al. 2013) to compute the gradient. This works when the error introduced by STE is small, like with int 8 quantization, but does not suffice in compression regimes where the approximation made by the compression is more severe.
In this work, we show that quantizing only a subset of weights instead of the entire network during training is more stable for high compression schemes. Indeed, by quantizing only random fraction of the network at each forward, most the weights are updated with unbiased gradients. Interestingly, we show that our method can employ a simpler quantization scheme during the training. This particularly useful for quantizers with trainable parameters, such as Product Quantizer (PQ). for which our quantization proxy is not parametrized. Our approach simply applies quantization noise, called Quant-Noise, random subset of the weights, see Figure [ We observe that this makes network resilient to various types of discretization methods: significantly improves the accuracy associated with (a) low precision representation of weights like int 8; and (b) state-of-the-ar PQ. Further, we demonstrate that Quant-Noise can be applied to existing trained networks as a post-proces pess-processing step, to improve the performance network after quantization.
In summary, this paper makes the following contributions:
We introduce the Quant-Noise technique to learn networks that are more resilient variety of quantization methods such as int 4, int and PQ;
Adding Quant-Noise to PQ leads to new state-of-the-an trade-offs between accuracy and model size. For instance, for natural language processing (NLP), we reach 82.5% accuracy on MNLI by compressing RoBERTa to 14 MB. Similarly for computer vision, we report 80.0% top-1 accuracy ImageNet by compressing an EfficientNet-B3 to 3.3 MB;
By combining PQ and int B to quantize weights and activations for networks trained with Quant-Noise, we obtain extreme compression with fixed- precision computation and achieve 79.8% top-1 accuracy ImageNet and 21.1 perplexity on WikiText-103.
2 RELATED WORK
Model compression. Many compression methods focus on efficient parameterization, via weight
pruning (LeCun et al., 1990, Li et al. 2016 Huang et al., 2018 Mittal et al.) 2018), weight sharing
(Dehghani et al. 2018 Ture et al. 2019 Lan et al.] 2019) with dedicated architectures (Tan)
& Le 2019 Zhang et al. 2017, Howard et al. 2019). Weight pruning is implemented during training (Louizos et al. 2017)
fine-tuning post-processing step (Han al. 2015 2016). Many pruning methods are unstructured, i.c., remove individual weights (LeCun et al., [9900 Molchanoy et al. 2017). On the other hand, structured pruning methods follow the structure of the weights to
2

---

Published as a conference paper at ICLR 2021
reduce both the memory footprint and the inference time of a model (Li et al. 2016 Luo et al. 2017 Fan et al. 2019). We refer the reader to Liu et al. (2018) for a review of different pruning strategies.
Other authors have worked on lightweight architectures, by modifying existing models (Zhang et al.) 2018; Wu et al. 2019; Sukhbaatar et al., (019a) or developing new networks, such as MobileNet (Howard et al. 2019), ShuffleNet (Zhang al. 2017), and EfficientNet (Tan & Le 2019 in vision.
Finally, knowledge distillation (Hinton et al.
2015) has been applied to sentence representation (Ture
et al. 2019: Sanh al.) [2019a] [Sun et al.
2019 Zhao et al. 2019 Viao et al. 2019), to reduce the
size of BERT model (Devlin et al. 2018
Quantization. There extensive studies of scalar quantization train networks with low- precision weights and activations (Courbariaux et al. [013 Courbariaux & Bengio 2016 Rastegari ct al. 2016 McDonnell 2018). These methods benefit from specialized hardware also improve the runtime during inference (Vanhoucke et al. [011). Other quantization methods such as Vector
Quantization (VQ) and PQ (Jegou et al. [00 D) quantize blocks of weights simultaneously to achieve higher compression rate (Stock et 2019 Gong et al. 2014 Joulin et al. (2016] Carreira-Perpaee
& Idelbayev] 2017). Closer to our work, several works have focused simultaneously training and quantizing a network (Jacob et al. 2018; Krishnamoorthi 2018 Gupta et al. 2015 Dong et al. 2019). Gupta et al.) (2015) assigns weights to a quantized bin stochastically which specific to scalar quantization, but allows training with fixed point arithmetic. Finally, our method can be interpreted as a form of Bayesian compression (Louizos et al. 2017), using the Bayesian interpretation of Dropout (Gal & Ghahramani 2016). As opposed to their work, we select our noise to match the weight transformation a target quantization methods without restricting it to a scale mixture prior.
QUANTIZING NEURAL NETWORKS
In this section, we present the principles of quantization, several standard quantization methods, and describe how to combine scalar and product quantization. For clarity, we focus on the case of a fixed real matrix W E R=xp, We suppose that this matrix is split into m q blocks bys
b
big
W =
(1)
b
Dmgg
where the nature of these blocks is determined by the quantization method. codebook is set of K vectors, i.e., C = [c[l]..... c[]]]. Quantization methods compress the matrix W by assigning to each block bμ an index that points to codeword e in codebook C, and storing the codebook C and the resulting indices (as the entries I of an index matrix I) instead of the real weights. During the
inference, they reconstruct an approximation W of the original matrix W such that b c[Lx]--
We distinguish scalar quantization, such as i intt, where each block byr consists of a single weight, from vector quantization, where several weights are quantized jointly.
3.1 FIXED-POINT SCALAR QUANTIZATION
Fixed-point scalar quantization methods replace floating-poin number representations by low- precision fixed-point representations. They simultaneously reduce model's memory footprint and accelerate inference by using fixed-point arithmetic on supporting hardware.
Fixed- Fixed-point scalar quantization operates blocks that represent single weight, i.e. bur Wa. at- Floating-poin weights are replaced by N bit fixed-point numbers (Gupta et al. 2015). with the extreme case of binarization where N' = (Courbariaux et al.) 2015). More precisely, the weights are rounded to one of 2 possible codewords. These codewords correspond to bins evenly spaced by scale factor and shifted by bias Each weight W is mapped to its nearest codeword by successively quantizing with z + round(W wa// + 2) and dequantizing with the inverse operation:
(round (Wtr/ + 2) =) x 8,
(2)
where we compute the scale and bias as:
W
min W
and round W/s).
2N
3

---

Published as a conference paper at ICLR 2021
We focus this uniform rounding scheme instead of other non-uniform schemes (Cho et al. 2018 Li et al. 2019), because it allows for exedtpoint arithmetic with implementations in Py Torch and Tensorflow (see Appendix). The compression rate is x32/N. The activations are also rounded to N-bit fixed-point numbers. With int for instance, this leads to x2 to x4 faster inference on dedicated hardware. In this work, consider both int and int quantization.
3.2 PRODUCT QUANTIZATION
Several quantization methods work on groups of weights, such as vectors, to benefit from the correlation induced by the structure of the network. In this work, we focus on Product Quantization for its good performance at extreme compression ratio (Stock et al. 2019).
Traditional PQ. In vector quantization methods, the blocks are predefined groups of weights instead of single weights. The codewords are groups of values, and the index matrix I maps groups of weights from the matrix W to these codewords. In this section, we present the Product Quantization framework as it generalizes both scalar and vector quantization. We consider the case where we apply PQ to the columns of W and thus assume that q = p.
Traditional vector quantization techniques split the matrix W into its p columns and learn a codebook on the resulting p vectors. Instead, Product Quantization splits each column into m subvectors and learns the same codebook for each of the resulting m p subvectors. Each quantized vector is subsequently obtained by assigning its subvectors to the nearest codeword in the codebook. Learning the codebook is traditionally done using l- k-means with a fixed number K of centroids, typically K 256 to store the index matrix using int 8. Thus, the objective function is written as:
W
Σ
||b c[[m]] 2
PQ shares representations between subvectors, which allows for higher compression rates than intN.
Iterative PQ. When quantizing full network rather than single matrix, extreme compression
with PQ induces a quantization drift as reconstruction error accumulates (Stock et al., 2019). Indeed, subsequent layers take as input the output of preceding layers, which modified by the quantization of the preceding layers. This creates a drift in the network activations, resulting in large losses of performance. A solution proposed by Stock al. (2019). which we call iterative PQ (iPQ), is to quantize layers sequentially from the lowest to the highest, and finetune the upper layers as the lower layers are quantized, under the supervision of the uncompressed (teacher) model. Codewords of each layer are finetuned by averaging the gradients of their assigned elements with gradient steps:
1
Σ
OL
c c -1 [Jol
(4)
(k.1)EJ
where Je = 1(k,1) c[Ta]] c). L is the loss function and r) > a learning rate. This adapts the upper layers to the drift appearing in their inputs, reducing the impact of the quantization approximation on the overall performance.
3.3 COMBINING TIXED-POINT WITH PRODUCT QUANTIZATION
fixed-point quantization and Product Quantization often regarded competing choices, but can be advantageously combined. Indeed, PQ/IQQ compresses the network by replacing vectors of weights by their assigned centroids, but these centroids are in floating-poin precision. Fixed-point quantization compresses both activations and weights to fixed-point representations. Combining both approaches means that the vectors of weights are mapped to centroids that are compressed to fixed-point representations, along with the activations. This benefits from the extreme compression ratio of iPQ and the finite-precision arithmetics of intN quantization.
More precisely, for a given matrix, we store the int representation of the K centroids of dimension along with the log, K representations of the centroid assignments of the m x p subvectors. The int 8 representation of the centroids is obtained with Eq. (2). The overall storage of the matrix and activations during a forward pass with batch size (recalling that the input dimension n) writes
M Kd + log, K mp 8 bits.
(5)
4

---

Published as a conference paper at ICLR 2021
In particular, when K 256, the centroid assignments are also stored in int which means that every value required for a forward pass is stored in an int format. We divide by the float 32 overhead of storing the centroids, although the storage requirement associated with the centroids is small compared to the cost of indexing the subvectors for standard networks. In contrast to iPQ alone where we only quantize the weights, also quantize the activations using int8. We evaluate this approach on both natural language processing and computer vision tasks in Section 5
4
METHOD
Deep networks are not exposed to the noise caused by the quantization drift during training, leading to suboptimal performance. A solution make the network robust quantization is introduce it during training. Quantization Aware Training (QAT) (Jacob et al.) 2018) exposes the network during training by quantizing weights during the forward pass. This transformation is not differentiable and gradients are approximated with a straight through estimator (STE) (Bengio et al. 2013) Courbariaux Bengio 2016). STE introduces a bias in the gradients that depends on level of quantization of the weights, and thus, the compression ratio. In this section, we propose a simple modification to control this induced bias with stochastic amelioration of QAT, called Quant-Noise. The idea is to quantize a randomly selected fraction of the weights instead of the full network as in QAT, leaving some unbiased gradients flow through unquantized weights. Our general formulation can simulate the effect of both quantization and of pruning during training.
4.1 TRAINING NETWORKS WITH QUANTIZATION NOISE
We consider the case of a real matrix W as in Section 3 During the training of a network, our proposed Quant-Noise method works follows: first, compute blocks ьạ related to target quantization method. Then, during each forward pass, we randomly select subset of these blocks and apply some distortion to them.
More formally, given a set of tuples of indices J C {(kk)) 1)) for 1 S k S m, 1 < | < and a distortion or noise function yp acting on block, we define an operator e.( J) such that, for each block by, we apply the following transformation:
(bb)) [)) ba
if (k; J. otherwise.
(bbal J) =
(6)
The noise function y simulates the change in the weights produced by the target quantization method (see Section 4.2 for details). We replace the matrix W by the resulting noisy matrix W Waoise during the forward pass to compute a noisy output Yuoises i.c.,
W moise = (bb(ba)) J))st and
y n'nise-
x
W
noise
where x an input vector. During the backward pass, we apply STE, which amounts to replacing the distorted weights W noise by their non-distorted counterparts. Note that our approach is equivalent to QAT when / containts all the tuples of indices. However, an advantage of Quant-Noise over QAT is that unbiased gradients continue to flow via blocks unaffected by the noise. As these blocks are randomly selected for each forward, we guarantee that each weight regularly sees gradients that are not affected by the nature of the function yo. As side effect, our quantization noise regularizes the network in a similar way as DropConnect (Wan et al. 2013) or LayerDrop (Fan et al. 2019)..
Composing quantization noises. As noise operators compositionally commutative, we can make a network robust to a combination of quantization methods by composing their noise operators:
(bb JJ) 1/1 4/2(bbs/ .).
This property is particularly useful to combine quantization with pruning operators during training, as well as combining scalar quantization with product quantization.
4.2 ADDING NOISE TO SPECIFIC QUANTIZATION METHODS
In this section, we propose several implementations of the noise function op for the quantization methods described in Section g We also show how handle pruning with it.
5

---

Published as a conference paper at ICLR 2021
Quantization Scheme
Language Modeling 16-layer Transformer
Image Classification EfficientNet-BB
Wikitext-105
ImageNet-Ik
Size
Compression
PPL
Size
Compression
Top-1
Uncompressed model
942
18.3
46.7
1
81.5
int4 quantization
118
8
39.4
5.8
8
45.3
trained with QAT trained with Quant-Noise
118
34.1
5.8
8
59.4
118
X 8
21.8
5.8
x 8
67.8
int 8 quantization
236
19.6
11.7
80.7
trained with QAT
236
X 4
21.0
11.7
X 4
80.8
trained with Quant-Noise
236
4
18.7
11.7
X 4
80.9
iPQ
38
x 25
25.2
3.3
X 14
79.0
trained with QAT trained with Quant-Noise
38
X 25
41.2
3.3
X 14
55.7
38
25
20.7
3.3
X 14
80.0
iPQ & 8 + Quant-Noise
38
x 25
21.1
3.1
X 15
79.8
Table 1: Comparison of different quantization schemes with and without Quant-Noise on language mod- eling and image classification. For language modeling, we train Transformer on the Wikitext-103 benchmark and report perplexity (PPL) on test. For image classification, we train a EfficientNet-B3 on the ImageNet- benchmark and report top-1 accuracy original implementation of Tan & Le (2019) achieves uncompressed Top-1 accuracy of 81.9%. For both settings, report model size megabyte (MB) and compression ratio compared the original model.
validation and use re-implementation of EfficientNet-B3. The
Fixed-point scalar quantization. In intN quantization, the blocks are atomic and weights are rounded their nearest neighbor in the codebook. The function y replaces weight War with the output of the rounding function defined in Eq. ( i.e.,
PLnte (w) (round (// 2)
where & and z are updated during training. In particular, the application of Quant-Noise int8 scalar quantization is stochastic amelioration of QAT.
Product quantization. As opposed to intN, codebooks in PQ require a clustering step based on weight values. During training, learn codewords online and use the resulting centroids to implement the quantization noise. More precisely. the noise function pro assigns selected block to its nearest codeword in the associated codebook C:
(v) )- = argmin 1 cl1--
(10)
Updating the codebooks online works well. However, empirically, running k-means once per epoch faster and does not noticeably modify the resulting accuracy.
Note that computing the exact noise function for PQ is computationally demanding. We propose simpler and faster altermative approximation Pprxx to the operational transformation of PQ and iPQ. The noise function simply zeroes out the subvectors of the selected blocks, i.c., Ppraxy (v) = 0. As sidenote, we considered other alternatives, for instance one where the subvectors are mapped to the mean subvector. In practice,
found that these approximations lead to similar performance, Section 7.2 This proxy noise function is form of Structured Dropout and encourages correlations between the subvectors. This correlation is beneficial the subsequent clustering involved in PQ/iPQ.
Adding pruning to the quantization noise. The specific form of quantization noise can be ad- justed incorporate additional noise specific to pruning. We simply combine the noise operators of quantization and pruning by composing them following Eq. (8) We consider the pruning noise function of Fan et al. (2019) where they randomly drop predefined structures during training. In particular, we focus
LayerDrop, where the structures are the residual blocks of highway-like layers (Srivastava et al. 2015), as most modern architectures, such as ResNet or Transformer, are composed of this structure. More precisely, the corresponding noise operator over residual blocks v is PoayerrDop = 0. For pruning, we do not use STE to backpropagate the gradient of pruned weights, dropping them entirely during training has the benefit of speeding convergence (Huang

---

Published as a conference paper at ICLR 2021
Languago Modeling
ImageNet Top-1
24
X XL
85
Ideal
Dropx
Videal
X Xi
Accurny %
MobileBER MBERT
Perplexity
22
OOurs+
80
Ours
Aecuracy
Quts
FeNer BO
Ours
Mem.
Sharex
XHIA x Net-5
20
83
XTn BER
IPQ Resh Denso-16y
Tens core
X Layer XTr XL
OOurs+
Distil
Xcoobile-22
18
XDISTBERT
70
[deal
Corpp Try
AdaBERT
81
10
30
100
300
1000
10
30
100
300
1000
10 30 100
300 1000
Size (MB)
Size (MB)
Size (MB)
Figure 2: Performance as function of model size. We compare models quantized with PQ and trained with the related Quant-Noise to state the art. (a) Test perplexity on Wikitext- 103 (b) Dev Accuracy on MNLI (c) ImageNet Top-1 accuracy. Model size shown in megahytes on a log scale. Red gray coloring indicates existing work, with different colors for visual distinction.
Language modeling
Sentence Representation
Image
Classification
Comp.
Size
PPL
Comp.
Size
Acc.
Comp.
Size
Acc.
Unquantized models
Original model
X
1
942
18.3
X 1
480
84.8
1
46.7
81.5
Sharing
L.8
510
18.7
1.9
250
84.0
X 1.4
34.2
80.1
+ Pruning
3.7
255
22.5
x 3.8
125
81.3
X 1.6
29.5
78.5
Quantized models
iPQ
24.8
25.2
X
38
82.5
3.3
79.0
+ Quant-Noise
X
24.8
38
20.7
X 12.6
38
83.6
X
14.1
3.3
80.0
+ Sharing
X 49.5
19
22.0
X 34.3
14
82.5
X
18
2.6
78.9
+ Pruning
94.2
10
24.7
X 58.5
8
78.8
20
2.3
77.8
Table 2: Decomposing the impact of the different compression schemes. (a) we train Transformers with Adaptive Input and LayerDrop on Wikitext-109 103 (b) we pre-train RoBERTA hase models with LayerDrop and then finetune MNLI (e) train EfficientNet-BB -3 on ImageNet. We report the compression ratio w.r.t. to the original model ("comp.") ") and the resulting size in MB.
al. 2016). Once model is trained with LayerDrop, the number of layers kept inference can be adapted to match computation budget or time constraint.
RESULTS
We demonstrate the impact of Quant-Noise on the performance of several quantization schemes in variety settings (see Appendix Sec. 7.5)
5.1 IMPROVING COMPRESSION WITH QUANT-NOISE
Quant-Noise is regularization method that makes networks more robust the target quantization scheme or combination of quantization schemes during training. We show the impact of Quant-Noise in Table for variety of quantization methods: int 4int and iPQ.
We experiment in 2 different settings: a Transformer network trained for language modeling on WikiText-1033 and EfficientNet-B3 convolutional network trained for image classification on ImageNet- Our quantization noise framework is general and flexible Quant-Noise improves the performance of quantized models for every quantization scheme both experimental settings. Importantly, Quant-Noise only changes model training by adding regularization noise similar to dropout, with no impact on convergence and very limited impact on training speed (< 5% slower).
This comparison of different quantization schemes shows that Quant-Noise works particularly well with high performance quantization methods, like iPQ, where QAT tends to degrade the performances, compared to quantizing as a post-procee post-processing step. In subsequent experiments in this section, focus on applications with iPQ because it offers the best trade-off between model performance and compression, and has little negative impact FLOPS.

---

Published as a conference paper at ICLR 2021
Language Modeling
PPL
RoBERTa
Acc.
Train without Quant-Noise
25.2
Train without Quant-Noise
82.5
Finetune with Quant-Noise
20.9
+ Finetune with Quant-Noiss
83.4
Train with Quant-Nooie
20.7
Train with Quant-Noise
83.6
Table Quant-Noiss Finetuning vs training. We report performance after iPQ quantization. train with the Openy noise and finetune with Quant-Noise, and use it during the transfer to MNLI for each RoBERTa model.
Fixed-Poing Product Quantization. Combining iPQ and int 8 as described in Section 3.3 allows us take advantage of the high compression rate of iPQ with fixed-point representation of both centroids and activations. As shown Table [ this combination incurs little loss in accuracy with respect to iPQ Quant-Noise. Most of the memory footprint of iPQ comes from indexing and storing centroids, so the compression ratios are comparable.
Complementarity with Weight Pruning and Sharing. We analyze how Quant-Noise is compati- ble and complementary with pruning (* ("+Prune") and weight sharing ("+Share"), ) see Appendix for details on weight sharing. We report results for Language modeling on WikiText- 103, pre-trained sen- tence representations on MNLI and object classification on ImageNet- Ik Table [ The conclusions are remarkably consistent across tasks and benchmarks: Quant-Noise gives a large improvement over strong iPQ basclines. Combining it with sharing and pruning offers additional interesting operating points of performance vs size.
5.2 COMPARISON WITH THE STATE OF THE ART
We now compare our approach on the same tasks against the state of the art. We compare iPQ + Quant-Noise with 6 methods of network compression for Language modeling, 8 state-of-the-art methods for Text classification, and recent methods evaluate image classification on Imagenet with compressed models. These comparisons demonstrate that Quant-Noise leads to extreme compression rates a reasonable cost in accuracy. We apply our best quantization setup on competitive models and reduce their memory footprint by x20 94 when combining with weight sharing and pruning, offering extreme compression for good performance.
Natural Language Processing. In Figure 2 we examine the trade-off between performance and model size. Our quantized RoBERTa offers competitive trade-off between size and performance with memory reduction methods dedicated to BERT, like TinyBERT, MobileBERT, or AdaBERT.
Image Classification. We compress EfficientNet-B3 from 46. 7Mb to 3.3Mb (x14 compression) while maintaining high top-1 accuracy (78.5% versus 80% for the original model). As shown in Figure 2 our quantized EfficientNet-B3 is smaller and more accurate than architectures dedicated to optimize on-device performance with limited size like MobileNet or ShuffleNet. We further evaluate the beneficial effect of Quant-Noise on ResNet-50 to compare directly with Stock et al.) (2019). Results shown in Table 4 indicate improvement with Quant-Noise compared to previous work.
Incorporating pruning noise into quantization is also beneficial. For example, with pruning iPQ+Quant-Noite reduces size by x25 with only a drop of 2.4 PPL in language modeling. Further, pruning reduces FLOPS by the same ratio its compression factor, in our case, x2. By adding sharing with pruning, in language modeling, we achieve an extreme compression ratio of x94 with a drop of 6.4 PPL with FLOPS reduction from pruning entire shared chunks of layers. For comparison, our 10 MB model has the same performance as the 570 MB Transformer-X base.
5.3 FINETUNING WITH QUANT-NOISE FOR POST-PROCESSII QUANTIZATION
We explore taking existing models and post-processing with Quant-Noise instead of training from scratch. For language modeling, we train for 10 additional epochs. For RoBERTa, we train for 25k additional updates. Finetuning with Quant-Noise incorporates the benefits and almost matches training from scratch (Table In language modeling, there only a 0.2 PPL difference. We further examine how to incorporate Quant-Noise more flexibly into pretraining RoBERTa. We take an already
8

---

Published as a conference paper at ICLR 2021
trained RoBERTa model and incorporate Quant-Noise during sentence classification finetuning. This is effective at compressing while retaining accuracy after quantization.
CONCLUSION
We show that quantizing a random subset of weights during training maintains performance in the high quantization regime. We validate that Quant-Noise works with a variety of different quantization schemes on several applications in text and vision. Our method can be applied to a combination of iPQ and int to benefit from extreme compression ratio and fixed- fixed-point arithmetic. Finally, we show that Quant-Noise can be used as post-processing step to prepare already trained networks for subsequent quantization, to improve the performance of the compressed model.
REFERENCES
A. Adcock, V. Reis, M. Singh, Z. Yan, L. van der Maaten, K. Zhang, S. Motwani, J. Guerin, N. Goyal, Mísra, L. Gustafson, C. Changhan, and P. Goyal. Classy vision. 2019.
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. arXiv preprint arXiv: 1085, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 2013.
James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. Quasi-recurrent neural networks. arXiv preprint
2016.
Qingqing Cao, Harsh Trivedi, Aruna Balasubramanian, and Niranjan Balasubramanian. Faster and just as accurate: A simple decomposition for transformer models.
Miguel A. Carreira-Perpiñánn and Yerlan Idelbayev. Model compression constrained optimization, with application to neural nets. part ii: quantization, 2017.
Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang, Bofang Li, Bolin Ding, Hongbo Deng, Jun Huang, Wei Lin, and Jingren Zhou, Adabert: Task-adaptive bert compression with differentiable neural architecture search. arXiv preprint Xix::2001.04246, 2020.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv: 00005, 2018.
Matthieu Courbariaux and Yoshua Bengio. Binarynet: Training deep neural networks with weights and activations constrained +1 or -1. CORR, 2016.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. CoRR, 2015.
Zihang Dai, Zhilin Yang. Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Lc, and Ruslan Salakhutdinov. Transformerxl:: Attentive language models beyond fixed-length context. arXiv preprint ar X 028600 2019.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proc. of ICML, 2017.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Fukasz Kaiser. Universal transformers, 2018.
J. Deng, W. Dong, R. Socher, L.-J. Li. K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint / 1810004855 2018.

---

Published as a conference paper at ICLR 2021
Yinpeng Dong, Renkun Ni, Jianguo Li, Yurong Chen, Hang Su, and Jun Zhu. Stochastic quantization for learning accurate low-bit deep neural networks. International Journal of Computer Vision, 127 (11-12):1699-1642,, 2019.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with structured dropout. arXiv preprint arXiv:7909, 1556, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014.
Edouard Grave, Armand Joulin, Moustapha Cisse. David Grangier, and Herve Jegou. Efficient 2016.
softmax upproximation for gpus. arXiv,
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In ICML, 2015.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NIPS, pp. 1135-1143, 2015.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. ICLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0255: 2015.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoe V. Le, and Hartwig Adam. Searching for mobilenetv3. arXiv e-prints, 2019.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
Gao Huang, Shichen Liu, Laurens Van der Maaten, and Kilian Q Weinberger. Condensenet: An efficient densenet using learned group convolutions. In CVPR, 2018.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-nn inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704-27777 2018.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. PAMI, 201
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: 10351, 2019.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.0366 2016.
Raghuraman Krishnamoorthi, Quantizing deep convolutional networks for efficient inference: whitepaper. arXiv preprint 8rX0v:1806.08334, 2018.
Guillaume Lample and Alexis Conncau. Cross-lingual language model pretraining, arXiv preprint
/ 2019.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations, 2019.
10

---

Published as a conference paper at ICLR 2021
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In NIPS, 1990.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 2016.
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: A non-uniform discretization for neural networks. arXiv preprini arXiv:1909.1314: 2019.
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprini arXiv: 1907. 1692, 2019.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell, Rethinking the value of 2018.
network pruning. arXiv preprint
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXi::1608.0390 2016.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. arXiv preprint arXiv: 1712.0131, 2017.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. ICCV, 2017.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet V2: practical guidelines for efficient CNN architecture design. CoRR, 2018
Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Dawei Song, and Ming Zhou. A tensorized transformer for language modeling. arXiv preprint arXiv:1906 097777 2019.
Mark D. McDonnell. Training wide residual networks for deployment using a single bit for each weight, 2018.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer Sentinel Mixture Models. arXiv, (oss/6699,784,, 2016.
Deepak Mittal, Shweta Bhardwaj, Mitesh M Khapra, and Balaraman Ravindran. Recovering from random pruning: On the plasticity of deep convolutional neural networks. In WACV, 2018.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In ICML, 2017.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. How to construct deep recurrent neural networks. In Proceedings of the Second International Conference on Learning Representations (ICLR 2014), 2014.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for rongerange sequence modelling, arXiv preprint arX1v:1911.05507, 2019.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xeor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
11

---

Published as a conference paper at ICLR 2021
Liang-Chieh Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Conference on Computer Vision and Pattern Recognition, pp.
2018.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. 2019a.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprini arXiv: arXiv:1990.01 1910.01 2019b.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint 0rXiv:1505.00388, 2015.
Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou, And the bit goes down: Revisiting the quantization of neural networks. CoRR, abs/1907. (bss//99000686, 2019.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint urXiv:1905.07790 2019a.
Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. Aug- menting self-attention with persistent memory. arXiv preprint 1907,01470 2019b.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. EMNLP, 2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: Task-agnostic compression bert for resource limited devices.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. International conference on machine learning, pp. 1139-1147, 2013.
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks, 2019.
Julia Ture, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better:
The impact of student initialization on knowledge distillation. arXiv preprint 1908::908.00962 2019.
Vincent Vanhoucke, Andrew Senior, and Mark Z Mao. Improving the speed of neural networks on cpus. 2011.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using DropConnect. In ICML, 2013.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. 2019. ICLR.
Kuan Wang, Zhijian Liu, Yujun Lin andx Ji Lin, and Song Han. HAQ: hardware-aware automated quantization. CoRR, 2018.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACL-HLT, 2018.
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In ICLR, 2019.
Biao Zhang, Deyi Xiong, and Jinsong Su. Accelerating neural transformer via an average attention network. arXiv preprint arXiv: /805000633 2018.
12

---

Published as a conference paper at ICLR 2021
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. CoRR, 2017.
Sanqiang Zhao, Raghav Gupta, Yang Song, and Denny Zhou. Extreme language model compression with optimal subwords and shared projections. arXiv preprint arXiv: 2019.
13

---

Published as a conference paper at ICLR 2021
Setting
Model
Compression
Top-1 Accuracy
Stock et al. (2019)
19x
73.8
Small Blocks
Quant-Noise
19x
74.3
Stock al. (2019)
32x
68.2
Large Blocks
Quant-Noise
32x
68.8
Table Compression of ResNet-50 with Quant-Noise. We compare to Stock et al. (2019) both the small and large blocks regime. For fair comparison, we hold the compression rate constant. Quant-Noist provides improved performance in both settings.
iPQ
21
Non-Quantized
35
Non-Quantized Quantized
20
Quantized
Valid PPL
30
Valid
19
18
20
0.2
0.4
0.5
Quist-Noise rate
Quant-Noise rate
Figure 3: Effect of Quantization Parameters. We report the influence of the proportion of blocks to which we apply the noise. We focus on Transformer for Wikitext-10 103 language modeling. We explore two settings: iPQ and 3. For iPQ, we use SPproxy-
7 APPENDIX
7.1 QUANTIZATION OF ADDITIONAL ARCHITECTURES
ResNet-50. We explore the compression of ResNet-50, a standard architecture used Computer Vision. In Table compare Quant-Noise iPQ Compression from Stock al. (2019) and show that Quant-Noise provide consistent additional improvement.
7.2 ABLATIONS
In this section, we examine the impact the level of noise during training as well as the impact of approximating iPQ during training.
7.3 IMPACT NOISE RATE
We analyze the performance for various values of Quant-Noise in Figure 3 on Transformer for language modeling. For iPQ, performance impacted by high rates of quantization noise. For example, Transformer with the noise function Pproxy degrades with rate higher than 0.5, i.e., when half of the weights are passed through the noise function Prroxy We hypothesize that for large quantities of noise, larger effect of using proxy rather than the exact PQ noise is observed. For int8 quantization and its noise function, higher rates of noise are slightly worse but not as severe. A rate of for int 8 quantization is equivalent to the Quantization Aware Training of (Krishnamoorthi 2018), as the full matrix is quantized with STE, showing the potential benefit partial quantization during training.
7.4 IMPACT OF APPROXIMATING THE NOISE FUNCTION
We study the impact of approximating quantization noise during training. We focus on the case of iPQ with the approximation described in Section 4.2 In Table 5 we compare the correct noise function for iPQ with its approximation Pproxy This approximate noise function does not consider cluster assignments or centroid values and simply zeroes out the selected blocks. For completeness, include intermediate approximation where consider cluster assignments to apply noise
14

---

Published as a conference paper at ICLR 2021
Noise
Blocks
PPL.
Quant PPL.
Table Exact versus proxy noise function for
different block selections with iPQ. We com- pare exact Φro and the approximation @proxy with blocks selected from all subvectors or subvectors from the same cluster.
Subvectors
18.3
21.1
prQ
Clusters
18.3
21.2
Pproxy
Subvectors
18.3
21.0
Sppray
Clusters
18.4
21.1
within each cluster, but still zero-out the vectors. These approximations do not affect the performance of the quantized models. This suggests that increasing the correlation between subvectors that are jointly clustered is enough to maintain the performance of model quantized with iPQ. Since PQ tends work well on highly correlated vectors, such as activations convolutional networks, this is not surprising. Using the approximation Sproxy presents the advantage of speed and practicality. Indeed, one does not need to compute cluster assignments and centroids for every layer the network after each epoch. Moreover, the approach (Proxy is less involved in terms of code.
7.5 EXPERIMENTAL SETTING
We assess the effectiveness of Quant-Noise on competitive language and vision benchmarks. We consider Transformers for language modeling, RoBERTa for pre-training sentence representations, and EfficientNet for image classification. Our models are implemented in PyTorch (Paszke al.) 2017). We use fai rsir (Ott et al.) 2019) for language modeling and pre-training for sentence representation tasks and Classy Vision (Adcock et al. 2019) for EfficientNet.
Language Modeling. We experiment on the Wikitext 103 benchmark (Merity et al. 2016) that contains 100M tokens and vocabulary of 260k words. We train a 16 layer Transformer following Baevski & (218) with a LayerDrop rate of 0.2 (Fan et al. 2019). We report perplexity (PPL) the test set.
Pre- Pre-Training of Sentence Representations. We pre-train traitrin the base BERT model (Devlin et al. 2018) on the BooksCorpus + Wiki dataset with LayerDrop rate of 0.2. We finetune the pre-trained models on the MNLI task (Williams et al. 2018) from the GLUE Benchmark (Wang et al.) 2019) and report accuracy. We follow the parameters in Liu al. 2019) training and finetuning.
Image Classification. We train an EfficientNNr-B3 model (Tan & Le 2019) on the ImageNet object classification benchmark (Deng et al., (2009). The EfficientNet-B3 of C1 assy Vi a si achieves Top-1 I accuracy of 81.5%, which is slightly below than the performance of 81.9% reported by Tan &
Le 22119)
7.6 TRAINING DETAILS
Language Modeling To handle the large vocabulary of Wikitext-103, we follow (Dauphin et al. 2017) and (Baevski & Auli, 2018) in using adaptive softmax (Grave et al. 2016) and adaptive input for computational efficiency. For both input and output embeddings, we use dimension size 1024 and three adaptive bands: 20K, 40K, and 200K. We use cosine learning rate schedule (Baevski & Auli 2018 Loshchilov & Hutter 2016) and train with Nesterov's accelerated gradient (Sutskever et al.
2013 We set the momentum 0.99 and renormalize gradients if the norm exceeds 0.1 (Pascanu ct al. 2014). During training, we partition the data into blocks of contiguous tokens that ignore document boundaries. At test time, we respect sentence boundaries. We set LayerDrop to 0.2. We set Quant-Noise value to 0.05. During training time, we searched over the parameters (0.05, 0.1, 0.2) to determine the optimal value of Quant-Noise. During training time, the block size of Quant-Noise
RoBERTa The base architecture is 12 layer model with embedding size 768 and FFN size 3072. We follow (Liu et al.) 2019) in using the subword tokenization scheme from (Radford et al. 2019), which uses bytes as subword units. This eliminates unknown tokens. We train with large batches of size 8192 and maintain this batch size using gradient accumulation. We do not use next sentence prediction (Lample & Conneau] 2019). We optimize with Adam with a polynomial decay learning rate schedule. We set LayerDrop to 0.2. We set Quant-Noise value O. 1. We did not hyperparameter
15

---

Published as a conference paper at ICLR 2021
Model
MB
PPL
Trans XL Large Dai al.
2019)
970
18.3
Compressive Trans [Rac al
2019
970
17.1
GCNN (Dauphin et al. 2017
870
37.2
4 LLyer QRNN (Bradbury ct al
2016
575
33,0
Trans XI Base (Da el al. 2019
570
24.0
Persis Mem (Sukhbaatar
2019b)
506
20.6
Tensorized core-2 (Ma et al.
2019)
325
18.9
Quant-Noise
38
20.7
Quant-Noise + Share Prune
10
24.2
Table 6: Performance on Wikitext-103. We report test set perplexity and model size in megabytes.
Lower perplexity better.
search to determine the optimal value of Quant-Noise training RoBERTa is computationally intensive. During training time, the block size of Quant-Noise is 8.
During finetuning, hyperparameter search three learning rate options (1e-5, 2e-5, 3e-5) and batchsize (16 or 32 sentences). The other parameters are set following (Liu et al., 2019). We do single task finetuning, meaning only tune the data provided for the given natural language understanding task. We do not perform ensembling. When finetuning models trained with LayerDrop, apply LayerDrop and Quant-Noise during finetuning time well.
EfficientNet We use the architecture of EfficientNet-B3 defined in Classy Vision (Adcock al. 2019) and follow the default hyperparameters for training. We set Quant-Noise value to 0.1. During training time, we searched over the parameters (0.05, 0.1, 0.2) to determine the optimal value of Quant-Noise. During training time, the block size of Quant-Noise set to 4 for all x 1 convolutions, 9 for depth-wise 3 x 3 convolutions, 5 for depth- wise x 5 convolutions and 4 for the classifier. For sharing, we shared weights between blocks 9-10, 11-12, 14-15, 16-17, 19-20-21, 22-23 and refer to blocks that share the same weights chunk. For LayerDrop, drop the chunks of blocks defined previously with probability 0.2 and evaluate only with chunks 9-10, 14-15 and 19-20-21.
7.7 SCALAR QUANTIZATION DETAILS
We closely follow the methodology of PoTorch 1.4. We emulate scalar quantization by quantizing the weights and the activations. The scales and zero points of activations are determined by doing a few forward passes ahead of the evaluation and then fixed. We use the Histogram am method compute s and z, which aims at approximately minimizing the L2 quantization eтror by adjusting s and z. This scheme is refinement of the Mi nMax scheme. Per channel quantization also discussed in Table 10
7.8 IPQ QUANTIZATION DETAILS
Language Modeling We quantize FFN with block size 8, embeddings with block size 8, and attention with block size 4. We tuned the block size for attention between the values (4. 8) to find the best performance. Note that during training with apply Quant-Noise to all the layers.
RoBERTa We quantize FFN with block size 4, embeddings with block size 4, and attention with block size 4. We tuned the block size between the values (4, 8) to find the best performance. Note that during training with apply Quant-Noise to all the layers.
EfficientNet We quantize blocks sequentially and end up with the classifier. The block sizes are 4 for all convolutions, for depth-wise x 3 convolutions, 5 for depth-wise wise 5 x convolutions and 4 for the classifier. Note that during training with apply Quant-Noise InvertedResidual Blocks (except the Squeeze-Exee Excitatiin subblocks), the head convolution and the classifier.
all the weights in
16

---

Published as a conference paper at ICLR 2021
Model
MB
MNLI
RoBERTa Base LD Fan al.
2019
480
84.8
BERT Basc (Devlin et al. 2018
420
84.4
PreTraineed Distil Ture ct al. 2019
257
82.5
DistilBERT Sanh ct al. (2019b)
250
81.8
MobileBERT* et al.
96
84.4
TinyBERT [iao
[2099]
55
82.8
ALBERT Base (L.n
2019
45
81.6
AdaBERTY (Chen et al. 2020
36
81.6
Quant-Noiss
38
83.6
Quant-Noise Noint-Noiee-oa Share + Prune
14
82.5
Table 7: Performance on MNLI We report accuracy and size in megabytes. * indicates distillation using BERT Large. indicates training with data augmentation. Work from Sunn et al. (2019) and Zhao et al. (2019) do not report results the dev set. Cao et al. do not report model size. Higher accuracy is better.
Model
MB
Acc.
EfficientNet-B7 (Tan & Le
2019
260
84.4
ResNet-50 [ee 2013
97.5
76.1
DenseNet-169) [uang et a.
2018
53.4
76.2
EfficientNet-Bl/ lan & Lc 2019
20.2
77.3
MobileNet-v2 (Sandler ct al
2018
13.4
71.9
Shufflenet-v2 x (Ma ct
2018)
8.7
69.4
HAQ 4 bits (Wang et al.
2018
12.4
76.2
iPQ ResNet-SU (Stock et al.
2019
5.09
76.1
Quant-Nar Noise
3,3
80.0
Quant-Noise Share Prune
2.3
77.8
Table 8: Performance on ImageNet. We report accuracy and size in megabytes. Higher accuracy is better.
7.9 DETAILS OF PRUNING AND LAYER SHARING
We apply the Every Other Layer strategy from Fan et al. 2019). When combining layer sharing with pruning, we train models with shared layers and then prune chunks of shared layers. When sharing layers, the weights of adjacent layers are shared in chunks two. For a concrete example, imagine we have model with layers A, B, C, D, E, F, G, H. We share layers A and B, C and D, E and F, G and H. To prune, every other chunk would be pruned away, for example we could prune A, B, E,
7.10 NUMERICAL RESULTS FOR GRAPHICAL DIAGRAMS
report the numerical values displayed in Figures Table for language modeling, Table for BERT, and Table for ImageNet.
7.11
FURTHER ABLATIONS
7.11.1 IMPACT OF QUANT-NOIS FOR THE VISION SETUP
We provide another study showing the impact of the proportion of elements which to apply Quant-Noise in Table
7.11.2 IMPACT OF THE NUMBER OF CENTROIDS
We quantize with 256 centroids which represents a balance between size and representation capacity. The effect of the number of centroids performance and size is shown in Figure (a). Quantizing
17

---

Published as a conference paper at ICLR 2021
P
0.2
0.4
0.6
0.8
Top-1
80.66
80.83
80.82
80.88
80.92
80.64
Table 9: Effeet of Quantization Parameters. We report the influence of the Quant-Noise rate p with Scalar Quantization (int8). We focus on EfficientNet for ImageNet classification.
Efffcct of Number of Centrodds
Valid Perplexity
22
20
1000
2000
Number of Centroids
Figure 4: Quantizing with larger number of centroids. Results are shown on Wikitext-100 valid.
with more centroids improves perplexity - storage constraints.
this parameter could be adjusted based on the practical
7.11.3 EFFECT OF INITIAL MODEL SIZE
Large, overparameterized models more easily compressed. In Figure explore quantizing both shallower and skinnier models. For shallow models, the gap between quantized and non non-quantized perplexity does increase as layers removed (Figure[5) left). In contrast, there is larger gap in performance for models with smaller FFN (Figure right). As the FFN size decreases, the weights less redundant and more difficult quantize with iPQ.
7.11.4 DIFFICULTY OF QUANTIZING DIFFERENT MODEL STRUCTURES
Quantization is applied various portions of the Transformer architecture the embedding, attention, feedforward, and classifier output. We compare the quantizability of various portions of the network in this section.
Is the order of structures important? We quantize specific network structures first this is important as quantizing weight matrices can accumulate reconstruction error. Some structures of the network should be quantized last the finetuning process can better adjust the centroids. We find that there are small variations in performance based on quantization order (see Figure 6). We choose quantize FFN, then embeddings, and finally the attention matrices in Transformer networks.
Which structures be compressed the most? Finally, analyze which network structures can be most compressed. During quantization, various matrix block sizes can be chosen as parameter- the larger the block size, the more compression, but also the larger the potential reduction of performance. Thus, it is important to understand how much each network structure can be compressed to reduce the memory footprint the final model as much as possible. In Figure we quantize two model structures with a fixed block size and vary the block size of the third between and 32. As shown, the FFN and embedding structures are more robust to aggressive compression, while the attention drastically loses performance as larger block sizes are used.
7.11.5 APPROACH TO INTN SCALAR QUANTIZATION
We compare quantizing per-channel to using a histogram quantizer in Table I0 The histogram quantizer maintains a running min/max and minimizes L2 distance between quantized and non- quantized values to find the optimal min/max. Quantizing per channel learns scales and offsets as vectors along the channel dimension, which provides more flexibility since scales and offsets can be different.
18

---

Published as a conference paper at ICLR 2021
Model Depth
Model Width
Valid Perplexity
Valid Perplexity
30
23
20
14 16
1024
2048
3072
4096
Number of Layers
FFN Size
Valid PPL.
Quant Valid PPL
Figure 5: (a) Effect of Initial Model Size for more shallow models (b) Effect of Initial Model Size more skinny models
Order Quantization
Quantization Block Size
25.0
24.0
Valid Perplexity
Valid Perplexity
20.0
23.0
17.5
22.5
15.0
22.0
embemb fin ffn
16
32
emb fin att fin att emb fin emb ffn att emb all
Block Size
-O+ Attn
-O- Emb
- FFN
Figure 6: Effect of Quantization Model Structures. Results are shown the validation set of Wikitext-103. (a) Quantizing Attention, FFN, and Embeddings in different order. (b) More Extreme compression of different structures.
7.11.6 LAYERDROP WITH STE
For quantization noise, apply the straight through estimator (STE) remaining weights in the backward pass. We experiment with applying STE to the backward pass of LayerDrop's pruning noise, Results shown in Table I and find slightly worse results.
19

---

Published as a conference paper at ICLR 2021
Quantization Scheme
Language Modeling 16-layer Transformer
Image Classification EfficientNet-Bi
Wikitext-100
ImageNet-1K
Size
Compress
Test PPL
Size
Compress
Top-1 Acc.
Uncompressed model
942
x1
18.3
46.7
x1
81.5
Int4 Quant Histogram
118
x8
39.4
5.8
x8
45.3
Quant-Noise
118
x8
21.8
5.8
x8
67.8
Int4 Quant Channel Quise
118
x8
21.2
5.8
x8
68.2
118
19.5
5.8
x8
72.3
Int8 Quant Histogram
236
x4
19.6
11.7
x4
80.7
Quant-Noise
236
x4
18.7
11.7
80.9
Int8 Quant Channel
236
X4
18.5
11.7
X4
81.1
Quant-Noise
236
X4
18.3
11.7
81.2
Table 10: Comparison of different approaches to int4 and int with and without Quant- Noise on language modeling and image classification. For language modeling, we train Transformer on the Wikitext-103 benchmark. We report perplexity (PPL. on the test set. For image classification, we train a EfficientNet-B3 on the ImageNet-Ie 1K benchmark. We report top-1 accuracy on the validation set. For both setting, we also report model size in megabyte (MB) and the compression ratio compared to the original model.
Model
MB
PPL
Quant-Noise Share Prune
10
24.2
Quant-Noise Share + Prune with STE
24.5
Table 11: Performance on Wikitext-103 when using STE in the backward pass of the Layer- Drop pruning noise.
20


Reviews:
[{"review_id": "zHKXS4gw9-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a new compression-aware training method utilizing quantization noise as a regularization technique. During training, only a different random subset of weights is quantized for forward propagation such that controlling the amount of noise is a way to improve model accuracy when extreme compression is applied.\n\nEven though the motivation is interesting and the selection of recently developed models (including EfficientNet-B3 and RoBERTa) is reasonable, this reviewer has the following critical concerns:\n \n- It is hard to understand what the paper aims to describe because several equations in the paper seem to be very confusing or incorrect.\n1) Since $z$ is an integer in equation (2) (i.e. $z$ is a rounded value), $(round(W/s+z)-z)*s$ is equal to $c=round(W/s)*s$. Then, it seems that we just round the real-valued weights with only a scaling factor without zero-point. This reviewer is wondering whether $z$ means 'zero-point'. It is very confusing.\n2) In equation (5), as far as I understand, $n$ seems to be the term related to activation, but there is no explanation about $n$.\n3) The term in equation (8), $y_{noise} * x^T$, needs to be explained in the aspect of gradients. If we follow equation (8), then we may update the weights with only the product of input and output of a layer. This reviewer cannot understand what the authors mean by equation (8). To understand the training process of the proposed quant-noise method, equation (8) seems to be important. However, there is no explanation for how it was derived.\n4) Some scalar values (e.g. $W_{kl}$ or $I_{kl}$) are denoted by using matrix notations.\n\n- There is no enough analysis on why quant noise can improve the quantization accuracy. The training process can be affected by various hyperparameters such as learning rates, dropout prob., weight decay, and so on. The authors are encouraged to present how regularization-related hyper-parameters are affected and to clarify the effects of the quant-noise method on the model accuracy. The current manuscript lacks such an analysis. In addition, there are no graphs showing the training process with the proposed quant-noise method. The training graph with various learning rates or regularization parameters could have elaborated on the contributions of this paper.\n \n- The comparisons in Table 1 are not fair and detailed analysis and discussion of the experimental results are required. The authors show poor results trained with QAT and claim that the proposed method can improve the accuracy significantly. But, this reviewer cannot understand why the quantization-aware training method is not better than the post-processing method (i.e., intN quantization). Also, it should be clarified whether int4 quantization and int8 quantization are post-processing quantization or not.\nWhen training with QAT, it will be affected by many hyper-parameters and there may be a decrease in performance (PPL and Top-1). It should be discussed that the QAT results are well-trained results.  \n \n- Moreover, it is difficult to compare the effects of quant-noise because there have been no previous works that have performed quantization in the models described in the manuscript. The authors select three models to show the effects of the method those are Transformer LM, EfficientNet-B3 (why not B1/B2?), and Roberta. But, the rather well-known models (e.g., ResNet, MobileNet, BERT, and so on) could be better than the selected models in the manuscript. For example, a highly related paper, iPQ paper (Stock et al., 2019), used the ResNet-18 and 50 models. This reviewer is wondering why the authors do not choose the models for easy and fair comparisons. \n\n \n\nIn summary, this reviewer cannot estimate the advantages of the proposed technique because the training process is not described in detail and the experimental results are not fair.\n\n \n <Minor comments>\n\n- In Table 3, I think the specific model name should come instead of \"adaptive input\".\n\n- typos\n 3page:\n     (weights) simulatneously -> simultaneously\n     (Bayesian) intepretation -> interpretation\n 4page:    (and) learns -> learn\n 8page:     (These) comparison -> comparisons\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XtGXMln4whC", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Reasons for Score:\nThe work proposed in this paper is novel and very well presented. The claims are supported by experimental results on different neural network models and applications, showing a good trade-off between accuracy and neural network compression.\n\nStrengths:\n1.\tThe Quant-Noise idea for quantizing the weights and activations of neural networks is innovative, and it can be easily applied to various quantization methods for compressing the neural networks and saving energy. \n2.\tThe results are very good in terms of compression rate and accuracy, and the experiments are obtained on complex state-of-the-art neural network models and applications.\n3.\tThe combination of int8 and Product Quantization results in hardware-deployable compressed models.\n4.\tThe background and the methods used are clearly explained, also from the mathematical and formal point of view. \n5.\tThe paper is overall clear and well written. \n\nMinor Comments:\n1.\tWhich is the training time overhead of having quantization noise during training?\n2.\tThe percentage (%) should be indicated in the vertical axes of Figure 2.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KCB4cN6LlH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contribution:\n1. conduct Quantization-Aware-Training via introducing quantization noise during optimization. Further, Quant-Noise can be applied to existing trained networks\n2. The proposed method is validated on multiple models like Transformer, ConvNet, and more challenging EfficientNet-B3. Experiential results show that the proposed method is practical.\n\n\nCons:\nProduct Quantization (PQ) is not of much novelty since it was studied by Stock et al. This paper combines fixed-point and PQ and tries to claim that 'Fixed-point quantization and Product Quantization are often regarded as competing choices'. However, this claim is under-explained. \n \n\nOne of important novel points in this paper is subset quantization. A potential advantage of subset quantization is to mitigate the Gradient Mismatch problem caused by STE. This motivation makes sense to me because the gradient of unselected parameters is unbiased. This unbiased gradient will improve the quality of the gradient for the previous layers. However, a similar training scheme for both pruning and quantization was studied in related work [1, 2].\n\n[1] Guo, Yiwen, Anbang Yao, and Yurong Chen. \"Dynamic network surgery for efficient dnns.\" Advances in neural information processing systems. 2016. \n[2] Zhou, Aojun, et al. \"Incremental network quantization: Towards lossless cnns with low-precision weights.\" arXiv preprint arXiv:1702.03044 (2017). \n\nThere are some technical differences among these methods. However, considering such a similarity, more analysis is necessary to help us understand why subset quantization benefits (and helps reduce gradient mismatch). Otherwise, the novelty of this submission will be incremental. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "9cCz030j9v0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I agree with the three key contributions listed in the paper. The paper is well written and clearly articulates a contribution to the literature. The proposed Quant-Noise is intuitive and straightforward. The experimental evidence is provided for both image classification and language modeling tasks. Most of the related works are cited. The paper does not contain a theory part, but wherever possible, equations are provided to illustrate how the method works.\n\nConcerns:\nFor the experimental results, only one network is used to evaluate the proposed method. With different architecture, the accuracy gap between the full precision and quantized models could be different. It will be helpful to validate the proposed Quant-Noise on multiple networks. Moreover, the experimental evidence only shows results for INT4, INT8, and iPQ. As the proposed method should also work for fixed-point quantization and even binarization. The effectiveness of the proposed approach can be better justified by comparing the results with other SOTA fix-point quantization and binarization schemes in [1-4].\n\nI also have a question about the selection of the weights. The current scheme selects a random subset of the weights to add the proposed noise, which means the weights in different layers are chosen with the same probability. For pruning, existing work [5] finds that it is desired to prune different fractions of weights for each layer. Is there a better way to select the subset of the weights?\n\nReasons for score: Overall, I am leaning towards accepting the paper. I like the simple idea of controlling the imprecise gradients in BNN training by only quantizing a subset of weights at each iteration. However, fewer comparisons are made with other SOTA quantization/binarization methods. I would consider raising my score if the authors could address the aforementioned concerns.\n\n[1] Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm\n\n[2] Searching for Low-Bit Weights in Quantized Neural Networks\n\n[3] Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks\n\n[4] LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks\n\n[5] Pruning neural networks without any data by iteratively conserving synaptic flow", "extra_scores": {}, "extra_text": {}}, {"review_id": "0fiTY9cDs-o", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces Quant-Noise that quantizes a random fraction of the network at each step instead of quantizing the entire network. The experiments show that the proposed Quant-Noise can improve the accuracy of quantized neural networks. \n\nPros:\n1. The proposed technique is simple and easy to use.\n2. The proposed Quant-Noise shows very impressive accuracy improvements for int4 quantization. \n\t \nCons: \n1. The novelty of the proposed method is limited. Especially when combined with product quantization, Quant-Noise is essentially a form of structured dropout. I agree adding structured dropout may lead to some accuracy improvements. But I do not think this is one of the contributions of this paper. Besides, why Quant-Noise works for int4 quantization is not clear. \n2. The improvement of Quant-Noise for int8 quantization looks very limited. The proposed method seems to be useful for low-bit quantization, but not very effective for int8 quantization. If so, I think the authors should focus on low-bit quantization in the experiment section and test the proposed method on more low-bit quantization settings  (e.g., 2bits quantization, 3bits quantization, mix-precision quantization, etc).\n\nOverall, I think the novelty of the proposed method is limited and why it works is not very clear. Besides, I think the proposed method should be tested on more low-bit quantization settings.  I recommend \"Marginally below acceptance threshold\".", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Quantization is an important practical problem to address.  The proposed method which quantizes a different random subset of weights during each forward is simple and interesting. The empirical results on RoBERTa and EfficientNet-B3 are good, in particular, for int4 quantization.  During the rebuttal, the authors further included quantization results on ResNet which were suggested by the reviewers. This additional experiment is important for comparing  this proposed approach with the existing methods which do not have quantization results on the models in this paper. 
Decision: Accept (Poster)