MixQuant: A Quantization Bit-width Search that Can Optimize the Performance of your Quantization Method
Authors: Eliska Kloberdanz
Wei Le
Venue: Submitted to ICLR 2023 (2023)
Keywords: neural network quantization
rounding error
bit-width search

Abstract: Quantization is a technique for creating efficient Deep Neural Networks (DNNs), which involves performing computations and storing tensors at lower bit-widths than f32 floating point precision. Quantization reduces model size and inference latency, and therefore allows for DNNs to be deployed on platforms with constrained computational resources and real-time systems. However, quantization can lead to numerical instability caused by roundoff error which leads to inaccurate computations and therefore, a decrease in quantized model accuracy. In this paper we focus on simulated quantized inference, where the quantized model parameters are stored in low-precision, but the mathematical operations on them (e.g. matrix multiplications and additions) are performed with floating point arithmetic. This means that the DNN parameters are first quantized from f32 to, for example, int4, and then dequantized back to f32 to perform computations. We show that the roundtrip process of quantizing and dequantizing the model parameters leads to roundoff error, which may lead to numerical instability. Similarly to prior works, which have shown that both biases and activations are more sensitive to quantization and are best kept in full precision or quantized with higher bit-widths, we show that some weights are more sensitive than others which should be reflected on their quantization bit-width. To that end we propose MixQuant, a search algorithm that finds the optimal custom quantization bit-width for each layer weight based on roundoff error and can be combined with any quantization method as a form of pre-processing optimization. We show that combining MixQuant with BRECQ, a state-of-the-art quantization method, yields better quantized model accuracy than BRECQ alone. Additionally, we combine MixQuant with vanilla asymmetric quantization to show that MixQuant has the potential to optimize the performance of any quantization technique.

Under review as a conference paper at ICLR 2023
MIXQUANT: A QUANTIZATION BIT- WIT-WIDTH SEARCH THAT CAN OPTIMIZE THE PERFORMANCE OF YOUR QUANTIZATION METHOD
Anonymous authors Paper under doublc-blind review
ABSTRACT
Quantization a technique for creating efficient Deep Neural Networks (DNNs), which involves performing computations and storing tensors at lower bit-widths than f32 floating point precision. Quantization reduces model size and inference latency, and therefore allows for DNNs to be deployed on platforms with con- strained computational resources and real-time systems. However, quantization can lead to numerical instability caused by roundoff error which leads to inaccu- rate computations and therefore, a decrease in quantized model accuracy. In this paper we focus on simulated quantized inference, where the quantized model pa- rameters are stored in low- lowpprccision, but the mathematical operations on them (c.g. matrix multiplications and additions) are performed with floating point arithmetic. This means that the DNN parameters are first quantized from 132 to, for example, int4, and then dequantized back f32 to perform computations. We show that the roundtrip process of quantizing and dequantizing the model parameters leads to roundoff error, which may lead to numerical instability. Similarly to prior works, which have shown that both biases and activations are more sensitive to quanti- zation and are best kept in full precision or quantized with higher bit-widths, we show that some weights are more sensitive than others which should be reflected on their quantization bit-width. To that end we propose MixQuant, a search algo- rithm that finds the optimal custom quantization bit-width for each layer weight based on roundoff error and can be combined with any quantization method as a form of pre-processing optimization. We show that combining MixQuant with BRECQ, a state-of-the-art quantization method, yields better quantized model accuracy than BRECQ alone. Additionally, we combine MixQuant with vanilla asymmetric quantization to show that MixQuant has the potential to optimize the performance of any quantization technique.
INTRODUCTION
Quantization is a method for mapping continuous values to a set of discrete values. The goal of neural network quantization is perform computations and store tensors lower bit-widths than floating point precision to reduce model size and inference latency while maintaining model ac- curacy, which allows for deploying DNNs on platforms with constrained computational resources, e.g.: real time inference on mobile devices. Quantization can be performed during training or infer- ence. In this paper we focus on quantized inference, specifically post-training quantization, which quantizes a full precision trained model without the need for re-training or fine-tuning.
Quantized inference can be either simulated or integer-only, and this paper we focus on simulated quantization, where the quantized model parameters are stored in oowprrcciiion, but the mathemati- cal operations on them (e.g. matrix multiplications and additions) are performed with floating point arithmetic (Gholami et al., 2022). In Tensorflow, PyTorch, and HuggingFace (QDQBERT model), simulated quantization is referred to as fake quantization. This means that the DNN parameters first quantized from f32 to, for example, int4, and then dequantized back to f32 perform the forward pass executed during inference. We show that the roundtrip process of quantizing and dequantizing the model parameters leads to roundoff error, which may lead to numerical instability.

---

Under review as a conference paper at ICLR 2023
Similarly to prior works, which have shown that both biases and activations are more sensitive to quantization and are best kept in full precision or quantized with higher bit-widths (Zhou et al, 2016), we show that some weights are more sensitive than others which should be reflected on their quantization bit-width. To that end we propose MixQuant, a search algorithm that finds the optimal quantization bit-width from int2, int3, int4, int5, int6, int7, and int8 for each layer weight based on roundoff error and can be combined with any quantization method as a form of pre-processing optimization. We show that combining MixQuant with BRECQ (Li al. 2021), state-of-the-art quantization method, yields better quantized model accuracy than BRECQ alone. Additionally, we combine MixQuant with vanilla asymmetric quantization to show that MixQuant has the potential to optimize the performance of any quantization technique.
MixQuant has three main benefits. First, MixQuant component of the quantization process, which can be leveraged to find optimal quantization mixed precision bit-widths that can be plugged into any quantization method to optimize its performance. Second, MixQuant is linear and runs in matter of seconds, which makes it practical. Third, combining MixQuant with BRECQ, state- of-the-art quantization method yields better quantized model accuracy than BRECQ alone, OMSE [Choukroun et al.] 2019). AdaRound (Nagel et al. 2020). AdaQuant (Hubara al. 2020 and Bit-Split (Wang et al. 2020).
2 RELATED WORK
Neural Network Quantization Neural network quantization can be applied to training Guutt
et al. 2015 Zhou et al. 2016 Hubara et al. 2017 Bartan & Pilanci| 2021 Elthakeb et al. 2020) or
inference. There are two paradigms in quantized DNN inference: post- tost-trainig quantization (PTQ) and quantization-aware training (QAT) (Jacob et al., 2018 Tailor et al., 2021). In contrast PTQ, QAT requires that the f32 model is retrained while simulating quantized inference in the forward pass. While MixQuant can be integrated with either, we focus on PTQ which does not require any re-training.
Hubara et al. (2021) and Li al. (2021) are amongst the current state-of-the-art post training quan- tization works. Hubara et al. (2021) introduce AdaQuant, which finds optimal quantization for both weights and activations and based on minimizing the error between quantized layer outputs and f32 layer outputs. This approach is similar to MixQuant; however, MixQuant finds the optimal quantization bit-widths based on quantization error (QE) minimization, while AdaQuant treats the bit-width as a constant and quantizes all weights and activations using the same bit-width (either int8 or int4). Li et al. (2021) 1) propose BRECQ, a quantization method based on DNN block reconstruc- tion. Nagel et al. 2020) propose AdaRound, adaptive rounding for weights, which achieves better accuracy than rounding to the nearest. They formulate the rounding procedure as an optimization problem that minimizes the expected difference between model loss with and without weights quan- tization perturbation. [iet al. (020)) develop a method based on constraining all quantization levels the sum of Powers-of-Twoo terms, Wang al. (2020) propose Bit-Split and Stitching framework (Bit-split), Nahshan Saisssp et al. (2021) study the effect of quantization on the structure of the loss land- scape, [Banner et al. (201)) develop ACIQ-Mix, a bit convolutional neural network quantization, and Cai et al. (2020) perform zero-shot quantization ZeroQ based on distilling a dataset that matches the input data distribution.
Quantization originated with convolutional neural networks, but it has been extended to natural language processing neural networks as well. Chen & Sun (2020) propose differentiable product quantization, a learnable compression for embedding layers in DNNs. Kim et al. (202T) study an integer-only quantization scheme for transformers, where the entire inference performed with pure integer arithmetic.
Other works studied hardware optimization for quantization or the relationship between quantiza- tion and adversarial robustness. Han et al. (2020) focus on performance optimization for Low-bit Convolution on ARM CPU and NVIDIA GPU. Fu et al. (2021) investigate quantized models' ad- versarial robustness. They find that when adversarialy trained model is quantized to different precisions in post-training manner, the associated adversarial attacks transfer poorly between dif- ferent precisions.
2

---

Under review as a conference paper at ICLR 2023
Mixed Precision Quantization In this paper we focus mixed precision quantization. There are only a few prior works that focus on mixed precision quantization since most focus on single precision quantization, where the quantization bit-width of all weights uniform and therefore; treated as a constant. Waan et al. (2019) propose a framework for determining the quantization policy with mixed precision and reinforcement learning, but compared MixQuant it requires sig- nificantly more overhead (hardware simulators and reinforcement learning). Liang (020) focuses on mixed precision quantization of activations and distinguishes between key and non-key activa- tions to assign 8- 8-bit and 4-bit precision respectively. In contrast to MixQuant, which searches for weights mixed precision from 8 to 2 bits, [Liang] (2020) is limited choice between 4 and 8 bits and applies only to activations while all weights are quantized with 8-bit precision. The primary focus of Wu al. (018) is neural architecture search, which can also be used for mixed precision quantization. However, their search on ResNet 18 for ImageNet takes 5 hours, while MixQuant runs in order of a few seconds. Liu et al. (202I) use single precision for weights, where the mixed pre- cision is represented only by selecting a different bit-width for weights than activations. Liu et al. (021 is the most most recent, and we show that MixQuant yields better accuracy.
Another mixed precision quantization work that build is Lin et al. (2016), who identify op- timal bit-width allocation across DNN layers. However, there are two primary differences between Lin al. (2016) and our work: (1) Lin et al.] (2016) focus fixed- point precision, not integer precision, Lin et al. (2016) different method for finding layer bit-widths based on predicted -ratio. Moreover, while they find that on CIFAR- convolutional DNN is able to achieve 20 % model size reduction; their AlexNet experiments on ImageNet- -e00eett-0000000 achieve less than 1% model reduction. In this work we are able successfully leverage mixed precision optimal bit-width allocation on ImageNet- 1000 models.
signal-quamtization-oooo
QUANTIZATION AND NUMERICAL INSTABILITY
Quantization involves lowering the bit-width numeric tensor representation, which can cause numerical instability that leads inaccurate outputs (Kloberdanz al. 2022). In general, numerical instability arises due to two types of numerical errors: (1) roundoff errors and (2) truncation errors. Roundoff errors are caused by approximating real numbers with finite precision, while truncation errors are caused by approximating an iterative mathematical process with only finite number of iterations. We argue that quantization can significantly amplify the roundoff error, which leads to degradation in quantized DNN accuracy.
DNN training and inference is typically performed in f32 precison, which already introduces round- off errors, because it has only 32 bits to represent real numbers. Specifically, f32 can represent a zero and numbers from -3.40282347E++38 3.40 3.40082347E+38, but numbers outside of this range are not representable in f32. In simulated quantization the process quantizing DNN parameters from f32 to int (e.g.: int4) and dequantizing them back to f32 to perform matrix multiply and add (e.g.: inputs weights biases) can lead to loss of precision.
-1.17549435B-38 and from 1.17549435E-38 to
Listing shows example of simple simulated quantized inference, where the weights tensor is quantized to int2 and its subsequent dequantization back to f32 has a roundoff error. The roundoff occurs in the second element of the weight tensor, which becomes 0.0 (line 40) while its true original value is 0.01 (line 38). This error caused by quantization then propagates further the computation inputs weights biases returns 1 1.0000e-05 (line 43) instead of 1.2000e-05 (line 42) in the second element of the result tensor.
Listing 1: Loss of Precision due to Quantization Example
2 def scale (r, bits ):
3 min_r r.
4
r. ()
5
qmin =I1 (2 ** (bits 1))
6
qmax = 2 a* (bits - 1)
7
scale_r = (max_r min.r) / (qmax - qmin)
8
return scale_r
9
3

---

Under review as a conference paper at ICLR 2023
10 def zero-point bits
seale_r scale (r, bits)
min_r min
13
qmin = -I (2 se (bits - 1))
14
zpt.r qmin (min_ seale_r)
return zpt_r
def quant bits):
18
2 (r bits)
19
scale bits)
20
q round round (T/s) z). zn.int(
21
return 9
22
23
def dequant (q, s): (q 2)
24
return r. float
26
27
input torch tensor (10.005, 0.0002, 0.01, 0.003])
28 bias torch tensor (1O. (10.00001)
29 weight torch tensor 0.01, 1.0, 2.0]) # original weight
31 S scale (weight, 2) quantization scale
32 Z zero point (weight. 2) # quantization zero point
33 q geeeghht quant 2) quantized weight
dq-weight dequant
S) dequantized weight
36 result = input * weight + bias dq_result input deigh + bias
37
38
B32 weight: tensor ([=.0000000000 0.0100, 1.0000, 2.0000])
40
quantized weight: tensor ([-,, -1, 0, 1]. dtype=torch. .nt.o) in132)
41
dequantized weight: censor([-1. 0., 1.. 2.))
42
f32 result: (cnsor((-4.9900e-0) 1.2000e-05, 1.0010c-02,
6.0100e-03)
simulated quantization result: tensor (I-4.9900c-00) 1.0000e-05, 1.0010c-02, 6.0100c-03)
4 MIXQUANT
MixQuant is quantization scheme that relies on mixed precision to find the bit-widths of individual layer weights that minimize roundoff error and therefore, minimize model accuracy degradation due to quantization. Specifically, MixQuant is search algorithm that finds optimal bit-widths that min- imize model accuracy degradation caused by quantization. Prior works have shown that biases and activations
sensitive to quantization than weights, and therefore typically kept in higher precision. In this paper we argue some weights are more sensitive to quantization than others, which show in our ablation studies. This warrants careful bit-width allocation individual weights and serves as motivation for MixQuant. In essence, MixQuant can be viewed as an additional pre- processing optimization component of the quantization process, which can be combined with any quantization method optimize its performance.
MixQuant described in Algorithm|] | The optimal weight layer bit-widths search has two primary components: layer-wise QE minimization and QE multiplier (QEM). The layer-wise QE is cal- culated as the mean squared error (MSE) between the f32 model weights and the weights that have been dequantized following an int quantization (any quantization method can be used at line 8 in Algorithm [ to capture the information loss due to roundoff error caused by quantization. This error calculated for each layer for each bit-width from the following list: 8, 7, 6. 5, 4, 3, and (lines 4-11 in Algorithm|]] T). Following that, MixQuant searches for the optimal bit bit-width for each layer by comparing the QE of each bit- witth from this list with int8 error, which serves I). To push MixQuant to select bit-widths lower than int8, MixQuant leverages the QEM. If the QE at width b is less equal int8 QE multiplied by the QEM, b becomes
baseline (lines
12-13 3 in Algorithm|]
4

---

Under review as a conference paper at ICLR 2023
the optimal bit-width for that layer. This can be expressed as optimization problem:
Bit = arg mit QuantErrors
Subject to quantErrors I 8bit, * QEM
(1)
opt.Bit E B
Because the QEM is an imput parameter into MixQuant, it allows the user to specify a custom trade- off between quantization bit-width and model accuracy; and therefore, it allows the user to find their optimal layer bit-width.
Algorithm MixQuant
1: Input: full precision weights W, bit-widths B, QE multiplier QEM
2: Initialize optimal BitW idths
1* Iterate over all layers */
3: for in layers do
4: 8bitw Quantize(W', bitWidth 8)
1* Compute int8 quantization error in layer */
5: Sbitr W Dequantize(Sbitty)
1* For every bit-width in B compute quantization error in layer */
6: Initialize muan--Error for bilW idth in B do
7:
8: guantizedW Quantize(W, bitWidth)
9:
gError W Dequantiz(quantizeeWW)
10:
Append g Error to quant end for
11:
1* Select optimal bit-width at layer 1 */
12:
Bit arg min quantErrorrs quantErroors < 8hily Error QEM,
Bit
13: Append opt to optimal idths end for
15:
return optimal Bit idths
Weights Mixed Precision Quantization We focus on weights quantization for three reasons. First, weights account for majority of parameters in DNN and therefore, have the greatest im- pact on model size and inference time. Second, model accuracy is more sensitive to quantized activations than weights (Zhou al.) 2016). Third, we guided our algorithm design with the state- of-the art results in table 2 in Li et al. 2021). who introduced BRECQ which shows weight-only quantization.
Approximating Roundoff Error We use the QE (measured as the MSE between f32 and dequan- tized weights) to approximate the impact of quantization on model accuracy for three reasons. First, prior works have leveraged quantization error proxy for quantized model accuracy |Banner et al. 2019) used quantization MSE to approximate optimal clipping value (ACIQ) and optimal bit-width for each channel. Second, provide empirical evidence that there is negative relation- ship between model accuracy and quantization error (see Figure 6] Appendix). Third, computing layer-wise quantization error instead of determining the model accuracy with respect to each layer and each possible layer bit-width has the advantage of linear time complexity. An exhaustive com- binatorial search runs in exponential time (Wu et 2018).
We analyze the algorithm's 's' time complexity by considering its two logical components the error calculations and the bit-width search based on them. Let be the total number of layers, B the total number of bit-widths, and M the total of QEMs. We calculate the QE of each layer for each widtw..th. Thus, the time complexity of MixQuant's error calculations (line 4-11 in Algorithm|] [] is O(L B). The bit-width search (line 12- 13 in Algorithm [1] compares the QE of each bit-width to the baseline int8 QE for each layer and can be performed for number of QEMs, which takes (I. (B M)).
Time Complexity Analysis
5

---

Under review as a conference paper at ICLR 2023
Table 1: Model accuracy comparison of MixQuant combined with BRECQ and BRECQ alone
Bits W/A
MixQuant + BRECQ
vs MixQuant
Bits
BRECQ
vs BRECQ
+ BRECQ
W/A
32/32
69.76
32/32
71.08
5, 6/32
70.69
0.93
4/32
70.7
-0.38
R8sNet-18
4, 5, 6/32
70.69
0.93
3/32
69.81
-1.27
2, 6/32
68.93
-0.83
2/32
66.3
-4.78
32/32
71.88
32/32
72.49
4,5, 7/32
71.92
0.04
4/32
71.66
-0.83
MobileNetV2
4, 7/32
71.92
0.04
3/32
69.5
-2,99
2,5, 6/32
59.53
-12.35
2/32
59.67
-12,82
Table 2: Model accuracy comparison MixQuant combined with BRECQ and Liu et al. (2021)
Bits
MixQuant BRECQ
[32 vs MixQuant
Bits
Liu et al. (2021)
132 vs Liu al. (2021)
W/A
BRECQ
WAA
32/32
69.76
32/32
74.24
4. S, 6/32
70.69
0.93
4/8
61.68
-12.56
ResNet-
6/3/2 6/32
70.69
0.93
4/8
61.68
-12.56
5,
68.93
-0.83
4/8
61.68
-12.56
32/32
71.88
71.78
4, S, 6, 7/32
71.92
0.04
8/8
70.7
-3.54
MobileNetV2
4, 5, 6, 7/32
71.92
0,04
8/8
70.7
-3.54
2, 6/32
59.53
-12.35
8/8
70.7
-3.54
Therefore, the overall time complexity of MixQuant is, which is linear with respect to the number of layers:
OL * B) + O * M)) = (
M))
(2)
we used model loss instead of layer QE to search for optimal bits, we would need to consider all the models generated via the combinations of number of bit-widths over L number of layers. The time complexity would be O(B)), which is exponential.
5 RESULTS
We implement MixQuant using Python and combine it with two types of quantization techniques: (1) BRECQ (Li al. (02T), state-of-the-art quantization method, and (2) vanilla asymmetric quantization (Jacob 2018)) and evaluate it on the validation set of the Imagenet ILSVRC2012 dataset. Our results demonstrate that MixQuant can optimize the performance of cxisting quantiza- tion techniques.
MixQuant with BRECQ BRECQ is state-of-the art quantization method that has been shown to outperform OMSE [Choukroun et al., 2019). AdaRound (Nagel et al., (020), AdaQuant (Hubara al.] 2020). and Bit-Split (Wang et al. (020), and in Table [ we demonstrate in that when MixQuant is combined with BRECQ, we achieve better quantized accuracy than BRECQ alone. Additionally, in Table 2 we compare results with (Liu et al., (021), state of the art mixed pre- cision quantization technique, and show that the accuracy degradation is significantly greater in Liu
et al. 20021)
MixQuant with Asymmetric Quantization In addition to BRECQ, we combine MixQuant with asymmetric quantization and compare its quantized model accuracy with f32 and int8 baselines. Table 3 shows the set of bit-widths found via MixQuant for various QEMs and various ResNet ar- chitectures along with model top-1 and top-5 accuracy. A user can flexibly select the quantization solution based their requirements with the QEM. For higher QEMs the bit-widths are lower and the model accuracy decreases, while for lower QEMs the bit-widths and quantized model accu- racy higher. Therefore, MixQuant allows its user flexibly select the trade-off between model

---

Under review as a conference paper at ICLR 2023
Table 3: MixQuant Results: quantization bit-widths, quantized model accuracy, loss and quantiza- tion mean squared error for various quantitative enror multipliers
Architecture
Experiment
QEM
layers bit widths
Acc@
Acc@
loss avg
QMSE
resnet18 8
baseline: 132
N/A
all layers are float 32
69.76
89.08
1.25
N/A
resnet18 8
baseline: int8
N/A
all layers are int 8
69.63
89.07
1.25
N/A
resnet18
6.
68,20
88.30
1.31
0.23
resnet18
3
6.
63.96
85.58
1.51
0.36
resnet 8
MixQuant
3.25
5, 6
64.00
85.54
1.51
0.37
resnet18 18
3.3
4,
61.29
83.81
1.64
0.37
resnet18
3.5
4,
53.67
77.78
2.04
0.38
resnet34
baseline: f32
N/A
all layers float 32
73.31
91.42
1.08
N/A
resnet34
baseline: int8
layers are int
73.24
91.39
N/A
resnet34
2
6.
72.35
90.91
1.12
0.24
resnet34
3
6,
61.21
82.93
1.70
0.39
MixQuant
resnet34
3.25
4,
61.36
83.05
1.68
0.40
resnet34
3.3
4,
61.36
83.05
0.40
resnet50
baseline: f32
layers are float
76.13
92.86
N/A
resnet50
baseline: int8
N/A
all layers are int 8
75.99
92.81
0.97
N/A
resnet50
2
6,
75.18
92.52
1.00
0.28
resnet50
MixQuant
3
5,
70.58
90.04
1.19
0.43
resnet50
4, 6,
50.13
74.29
0.45
resnet101
baseline: f32
all layers are float
77.37
93.55
0.91
N/A
resnet101
baseline: int8
N/A
all layers are int 8
77.21
93.51
0.92
N/A
resnet101
1.3
5, 6,8
76.96
93.42
0.92
0.22
resnet101
1.5
2, 7,
59.23
81.74
1.83
0.24
resnet101
MixQuant
1.7
4, 7,8
58.86
0.30
resnet101
1.8
2, 6,
52.32
75.61
2.25
0.33
resnet101
1.9
2, 6.
49.36
72.63
2.44
0.34
resnet152
baseline: f32
N/A
all layers are float 32
78.31
94,05
0.88
N/A
rcsnet152
baseline: int8
N/A
all layers int 8
78.31
94.02
0.88
N/A
resnet152
8.8
78.20
94.01
0.20
resnet152
1.3
6,
78.15
94.01
0.89
0,20
resnet152
1.5
77.55
93.76
0.91
0.23
MixQuant
resnet152
1.7
3, 7,
70.68
90.11
1.22
0.28
resnet152
1.8
5.
71.48
90.16
0.31
resnet152
1.9
2, 2, 6,
62.99
85.01
1.66
0.32
resnext50_322 33x44
baseline: f32
N/A
all layers are float 32
77.62
93.70
0.94
N/A
resnext50_324x 3324d
baseline: int8
N/A
all layers are int 8
77.40
93.63
0.95
N/A
cesnext50322x44
1.3
7,8
77.43
93.52
0.95
0.19
resnext50_32xce
1.5
7,8
77.21
93.51
0.95
0.20
resnext50_32xee _32x4d
1.7
5, 7,8
76.93
93.29
0.27
2aanext5_33xx444 MixQuant
1.8
5, 6,
75.43
92.60
1.05
0.30
resnext50 32x4d
1.9
5, 6,
75.43
92.60
1.05
0.30
resnext50 32x4d
2
6,
72.60
90.79
1.18
0.30
resnext101 _322xd
baseline: f32
N/A
all layers float 32
79.31
94.53
0.93
N/A
resnext101 33xxx baseline: int8
N/A
all layers are int
79.11
94.51
0.93
N/A
Resnext101. _32x8dd
1.1
7,8
79.12
94.51
0.93
0,31
resnext101 3328dd
1.3
4, 7,8
76.61
93.26
1.04
0.33
resnext101 33x8d MixQuant
1.5
2,4,
7,8
59.91
81.46
2.05
0.39
resnext101 32x8d
1.7
2,4.
37.65
59.52
3.84
0.46
@ssnex101_32xx6
1.8
2.
6.
26.14
45.57
5.02
0.49
accuracy and lowering the quantization bit-width. For example, the highlighted lines in Table
satisfy the requirement of selecting the minimum quantization bit-widths such that the model top-1 accuracy degradation is 3%.
Runtime Analysis Table reports the runtime in seconds of MixQuant for various ResNet archi- tectures, where MixQuant considers the bit-widths of 8, 7, 6, 5, 4, 3, and 2, and one or ten different QEMs. It can be observed that the runtime grows with the number of layers since higher number of layers imply a larger search space. For one QEM, the MixQuant search takes between 0.1 and 0.5 seconds. If it combined with asymmetric per-layer quantization using the optimal bit-widths

---

Under review as a conference paper at ICLR 2023
returned by the search, it takes between 1.0 and 3.2 seconds. If the number of QEMs is increased from one to ten the MixQuant search takes between 0.9 and 5.5 seconds, which represents a linear increase in runtime.
QUANTIZATION SENSITIVITY OF WEIGHTS ABLATION STUDIES
To demonstrate that quantizing DNN weights warrants a search for optimal bit-widths as opposed to uniform precision quantization, perform two ablation studies show that different weight layers have different sensitivity to quantization based on their type and position.
Weights Quantization Sensitivity by Layer Type First, we investigate if different layer types have different sensitivity quantization. We consider four layer types in the ResNet architecture: (1) first conv layer, (2) conv layers with a 3x3 kernel, (3) conv layers with a 1xl kernel, and (4) final fully connected layer. For each type of layer, we perform asymmetric quantization and vary its bit-width while keeping the bit-width of all other layer types constant at int8, We calculate the model accuracy, loss, and quantization error for the following quantization bit-widths: 8, 7, 6, 5, 4, 3, and 2.
In Figure 1 we show the impact of varying the bit-width of one layer type at a time on the model top-1 accuracy. Lowering the quantization bit- bit-width of conv layers with a 3x3 kernel has the most adverse impact on top-1 accuracy shallower ResNet architectures, while in deeper ones conv layers with Ixl kernel followed by conv layers with 3x3 kernel that impacts model accu- racy the most. The first conv layer and conv layers with Ixl kernel have approximately the same sensitivity to varying bit-width in the shallower architectures. Finally, the quantization bit- wittwidth of the final fully connected layer has the smallest impact on model accuracy for all ResNet archi- tectures. In general, starting at 5 bits the model accuracy begins degrade; however, the deeper architectures are less sensitive to decreasing bit-width. While the reason that the conv layers with 3x3 kernel and Ixl kernel are the most sensitive is the fact that those layer types account for the highest number of layers in ResNet, we can still conclude that different layer types have different sensitivity quantization bit-width measured as the impact on the overall model quality. Therefore, different layer types will benefit from different quantization bit-widths, which motivates MixQuant. Similar results can also be found by measuring layer type sensitivity using the model average loss and quantization mean squared error (Figures and 3 in Appendix).
the
Weights Quantization Sensitivity by Layer Position In addition to the layer type, we investigate the position of a layer has an impact on quantization sensitivity of weights. We measure the relative quantization error (RQE) of individual layers for the following bit-widths: 8, 7, 6, 5, 4, 3, dequantized w f22 w), where iē is the weights vector and the arg operation returns a scalar that represents the mean of all elements in a vector.
2, and define the RQE as RQE- aog((f3)
Table 5 identifiis the most sensitive layers across various bit-widths and architectures, where layers are indexed from 0 through n, and equals is the total number of layers in an architecture minus one. For example, for int8, it is the 1st layer in resnet18 8 that has the highest relative quantization eтror compared to all other resnet 18 layers while for resnet50 it is the 46th layer. We can see that the quantization bit-width has a significant impact on the position of the most sensitive layer with the exception of ResNet50. While ResNet50's 's most sensitive layer is located towards the end of
Table 4: Runtime of MixQuant search and MixQuant combined with asymmetric quantization re-
ported in seconds for (a) QEM and (b) 10 QEMs
(b)
Architecture
search
search + quantization
Architecture
search
search--uanntizttion
resnet18 18
0.1
resnet18
0.9
1.8
resnet34
0.2
1.1
resnet34
1.5
2.5
resnet50
0.2
1.3
resnet50
2
3.1
resnet101
1.7s
resnet101
3,6
4.9
resnet152
0.5
2
resmet152
5.2 s
6.8
resnext50.322.0
0.2
1.4s
resnext50 33x4d
2
3.2
resnext101.3240 32xxdd
0.5 s
3.2
resnext101 _2xx8d
5.5
8.2
8

---

Under review as a conference paper at ICLR 2023
Readdt
(a) ResNet18
ResNet34
ResNer50
meppel 52
Layer Bt Wasth
Layer Bt: Witth
(d) ResNet101
(e) ResNet152
resnexd50_32xld
resnextll133xx6
Layer
Laper
(1) ResNeXX50 32x4d
(g) ResNeXt101 32x8d
Figure Sensitivity different layer types to quantization measured as quantized model top-1 accuracy with respect to varying bit-width of one type of layer while holding all other layer types bit-widths constant at int8
The most sensitive layer positions in DNN measured as a relative quantization error with respect to varying quantization bit-width
Table 5:
Most sensitive layer position at various quantization bit-widths
Architecture
int 8
int 7
int 6
int 5
int 4
int
int
resenet18 18
17
17
17
16
resnet34
1
1
20
20
20
33
35
resnet50
47
44
resnet101
6
6
6
97
97
99
resnet152
1
45
45
148
148
148
152
resnext50 33x4d
1
1
52
45
45
45
resnext101_32x88
49
96
96
the network for all quantization bit-widths, other architectures's most sensitive layer position varies based on the bit-width. For higher bit-widths and 6 it is located at the beginning while for lower bit-widths 2, 3, and it is at the end. The most sensitive layers of ResNet34 and Re22eXt101332x88 bit-widths 4, 5, and 6 are positioned in the middle the network. Based on these experiments, we can conclude that different layer positions have different sensitivity to varying bit-width. Ad- ditionally, we can see that the position of sensitive layers depends on the bit-width and network architecture.
7 CONCLUSION
In this paper we propose MixQuant, search algorithm that finds the optimal quantization bit- bit-width for each layer weight and can be combined with any quantization method as a form pre-processing optimization. We show that combining MixQuant with BRECQ (Li et al. 2021), state-of-the-art quantization method, yields better quantized model accuracy than BRECQ alone. Additionally, we combine BREQ with asymmetric quantization (Jacob et al. 2018) to show that MixQuant has the potential to optimize the performance of any quantization technique. Our code is open-sourcee and available at: https:: haap::////noomoouss.oo 4open. sci scwince/r//aaaenee zed maggeett 43C5

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. In NeurlPS, 2019.
Burak Bartan and Mert Pilanci. Training quantized neural networks to global optimality via semidef- inite programming. In ICML, 2021.
Yaohui Cai, Zhewei Yao, Zhen Dong. Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 13166-13175, 2020.
Ting Chen and Yizhou Sun. Differentiable product quantization for end-to-end embedding com- pression. In ICML, 2020.
Yoni Choukroun, Eli Kravchik, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. 2019 IEEF/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3009-3018, 2019.
Ahmed T. Elthakeb, Prannoy Pilligundla, and Hadi Esmaeilzadeh. Divide and conquer: Leveraging intermediate feature representations for quantized training of neural networks. In ICML, 2020.
Yonggan Fu, Qixuan Yu, Meng Li, Vikas Chandra, and Yingyan Lin. Double-win quant: Aggres- sively winning robustness of quantized deep neural networks via random precision training and inference. In ICML., 2021.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. ArXiv, abs/2103.13630) 2022.
Suyog Gupta, Ankur Agrawal, K. Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. ICML,, 2015.
Qingchang Han, Yongmin Hu, Fengwei Yu, Hailong Yang. Bing Liu, Peng Hu, Ruihao Gong, Yanfei Wang, Rui Wang, Zhongzhi Luan, and Depei Qian. Extremely low-bit convolution optimization for quantized neural network on modern computer architectures. 49th International Conference on Parallel Processing ICPP, 2020.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv,, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. ArXiv, abs/1609.077 1006609009.6..... 2017.
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Improving post training neural quantization: Layer-wise calibration and integer programming. ArXiv, abs/2000. 10518, 2020.
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. In ICML, 2021.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-aritee weteerrarittmetc-onllll inference. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2704-2713, 2018.
Schoon Kim, Amir Gholami, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. 1-bert: Integer- only bert quantization. In 2021.
Eliska Kloberdanz, Kathrin Kloberdanz, and Wei Le. Deepstability: A study of unstable numerical methods and their solutions in deep learning, 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE), pp. 586-597, 2022
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient non- uniform discretization for neural networks. In ICLRR 2020.
10

---

Under review as a conference paper at ICLR 2023
Yuhang Li, Ruihao Gong. Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. ArXiv, abs/2102.05426,, 2021.
Lingyan Liang. Post training mixed-precision quantization based key layers selection. In ECCV Workshops, 2020.
Darryl Dexu Lin, Sachin S. Talathi, and V. Sreekanth Annapureddy. Fixed point quantization of deep convolutional networks. ArXiv, abs/1511.06394 2016.
Xingchao Liu, Mao Ye, Dengyong Zhou, and Qiang Liu. Post-training quantization with multiple points: Mixed precision without mixed precision. In AAAI, 2021.
Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. ArXiv, abs/2004. 105688 2020.
Yury Nahshan, Brian Chmiel, Chaim Baskin, Exgenii Zheltonozhskii, Ron Banner, Alexander M.
Bronstein, and Avi Mendelson. Loss aware post-training quantization. Mach. Learn., 110:3245- 3262, 2021.
Shyam A. Tailor, Javier Fernández-Marqué,, and Nicholas D. Lane. Degree-quant: Quantization- aware training for graph neural networks. ArXiv, abs/2008.05000, 2021.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quan- tization with mixed precision. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 8604-8612, 2019.
Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantization via bit-split and stitching. In ICML, 2020.
Bichen Wu, Yanghan Wang. Peizhao Zhang, Yuandong Tian, Péter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search. ArXiv, Abs//1812,0009,, 2018
Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. ArXiv, 2016.
A APPENDIX
11

---

Under review as a conference paper at ICLR 2023
110
(a) Resnet18
(b) Resnet34
(c) Resnet50
tssee112
Layer
Layer
(d) Resnet101
(e) Resnet152
resteedito 3222dd
reoneat _22ddd
1250
3000
730
803
300
Laye www.owaa
www.
(l) Ressnet50.2244
(g)
Resnet101132x30
Figure 2: Sensitivity of different layer types to quantization by architecture measured quantized model average loss with respect to varying bit-width of one type of layer while holding all other layer types bit-widths constant at int 8
(a) Resnet18 8
(b) Resnet34
Resnet50
resnet10]
resnet152
3.55
Layer Wadh
Layer Bit
(d) Resnet101
(e) Resnet152
Layer
Iwer:0ta Sat.com
combic
st leyer kity
Mo
cervial
condic
(I) Resnet50.32
(g) Resnet 32x8d
Figure 3: Sensitivity of different layer types to quantization by architecture measured as quantized model total quantization mean squared error with respect to varying bit-width of one type of layer while holding all other layer types bit-widths constant at int 8
12

---

Under review as a conference paper at ICLR 2023
1.0
0.8
Relative Quantization
0.6
0.4
2
3
7
Layer Bit Width
3
6
12
15
18
4
7
10
13
16
19
5
8
11
14
17
20
Figure 4: Resnet18 layer sensitivity with respect quantization bit-width
2am
LEB Auroap
Fortgg Aroory
Annnay
Resnet18
(b) Resnet34
(c) Resnet50
resnet101
resnet152
0.04
Normalized Values
6.03
0.03
6.02
0.01
Quantaration
Quantizaton Multiplirr
Narmall lood Average Lods
Normalized Average Loss
Normal Average Acturacy
kormaiized Aveerage Accuracy Normalical Quartization MSS
Normal lued Quantiaziion MSE
(d) Resnet101
(c) Resnet 152
resnext50 32v4d
resnea1111 01 32x8d
0.03
Normaized Valuee
0.02
0.03
8.02
0.01
Quantication Error Multipiier
Quantication Multiplier
Normall alodd Aceeage Loss
Normelized Average Lows
Norm lied Rver age Accuratc
hormalized Avee age Accuracy kormsiized Quantization MSE
Normalber bed Qusnitzation MSE
(i) Resnet50.32x4d
(g) Resnet101_32xk6
Figure 5: Relationship between quantization error multiplier (QEM) and model accuracy by archi- tecture
13

---

Under review as a conference paper at ICLR 2023
Table 6: MixQuant results for ResNet18, ResNet34 and ResNet50: individual layer quantization bit- width assignments, quantized model accuracy, loss and quantization mean squared error for various quantitative multipliers
arch
QEM
layers bit widths
Acc@
Acc@5
loss avg
QMSE
resnet18
N/A
all layers float 32
69,76
89.08
1.25
N/A
resnet18
N/A
all layers
69.63
89.07
1.25
N/A
resnet18
2
[6, 6, 7, 7, 7, 7,
7, 7,
7,
7,
7,
7,7,
68.20
88.30
1,31
0.23
7. 7,
resnet18 8
3
[5, 5,
6,
6. 6.
6.
6.
6.
6.
6,
6,
63.96
85.58
1,51
0.36
6, 6, 6, 6, 7]
resnet18
3.25
[5,
6, 6, 6
6,
6,
6, 6,
64.00
85.54
1.51
0.37
6, 6, 6, 6,
6]
resnet18
3.3
[4. 5, 6, 6,
6,
6. 6. 6. 6.
6
6.
6,
6,
61.29
83.81
1.64
0.37
6. 6, 6.
6]
resnet18
3.5
[4,
6, 6.
6. 6. 6,
53.67
77.78
0.38
6, 6, 6, 6, 6]
resnet34
N/A
all layers are float 32
73.31
91.42
1.08
N/A
resnet34
N/A
all layers are int
73.24
91.39
1.08
N/A
resnet34
2
[6,
7.
7. 7.
72.35
90.91
1.12
0.24
7. 7. 7.
7.
7.7.
7. 7.7.7.777
7.
resnet34
3
[4, 5, 6, 6,
6,
6,
6,
6,
6,
6,
6,
6,
6,
6,
6,
61.21
82,93
1.70
0.39
6, 6, 6, 6, 6,
6,
6,
6,
6
6,
6,
6,
6, 6. 6,
resnet34
3.25
[4,
6.
6.
6.
6.
6,
6,
6,
6,
61.36
83.05
0.40
6, 6, 6, 6,
6,
6,
6,
6,
6,
6. 6. 6. 6.
resnet34
3.3
[4,
6.
6.
6, 6,
61.36
83.05
1.68
0.40
6, 6, 6. 6.
6,
6, 6,
6, 6, 6]
resnet50)
N/A
all layers are float 32
76.13
92.86
0.96
N/A
resnet50
N/A
all layers are int
75.99
92.81
0.97
N/A
resnet50
2
[7,
7, 7,
7. 6,
75.18
92.52
1.00
0.28
6,
7,
7, 7.
7. 7, 7,
7,
7,
7,
7.
7,
7,
7. 7, 7,
resnet50
3
[6, 5,
5,
5,
6,
6,
6,
6,
6, 6,
5, 5,
70.58
90.04
1.19
0.43
5, 6,
6,
6,
6,
6,
6.
6.
6,
6,
6.
6.
6, 6, 6,
6,
6
resnet50
3.25
[6. 5, 4, 4, 4,
6.
5,
6,
6. 6. 6, 6, 5,
50.13
74.29
2.30
0.45
5, 6.
6,
6.
6.
6.
6,
6, 6. 6.
6, 6,
6,
6,
6,
15.0
63
12.5
hogg3
210.0
22
20
0.A
0.8
0.
Ousmizatin
Ouantiation Eror
(a) Top accuracy function of mean squared quantization enror
(b) Top accuracy a function of mean squared quantization eтror
(c) Average model as function of mean squared quantization error
Figure 6: Relationship between quantization error and model accuracy loss
14

---

Under review as a conference paper at ICLR 2023
Table 7: MixQuant results for ResNet101 and ResNetl 52: individual layer quantization bit- width assignments, quantized model accuracy, loss and quantization mean squared error for various quan- titative error multipliers
arch
QEM
loyers hit widths
Aced1 1
Acede5
less avg
QMSER
resnetiO1
NIA
Boot 32
77.57
93.35
0.91
N/A
resnet101
NIA
layers
71:21
7. 6,6,
B.
76.96
93.42
0.92
0.22
K. &
H. H. 8, 8, B,
10sneti01
1.
77 6 7. 6,
2 6
&
R,
S, 8, 8, 7.
59.23
81.74
1.83
0.24
8. 8, 8, 8,
R
10
17
[6
S. G.
7. 6.
8, 7.
58.86
81.05
1.86
0.30
S.
7, T.
R
res0er101
188
[6, S. 6.
2
T. 7. 6,6,7 1, T. 5.
52.32
75.61
2.25
0.33
7.7.
T.T.6
7,
7.
7
7
7
101
1.9
16 4.
6,
5.
7.
7.
7.4 4.T. T.T. T.7.
7.
49.36
72.63
2.44
0.34
7,
7.
7. 7.
7.
resnet
NIA
all layers are Boot 32
78:51
94.05
154021152
NIA
all layers are
78. 51
94.02
0.88
N/A
K.
R,
re502t152
1.5
16 S. 6. &
7, 7.
T.T.T.K 8.7.8.7. 7. 7.
77.55
93.76
0.91
0.23
7. &
T.
T.
8,
R. R, R R.
R, R,
R. R. 8, 8, 8,
K.
R R R, R, B.B.
resnet152
70.68
90.11
8.8.
7.8.7.88
152
L88
E.
T
7.
7.
7.
7. 6 T. 7. 7. 7. 7. 7, 6.6
71.48
90.16
1.19
0.31
T.
T.
7. 6.
7.7.
7
7.
T. T. T. T. T. T T.7. 7.
T. T. 7. 7. 7. 7. 7. 7.
7.
7.
1.7.
7, T.
7. 7.
7. T. 7. 7.
7.
7.
7. 7. 7
7.
7.7.7
15

---

Under review as a conference paper at ICLR 2023
Table 8: MixQuant results for ResNeXt50 32x4d and ResNeXt101 32x8d: individual layer quanti- zation bit-width assignments, quantized model accuracy, loss and quantization mean squared error for various quantitative error multipliers
QEM
widchs all layess are Soat 32
Ace@1
Acc@5
QMSE
resnext50
32x4d
NIA
71.62
93.70
0.94
N/A
resnext50
NIA
layers
77.40
93.63
resnoxt50)
32x4d
1.3
IS 7.7.
7. 7, R R.R.S.
8, 8, 8, B.
X]
77.43
93.52
0.95
0.19
32x4d
77.21
R. R. R. R R R,
R, S,
R, B,
8]
32x4d
resnou15
6,
7.
76.93
93.29
resnext50)
32x4d
18
57
7. 7.
7.
5.
6, 7. 6, 7.
7
7.
7
7, 7
7.
7,
7. 7.
T.
7.7.
75.43
92.60
L.05
0.30
resnes150
32x4d
17.5
6,
T.
7.
92.60
7. 7.
7.
7. 7. 7. 7.
7,
7.
7.
7 7 T.
32x4d
resnau150
17
71
72.60
90.79
0.30
resnextl0
NIA
Layors Boot
79.31
94.53
32uad
32x8d
resnest101
NIA
all int
79.11
94.51
0.93
N/A
resnext
92s8d
1S. 7. 8.
79.12
94.51
0.93
0.31
R.
S.
32x8d
resnext101
1.3
17.
7 6, 6.
7. R.
8,
76.61
93.26
1.04
0.33
S S E R. S. N, S,
S1
32x8d
resnexti01
1.5
[6 2, 7. 6,
8,
8,
59.91
81.46
2.05
0.39
7.
7.
R. S. 8 R, R. S. R, R,
8, R,
resnexT1B1
6
37.65
7, 6.
7. 7.
7.
7.7.
7. T. 7.
T. T.
M.
15. 2,
5. 7, 6.
6
6,
7, 7.
7, 7.
16


Reviews:
[{"review_id": "pmILEAdlMT", "reviewer": "Reviewer_6kQQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a search algorithm named MixQuant. MixQuant can search for the optimal bit-width for each layer weight based on the the l2 quantization error. Combined with BRECQ, MixQuant + BRECQ achieves the  state-of-the-art performance across multiple datasets.\n", "summary_of_the_review": "Overall, I think this idea is too obvious and this problem is either not new. Originality is not enough to get this work accept.\nIn addition, the evaluation also lacks a lot of important baselines.\nMinor problem: in forth row of section 4: \"Prior works have shown ...\", please add some reference for these works.", "strength_and_weaknesses": "+ Algorithm 1 is basically the core of the paper. The method is technically sound and clear \n+ The writing is clear in general\n- The method is very simple and lack of originality. Algorithm 1 simply performs exhaustive search on all the weight precisions, and find the optimal precision that can achieve comparable quantization error with the 8-bit quantization. \n- The evaluation lacks some important baseline algorithms. This layerwise mix quantization has been studied extensively by the early work.\n- Implementation of MixQuant DNN is difficult. How to design the hardware platform to have this mixQuant DNN working?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "k9adF9m9MH", "reviewer": "Reviewer_Z2i6", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose MixQuant, a simple mixed precision search algorithm. MixQaunt evaluates the quantization error per layer (thus assumes independence) and compares the relative increase of lower bit-widths wrt the 8 bit error. They combine their search algorithm with BRECQ to improve the overall quantization accuracy and claim their search can be combined with any quantization algorithm. The authors evaluate MixQuant for Resnets and MobileNet v2 on ImageNet.", "summary_of_the_review": "The paper has major flaws in the empirical evaluation and also some mistakes in the proposed algorithm. Given the idea is simple and not novel, the paper clearly does not meet the expected quality for ICLR.\n", "strength_and_weaknesses": "\nStrength:\n* The search algorithm is very simple and fast (just a few seconds).\n* Can be combined with other PTQ methods such as BRECQ etc.\n* The paper is language wise well written and easy to follow.\n\nWeaknesses:\n* Their bit-width selection seems wrong (eq 1 and line 12 in algorithm 1): the quantization error for the highest bit-width (here 8 bits) should almost always give the lowest quantization error, thus the argmin will select this. And 8 bits will always satisfy the condition for QEM >= 1 which is the case in the experiment section. (Based on the text my guess would be that this equation might rather be $optBit = argmin_{optBit \\in B} optBit$).\n* The paper wrongly claims that MixQuant finds the optimal bit-widths. There is no proof for this and in general finding the optimal bit-width is NP-hard and any linear algorithm will be unable to find the optimal solution for all cases.\n* Empirical evaluation is very bad and no conclusions can be drawn from them. A few examples:\n    * No average bit-width or BOPs are reported for MixQuant in any table making it impossible to compare with fixed bit-widths. The alignment in the table is even misleading, e.g. in table 1 and 2 they put MixQuant with 4,5,6 bits next to fixed 4 bits and fixed 3 bits, which are clearly not comparable in size or BOPs.\n    * In several cases MixQuant is better than the reported baseline (both MobileNet and Resent). This can not be true given that we do post training quantization in which only noise/error is added to the model but no training is performed, especially not almost 1% as it is for Resnet18 (70.69% vs 69.76%).\n    * Several rows in table 1 and 2 are duplicated with no additional value (some rows in table 2 are literarily identical). Full table 2 could be omitted as it does not add new results, just compares to Liu et al. which could be added to table 1 (and Liu et al. seems a irrelevant baseline as it completely underperforms, even older literature such as AdaRound or BitSplit performs significantly better than that).\n* Quantization and numeric instability and listing 1: The example is trivial and it does not show more than that quantization error (here rounding error) does exist in practice and also leads to and error after the matmul. That quantization error exists is logical and widely known in the field and does definitely not justify an example of over 1 page in the main paper.\n* They do not do activation quantization and I do not fully agree with their reasoning. I agree somewhat with argument one, argument two is actually a reason why we should do activation quantization as well and have mixed precision for activations. The third reason is misleading as BRECQ is also doing activation quantization except in one ablation study.\n* Is W a single weight or a tensor? Assuming the latter, then in line 9 of algorithm one (the quantization error) would also be a tensor. How is this error aggregated and compared? MSE?\n* For the layer-wise error alternatives should be discussed and empirically considered. E.g. BRECQ/AdaRound which the authors heavily cite and compare to show that the MSE of the output activations ($||Wx - W_q x||^2_F$) is a better task loss estimate than the MSE on the weights ($||W - W_q||^2_F$) for weight quantization error. Should that not help for mixed precision as well?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "_e1UUzLeNG", "reviewer": "Reviewer_Kh8z", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an optimization algorithm for mixed-precision weight quantization. This algorithm decides the bit-width of each layer in forward order. The optimization target is the amount of quantization errors. Their method can make mixed-precision quantized networks (weight only) using various bit-widths with higher accuracy.", "summary_of_the_review": "I think this paper and this method should be refined in the aspects of quantization format and optimization algorithm.", "strength_and_weaknesses": "Strength\n - This method is simple, straightforward and easy to adopt. \n - Assigning varied bit-widths to each layer is not a time-consuming process.\n\nWeakness\n - The quantization method is not quite novel because this method is not complex or an intelligence optimization algorithm. It seems to be just a kind of heuristic algorithm with layer-wise bit-selection with a threshold of quantization errors. There is less theoretical reason for this algorithm, and it is not proved that this algorithm can find optimal set of bit-widths to minimize quantization errors. Indeed, minimizing quantization doesn’t exactly mean the best performance of a quantized model (e.g. recent PTQ papers have pointed out MSE is not an optimal solution for quantization). \n - In the practical side of quantization, I don’t understand why this paper adopts such a quantization scheme. They quantize only weights with various bit-widths from 2-bits to 8-bits. Because there are no such processing units for that, the quantized weights should be dequantized into full-precision format during inference. So, there is no gain on computation due to their quantization. Then, they can reduce only memory-bounded problems. But, the target networks of this paper are only CNNs, which are computation-bounded models (i.e. the weights of CNNs are not a big portion of memory during inference.) Many quantization papers also include weight-only quantization results, but they also provide these results as a bridge to show their final results when quantizing both weight and activation. Since this paper seems not to be applicable to activation quantization, it is hard to extend to real computation units (INT8/4).\n - In addition, I’m very curious why this paper compares 4/8 results from (Liu, 2021) while this paper has no idea about activation quantization. Indeed, recent PTQ papers have improved the w4a8 quantization results, which are close to full-precision. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "BxwZt-YaEt", "reviewer": "Reviewer_MUWW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a mixed-precision quantization method, called MixQuant, to identify the bit-widths of individual layer weights. In particular, the paper proposes to quantize model weights layer-by-layer by greedily minimizing the quantization error from using low bit-width values. Evaluation of the proposed method on ResNets over the Imagenet dataset shows that the proposed method can find mixed-precision quantization solutions that achieve comparable accuracy in a relatively short amount of time. ", "summary_of_the_review": "The paper has done some good evaluations and ablation studies on mixed-precision quantization using its proposed method and demonstrated how the method can be composed with existing ones. However, there are some major concerns on the technical novelty and small scope of evaluation. ", "strength_and_weaknesses": "Strengths:\n- Thorough evaluation of mixed-precision quantization on variants of ResNets.\n- Demonstrated that the proposed method can be composed with other methods such as BRECQ while leading to improved performance. \n\nWeaknesses:\n- Most findings from this paper seem to be well-known facts that have been explored by prior works. So the novelty of the proposed method is limited. For example, why is it a new observation that \"quantizing and dequantizing the model parameters lead to roundoff errors\" and \"some weights are more sensitive than others which should be reflected on their quantization bi-width\"? The former phenomenon is pretty much observed by all quantization-based studies, and the latter one is what motivates all prior studies on mixed-precision quantization.  The main idea is to quantize the model layer-by-layer by solving a minimization problem of quantization error for each individual layer. Such a scheme has been explored by prior works such as [1] and [2]. The idea that different weights may be replaced with values of varying bit-width is also not new, as it has been studied by [3] and [4]. Several of these prior works are not references, which also indicates that the authors did not do a thorough literature search. \n- The proposed method only considers mixed precision for weights whereas the activations remain to be 32. As such, there is no actual performance improvement from the proposed method because the actual computation still happens in FP32. \n- The evaluation scope is quite limited, focusing primarily on image classification tasks and particularly ResNet variations. The authors are encouraged to evaluate its approach on a wider range of tasks/datasets, such as including NLP models and Transformer-based models such as vision transformers.  \n- Quantizing bits to INT2, INT3, INT5, INT6, and INT7 cannot lead to any performance gains, to the best of the reviewer's knowledge, as there is no real hardware that can benefit from this irregular bit-width. \n\n[1] Hassibi et. al. \"Optimal Brain Surgeon and general network pruning\", 1993\n[2] Frantar et. al. \"Optimal Brain Compression\", 2022\n[3] Shen et al. \"Q-BERT: Hessian Based Ultra Low Precision Quantization of BER\". 2020\n[4] Zhao et. al. \"Automatic Mixed-Precision Quantization Search of BERT\", 2021\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject