Mixed-Precision Inference Quantization: Radically Towards Faster inference speed,  Lower Storage requirement, and Lower Loss
Authors: Daning Cheng
Venue: Submitted to ICLR 2023 (2023)
Keywords: Mixed-Precision Quantization
Inference
Neural networks
Noise Robustness
Residual Network

Abstract: Model quantization is important for compressing models and improving computing speed.  However, current researchers think that the loss function of quantizated model is usually higher than full precision model. This study provides a methodology for acquiring a mixed-precise quantization model with a lower loss than the full precision model. In addition, the analysis demonstrates that, throughout the inference process, the loss function is mostly affected by the noise of the layer inputs. In particular, we will demonstrate that neural networks with massive identity mappings are resistant to the quantization method. It is also difficult to improve the performance of these networks using quantization.

Under review as a conference paper at ICLR 2023
MIXED-PRECISION INFERENCE QUANTIZATION: RAD- ICALLY TOWARDS FASTER INFERENCE SPEED, LOWER STORAGE REQUIREMENT, AND LOWER Loss
Anonymous authors Paper under doublc-blind review
ABSTRACT
Model quantization is important for compressing models and improving computing speed. However, current researchers think that the loss function value of the quantized model is usually higher than the full-precision model. This study provides a methodology for acquiring a mixed-precision quantization model with a lower loss without "fine-tuning" than the full-precision model. Using our algorithm in different models on different datasets, we gain lower loss quantized models than full-precision models.
INTRODUCTION
Neural network storage, inference, and training are computationally intensive due the massive parameter sizes of neural networks. Therefore, developing compression algorithm for machine learning models necessary. Model quantization, based on the robustness of computational noise, one of the most important compression techniques. The primary sources of noise are truncation and data type conversion errors. the quantization process, the initial figh-precision data type used for model's 's parameters is replaced with lower-precision data type. Both PyTorch and TensorFlow have quantization techniques that translate floats to integers. Various quantization techniques share the same thcoretical foundation, which is the substitution of approximation data for the original data in the storage and inference processes. A lower- lower-precision data format requires less memory, and using lower-precision data requires fewer computer resources and less time. In quantization, the precision loss different quantization level conversions and data type conversions is the source of the noise.
Current works, the other hand, raise the following issues:1. No study examines how to reduce the loss function value of a model using quantization technology. There is the myth that the quantized model's loss is higher than the full-precision model. 2. The background of some work is against current computation device requirements: current computation devices have to use the same two data types one computation process, which means the layer's weight and the input of the layer have to be the same quantization level. 3. No one has examined which types of models are stable in the quantization process and why.
The purpose of this paper is mainly to discuss the question of whether quantization technology always leads to the model's loss function increasing and how to gain a better performance quantized model by quantization method. In current papers, the main target of the current algorithm is to gain quantized model whose loss function value not much higher than a full-precision model. However, we want to give algorithm that can find the quantized model that is better than the full precision model, ie.., the quantized model's 's loss function value is lower than the full precision model, based on the current computation device's 's requirements.
This research provides a basic analysis of the computational noise robustness of neural networks. Furthermore, we present method for acquiring quantized model with a lower loss than the model with full precision by using the floor and ceiling functions in different layers, with a focus on layerwise post-training static model quantization.

---

Under review as a conference paper at ICLR 2023
As added benefit in algorithm analysis, give the theoretical result to answer the question that which types of models are stable the quantization process and why when the noise introduced by quantization process can be covered by the neighborhood concept.
RELATED WORK
Model compression methods include pruning methodşHan et al. (2015); Li et al. (2016); Mao
et al. (0177) knowledge distillatiorHinton et al. 2013), weight sharing UlIrich et al. (2017) and quantization methods. From the perspective of the precision layout, post-training quantization methods can be mainly divided into channelwise Li al. (2019); Qian et al. (2000) groupwise Dong al. (2019b) and layerwise Dong al. (2019a) methods. Layerwise mixed-precision layout schemes are more friendly to hardware. Parameters of the same precision are organized together, making full of program's temporal and spatial locality. Some works give the relationship between the weight and input of layer's s best quantization analysisSakr et al. (2017); [akkr & Shanbhag (2018). But in current computation architectures, the quantization level for weight and input should be the same. common problem definition for quantizatior et al. (2019a); Morgan et al. (1991): Courbariaux cet al. 2015); Yao et al. 2020 is as follows Gholami et al. (2021).
Problem 1 The objective of quantization is solve the following optimization problem:
min
[l wil?
qEQ
where is the quantization scheme, ((w) is the quantized model with quantization q. and w represents the weights, i.e., parameters, the neural network.
Although problem gives researchers target to aim for when performing quantization, the current problem definition has two shortcomings: 1. The search space of all possible mixee-precision layout schemes is discrete space that is exponentially large in the number of layers. There is no effective method to solve the corresponding search problem. 2. There is gap between the problem target and the final task target. As we see, no terms related to the final task target, such as the loss function or accuracy, appear in the current problem definition.
3 BACKGROUND ANALYSIS
3.1 MODEI. COMPUTATION, NOISE GENERATION AND QUANTIZATION
Compressed models for the inference process computed using different methods depending the hardware, programming methods and deep learning framework. All of these methods introduce noise into the computing process. One reason for this noise problem is that although it is common practice to store and compute model parameters directly using different data types, only data of the same precision can support precision computations in a computer framework.
Therefore, before performing computations on nonuniform data, a computer will convert them into the data type. Usually, lower-precision data type in standard computing environment will be converted into a higher- more computational resources and time. However, to accelerate the computing speed, some works on artificial intelligence (AI) computations propose converting precision data types based on the premise that Al models are not sensitive to compression noise. The commonly used quantization technology is converting data directly and using a lower-precision data type to map to figgher-precisiie data type linearly.
data type; this ensures that the results are correct but require
preccisoo data types into lower-
We use the following example to illustrate quantization method, which is presented in Yao et al. (2020). Suppose that there are two data objects impul,, and impulz are to subjected to computing operation, such as multiplication. After the quantization process, we have Q1 int( imputs and Q2 int( (mpudz) ), and we can write
scate]
Qoutput = int( input, * imputz ( = intQQQQ. scale scalez
scalevutput
scaleoutpas
scaleoutputs scale, and scale, are precalculated scale factors that depend on the distributions of impudi, inpula and the output; Q; is stored lower-precision data type, such an integer. All
2

---

Under review as a conference paper at ICLR 2023
scale terms can be precalculated and established ahead of time. Then, throughout the whole inference process, only computations on the Q; values are needed, which are fast. In this method, the noise is introduced in the int(-) process. This basic idea gives rise to several variants, such as (non)uniform quantization and (non)symmetrical quantization.
When focus on quantization strategy, i.e. round function in quantization framework like Micronet, we can have at least three strategy: round up, i.e., ceil function in python, round down, i.e., floor function in python and rounding, i.e., round function in python. usually, rounding is the most common method to deal with quantization. But, in this paper, we will show that how to mixed use round up/round down gain mixed precision quantized model which is better than full precision model.
3.2 NEURAL NETWORKS
In this paper, we mainly use the mathematical properties of extreme points analyze quantization methods. This approach is universal to all cases, not only neural networks. However, there is myth in the community that it the neural network properties that guarantee the success of quantization method Wanh et al. (2019); Morgan et al. (1991); Demidovskij & Smirnov (2020). To show that the properties of the extreme points, not the properties the neural network, are what determine the ability to quantize, i.c. the ability to handle noise, we must first define what a neural network is.
The traditional definition of neural network Denilson & Barbosa (016) as a human brain simulation is ambiguous; it is not rigorous mathematical concept and cannot offer any analyzable information. The traditional descriptions of neural networks Denilson & Barbosa 2016) focus on the inner products of the network weights and inputs, the activation functions and directed acyclic graphs. However, with the development of deep learning, although most neural networks still consist of weighted connections and activation layers, many neural networks network architectures for position embedding and layer norm operations in Transformers. Moreover, current deep learning frameworks, such as PyTorch and TensorFlow, offer application programming interfaces (APIs) to implement any function in a layer. Therefore, we propose that the definition of neural network adheres to the engineering concept indicated by the definition [ rather than a precise mathematical definition; that is, neural network is a way for implementing a function.
longer obey these rules, such as the
Definition The neural network is the function which is implemented in composite function form.
A neural network can be described in the following Eq. form.
model (x)) hu (12,, (hg,1 (...), .... J3 12, (13,.++ W3), W2,2), wi) where ha.j. i E [2..... n²], are the (n 1)th layers in the neural network; ws.j the parameter in (-)
(1)
Definition T means that a neural network, without training, can be any function. With definition I neural network is no longer mathematical concept, but this idea is widely used in practice Roesch al. 2019). We can see from definition that the requirement that a neural network is in composite function form is the only mathematical property of neural network that can be used for analysis.
practiceRRoechh
In practice the loss function is one method to evaluate a neural network. A lower loss on a dataset means a better performance neural network. For example, the training process optimises the model's loss, i.e., following Eq. 2
min (w) = Esample llw, sample)
Σ (LL.)))
l(w, (w,,,,,1 Vi)
where f(-) is the loss for model dataset, represents the model parameters, D is the dataset, m is the size of the dataset, E(-) is the loss function for a sample and (x;, y:) represents a sample in the dataset and its label.
In this paper, we mainly use the sequential neural network to describe the conclusion for the sequential neural network is easily described, and the whole conclusion non-related to the structure of the neural network. For sequential n-layer neural network, E(-) can be described in the following Eq form.
2s, v)) L(modela (2i, w), v1) model. hi hi(ha(hs(ha(- An (hi+1,, wo) wa), ws), w2), wi)
(3)
3

---

Under review as a conference paper at ICLR 2023
where [.) is the loss function, such as the cross-entropy function; h4, [l,..., n²|, is the (n (n-i+1) 1)th
layer in the neural network; (w - " "T wy is the parameter in hy(-); and for unified format, hn+1 stands for the sample z.
4 ALGORITHM AND BASIC ANALYSIS
4.1
START POINT
4.1.1 ANALYSIS BASE
Quantization methods for inference complex. Different algorithms use different assumption to solve the problem. Most of them pay much attention to the noise on parameters in NNDong al. (2019))] Yao al. (2020); Gholami et al. (202T); Nagel et al. (2020). However, in addition to the noise added to the parameters directly, noise is also introduced between different layers in the inference process because different quantization levels or data types of different precisions are used in different layers.
After quantization, the quantized loss for sample, i.e.(()). ) in the inference process as follows.
E(w, 24, 91) L(h1 (h2(--- hn (hn+1 Gns W2 + 6,, En-1 W2 62) c1, W1 + 61),94) where 8, 1,... n, and 6,, i E [I,..., nl), are the minor errors that are introduced in model parameter quantization and in data type conversion in the mixed-precision layout scheme, respectively.
Thus, obtain the following expression based on the basic total differential calculation.
a
a
E(w, Ni, ya) Elw, Ne, Vi)
0h;41
63
Ow:
6,
where is inner product and is the scalar product in following parts. For the loss whole dataset, we can gain
min f(w) - f(w) = 1 .ER
m (#)
Σ
aha41
de
e, + Oe
Ow:
5.
m
0he41
de
Ex
(5)
where f(w) = 1 Σ()). The reason for second equation in Eq.
5 is for a well-trained model, at components,
m the expectation of E(-)'s gradient for parameters is zero, i.e., for the Σ (m)wa))D Zer
The O.
4.1.2 TARGET AND ALGORITHM GUARANTEE
The key is to choose the appropriate vector to gain a lower loss model. When the loss of the inner product, i.e., Σ E(x.D.)ED ah;1 is negative, the loss for the quantized model, i.e., f. is lower
de
is our at than for the full precision model. An appropriate to produce negative E E(am)eD Thet algorithm target.
at frequently asked question is why Ee Sw zero but ((rg)) The41 non-zero. The at random in the final model except for the layers with bias terms like the batch norm layer. The bias term will absorb the gradient and train them in the optimization process. What is more, in the model, which de is close to zero vector, and we will show this mainly consists of identity mapping, Ex..,)) 0hat1
optimization algorithm to optimize w the training process. Thus, Σ (ED Thi41
in the next chapter.
Our problem setting for quantization is different from previous works like HAWQDong et al. (0119)); Yao et al. y Dong et al. (2019b); Nagel et al. (2020) because these methods do not take the emror in the layer's s input into consideration, which prevents their work and analysis in the mixed-prec -rexcssin computing area. As result, these works can only be used to store compressed neural network disk. When the compressed model is stored in memory for inference, these compressed models have to be recovered into the full precision model.
4

---

Under review as a conference paper at ICLR 2023
Figure 1: When the predicted point(x,) is out of the neighborhood range but pretty far from To, secant lines between the Tg and 21 Figure 2: Different directions have different perform significantly better than tangent lines. secant line in prediction process.
The choice of 2 is the maximum quantization noise in practice.
4.2
THE MAP FROM MATHEMATICAL ANALYSIS TO REAL ENGINEERING
In the above analysis, the whole process is under the condition that vector is small enough, which can be used in the total differential method. However. in practice, the scope of may be within [-0.1,0.1], which would escape the concept of neighborhood. What is more, mapping e vector into round operation should be fully discussed. This part will show how to deal with the above gap between analysis and engineering.
4.2.1
ROUND FUNCTION CHOICE
We use the convenient language of probability theory describe Thet1 for is stochastic vector naturally. We set = [ei, e,, and e; is i.i.d. random variable. We also set that P2,,P.,...PPP and p; is i.i.d. random variable and p are independence to each other.
DP
Th.-1
Then, we have 88 Th.t1
= Ei= e, = kep and following Eq,
E de
E
Ekep kEeEp
(6)
0h;t1
For a trained model, the Ep can be computed as Ep &
88 I. Th:
Then to gain negative
That1 ee, the Ee should be different signs with Ep.
To gain the suitable vector, we use the different round functions to ensure the sign of Ee. The roundup function, i.e., the ceil function in python, will produce an error vector whose all elements are positive. The round down function, i.c., the floor function, will produce an error vector whose all elements are negative. Thus, we are sure that the Ee positive and negative by round methods. Although the parameters in layers have strong noise robustness, we still try to add less noise to them. Thus, the parameters quantization process, we use the rounding method, i.e., the round function in python, quantize parameters for the rounding method exerts less noise original data.
4.2.2 REPLACE GRADIENT WITH SECANT LINE SLOPE
Although the elements in the vector are not small enough to use the total differrnniall directly, the elements in the vector are still small. For example, when using INT8 to quantize the res 14 model without identity mapping, the element in the vector is less than 0.01. The above fact shows that ||c]]| ||all| is small, which has a tiny influence on the final loss function. Thus, we can use the slope of the secant line to replace the gradient in the total differential, which is shown in figure
We also can treat p. as the random variable with different distributions or directly use E ah, following analyses. The conclusions the same close with current analysis.
vector in
5

---

Under review as a conference paper at ICLR 2023
Algorithm I: Radical Precision Inference Layout Scheme
Input: Neural network M, quantization levels [41 .2, qn]s errormins ETT erTormaz- H4.
calibration dataset D
Output: Quantized neural network
Arrange Q 191- 12.--..>.. In in ascending order Q (9: Iey..- li.. based the the size of
parameters under q::/FFor example Q=[INT8,NT4,,NTI into Q= for qi in Q do
for Layer; in M do if Layer, quantized then continue
end
Compute the error A of h5+1 under qi quantization level on if A <, min then
quantize Layer, parameters and input by rounding method on q. quantization level. end
errormaz then continue end
Compute secant (h4 A) and secant" (h; A) if ||cc||tt (-)ll < then
continue
end
Choice = max( max(secant A,, A), min(secant A)0)[I)
if Choice secant" (hę, A) then
quantize Layer; 's input by round down method on % quantization level. end
if Choice == secant" (h,, A) then
quantize Layer,'s 's imput by round up method 9, quantization level. end
if Choice != 0 then
quantize Layer;'s 's parameters by rounding method on z quantization level.
end
end end
return M
Because in different direction, the secant line is different, which is shown in figure so we have define the following secant* (h4, A), R'+ and secant" (h4, A), +. A is the maximum етrоr which is introduced by quantization. For example, the scale parameter in Section 3.1 I's example is the max error introduced by quantization.
(w,, w. v.) ) (1a( (1g4447 + A w.) w2). wn w)).
J (w)) 1 Σ (),,,, secanthh,, A)= -
+A
Dataset
+ T. We use this definition because 1. Compared to computing by the definition of secant, the secant function is easy to be computed. 2. If slope of secant line is [scc,, secg..., seck], secant (h,, A) Esec.
In the algorithm, we will use scant+(-) to replace Th.t1
Although we know the element in vector is less than 0.01 empirically, we still have to set mechanism in real algorithm design to keep the analysis map into algorithm practice. Thus, we have to set a value errormaz maz, which is small enough for the final loss function. When A > error max. we can choose more bits quantization level or full precision in this layer.
When is close to zero, i.e., we use more bits quantization level. For |l Ilr 26_ -
J |||e
the performance loss improvement small this layer. So, directly quantize these parameters

---

Under review as a conference paper at ICLR 2023
and layer's input with this quantization level to reduce computation resources. In algorithm design, we can use errormsn to control this case.
4.2.3
THE PROBABILITY OF GETTING BETTER MODEL
To show the probability of getting positive Eq.
at
use chebyshev's theorem, we have following
0h++1
0e
de 0l4+1
Var(kep)
Var(e)Vor())
Var(e)
Var(p)
P( 0l441
1220 0)
P(I
Ekepl
2 [keep]] <
Ekepl EeEp|]
Ee|
|ppl
(7)
Based on Eq 7 we know that to gain a better model performance, for the layer whose ab.+1
8r_
Ill is
large and V Var(p) is small, we can use high quantization level to gain model which is better than full precision model with high probability. To guarantee the success probability is high, we can set a algorithm parameter μu. Algorithm quantize Layer; only when |Epl μ..
4.3 ALGORITHM DESCRIPTION
Based on the above map between analysis and engineering, we can get algorithm [ Algorithm T is radical probability algorithm. In algorithm [ we small quantized model. Under the appropriate μ setting, algorithm I would give a better model with high probability.
high quantization level priority gain
THE LIMITATION OF ALGORITHM AND MODEL ROBUSTNESS
Although algorithm D provides better model, yet in experiments, we find that ResNet50 / ResNet101 are hard to gain a significant improvements, which makes us think our algorithm have the limitations. To show this limitations is rooted in the model properties, will prove stronger conclusion in this section.
The neural network is under the description of the probably approximately correct (PAC) learning frameworkDemilloon & Barbosa (2016). A neural network hypothesis class H consists of the neural networks which share the same structure. The learning algorithms, s, are SGD and SGD's variants for the neural network hypothesis class. Identity mapping is when the input to some layer is passed directly or as a shorteut to some other layer. The neural networks, which mainly consist of identity mappings, like ResNet or DenseNet, succeed in the CV area. Then, we can gain the following propositions.
Proposition 1 There is a set function G. For any random variable vector and any random variable vector y. 3g l which satisfies Eg(x) Ey S and g(x) belongs Ô's neighborhood.
For a well-trained neural network model", E Hn by learning algorithm st there exists modela+1 E HYb-1 which is slightly better than model", The difference between Hn and He+1 is the modela+1 E Ha+1 have one more residual block than model,+1 E He and the function in residual block G.
Brief proof: From the analysis in algorithm we can find an appropriate Ee that E can use E 3 to replace e. Then, proposition Tis proved, which also shown in figure 3
S O. We
Shar
The set, which consists of Relu (e)uu(Conw)))), satisfies the requirements of 9. Proposition tells us how to structure a deep residual network. Repeatedly using proposition [ and retraining the new model would show that for the neural networks consisting of residual blocks like ResNet, the deeper, the better. It is shown in figure 4 Using proposition in a different place, we can get different networks, like Resnet or DenseNet.
Based on the proposition
Is structure process, we can prove the following proposition
Proposition 2 For a dataset's SOTA or close to SOTA residual network, all E 88 Th.41
are close to zero.
7

---

Under review as a conference paper at ICLR 2023
Brief proof: The SOTA model implies that adding new layers will not improve model performance, i.e., for well-trained model", E Hr and well-trained model" E In+11 Esample L (model",, sample) samplle sample) O. So for any and any appropriate Ec, we have the following Eq
E
de E
sample) Esample (modeln sample)
0hi+1
EsampleLimmoddd sample) DeampteL sample) 0
Because and Ee can be chosen at random, tell that E at is zero very close
0h++1
Proposition shows one of the residual network's SOTA criterion. Then, we can prove the following theorem
Theorem
When quantization noise is under the concept of neighborhood, SOTA or near to SOTA
residual networks dataset exhibit high noise robustness.
Brief proof: Based on Eq and proposition
2
we know
de
7 Σ
Of E
llf(w) - f(w)l
E
all
Ee;
(9)
ah,+1
Sh,+1
which means the noise would have higher-order infinitesimal, i.c., o(noise), influence. Thus, SOTA or near to SOTA residual networks in a dataset exhibit high noise robustness
Hurt
Layer..
Layer
Layoni
Fostt
Outbak
Layor
Layer
Layer
Figure 4:
The deeper the neural
Figure Proof of proposition
network is, the better.
Theorem [ shows that the robustness is stronger with increase of number of layers and identity mapping when quantization noise is under the concept of neighborhood. Based on Theorem [ we can gain followint corollary.
Corollary When the quantization noise is small, algorithm cannot improve SOTA SOTA model's 's performance too much.
EXPERIMENT
In this section, we evaluate the performance of algorithm ! model gained by algorithm is better than the full precision model without "fine-tuning" technology.
Our objective is to show that the quantized
6.1 EXPERIMENT SETTING
6.1.1 DATASET AND MODEL
We make use of datasets from MNIST, CIFAR 10, CIFAR 100, and ImageNet-100.. The calibration and training datasets are separated from the training dataset. The calibration dataset's size also the same as the test dataset's.
We employ a DNN as benchmark the MNIST dataset that accordance with the Sark al. (211)) ReLU layer is between each layer in the model's apply ResNet8/14 and VGG11/13 to the CIFAR 10 dataset. We employ VGGI3, ResNet34, and mobilnet in the CIFAR 100 dataset. The mobilenet dataset for ImageNet- 100 is used. We remove the identity mapping structure from the Resnet model in our experimental models to magnify the outcomes of the tests.
design. We
8

---

Under review as a conference paper at ICLR 2023
model_dataset dataset
full model's s loss
quantized model
quantization level
vgg13_cifar10 cifar10
0,0091
0.0086
INS&FLLOAT
0.0019
0.0017
INTS&FLOAT
res8 cifar10
0.3896
0.3782
INTS&FLOAT
aas14_cifarle
0.3634
0.3576
INTS&FLOAT
CNN_mnist
0.0792
0.0774
INTS&INT4&FLOAT
CNN_mnist
0.0792
0.0786
INTS&FLOAT
CNN_mnis
0.0792
0.0813
INTS&INT4
vggl3_ _
1.2726
1.2503
INT&&FLOAT
"ggl3_cifarl00
1,2726
1.2384
INTS&INT&&FLOAT
[egl3_cifarloe
1.2726
2.4891
INTBLINTH&KiNI&FLONT
mobilenet_eifan eifar1000000
1.5653
1.5631
INTS&FLOAT
mobilenet_cifarl
1.5653
2.7669
INTS&INT4&FLOAT
res34_cifar100
1.3383
1.3293
INTS&FLOAT
res34_cifar100
1.3383
1.7104
INTS&INT4&FLOAT
mobilenet imagenet
1.6358
1.6245
INTS&FLOAT
cobilenet_imagende 1.6358
1.7313
INTS&INT4&FLOAT
Table 1: Experimental results
6.1.2 ALGORITIM SETTING
Before quantization process, we will process whole calibration dataset in full precision and find the min and max value in dataset for layer's input and compute A. We use this setting because we want enlarge the noise and get obvious experimental results. CIFAR VGG experiments, under this setting, cannot find an appropriate layer to quantize because all As are large Choices variable are zero. Thus, we use the min/max on current quantization vector like HAWQ Dong et al.) (2019a) experiments to compute A. To gain a high performance model radically possible, we set error = O, errormaz = 0.1. For CIFAR 100 and ImageNet- 100 experiments, we uses μ = 0.6 in experiments, in MNIST and CIFAR10 10 dataset, the μ is 0.4. And the value of μ is adjusted during our experiments.
6.2 EXPERIMENTAL RESULTS
In this part, we also show the range of noise which is introduced by the different quantization level in the input of the layer. In experiments, Only few layers would be larger than this level (usually less than 5e-1), For INT4, the quantization noise is less than 5e-2 and few layers would larger than 1, and these layers whose noise larger than should omit in quantization level.
find that INT8 quantization level brings less than 1e-2.
In our algorithm, we substitute secant line for gradient in Eq. which expands the applicability of Fig 5 beyond the mathematical neighborhood concept into the le-2 ball. Actually we find that the gradient can used in Eq.5 only when the noise is less than 3e-4. However, secant line would fail when the noise is larger than 5e-2, which is smaller than the noise by INT4.
In INT8 level quantization, our algorithm can find model with a lower loss function value than full-pre uull-preiisoon models. When the quantization noise is larger than 5e-2, secant line would fail in some cases.
7 CONCLUSION
This paper shows that quantization technology can improve the model's performance, i.e., gain lower loss. Moreover, based on our analysis, we propose a Radical Mixed-Precisse Inference Layout Scheme, which could produce quantized model which is better than the full- precision model. We also show that residual networks are very resistant to noise. This means that the performance of SOTA residual network is stable for any quantization algorithms.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Matthicu Courbariaux, Yoshua Bengio, and Sean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances neural information processing systems, pp. 3123-3131, 2015.
A. Demidovskij and E. Smirnov. Effective post-training quantization of neural networks for inference on low power neural accelerator. In 2020 International Joint Conference on Neural Networks (IJCNN), 2020.
Denilson and Barbosa. Understanding machine learning: from theory to algorithms. Computing reviews, 5744)2238-238, 2016.
Z. Dong, Z. Yao, A. Gholami, M. Mahoney, and K. Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. IEEE, 2019a.
Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael Mahoney, and Kurt Keutzer. H2wq-v2: Hessian aware trach-weighted quantization of neural networks, arXiv preprint 1rXiv:1911.000 2019b.
Amir Gholami, Schoon Kim, Zhen Dong. Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103. 13330 2021.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. arXiv preprint arXiv:1506.000 026226, 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv 2015.
preprint arXiv:
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 1rXiv:1608.08710 2016.
R. Li, Y. Wang, F. Liang, H. Qin, and R. Fan. Fully quantized network for object detection. In 2019 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, and William J Dally. Exploring the regularity of sparse structure in convolutional neural networks. arXiv preprint arXiv: /705 08922, 2017.
Nelson Morgan et al. Experimental determination of precision requirements for pack-propagation training of artificial neural networks. In Proc. Second Int'l. Conf. Microelectronics for Neural Networks, pp. 1-16. Citeseer, 1991.
Markus Nagel. Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pp. 7197-7206. PMLR, 2020.
X. Qian, V. Li, and Darren. Channel-wise hessian aware trace-weighted quantization of neural networks. 2020.
Roesch, Lyubomirsky, M. Kirisame, J. Pollock, and Z. Tatlock. Relay: A high-level ir for deep learning. 2019.
C. Sakr and N. Shanbhag. An analytical method to determine minimum per-layer precision of deep neural networks. In 2018 IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), 2018.
C. Sakr,, Y. Kim, and N. Shanbhag. Analytical guarantees on numerical precision of deep neural networks. In International Conference on Machine Learning, 2017.
Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression. arXiv preprint arXiv: 7707000200000 2017.
10

---

Under review as a conference paper at ICLR 2023
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware- aware automated quantization with mixed precision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8612-8620, 2019.
Z. Yao, Z. Dong, Z. Zheng, A. Gholami, and K. Keutzer. Hawqv3: Dyadic neural network quantization. 2020.


Reviews:
[{"review_id": "lljjercRQ6S", "reviewer": "Reviewer_avng", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tries to provide a method for mixed-precision quantization. A statistical analysis is provided to quantify the impact of quantization on model accuracy. This method is not novel.", "summary_of_the_review": "As far as I am aware, this paper has used results from prior works without proper credit. On top of that, there is nothing really novel presented in the paper. Finally, there are almost zero empirical results, only a few sentences provided on a supposed accuracy using quantization.", "strength_and_weaknesses": "The biggest concern I have with this paper is that, as far as I know, it is a serious case of using prior results without giving credit, nor adding any novel contribution on top. The analysis starting in eq. (4) has already been established in [1] who also used Chebyshev's inequality to obtain bounds on the accuracy in the presence of quantization similar to the way this paper arrived to eq. (8). Similarly, using this analysis to formulate a mixed-precision quantization strategy was proposed in [2].\n\n[1] Sakr, Charbel, Yongjune Kim, and Naresh Shanbhag. \"Analytical guarantees on numerical precision of deep neural networks.\" International Conference on Machine Learning. PMLR, 2017.\n[2] Sakr, Charbel, and Naresh Shanbhag. \"An analytical method to determine minimum per-layer precision of deep neural networks.\" 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "diEBABLTU2", "reviewer": "Reviewer_JF51", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a method, called \"Radical Mixed-Precision Inference Layout Scheme\" to obtain a mixed-precision quantized model that has lower loss than the full-precision model. The paper provides mathematical justification on why deep neural networks are robust to quantization. Evaluation of CIFAR10 and several residual networks show that the proposed method can obtain quantized models with lower loss than the corresponding full-precision models. ", "summary_of_the_review": "Although offering some interesting perspectives on why DNNs are robust to quantization and the proposed mixed-precision quantization scheme is able to show some promising results on small datasets, the paper still has a large room for improvements in its writing quality and evaluation. ", "strength_and_weaknesses": "Strengths:\n- The paper offers some interesting perspectives on why neural networks are robust to quantization from a computational noise perspective. \n- The paper also offers some mathematical justification for why residual networks are robust to noise.\n\nWeaknesses:\n- There is no clear definition or description in terms of the problem this paper tries to address. Is the main goal to provide a mathematical justification on why DNNs are robust to quantization, or is the main goal to propose a better method for mixed-precision of quantization. From the reviewer's perspective, the first one is a very hard problem and may not even be true because, in practice, different models can have very different sensitivity to quantization (e.g., some generative models are very sensitive to quantization), and even the same model can have drastically different quantization results depending on how many bits are used for quantization and the quantization schemes. Unfortunately, none of those are covered in the theoretical analysis. If the goal is the latter one, then clearly the paper also fails to achieve that goal because the evaluation is largely inadequate due to the small datasets and models. \n- Evaluation is very much done using toy datasets and small-scale models, making it hard to be convinced that the observations (e.g., lower loss from quantized models using the proposed method) can be generalized. \n- There is a big improvement room for the writing quality of this paper. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1WQ8zSfKsV", "reviewer": "Reviewer_kdep", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tried to explain a theoretical way to find a quantization model which is better than the anchor full precision model. My main concern is about the experimental results. There are only a few of results which are not persuasive. Moreover, there is no comparison with previous works.", "summary_of_the_review": "Please check the Strength And Weaknesses. I think the current version is hard to be accepted since the results are not comprehensive. However, I am willing to increase the grade if the author can provide more evidence to persuade me.", "strength_and_weaknesses": "Strength\n\nThe author tried to prove that for a specific full precision model, there always exists a quantized model which can reach or even outperform the accuracy of the full precision baseline.\n\nWeaknesses\n\n1. Some assumptions in the derivation are confusing for me. For example, in Eq. 7, why e and p are independent to each other? Also, above Section 4.1.2, the author said \"this is why channel-wise quantization methods are booming\", what is the relationship between \"channel-wise\" and equations 4-6?\n2. The author said \"Thus we have to choose the model which is far from the SOTA model\". Is this statement related with Theorem 1, to say, for the (near) SOTA, your method cannot improve model performance too much? If so, I cannot understand the logic. Looks like you have to select some non-SOTA models as the baseline. Then even your quantization can be better than non-SOTA fully precision baseline, it might still be worse than some other quantization methods based on SOTA baseline?\n3. The author said \"like V100 GPU, only support INT8,INT16 and INT32 computing in hardware\". If you did not intend to give the running time, you don't have to be restricted with the GPU runtime of some specific hardwares. Therefore, you can try more mixed precision such as lower bits (i.e. 2/4/6 bit).\n4. It is better to compare with some other network quantization methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject