Mixed-Precision Inference Quantization: Problem Resetting, Mapping math concept and Branch\&bound methods
Authors: Daning Cheng
Venue: Submitted to ICLR 2023 (2023)
Keywords: Mixed-Precision Quantization
Inference
Neural networks
Noise Robustness
NP hard

Abstract: Based on the model's resilience to computational noise, model quantization is important for compressing models and improving computing speed. Existing quantization techniques rely heavily on experience and "fine-tuning" skills.   This study shows that in inference process, the mixed-precision quantization is a NP-hard problem and we designed a set of method to map mathematical method into pratical method. And our problem setting can be solved by branch and bound method with less computing resouces. We also show that how to set the quantization parameters in theorical method.
 

Under review as a conference paper at ICLR 2023
MIXED-PRECISION INFERENCE QUANTIZATION: PROB- LEM RESETTING AND TRADITIONAL NP HARD PROB- LEM
Anonymous authors Paper under double-blind review
ABSTRACT
Based on the model's resilience to computational noise, model quantization is important for compressing models and improving computing speed. Existing quantization techniques rely heavily experience and "fine-tuning" skills. In this paper, we map the tixed-precisiog layout problem into a traditional NP hard problem and the problem can be solved by low cost methods like branch and bound method without "fine-tuning". In experiments, experimental results show that our method is better than HAWQ-v2, which is one of the current SOTA methods solve mixed peeeddd-eessions layout problem.
INTRODUCTION
Neural network storage, inference, and training are computationally intensive due to the massive parameter size of neural networks. Therefore, developing compression algorithm for machine learning models is necessary. Model quantization, based on the robustness of computational noise, is one of the most important compression techniques. The computational noise robustness measures the algorithm's 's performance when noise is added during the computation process. The primary sources of noise are truncation and data type conversion mistakes.
The initial high-precision data type used for a model's parameters is replaced with a lower-precision typical to replace FP32 with FP16, and both PyTTorch and
data type during model quantization. TensorFlow have quantization techniques that map floats to integers. Various quantization methods share the same theoretical foundation, which is the substitution of approximation data for the original data format requires less memory,
data in the storage and inference processes. A and using power-precistoe data requires fewer computer resources and less time. In quantization, the precision loss in different quantization level conversions and data type conversions is the source of the noise.
The primary issue with the model quantization methods is that naive quantization scheme is likely to raise the loss function. It is not easy to substitute massive-scale model parameters with extremely low-data precision without sacrificing significant precision. It is also not possible utilize the same quantization level, i.e., to introduce the same level of noise to all parameters for all model parameters and get good performance.
Utilizing mixxed-precisee quantization is one way to solve this issue. For more "sensitive" model parameters, figher-precision data is used, whereas lower- wwerr-peecsioon data is used for "nonsensitive" model parameters. Higher-precision data indicates that the original data adds small noise, while lower-precision data indicates that the original data adds large noise. But mixed-ppecisse also has its restrictions: In computing processes, for example, GPU and CPU must use the same data type in a computing process.
quantization
Moreover, the following facts challenge current algorithms for mixed-precision algorithms: 1. These algorithms are built on empirical experience and "tuning" skills. 2.Some algorithms forego neural network and dataset analysis. Some algorithms base model quantization hardware features. It impossible to show the bounds of algorithms' performance. 3. Some algorithms utilize Hessian data. The majority of them analyzable. However, obtaining Hessian information necessitates

---

Under review as a conference paper at ICLR 2023
considerable computing resources and time. Some of these methods are only useful for storing purposes.
The consensus among researchers is that quantization technology without "fine-tuning" is harmful to the performance of the model. Current quantization technology widely uses the "fine-tuning" method, but can explain why their algorithm cannot work without "fine-tuning", even though the basis of their algorithm is well-defined in math.
This paper establishes the model quantization problem setting for inference processes. Based analysis, we map layerwise PTQ problem into one of the traditional NP-hard problems, ixe.,extended 0-1 knapsack problems, which can be solved by the branch and bound method without too much computing resources or inference/training model too many times. Thus, our work gets rid of "fine- tuning" skills and has clear interpretability. Compared with the SOTA mixed-precisioe algorithm HAWQ-v2 without "fine-tuning" the quantization model based on our work is better than the model from HAWQ-v2 under the same computation resource limitation.
RELATED WORK
Model compression methods include pruning methodşHan al. 2015 ; Li (2016); Mao
et al. (2017). knowledge distillatiorHiiton et al. (2015), weight sharing UlIrich et al. (2017) and quantization methods. From the perspective of the precision layout, post- training quantization methods can be mainly divided into channelwise [i et al. (2019); Qian et al. (2020), groupwise Dong et al. (2019h) and layerwise (Dong et al. (20T9a) methods. Layerwise mixed-precision layout schemes more friendly to hardware. Parameters of the same precision organized together, making full of a program's 's temporal and spatial locality. A common problem definition for quantizatior al. 2019a) Morgan al. (1991) Courbariaux et al. (2015) Yao et al. (2020) is as follows Gholami et al. 2021).
Problem
The objective of quantization is to solve the following optimization problem:
min
[lq(w) w|l
qEQ
where is the quantization scheme, g(w) is the quantized model with quantization q. and w represents the weights, i.e,, parameters, the neural network.
Although problem gives researchers a target to aim for when performing quantization, the current problem definition has two shortcomings: 1. The search space of all possible mixed-precision layout schemes is a discrete space that is exponentially large in the number of layers. There is no effective method to solve the corresponding search problem. 2. There is gap between the problem target and the final task target. As we can see, no terms related to the final task target, such as the loss function or accuracy, appear in the current problem definition.
BACKGROUND ANALYSIS
3.1 MODEL COMPUTATION, NOISE GENERATION AND QUANTIZATION
Compressed models for the inference process are computed using different methods depending on the hardware, programming methods and deep learning framework. All of these methods introduce noise into the computing process. One reason for this noise problem is that although it is common practice to store and compute model parameters directly using different data types, only data of the same precision can be support precise computations. Therefore, before performing computations on nonuniform data, a computer will convert them into the same data type. Usually, nower-precision data type in a standard computing environment will be converted into higher-precisioe data type; this ensures that the results are correct but require more computational resources and time. However, to accelerate the computing speed, some works artificial intelligence (AI) computations propose converting higher precision data types into lower- precisiien data types based on the premise that Al models are not sensitive to compression noise. The commonly used quantization technology is converting data directly and using a lower power-precision data type to map to a linearly.
data type
2

---

Under review as a conference paper at ICLR 2023
We use the following example to illustrate quantization method, which is presented in Yao al. (2020). Suppose that there are two data objects imput, and inputy are to be subjected to computing operation, such as multiplication. After the quantization process, we have Q1 = int (mputs and
scaley
Q2 int (mputz) and we can write
scalez
Qoutput int( imput * imputz ) int(QQ scale1 scalez
scaleoutput
scale
scaleoutput scale, and scalez are precalculated scale factors that depend on the distributions of impula, impul2 and the output; Qi is stored scale terms can be precalculated and established ahead of time. Then, throughout the whole inference process, only computations on the Qi values needed, which are fast. In this method, the noise is introduced in the int(-) process. This basic idea gives rise to several variants, such as (non)uniform quantization and (non)symmetrical quantization.
lower-precision data type, such as an integer. All
When we focus on quantization strategy, i.e. round function in quantization framework like Micronet, can have least three strategy: round up. i.c., ceil function in python, round down, i.c., floor function in python and rounding, i.e., round function in python. usually, rounding is the most common method deal with quantization. But, in this paper, will show that how to mixed use round up/round down to gain mixed precision quantized model which is better than full precision model.
3.2 NEURAL NETWORKS
In this paper, we mainly use the mathematical properties of extreme points to analyze quantization methods. This approach is universal all cases, not only neural networks. However, there is myth in the community that it is the neural network properties that guarantee the success of quantization method Wang al. (0199; Morgan al. (1991) Demidovskij & Smirnov (2220). To show that the properties of the extreme points, not the properties of the neural network, are what determine the ability to quantize, i.e. the ability to handle noise, we must first define what a neural network is.
The traditional definition of neural network Denilson & Barbosa (2016) as a human brain simulation ambiguous; it not a rigorous mathematical concept and cannot offer any analyzable information. The traditional descriptions of neural networks Dennlsonn & Barbosa (2166) focus on the inner products of the network weights and inputs, the activation functions and directed acyclic graphs. However, with the development of deep learning, although most neural networks still consist of weighted connections and activation layers, many neural networks no longer obey these rules, such as the network architectures for position embedding and layer norm operations in Transformers. Moreover, current deep learning frameworks, such as Py Torch and TensorFlow, offer application programming interfaces (APIs) to implement any function in layer. Therefore, we propose that the definition of neural network adheres to the engineering concept indicated by the definition ratthr than a precise mathematical definition; that is, a neural network a way for implementing a function.
Definition The neural network is the function which is implemented in composite function form.
A neural network can be described in the following Eq. form. model(x) = (,2,,, ( ... h3,k W2,1), 12.2 (b.2(hs,x+1(..) where hi.j. i E .2....,, n),, are the (n i + 1)th layers in the neural network; way is the parameter in (-)
W/) ..., 22.2).... wE)
(1)
Definition) D means that a neural network, without training, can be any function. With definition M neural network no longer mathematical concept, and this idea is widely used in practice Roesch et al. 2019). We can see from definition that the requirement that neural network is in composite function form is the only mathematical property of neural network that can used for analysis.
In practice, the loss function is one method to evaluate a neural network. A lower loss on dataset means a better performance neural network. For example, the training process optimises the model's loss, i.e., following Eq. 2
min (ww Esampte (ww, sample)
8(w, T4, Uz)
(2)
3

---

Under review as a conference paper at ICLR 2023
where f()) is the loss for model dataset, represents the model parameters, D is the dataset, the size of the dataset, E(-) is the loss function for a sample and (T', y4) represents a sample in the dataset and its label.
In this paper, we mainly use the sequential neural network to describe the conclusion for the sequential neural network is easily described, and the whole conclusion is non-related the structure of the neural network. For sequential n-layer neural network, {(-) can be described the following Eq form.
f(w, Us, Me) = L(modela w), v/)
model, h1 (12(ha (h Wn) w4), Wa), w2), w1)
where L(-) is the loss function, such the cross-entropy function; hG, [I,..., m.), is the (n 1)th layer in the neural network; w (w w "", is the parameter in hę(:); and for unified format, ha+1 stands for the sample z.
START POINT AND THEORY PROBLEM SETTING
4.1 QUANTIZATION TARGET
The problem cannot expose any useful information. Thus, we have to setting another quantization target. Because of the natural of quantization the trade of the performance and the computation resource. From the current work, we know that generally, the less computation resource use, the worse the model's performance is. We want to give the model that has best quantization performance with limited computation resource, like limited memory cost. The performance of model can be measured by loss function. Thus, using memory cost as computational resource example. we have following problem setting.
Problem 2
min llf(w) f(w)ll, s.t.||w||
(4)
where C the limitation of memory cost, f((w) is the loss of model after quantization.
Again, problem 2 setting also cannot be solved. We need more description about f.
4.2 ANALYSIS BASE
Quantization methods for inference are complex. In addition to the noise added to the parameters directly, noise also introduced between different layers in the inference process because different quantization levels or data types of different precisions are used in different layers, which shown in figure
layer,
layer,
layer,.1
Icoa
(h.)
(h
(h,)
function (L)
Parameter
Parameter
Parameter (wr)
x(h
(Wo)
outpuf
Input
(Wa.)
output
input
output
Inuul
label
loss
The quantization models for use in the inference process. When quantized models are used in the inference process, the outputs of different layers suffer from noise due to the conversion between the different quantization levels of different layers in layerwise quantized models.
Figure 1:
After quantization, the quantized loss for sample, i.c.((-), in the inference process is as follows. f(w, Ft, 9:) L(ha ( -, (hu+1 cn, w., + 5n)) Cn-1"" ". ; W2 + 62) C1, wi + 51),
4

---

Under review as a conference paper at ICLR 2023
where d,. 1,... n, and C,, i E [1,...,n]] parameter quantization and data type conversion in the mixed peeedd--eission layout scheme, respectively,
n².. the minor errors that introduced in model
Thus, obtain the following expression based on the basic total differential calculation.
de
f(w, Izs vi) f(w, 24 24)
E
E, + Ow,
6,
0he41
where is inner product and the scalar product in following parts. For the loss on whole dataset, we can gain
de
de Ow.
min f(w)) - f(w) kEE
Σ
+
6
m
0hi+1
(B) i=
de
E
1 S
Σ (E)
Ol
Ei
S
1
Σ (D
de
||&|||
m
0hi+1
m
Thit1
m
0hit1
(E
(6)
where f(w) 1 Σ E(-). The reason for second equation in Eq. 6 is for well-trained model, the expectation of ((-)'s gradient for parameters is zero, i.e., for the > (x),;)) aan ae 0. Bic
at components,
As we can see, current works mixed discussed the quantization for storage and inference Dong et al. (2019a); Yao al. (220);: Dong et al. (2019b); Nahshan et al. (202T). Consequently, these works must add "fine-tuning" process, and they still fail in some cases. Moreover, this is why channel-wise quantization methods are booming. In a channel that uses the same data type at all times, the precision loss of the corresponding layer input is usually zero.
frequently asked question is why (x Bue zero but Σ (ED Bhet1 non-zero. The at random in
optimization algorithm is optimize in the training process. Thus, Σ E(e.g) Sh; i1 the final model except for the layers with bias terms like the batch norm layer. The bias term will absorb the gradient and train them in the optimization process. What is more, in the model, which Df close to zero vector.
mainly consists of identity mapping, Σ E(EW)EE ōh;11
4.3 THEORY PROBLEM SETTING
Eq. 6 makes it extremely clear that the noise supplied from the output of the previous layer to the input of the later layer in neural network exerts first-order effect the change of the final loss function, whereas noise on weights exerts a second-ordes effect on the loss function. In the quantization process, the input and output of the layer should take precedence over the quantization method, while the quantization of the weights should accommodate the impact brought by the quantization between layers.
ae is large, the input and the weight of that layer should When the gradient between layers, i.e., 0h; be quantized with a more precise approach, i... more bit data type. In this case, although it is possible that weight quantization in this layer does not require high-bit data type, for example, the distribution of weight is binary distribution, it cannot be stored with a simple boolean data type due the significant error generated by the input layer quantization. When the gradient between layers is small, we can select data type with the lowest possible precision to boost the speed of computing while still satisfying the mathematical concept of "neighborrood"" for the magnitude of the noise introduced in both the layer weights and the layer inputs.
Based on the above description, we can derive the main idea of the algorithm: we want to solve the following problem by using total differential as the benchmark and prediction function, adjusting the quantization 8h, in a mixxprrccsioon quantization scheme, and quantizing weight to accommodate the quantization of corresponding at We modified problem 2 into the problem 86,-
de
5

---

help narrow the Under review as a conference paper at ICLR 2023
Problem 3
min
de
|cs|| or
Σ (Zg.8))ED
de
1.t.||w1| <C
ah1+1
0h+t1
(D
where C the limitation of memory cost.
Our problem setting for quantization is different from previous work like HAWQDong et al. (2019a): Yao et al. (2020); Dong et al. ( Nagel et al. (2020) because these methods do not take the error the layer's 's input into consideration, which prevents their work and analysis in the mixed-preeision computing area. As a result, these works can only be used to store a compressed neural network on disk. When the compressed model is stored in memory for inference, these compressed models have to be recovered into the full precision model.
MAP MATHEMATICAL CONCEPT INTO PRACTICAL OPERATION AND REAL PROBLEM SETTING
Although we have a target problem 3 yet all variable in Eq. is purity mathematical concept and cannot be used indicate the practical operation like quantization level choosing. Thus, have to map mathematical concept into practical operation
5.1 MAP NOISE INTO QUANTIZATION LEVEL
Before using Eq. design algorithm, have deal with the problem that maps the math concepts into practical operations and concepts that can be implemented by a computation device.
Based the preceding discussion, the first consideration when mapping from the concept of noise to the quantization level or data type is how to describe the noise into data type and how to design a method in which noise can be covered by the mathematical concept of "neighborhood" And if the value of noise exceeds the mathematical concept of "neighborhood" it must be adjusted based on the practical environment, computation source, and target.
5.1.1 MAP e INTO QUANTIZATION METHOD PARAMETERS
Based on the above explanation, the fluctuations due quantization are almost linear function of the c. Thus in order to restrict the changes of the loss function caused by quantization, the c vector should have the smallest second norm value, i.e., the quantization parameter setting's objective is min 2. On the basis of the above objective function, it is simple to choose optimal parameter configurations for many quantization schemes.
Using symmetric and asymmetric quantization schemes as examples in this paper, we need to set a clip value for the (a)symmetric INT quantization scheme. When the value is either less than clip greater than clip+, treat them clip(+/-), Using the right clip setting range of data distribution and improve the accuracy of INT expression. Thus, the values of clip(+/-) parameters are very important. In this part, we will show why the ACIQ quantization methodBannee et al. (2019) is reasonable which is experimental results currently.
The traditional approach is to use the KL divergence to determine the clip value. The problem is that the approximation of the KL. divergence is not approximation in the aspect of the norm. It only represents the difference between the two distributions terms of information theory. But the difference in information theory cannot be directly reflected in the change of the loss function. In general, the mathematical property that directly reflects the change in the loss function is the loss function's gradient the value that is relative the gradient.
Simple proof of ACIQ Because of the Eq. 6 the fluctuation of loss function is related to the is constant number for a network on dataset. Thus, we have minimize the ||]]] ie Thus, when we use asymmetric quantization, then we should choose a clip value that satisfies the following conditions. When the data distribution is p(z)), and quantization datatype has
a ||and ||= || 3

---

Under review as a conference paper at ICLR 2023
B bits, the objective function is as following.
min ||
min
clp
(x (Ø-clip) clip - +
ciip
(x clip
x
x
(x- c'ppptpppppp ( (())d
clip
clip
Jdipt
The above problem can be solved by direct derivation after determining the form of p(x), which is the content of the workBanner et al. (2019).
5.1.2 MAP at Th. INTO THE SLOPE OF SECANT LINE
In our discussion, a prerequisite for the correct application of total differentiation is that the noise that we introduce must be small enough to permit error estimation using total differentiation. However, in practice, we always expect faster computation by employing coarser data storage methods and lower-bit data types. In practice, there may be instances where large noise is used in the quantization process, such as quantization with INT4 or 2-value, 3-value on a vector with a wide value distribution. In our experiments with ResNet8 and 14 the cifar10 dataset, the noise when quantization using the above quantization scales is roughly in the range of 0.1 to 0.3, indicating that the noise is outside the range that can be expressed in terms of "neighborhood".
Higher order differentials are useful for solving precise prediction problems, but a first-order approxi- mation is more practical due computational and memory constraints. As demonstrated in the figure 2 below, secant lines perform significantly better than tangent lines, whose slope is gradient, in the case of larger noise, particularly for points close to the extreme points. the work Cheng Chen (2022). it was shown that for the SOTA models and near SOTA models, the gradient of the inter-layer inputs, i.e.,,, tends to be zero.
MR
openes
Figure 2: When the predicted point(r,,) is out of the neighborhood range but not pretty far from Ho, secant lines between the 2g and Hı Figure 3: Different directions have different perform significantly better than tangent lines. secant line prediction process.
The choice of æ1 is the maximum quantization noise practice.
However, the secant line is different in different directions, which shown figure 3 The direction of the secant line on each feature is different, which leads having compute thousands of different secant lines. To avoid this issue, we only use floor or ceiling quantization methods in a vector to ensure that all data the vector has the same directions. The more computing method the content of the work Cheng & Chen (2022).
5.1.3 MODEL WEIGHT QUANTIZATION
Based analyses, we know that the real impact on the fluctuations in the loss function is the error in the quantization of the data between the layers, i.e value of c, and that the error in the weights in layer almost has no impact the loss within the "neighbourhood"" Ideally, the quantization, which is leaded by the 3
would expect Th. is accepted by the concept of the "neighborhood" of weights within layers. In this case, the problem can be set up to easily reach theoretical extreme points, and what important, the computation cost in this case is low. For the reason as in the
7

---

Under review as a conference paper at ICLR 2023
Algorithm I: Change Mixed-Precision Inference Layout Problem into Low computation cost NP hard problem
Input: Trained Neural Network M which has layers, Different Quantization Levels Calibration Output: Price Matrix P. Weight Matrix W, matrix sizes are n*k
(2,,02,... ... Qk .error mar. Callibbaatooo dataset F)
1 Forwark and Backward Network on Calibration Dataset, Collection the distribution of data inputs calibration.;
for qj in Q do
for Layer; in M do
Compute the W[]].. (For example, we quantizate model for the memory cost,
W[]][]] is the the memory cost of Layer; on yj quantization level);
3. Compute ||]] and scolesmput based on the input of Layer;
4. Compute snput = (M - finput (M; scaleinput' /ccaleippp | where
finpue M; scale; 1) is the loss on the M with the input of M ' Layer; add the scalesmput * [1, 1,,, 1] for ceil quantization or scaleinput [-1, -,,, -1] for floor quantization.:
Compute fluc (Mf weeight (M; scale i))ll, where fmeight (M; scalemoput; i) is the loss on the M with the weight of M's Layer; add the scale * 1,, 1] for ceil quantization or scale 1-1,- -1,,,,,, -1] 1] for floor quantization.;
fluc < error max then
6.P[d][ = slopeinput * ||e//// ssize(/) size(c;);
7.Continue;
end
Compute ||6,]| and scaleweight based on the weight of Layer;;
9. P slopeioput * ||eil// size(es) fluc/ scale * ||6:1// V size(ó:);
end end
return Price Matrix P. Weight Matrix W
above sections, in practice the noise introduced by quantization often becomes uncontrollable due to the need for speed/model size due the quantization needs of the layer input vectors passively. Therefore, the quantization level of this layer must be adjusted to match the weights. In practice, we will first determine whether the noise introduced by the quantization of the model parameters will have significant effect. The parameter setting in weight quantization is the same as in section 5.1.1. When the noise is significant, it must corrected by the secant line approach, which shown in 5.1.2.
5.2 PRACTICAL PROBLEM SETTING AND SOLVING
In the above sections, change the mathematical concepts in Eq. Ginto engineering computations one by one. Then we obtain the following algorithm [[] In this problem transformation, we map through the relationship between quantization level and the noise and the analysis stability of a model on the target data set. To extend the use of the gradient, we use the secant line for linear prediction in the case of larger noise. We convert the quantization problem to low computation cost NP hard problem, i.e., extended 0-1 knapsack problem as following problem 4
Problem 4 The Low Computational Cost Equivalence Problem of Mixed-precision layout problem
min
PiLL) s.t.
WiLL < C.j e [1, and j Z
where C is the constraint for quantization target. For example, if we quantization model for memory cost, the C' is the maximum memory size.
8

---

Under review as a conference paper at ICLR 2023
Problem 4 is the equivalence problem with the problem Compared with problem 3 all elements in problem and algorithm|] [ are corresponding to the quantization process parameters or operations. By choosing and selecting at least one P[ in each row of matrix price and one value [ the corresponding position in matrix weight, we guarantee that each layer knows which quantization level they should choose reach the quantization target. Problem A easy problem be solved by the pranch-and-boul method. This problem can be solved very quickly because it doesn't t need inference or training on the original neural network. Instead, only needs small number of additions and multiplications.
In traditional concepts, is generally believed that the less use the quantization bit, the larger the impact on performance, which can be expressed by the value of the loss function. Based on our analysis, i.e., the nature of the 0-1 knapsack problem, we verify that this expectation is correct when quantization introduces small noise. However, it often finds examples of quantization models with better performance with extremely small quantization bits in different papers. This is due to the inability to use the concept of the "neighborhood" of the points for the estimation of the prediction points, i.e., the use differential or Taylor expansions, which leads to unstable prediction results in the presence of large noise. Given the lack of techniques to perform Fourier expansions neural networks, the analysis and solution of the problem in the presence of large noise cannot be solved at the current stage.
EXPERIMENT
In this section, we evaluate the performance branch and bound method on problem Our objective is to show that branch and bound method based on proposed problem is better than the full precision model without "fine-tuning" technology, which we use HAWQ-v2 as benchmark.
We use CIFAR 10 dataset. The training dataset is split into calibration and training datasets. Further- more, the size of the calibration dataset is equal to the test dataset.
Based on the workCheng & Chen 2022), the model with many identity mapping structures has strong noise robustness. Thus we have to choose the model which contain less residual structure. For CIFAR 10, ResNet 20 is close to the SOTA ResNet model(ReNNet 110), which mains the layers' input gradient small. Thus, we choose ResNet 8 and ResNet 14 as our quantization models. Furtherly, to we delete the identity mapping structure in our experimental models.
enlarge the E D6
In quantization practice. quantizing data into INTS the most frequently used and ripe choice because current computation devices, like V100GPU, only support INT8, INT16 and INT32 computing in hardware. Thus, we use mixxe-ppreciioon INTS and full (all-rrecisoonFFP33)) in our experiments. We will compress these model, and our target is to reduce memory consumption by 20 percent. In this experiments, we set errormes = 0.1 to reduce the computing time.
The loss for ResNet8 with the full precision is 0.3896. The loss for the quantization ResNet8 model 0.4168 by our method. The loss for the quantization ResNet8 model is 0.4418 by HAWQ-v2. The loss for ResNet14 4 with the full precision is 0.3634. The loss for the quantization ResNet8 model is 0.3892 by our method. The loss for the quantization ResNet14 model is 0.3964 by HAWQ-v2. The experimental results match our analysis.
CONCLUSION
In this paper we propose the algorithm that change mix-precision problem, i.c. problem 2 into low computation cost NP hard problem, i.e., problem 4 Based the total differential, we show that the fluctuation of loss function mainly related to the gradient of layer's input and the noise of quantization. We map mathematical concept into practical operations. After gain the NP hard problem, branch and bound methods to solve the problem efficiently and our quantized model is better than the model from HAWQ-v2 with the same quantization limitation.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. Advances in Neural Information Processing Systems, 32, 2019.
Daning Cheng and WenGuang Chen. Quantization in layer's input is matter. arXiv preprint arXiv:2202.005137. 2022.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pp. 3123-3131, 2015.
A. Demidovskij and E. Smirnov. Effective post- post-training quantization of neural networks for inference on low power neural accelerator. In 2020 International Joint Conference on Neural Networks (IJCNN), 2020.
Denilson and Barbosa. Understanding machine learning: from theory to algorithms. Computing reviews, 57(4):238 238, 2016.
Dong, Z. Yao, Gholami, M. Mahoney, and Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. IEEE, 2019a.
Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. arXiv preprint a9 1.03852, 2019b.
Amir Gholami, Sehoon Kim, Zhen Dong. Zhewei Yao, Michael W Mahoney, and Kurt Keutzer.
survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2/03. 13630, 2021.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. arXiv preprint arXiv:1506.02622 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1003 2015.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv: (60.:::666..07110000 08700, 2016.
R. Li, Y. Wang, F. Liang, H. Qin, and R. Fan. Fully quantized network for object detection. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, and William J Dally, Exploring the regularity of sparse structure in convolutional neural networks. arXiv preprint 10X5::1705.08922. 2017.
Nelson Morgan et al. Experimental determination of precision requirements for back- proppppppgntioon training of artificial neural networks. In Proc. Second Inf'l. Conf. Microelectronics for Neural Networks, pp. 9-16. Citeseer, 1991.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pp. 7197-7206. PMLR, 2020.
Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M Bronstein. and Avi Mendelson. Loss aware post- pust-training quantization. Machine Learning. 110(11):325532622 2021.
X. Qian, V. Li, and C. Darren. Channel-wise hessian aware trace-weighted quantization of neural networks. 2020.
Roesch, Lyubomirsky, M. Kirisame, J. Pollock, and Z. Tatlock. Relay: A high-level ir for deep learning. 2019.
10

---

Under review as a conference paper at ICLR 2023
Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression. arXiv preprint arXiv: 1 170.004008 2017.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8612-8624 2019.
Z. Yao, Z. Dong, Z. Zheng, A. Gholami, and K. Keutzer. Hawqv3: Dyadic neural network quantization. 2020.


Reviews:
[{"review_id": "nBBmQicXFn", "reviewer": "Reviewer_hSyo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper lacks clarity, so I reproduce here my best understanding of its contents.\n\nThe paper proposes to search for the optimum bit width (or more generally, numerical precision) for each layer in a\nneural network during post-training quantization. To this end, the authors assume quantization induces an additive\nprice model -- <i>i.e.</i> each layer's quantization incurs an accuracy cost with respect to the full precision version,\nindependent on the quantization parameters of the other layers -- and an additive cost model -- i.e. layer computational\ncosts are independent on each other. Under these assumptions, finding the optimal bit widths can be cast as a 0-1\nknapsack problem. The price for quantizing each layer is estimated via a first-order linear approximation of the loss\nfunction with respect to the quantization-induced noise. The only experiment the authors report is the classification\nloss (not clear which loss is being reported) for ReseNet8 on CIFAR-10.\n", "summary_of_the_review": "This work does not seem to be technically novel. The presentation style and mathematical rigour needs to be\ngreatly improved. To me, there are many unclear points and questions regarding correctness. Empirical support for the\nmethod and hypotheses is lacking (and I would expect that the additive model does not hold in practice). Overall,\nI do not think this work meets the minimal standards for publication at ICLR. \n\n", "strength_and_weaknesses": "* Strengths:\n\n  * In cases where the independence assumptions hold, the proposed algorithm should be computationally efficient. \n\n* Weaknesses:\n\n  * The paper lacks clarity and notational rigour (please see notes below on clarity).\n  * The assumption of additive accuracy price and computational cost model may not hold in practice and the authors\n  do not conduct a study on this. The total accuracy degradation is likely not a linear function of layer-wise accuracy\n  losses, as layers interact via feature maps. Furthermore, if one considers running time as computational cost,\n  inter-layer interactions arise from cache locality and bandwidth effects, leading to non-additive models.\n  * The authors simply report the loss (not clear what loss is being reported) and compare against a single baseline,\n  on a single (and not very challenging) classification dataset with two network architectures. This is hardly\n  convincing. No ablation studies are shown and the additive model hypothesis is not validated.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "UvX9OhRGNa", "reviewer": "Reviewer_nDrD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose a neural network quantization method by mapping the mixed-precision layout problem into a traditional NP-hard problem, and solve the problem by some efficient implementation of relaxed solution to these NP-hard problem. ", "summary_of_the_review": "This paper is difficult to track, and the claimed contribution is not supported by the experiments.", "strength_and_weaknesses": "Weakness:\n\n\n- The paper is not well-written. \n\n- The empirical study is casual.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "C986ZR3nOs", "reviewer": "Reviewer_y7Tw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In  this paper,  the mixed-precision layer problem is formulated as a traditional NP hard problem and the problem can be solved by low cost methods without fine-tuning. The experimental results show that the proposed method is better than the current SOTA method.\n", "summary_of_the_review": "To solve the challenging mixed-precision inference quantization issue, this paper proposed a theoretical framework to convert the original problem into a traditional NP hard problem, the derivation and writing is clear. However, the experimental results are limited to show the benefits on popular neural networks.", "strength_and_weaknesses": "Strengths:\n1. Based on the total differential calculation, the paper proposed a theoretical framework incorporating bot the quantization error and data type conversion error. \n2. The paper also proposed the secant line method which fixed the large approximation error issue.\nWeaknesses:\n1. The method mainly focuses on the models which contain less residual structure. However, most of the popular models contain the residual structure such as transformer et al. \n2. The experimental results are not enough to show the strengths of the proposed framework: it only test simple architectures. It will be much better to show more results on more models, more baselines and more statistics such as accuracy and so on.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject