F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization
Authors: Qing Jin
Jian Ren
Richard Zhuang
Sumant Hanumante
Zhengang Li
Zhiyu Chen
Yanzhi Wang
Kaiyuan Yang
Sergey Tulyakov
Venue: ICLR 2022 Oral (2022)
Keywords: Neural Network Quantization
Fixed-Point Arithmetic

Abstract: Neural network quantization is a promising compression technique to reduce memory footprint and save energy consumption, potentially leading to real-time inference. However, there is a performance gap between quantized and full-precision models. To reduce it, existing quantization approaches require high-precision INT32 or full-precision multiplication during inference for scaling or dequantization. This introduces a noticeable cost in terms of memory, speed, and required energy. To tackle these issues, we present F8Net, a novel quantization framework consisting in only ﬁxed-point 8-bit multiplication. To derive our method, we ﬁrst discuss the advantages of ﬁxed-point multiplication with different formats of ﬁxed-point numbers and study the statistical behavior of the associated ﬁxed-point numbers. Second, based on the statistical and algorithmic analysis, we apply different ﬁxed-point formats for weights and activations of different layers. We introduce a novel algorithm to automatically determine the right format for each layer during training. Third, we analyze a previous quantization algorithm—parameterized clipping activation (PACT)—and reformulate it using ﬁxed-point arithmetic. Finally, we unify the recently proposed method for quantization ﬁne-tuning and our ﬁxed-point approach to show the potential of our method. We verify F8Net on ImageNet for MobileNet V1/V2 and ResNet18/50. Our approach achieves comparable and better performance, when compared not only to existing quantization techniques with INT32 multiplication or ﬂoating point arithmetic, but also to the full-precision counterparts, achieving state-of-the-art performance.

Published as a conference paper at ICLR 2022
F8NET: FIXED-POINT B-BIT ONLY MULTIPLICATION FOR NETWORK QUANTIZATION
Qing Jin
Jian Ren'
Richard Zhuang! Sumant Hanumante* Zhengang Li²
Zhiyu Chen Yanzhi Wang'
Kaiyuan Yang'
Sergey Tulyakov! Ricee University, USA
Snap Inc.
Northeasternn University, USA
ABSTRACT
Neural network quantization is promising compression technique reduce memory footprint and save energy consumption, potentially leading to real-time inference. However, there is performance gap between quantized and full- precision models. To reduce it, existing quantization approaches require high- precision INT32 or full-precision multiplication during inference for scaling or dequantization. This introduces noticeable cost in terms of memory, speed, and required energy. To tackle these issues, we present F8Net, a novel quantization framework consisting of only fixed-point 8-bit multiplication. To derive our method, we first discuss the advantages of fixed-point multiplication with different formats of fixed-point numbers and study the statistical behavior of the associated fixed- point numbers. Second, based on the statistical and algorithmic analysis, we apply different fixed-poine formats for weights and activations of different layers. We introduce a novel algorithm to automatically determine the right format for each layer during training. Third, analyze previous quantization algorithm- parameterized clipping activation (PACT)-and reformulate it using fixed-point arithmetic. Finally, we unify the recently proposed method for quantization fine- tuning and our fixed-point approach to show the potential of our method. We verify F8Net on ImageNet for MobileNet V1/V2 and ResNet 8/50. Our approach achieves comparable and better performance, when compared not only to existing quantization techniques with INT32 multiplication or floating- praini arithmetic, but also to the full- full-precision counterparts, achieving kate-of-the-ar performance.
INTRODUCTION
Real-time inference on resource-constraaied and efficiency-demandding platforms has long been desired and extensively studied in the last decades, resulting in significant improvement
the
trade-off between efficiency and accuracy (Han et al. 2015 Liu et al. 2018 | Mei et al. 2019 Tanaka
al. 2020 Ma al. 2020 Mishra al. 2020 Liang al. 2021 Jin et al. 2021
et (021).
As model compression technique, quantization is promising compared to other methods, such as
network pruning (Tanaka et al. 2020 Li al. 202 Ma al. 2020 2021a Yuan al., [02]] and
slimming (Liu et al. 2017 2018 1). as it achieves a large compression ratio (Krishnamoorthi 20188)
Nagel al. (2021) and is computationally beneficial for integer-only hardware. The latter one especially important because many hardwares (c.g.. most brands of DSPs
QCOM (2019)
only support integer or fixed-point arithmetic for accelerated implementation and cannot deploy
models with floating-point operations. However, the drop in performance, such as classification
accuracy, caused by quantization errors, restricts wide applications of such methods (Zhu et al. 2016).
To address this challenge, many approaches have been proposed, which can be categorized into simulated quantization, integer-only quantization, and fixed-point quantization (Gholami et al. 2021).
Fig. I shows comparison between these implementations, For simulated quantization, previous works propose to use trainable clipping-leved (Choi et al.) 2018), together with scaling techniques on activations (Jin al. 2020b) and/or gradients (Esser al.) (019), to facilitate training for the quantized models. However, some operations in these works, such as batch normalization (BN),
Work done during intersship Snap Inc. Code is available http://ghhhaa.ccommmmmrcccccccct

---

Published as a conference paper at ICLR 2022
Input (FP32) Weight (FP32)
Input [Int8] Weight (Int8)
Input (Int8) Weight (Int8)
Input (Int8) Weight [Int8]
Multiplication (FP32)
Multiplication (Int8)
Multiplication (Int8]
Multiplication (Inta)
Int32
Int32
Int32
Accumulation (FP32)
Accumulation (Int32)
Accumulation (Int32)
Accumulation (Int32)
Int32
Int32
Dequantization
Multiplication (Int32)
lnt32
FP32
FP32
Int32
Requantization
Bit Shifting (Int32)
Bit Shifting (Imt32)
Output (FP32)
Output (Inta)
Output (Ints)
Output (Inta)
(a) Full-precision.
(b) Simulated quant.
(c) Integer-only quant.
Fixed-point quant.
Figure 1: Inspired by Gholami et al. (2021), we show the comparison of full-precision model
(presented in (a)) and different quantizations settings: (b) simulated quantization; (c) integer-only quantization; and (d) exed-point quantization. Note the combination of last two operations in integer- only quantization is termed as dyadic scaling in literature (Yao et al. 2021).
are conducted with full- cill--reeision to stabilize training (Jin et al. 2020b; Esser et al. 2019), limiting
the practical application of integer-only hardware. Mcanwhile, integer-only quantization, where the model inference can be implemented with integer multiplication. addition, and bit shifting, has shown significant progress in recent studies (Jacob et al., 2018 Yao et al.] 2021 Kim et al. 2021). Albeit floating-poine operations are removed to enable models running on devices with limited support of operation types, INT32 multiplication is still required for these methods. On the other hand, fixed-point quantization, which also applies low-precision logic for arithmetic, does not require INT32 multiplication or integer division. For example, to replace multiplication by bit shifting, Jain ct al. (2019) utilize trainable power-of-2 scale factors to quantize the model.
In this work, we adopt fixed-point quantization. Our work differs from previous efforts (Jain it al., 2019) in three major aspects. First, to determine the minimum error quantization threshold, we conduct statistical analysis on fixed-point numbers. Second, we unify parameterized clipping activation (PACT) and fixed-point arithmetic to achieve high performance and high efficiency. Third, we discuss and propose quantization fine-tuning methods for different models. We dub our method as F8Net, as it consists in only Fixed-point 8-bit multiplication employed for Network quantization. We thoroughly study the problem with fixed-point numbers, where only INT8 multiplication is involved, without any INT32 multiplication, neither floating peating-pointton nor fixed-point types. Throughout this paper we focus on 8-bit quantization, the most widely supported case for different devices and is typically sufficient for efficiency and performance requirements. Our contribution can be elaborated as follows.
We show 8-bit fixed-poin number is able to represent a wide range of values with negligible relative error, once the format is properly chosen (see Fig. and Fig. 44. This critical characteristic enables fixed-point numbers a much stronger representative capability than integer values.
We propose a method determine the fixed-point format, also known as fractional length, for weights and activations using their variance. This is achieved by analyzing the statistical behaviors of fixed-point values of different formats, especially those quantized from random variables with normal distribution of different variances. The analysis reveals the relationship between relative quantization error and variance, which further helps us build an approximated formula to determine the fractional length from the variance.
We develop a novel training algorithm for fixed-poit -eint models by unifying fixed-point quantization and PACT (Choi al. 2018). Besides, we show the impact of fractional length sharing for residual blocks, which is also important to obtain good performance for quantized models.
We validate our approach for various models, including MobileNet V1/V2 and R on ImageNet for image classification, and demonstrate better performance than existing methods that resort to 32-bit multiplication. We also integrate the recent proposed fine-tuning method to train quantized models from pre-trained full-precision models with ours for further verification.
2 RELATED WORK
Quantization is one of the most widely-used techniques for neural network compression (Courbariaux
al. 2015 Han et al. 2015 Zhu et al. 2016 Zhou et al. 2016 2017 Mishra et al. 2017 Park et al.
2

---

Published as a conference paper at ICLR 2022
2017: Banner et al. 2018), with two types of training strategies: Post- Post-Training Quantization directly
quantizes a are-trained full- -ull-iiciiion model (He & Cheng 2018
Nagel et al. 2019 Fang et al.
2020a b Garg et al.] 202T): Quantization-Awer Aware Training uses training data to optimize quantized
models for better performance (Ciysel et al. 2018: Esser et al.) 2019 Hubara et al.
2020 Tailor et al.
2020). In this work, we focus on the latter one, which is explored in several directions. One area
uses uniform-prectiiei
quantization where the model shares the same precision (Zhou et al.
2018
Wang al. [2018] Choukroun et al. 2019 Gong et al. 2019 Langroudi et al. 2019 Jin al. 2020a
Bhalgat et al. al.0; Chen et al. [ et al.) 2020: Darvish Rouhani et al. 2020 Oh et al.
2021). Another direction studies mixed- preccssoeeisionnn that determines bit-width for each layer through
search algorithms, aiming at better accuracy-eficcieny trade-off (Dong et al.) 2019 Wang et al.
2019 Habi et al.) 2020) Fu et al., 2020 2021; Yang Jin 2020 Zhao et al. 202 bj Ma et al.
2b2 There is also binarization network, which only applies 1-bit (Rastegari et al. [2066] Hubara
ct al. 2016; Cai et al., 2017 Bulat et al., 2020 Guo et al. 2021). Despite the fact that quantization helps reduce energy consumption and inference latency, it is usually accompanied by performance degradation. To alleviate this problem, several methods are proposed.
One type of effort focuses simulated quantization. The strategy is to leave some operations, e.g.. BN, in full pallisieciinn for the stabilized training of quantized models (Choi et al. 20188] [Esse et al.) 2019. Jin et al., (2020b). Nevertheless, these methods limit the application of the quantized models on resource-demanding hardware, such as DSP, where full-precision arithmetic is not supported for accel- erated computing (QCOM 2019 Ho 2015). To completely climinate floating-poinl operations from the quantized model, integer-only quantization techniques emulate the full-precision multiplication by 32-bit integer multiplication followed by bit shifting (Jacob et al. 2018 Zhu al. 2020; Wu et al.)
2020: Yao et al. 202T Kim et al. (021). However, the calculation of IN IN132 multiplication in these works requires one more operation, which results in extra energy and higher latency (Gholami et al.,
2021). In parallel, recent work (Jain et al. (2019) proposes to restrict all scaling factors as power-of-2 values for all weights and activations, which belongs to fixed-point quantization methods (Lin et al.
2016: Jain et a [019 Kim & Kim 1 Mitschke al.] [2099; Enderich et al. [019b] Chen et al.
2017; Enderich et al. 20t Zhang et al. 2020 Goyal et al.] (021). This enables the model to only
incorporate INT8 even INT4 multiplications, followed by INT32 bit shifting. However, there still lack of a thorough study of the benefits of using fixed-point arithmetic. Also, the power-of-2 scaling factors are directly determined from the training data without theoretical analysis and guidance. In this work, we give an extensive analysis, especially on the potential and theoretical principle of using fixed-point values for quantized models, and demonstrate that with proper analysis and design, model quantized with only INT8 multiplication involved is able to achieve comparable and even better performance to the integer-only methods implemented with INT32 multiplication.
ANALYSIS OF FIXED-POINT REPRESENTATION
In this section, we first introduce the fixed-point multiplication (Smith et al. 1997: Tan & Jiang 2018) and analyze the distribution of weight from different layers in well-trained full-precision model (Sec. B.I. We then investigate the statistical property of fixed-point numbers, and demonstrate the potential of approximating full-precision values by 8-bit fixed-point numbers with different formats (Sec. B.2 After that, we study the relationship between standard deviation of random variables and the optimal fixed-point format with the smallest quantization error. Finally, we derive an approximated formula relating the standard deviation and fixed-point format, which is verified empirically and employed in our final algorithms (Sec. 3.3
3.1 ADVANTAGES OF LIXED-POINT ARITHMETIC
Fixed- point number is characterized by its format, which includes both the word length indicating the whole bit-width of the number and the fractional length (FL.) characterizing the range and resolution of the represented values (Smith et al. 1997p. Fixed-point
cruthnetic-especinn point multiplication--3 widely utilized for applications in, e.g., digital signal processing (Smith) et al. 1997 Tan & Jiang 2018). Compared with integer or floating point multiplication, fixed-point multiplication has two major characteristies: First, multiplying two fixed seedt numbers is more efficient than multiplying two floating point numbers, especially on resource-constraime devices such DSP. Second, it more powerful than its integer counterpart due to its versatility and the
fixed-
3

---

Published as a conference paper at ICLR 2022
Effective Weight Range (FP MobileNet V2)
Fractional Length vs Layer (8-bit MobileNet V2)
3
Effective Weight
Fractional Length
1
Layer
#Woight FL (pointwise)
#Activation FL [poinawise)
#WWeight FL (depthwise)
@Activation FL, [depthwise)
Layer
(a) Weight range.
(b) Weight and activation fractional length.
for pre-trained full-precin cell-precision (FP) model, and (b) fractional lengths of each layer for well-trained fixed-point model for MobileNet V2.
Figure 2: (a) Value range of effective weight (see Sec. 4.2
Signed 8-bit (Gaussian)
Unsigned (Rectified Gaussian) Undorlow
Unserfoww
Overflow
Ovarlow
1.0
0.7
0.8
0.6
0.5
Relative
0.6
0.4
0.4
0.3
0.1
0.0
0.0
10-3
10
10
10
102
10-3
10
10
10a
10a
(a) Signed quant. for Gaussian R.V.
Unsigned quant. for Rectified Gaussian R.V.
Figure 3: Representing potential for 8-bit signed (a) and unsigned (b) fixed-point numbers with different formats. The figures plot the relationship between relative quantization and the standard deviation for different fixed-point formats. Both are experimented on zero-mean Gaussian random variables (R. V.), with ReLU applied on (b).
representative ability of fixed-point numbers (there can be tens of different implementations for fixed-point multiplication but only for integer and floating-point ones (Smith al., 1997)). This efficiency and versatility make fixed-point quantization a more appealing solution than integer-only quantization. Specifically, as shown in Fig. [a the scales of weights from different layers pre-traine rre-traiee full- full-precision model can vary in orders. ranging from less than 0.1 to nearly 4. Direct quantization with only integers inevitably introduces considerable quantization error, unless more precision and more operations are involved, such as using INT32 multiplication together with bit shifting for scaling as shown in Fig. Ic. On the other hand, employing fixed-point numbers has the potential to reduce quantization error without relying high-precision multiplication, weights and activations from different layers have the extra degree of using different formats during quantization. Indeed, shown in Fig. 2b for well-trained MobileNet V2 with 8-bit fixed-poine numbers, the fractional lengths for weights and activations vary from layer to layer. This raises the question of how determine the formats for each layer. In the following, study this for 8-bit fixed-point models.
3.2 STATISTICAL ANALYSIS FIXED-POINE FORMAT
For predefined bit-width, integer, which is a special case of fixed-point numbers with zero fractional length, has a predefined set values that it can take, which severely constrains the potential of integer-only quantization. On the other hand, fixed-point numbers, with an extra degree of freedom, i.e. the fractional length, are able to represent a much wider range of full-precision values by selecting the proper format, and thus they are more suitable for quantization. As example, Fig. B shows the relative quantization error with 8-bit fixed-point values using different formats for a set of random variables. which sampled from normal distributions (both signed and unsigned, with the latter processed by ReLU before quantization) with zero-men mean and different standard deviations a (more experimental details in Appx. [..2p. From the experiments, make the following two observations.
Observation 1: Fixed-poite numbers with different formats have different optimal representing regions, and the minimum relative error and optimal standard deviation (annotated as star) varies
4

---

Published as a conference paper at ICLR 2022
Signed 8-bit (Gaussian)
Signed 8-bit (Gaussian)
1.0
Minimum Relative Error
104
0.8
Threshold
9
0.66
0.4
38.05 2
100
8
0.2
0
0.0
10-3
10-2
10"
104
10a
10a
Fractional Length (FL)
Opt. Frac. Len.
Min. Rel. Err.
Unear Fitting
Emplrical
(a) Optimal fractional length and minimum relative error for signed quant.
(b) Relationship between threshold standard devia- tion and fractional length for signed quant.
Unsigned 8-bit (Rectified Gaussian)
Unsigned 8-bit (Rectified Gaussian)
Minimum Relative Error
Opt. Frac. Len. (FL")
0.6
0.5
101
Threshold
0.4
0.3
69.0 2
0.2
100
0=
0.1
0.0
10-3 10-2 -1
10
10A
10
10a
Fractional Length (FL)
Opt. Frac Lon.
Min, Rel. Err.
Lnner Fitting
Empirical
(c) Optimal fractional length and minimum relative error for unsigned quant.
(d) Relationship between threshold standard devia- tion and fractional length for unsigned quant.
Figure Determining optimal fractional length from standard deviation. (a) and illustrate optimal fractional length and minimum relative quantization error against standard deviation for signed and unsigned 8-bit fixed-pe exed-point quantization for Gaussian and rectified Gaussian random variables. (b) and (d) show the relationship between threshold standard deviation and fractional length.
for different fractional lengths (Fig. 3). This is because the format controls the value magnitude and the representation resolution (the least significant bit).
Observation 2: Larger fractional lengths are robust represent smaller numbers, while smaller fractional lengths are more suitable for larger ones. For a given standard deviation, using small fractional length has the risk of underflow, while large fractional length might cause overflow issue. Specifically, integers (black curves in Fig, 4 are much more prone to underflow issues and have large relative emorr for small enough values to quantize.
3.3
CHOOSING OPTIMAL FIXED-POINE FORMAT
With the above observations, we are interested in answering two questions:
(I) Can we achieve small fixed-poin quantization error for a wide range of full-peccision values by always using the optimal fractional length corresponding to the smallest relative error?
To answer this, we first plot the smallest possible relative error amongst all the candidate fixed-point formats against the standard deviation. As shown in red lines from Fig. 4a and Fig. zero-mean normal distribution, by always choosing the optimal fixed-point format, we are able to achieve relative quantization error smaller than 1% for standard deviation with range of order of at least around 3. For example, for signed quantization, the standard deviation range from 0.1 to around 40 to achieve less than 1% error, and for unsigned guantization, the standard deviation can range from 0.1 100. The experiments verify our presumption that using fixed-point values with the optimal formats is able to achieve negligible quantization error.
(2) Can have simple way determine the optimal fractional length?
To answer this, we plot the optimal fractional length from the statistics of the full-precision values against the standard deviation, shown in the blue lines in Fig. 4a and Fig. 4c We find that the

---

Published as a conference paper at ICLR 2022
threshold value corresponding to the jumping point is almost equidistant on the log scale of the standard deviation. This is expected as the representing region of different formats are differed by factor of 2's exponents. Plotting the threshold standard deviation (on log-scale) against the corresponding optimal fractional length (Fig. [4b and Fig. 4d). we find their relationship is almost linear, leading the following semi-empirical approximating formulas to determine the optimal fractional length FL" from the standard deviation (more discussion in Appendix 7.7
40 I
70 logg
Signed: FL
loga
Unsigned:
FL
(1)
In the following, unless specifically stated, we use ①1 to determine the fractional length for both weight and activation quantization. Note that we only calculate the standard deviation during training.
METHODS
In this section, we diseuss proposed training technique for neural network quantization with fixed-point numbers, where the formats of weights and activations in each layer are determined based on (T) during training. We first analyze how unify PACT and fixed-point quantization (Sec. (4.I. Then we show how to quantize weights and activations, especially updating for BN running statistics and fractional lengths (Sec. 4.2). Finally, we discuss the necessity of relating scaling factors from two adjacent layers to calculate the effective weights for quantization, especially for residual blocks where some layers have several layers following them (Sec.
4.1 UNIFYING PACT AND FIXED-POII QUANTIZATION
To quantize positive value with unsigned fixed-point number of format (WL, FL), where WL and FL denotes word length and fractional length for the fixed-point number, respectively, we have the quantization function fix quant as:
fix quant(x) 2FL round (clip (a. 2x.2 (2.2%,,0,24 -2)-
(2)
where clip is the clipping function, and < FL S WL for unsigned fixed-point numbers. Note that fixed-point quantization has two limitations: overflow, which is caused by clipping into its representing region, and underflow, which is introduced by the rounding function. Both of these introduce approximation errors. To minimize the error, determine the optimal fractional length for each layer based on the analysis in Sec. 3.3
To achieve better way to quantize model using fixed-point numbers, take look of the most successful quantization techniques, PACT (Choi et al.) 2018), which clips on the full- pell-preciiion value with learned clipping-level before quantization:
a PACT(x) = M
round
a (z. (r.
where .M is pre-defined scale factor mapping the value from [0, 1] to [0, M). The formal similarity
between 2 and (3) inspires us to relate them with each other as (more details the Appx. 7.3):
2 2WL fix fix.quuan(((
2WL.
PACT(x)
z),
(4)
2FL
where have M 2WL 1, which is the typical setting. With this relationship, we can implement PACT and train the clipping-level a implicitly with fixed-point quantization.
4.2 UPDATING BN AND FRACTIONAL LENGTH
Double Forward for BN Fusion. To quantize the whole model with only 8-bit fixed-point multi- plication involved, we need to tackle the scaling factor from BN layer, including both the weight and rumning variance. Specifically, we need to quantize the effective weight that fuses the weight of convolution layers with the weight and running variance from BN (Jacob et al. 2018 Yao et al.) 2021I. This raises the question of how to determine the running statistics during training. To solve this problem, apply forward computation twice. For the first forward, apply the convolution

---

Published as a conference paper at ICLR 2022
using quantized input yet full-precision weight of the convolution layer, and use the output to update the running statistics of BN. In this way. the effective weight to quantize available. Note there is no backpropagation for this step. For the second forward. quantize the combined effective weight to get the final output of the two layers of convolution and BN and do the backpropagation.
Updating Fractional Length. Different from existing work that directly trains the fractional length (Jain et al.] 2019), we define the fractional length for weight on-the-fly during training by inferring from current value of weight, using (I. For the fractional length of activation, we use buffer to store and update the value with a momentum of 0.1, similar to how to update BN running statistics. Once the fractional lengths are determined after training, we keep them fixed for inference.
4.3 RELATING SCALING FACTORS BETWEEN ADIACENT LAYERS
As shown in 44, there are still two extra factors during the quantization operation, which we denote as a fix scaling factor Tlax:
2
Fix
2WL
(5)
Now is a trainable parameter with full- full-precision, which means the fix scaling factor is also in full-precision. To eliminate undesired extra computation, we absorb it into the above-mentioned effective weights for quantization (Sec. 4.2). However, the fix scaling factor occurs twice, one for rescaling after quantization (7ax) and the other for scaling before quantization (1/m)x). To completely absorb it, we need to relate two adjacent layers. In fact, for mapping that includes convolution. BÑ, and ReLU (more details are shown in Appx. 7.5 we apply PACT quantization to relate the activation between two adjacent layers
1+11 fix.quant d
29)
8
(6)
1141)
13+13 Tiix
Mix
Effective Weight
Effective Bias
where q is the fixed-point activation, W the full-precision weight of the convolution layer, and j the spatial indices, the total number of multiplication, and the superscript (1) indicates the I-th block consisting of convolution and BN. % B, a. μ are the learned weight, bias, running standard deviation, and running mean for the BN layer, respectively. Also, we set WT. 8 for all layers. As can be seen from (6. to obtain the final effective weight for fixed-point quantization, for the 1-th Conv-BN block, we need to access the fix scaling factor, or equivalently, the clipping-level a and the activation fractional length FL, from its following [+ 1)-th block(s). To achieve this, we apply two techniques.
Pre-estimating Fractional Length. As mentioned above, we determine the activation fractional length from its standard deviation. Also, (5) indicates that the fix scaling factor relies on such fractional length for each layer. However, in (6). need the fix scaling factor from the next layer to determine the effective weight under quantization, which we have not yet updated. Thus, when calculating the effective weights during training, we use the activation fractional length stored in the buffer, instead of the one for quantizing the input of the next layer.
Clipping-Level Sharing. As shown in Fig. 5 for residual blocks, some layers have two following layers (which we also name as child layer). Since we need the fix scaling factor from the child layer to calculate the effective weight for the parent (see (6)), inconsistent fix scaling factors between all children layers will be a problem. To this end, we define one layer as master and force all its siblings to share its clipping-level. In fact, the best way is to share both the clipping-level and the fractional length among siblings. but we find sharing fractional length leads to considerable performance drop, especially for deep models such as MobileNet V2 and ResNet50. This is because the fractional lengths play two roles here: one is for the fix scaling factor, and the other is for the representing region (or equivalently the clipping-level). Using different fractional lengths effectively enables different clipping-levels (although only differ by a factor of power-of-2, see Appx. 7.6). which can be beneficial because the activation scales might vary from layer to layer. Moreover, breaking the constraint of sharing activation fractional length does not introduce much computational cost, the value only differs in storing format, and typically the values are stored in 32-bit, i.c., the accumulation results are only quantized into 8-bit for multiplication. Note that when computing the effective weight
7

---

Published as a conference paper at ICLR 2022
FL
Child
Layer
a=3.54
RL=6
Parent
Child
Layer 11
Child
Layer 42
Parent
Layer
Child
Layer #1
Next Layer
(a) ResBlock with direct connection.
(b) ResBlock with downsampling.
Figure 5: The illustration of residual connections. For layer with several layers (named children layers) directly following it, we choose one to be master, and all its sibling layers use the master layer's clipping level. On the other hand, since using different fractional length only cause bit shifting or different fixed-point quantization formats, and the values are stored in 32-bit before quantized into 8-bit, we do share the fractional formats to allow more degrees of freedom. The figures show the case of direct residual connection (a) and that with downsampling convolution layer (b).
of the parent layer, only use the master child's activation fractional length. For effective weight of each child layer and fixed- fixed-point quantization on its input, we use its own fractional length.
EXPERIMENTS
In this section, we present our results for various mod- els on ImageNet (Deng et al. 2009) for classification task and compare the results with previous works that focus on quantization-awer aware training to verify the ef- fectiveness of our method. We show the results for two sets of training. First, we discuss the conventional train- ing method following Jiin et al. (2020b). Second, we unify our method with one recent fine-tuning method that quantizes full-precision models with high accu- racy Yao et al., (2021). More detailed experimental settings are described in Appx. 7.1
Table 1: 8-bit quantization with conven- tional training for ResNet18 and MobileNet VI/V2b. Following Yao et al. (2021), ab- breviate Integer-Only Quantization as "Int", INTS-MultiplicationnOO
Quantization as "8-bit", the Baseline Accuracy as "BI.", and Top-1 1 Accuracy "Top-1". All models for 8-bit weight and activation quantization. For MobileNet V2, we using MobileNet V2b version as it is the most typical one.
(a) ResNet18
Conventional training. We first apply our method using conventional training (Choi al.) 2018 Esser
Mothod
Int
8-bit
BL
Top-1
Baseline (FiP)
et al. 2019 Jin et al.) 2020b Fu et al.) 2021). where
RVQuant Park al 2018
X
X
69.9
70.0
the quantized model is trained with the simplest set- ting those for full-precision model (more details in Appx. To verify the effectiveness of our method, perform experiments
PACT Chor et at 2018 LSQ
X
70.2
69.8
Bxst e 2019
x
X
70.5
71.1
FSNert (ours)
70.3
71.1
several models including R8 and MobileNet V1/V2. As shown in Table | method achieves the state-of-the-art results for all models. Additionally, we obtain comparable or even better performance than the full-precision counterparts.
(b) MobileNer V1
Method
B-bit
Top
Baseline (FP)
x
72.4
72.4
PACT TQT
Choi et al.
2018
x
X
72.1
71.3
Compared with previous works on simulated quantiza- tion (Choi et al.
Jui 2019
/
/
71.1
71.1
SAT
Jan al. 20206
x
71.7
72.6
2018, Park al.) 2018 Esser al.,
F8Nel ToUr)
72.4
72.8
2019; Jin et al.) 2020b; Fu et al. 2021) that requires prel-precision rescaling after INT8 convolution, our approach is not only more efficient but also achieves better performance. On the other hand, compared with previous fixed-point quantization (Jain al., 2019). our approach gives better results. This might partially due to that method is based on more systematic analysis, as explained above in Section 3.3
(c) MobileNet V2b
8-hit
Top-1
Baseling (FP)
x
x
72.7
72.7
PACT Choi et al 2018 TQT Jam SAT
X
x
72.1
71.7
/
/
71.7
71.8
Jin et at. 20208
x X
71.8
72.5
F8Nel
72.7
72.6
To further understand the significance of our method, plot the fractional lengths for weight and activation for each layer. Illustrated in Fig. [br MobileNet V2, we find that the fractional lengths for both weight and activation vary from layer layer. Specifically, for weight quantization, since
8

---

Published as a conference paper at ICLR 2022
some layers have relatively large value range of effective weight, especially some depthwise layers, small fractional length is necessary to avoid overflow issue. On the other hand, for layers with small weight scale, large fractional length has more advantages to overcome the underflow problem. The same conclusion also applies for the fractional length for activation. Indeed, for some early layers in front of depthwise convolution layer, the activation fractional length needs to be small, yet for the later-stages, larger fractional length is desired. This further verifies our finding that using different fractional lengths for layers with the same parent is critical for good performance, because layers at different depths might be siblings and requires different fractional lengths (see Fig. 5
Tiny fine-tuning on full- full-precision model. Recent work (Yao et al. (021) focus on in- vestigating the potential of neural network quantization. To this end, they suggest to tiny fine-tune on a well-pretrained full-precision model with high accuracy. In this way, it might help to avoid misleading conclusion coming from improper comparison between weak full pell-precision models with strong quan- tized model. To further investigate the power of our method and compare it with these ad- vanced techniques, we also apply our method and fine-tune on several full-precision mod- els with high accuracy. Also, given the num- ber of total fine-tuing steps is very small, we apply grid search to determine the optimal fractional lengths for this experiment. The results are listed in Table and we can find that our method is able to achieve better per- formance than previous method (Yao et al.] 2021), without time- and energy-consuming high-precision multiplication (namely dyadic scaling shown in Fig. Ic).
Table 2: 8-bit quantization with tiny fine-tuning on well-trained full full-precision model. Follow- ing Yao al.) (20TI), we abbreviate Integer-Only Quantization as "Int". " INT8 NATT8-Muliiplicaton-Oni Quantization as "8-bit", Layer-Wier Wise Quantiza- tion as "Layer", the baseline accuracy as "BL", Top-1 Accuracy as "Top-1", and Top-1 Accuracy Drop with respect to the baseline "Drop". We use two baselines for ResNet50, one from Py- torchCV (Sémery) (021) (Baseline #1) and another from Nvidia (Nvidia (202T) (Baseline #2), and we use ResNet50b version. Note that the OMPQ (Ma) is mixed-precision quantization.
et al. 2021b
(a) ResNet18
Method
Int
8-bit
Layer
BL
Top
Drop
Baseline (FP)
71.5
71.5
HAWQ-VI
Yao 2002
21.5
71.6
0,1
HAWQ-V3
x
/
71.5
70.9
-0.6
OMPO (Ma
73.1
72.3
-0.8
PSNeI (cirs)
73.1
72.4
(b) ResNet500
Method
Int
8-bit
Layer
Top-1
Drop
Haseline #1 (PP)
x
x
77.6
77.6
HAWQ-V3 Yon al 2021
/
x
x
77.6
77.5
-0.1
Our method reveals that the high-precision rescaling, no matter implemented in full- precision, or approximated or quantized with INT32 multiplication followed by bit-shifting (a.k.a. dyadic multipiication,, is indeed un- necessary and not the key part for quantized model to have good performance. This is not well-understood in previous literature. Specifically, we demonstrate that by properly choosing the formats for weight and activation in each layer, we are able to achieve comparable and even better performance with 8-bit fixed-point numbers, which can be implemented more efficiently on specific hardwares such as DSP that only supports integer operation.
HAWQ-V3 aa 2027
/
77.1
F8Ner (ours)
/
77.6
77.6
0.0
Baseline (FP)
x
78.5
78.5
HAWQ-V! V3
Yan et 202
/ / x
x
x
78.5
78.1
-0.4
HAWO-VS V3
78.5
76.7
-1.8
(aurs)
78.1
CONCLUSION
Previous works on neural network quantization typically rely on 32-bit multiplication, either in full- precision or with INT32 multiplication followed by bit-shifting (termed dyadic multiplication). This raises the question of whether nigh-precision multiplication is critical guarantee high-performance for quantized models, or whether it is possible to eliminate it to save cost. In this work, we study the opportunities and challenges of quantizing neual networks with 8-bit only fixed-point multiplication, via thorough statistical analysis and novel algorithm design. We validate our method on ResNetl 8/50 and MobileNet V1/V2 on ImageNet classification. With our method, we achieve the state-of-the- performance without 32-bit multiplication, and the quantized model is able to achieve comparable or even better performance than their full- preccision counterparts. Our method demonstrates that high- precision multiplication, implemented with either floating-pon point dyadic scaling, is not necessary for model quantization to achieve good performance. One future direction is to perform an in-depth statistical analysis of fixed-point numbers with smaller word-lengths for neural network quantization.
9

---

Published as a conference paper at ICLR 2022
REFERENCES
Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Post-training 4-bit quantization of convolution networks for rapid-deployment. arXiv preprint arXiv: 2018.
Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In Proceedings of the IEEE/CVF Conference Computer Vision and Paltern Recognition Workshops, pp. 696-697, 2020.
Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. High-capacity expert binary networks. arXiv preprint arXiv:2010.03558, 2020.
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5918-5926, 2017.
Jianfei Chen, Yu Gai, Zhewei Yao. Michael W Mahoney, and Joseph E Gonzalez. A statistical framework for low-bitwidth training of deep neural networks. arXiv preprint arXiv:2010.14298, 2020.
Xi Chen, Xiaolin Hu, Hucheng Zhou, and Ningyi Xu. Expnet: Training a deep convolutional neural network in fixed-poin representation, In 2017 International Joins Conference on Neural Networks (IJCNN) pp. 2494-2501 IEEE, 2017.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv: 180500008, 2018.
Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In ICCV Workshops, pp. 3009-3018, 2019.
Matthieu Courbariaux, Yoshua Bengio, and Sean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pp. 3123-3131, 2015.
Bita Darvish Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, et al. Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point. Advances in Neural Information Processing Systems, 33, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian Proceedings of the IEEE/CVF
aware quantization of neural networks with mixed-precisioe International Conference on Computer Vision, pp. 293 30--302,222 2019.
Lukas Enderich, Fabian Timm, Lars Rosenbaum, and Wolfram Burgard. Fix-net: pure fixed-point representation of deep neural networks. 2019a
Lukas Enderich, Fabian Timm, Lars Rosenbaum, and Wolfram Burgard. Learning multimodal fixed-point weights using gradient descent. arXiv preprint 10x0v:190..07220, 2019b.
Steven K Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra Modha. Leamned step size quantization. arXiv preprint 19xiv:190..08153, 2019.
Jun Fang, Ali Shafiee, Hamzah Abdel-Aziz, David Thorsley, Georgios Georgiadis, and Joseph Hassoun. Near-lossless post-training quantization of deep neural networks via a piecewise linear approximation. arXiv preprint arXivv20022.00/04 pp. 2020a.
Jun Fang. Ali Shafiee, Hamzah Abdel-Aziz, David Thorsley, Georgios Georgiadis, and Joseph H Hassoun. Post- Post-training piecewise linear quantization for deep neural networks. In European Conference Computer Vision, pp. 69-86. Springer, 2020b.
10

---

Published as a conference paper at ICLR 2022
Yonggan Fu, Haoran You, Yang Zhao, Yue Wang, Chaojian Li, Kailash Gopalakrishnan, Zhangyang Wang, and Yingyan Lin. Fractrain: Fractionally squeezing bit savings both temporally and spatially for efficient dnn training. arXiv preprint arXiv:2012.1311. 2020.
Yonggan Fu, Han Guo, Meng Li, Xin Yang. Yining Ding, Vikas Chandra, and Yingyan Lin. Cpt: Efficient deep neural network training via cyclic precision. arXiv preprint S0Xiv:2201.09800 2021.
Sahaj Garg, Joe Lou, Anirudh Jain, and Mitchell Nahmias. Dynamic precision analog computing for neural networks. arXiv preprint arXiv:2102.06365. 2021.
Amir Gholami, Schoon Kim, Zhen Dong. Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. arXiv preprint 0rXiv:2103.136:00 2021.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4852-455555 4861,
2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in I hour. arXiv preprint arXiv: (t0:::170.0000 776707267, 2017.
Rishabh Goyal, Joaquin Vanschoren, Victor Van Acht, and Stephan Nijssen. Fixed-point quantization of convolutional neural networks for quantized inference on embedded platforms. arXiv preprint 1rXiv:2/02.021477 2021.
Nianhui Guo, Joseph Bethge, Haojin Yang, Kai Zhong, Xuefei Ning, Christoph Meinel, and Yu Wang. Boolnet: Minimizing the energy consumption of binary neural networks. arXiv preprint 10..922006.0691, 2021.
Philipp Gysel, Jon Pimentel, Mohammad Motamedi, and Soheil Ghiasi. Ristretto: A framework for empirical study of resource-efficient inference in convolutional neural networks. IEEE transactions on neural networks and learning systems, (1115))7784- 5789, 2018.
Hai Vietor Habi, Roy H Jennings, and Arnon Netzer. Hmq: Hardware friendly mixed precision quantization block for cnns. In Computer Vision- 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI 16, pp. 448-463. Springer, 2020.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint 1xXiv:1510.011 2015.
Xiangyu He and Jian Cheng. Learning compression from limited unlabeled data. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 752-769, 2018.
Joshua Ho. ing, 2015. URL httpss//www..aaanteeeeccc/////// in-
Qualcomm details hexagon 680 dsp in snapdragon 820 accelerated imag- details
hexagon - 680- -nndiiinnnaaaaoooo htapapprrcpoooe222aacccaaaaeeeee imaging
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.0466,, 2017.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks. Advances neural information processing systems, 29, 2016.
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Improving post training neural quantization: Layer-wise calibration and integer programming. arXiv preprint arXiv:2006.1 10518. 2020.
Benoit Jacob, Skinmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anteger-arithmetic-onn inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018
11

---

Published as a conference paper at ICLR 2022
Sambhav R Jain, Albert Gural, Michael Wu, and Chris H Dick. Trained quantization thresholds for ac- curate and efficient fixed- -oint inference of deep neural networks. arXiv preprint 1rXiv:1903.08066 2019.
Qing Jin, Linjie Yang, and Zhenyu Liao. Adabits: Neural network quantization with adaptive bit- widths. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2146-2156, 2020a.
Qing Jin, Linjie Yang, Zhenyu Liao, and Xiaoning Qian. Neural network quantization with scale- adjusted training. BMVC, 2020b.
Qing Jin, Jian Ren, Oliver J Woodford, Jiazhuo Wang. Geng Yuan, Yanzhi Wang. and Sergey Tulyakov. Teachers do more than teach: Compressing image-to-image models. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13600-13611, 2021.
Schoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-bert: Integer- only bert quantization. arXiv preprint 101.022001.0111, 2021.
Sungrae Kim and Hyun Kim. Zero-centered fixed-point quantization with iterative retraining for deep convolutional neural network-based object detectors. IEEE Access, 9:20828-20833 2021.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint 1806.v8806.0888 2018.
Hamed F Langroudi, Zachariah Carmichael, David Pastuch, and Dhireesha Kudithipudi. Cheetah: Mixed low-precision hardware & software co-design framework for dnns the edge. arXiv preprint arXiv::908. 0938.0236, 2019.
Zhengang Li, Geng Yuan, Wei Niu, Pu Zhao, Yanyu Li, Yuxuan Cai, Xuan Shen, Zheng Zhan, Zhenglun Kong, Qing Jin, al. Npas: A compiler-aware framework of unified network pruning and architecture search for beyond real-time mobile acceleration. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. (4255-14266 2021.
Tailin Liang. John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. Pruning and quantization for deep neural network accelcration: A survey. Neurocomputing, 461:370-44: 403,, 2021.
Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy. Fixed point quantization of deep con- volutional networks. In International conference machine learning, pp. 2849-2858. PMLR, 2016.
Ning Liu, Geng Yuan, Zhengping Che, Xuan Shen, Xiaolong Ma, Qing Jin, Jian Ren, Jian Tang, Sijia Liu, and Yanzhi Wang. Lottery ticket preserves weight correlation: Is it desirable or not? In International Conference on Machine Learning, pp. 7011-7020. PMLR, 2021.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pp. 2736-2744, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell, Rethinking the value of network pruning. arXiv preprint arXiv: 0a... 2018.
Ping Luo, Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. arXiv preprint wr.Xiv:100.0000 00846. 2018.
Xiaolong Ma, Wei Niu, Tianyun Zhang, Sijia Liu, Sheng Lin, Hongjia Li, Wujie Wen, Xiang Chen, Jian Tang, Kaisheng Ma, et al. An image enhancing pattern-based sparsity for real-time inference on mobile devices. In European Conference on Computer Vision, pp. 629-645. Springer, 2020.
Xiaolong Ma. Geng Yuan, Xuan Shen, Tianlong Chen, Xuxi Chen, Xiaohan Chen, Ning Liu, Minghai Qin, Sijia Liu, Zhangyang Wang, et al. Sanity checks for lottery tickets: Does your winning ticket really win the jackpot? Advances in Neural Information Processing Systems, 34, 2021a.
Yuexiao Ma, Taisong Jin, Xiawu Zheng, Yan Wang, Huixia Li. Guannan Jiang, Wei Zhang, and Rongrong Ji. Ompq: Orthogonal mixed precision quantization. arXiv preprint 2xXiv2219.00788 2021b.
12

---

Published as a conference paper at ICLR 2022
Jieru Mei, Yingwei Li, Xiaochen Lian, Xiaojie Jin, Linjie Yang, Alan Yuille, and Jianchao Yang. Atomnas: Fine-grae seee-graied end-to-end neural architecture search. arXiv preprint arXiv:1912.09640, 2019.
Asit Mishra, Eriko Nurvitadhi, Jeffrey Cook, and Debbie Marr. Wrpn: Wide reduced-preecisonn 2017.
networks. arXiv preprint
Rahul Mishra, Hari Prabhat Gupta, and Tanima Dutta. A survey on deep neural network compression: Challenges, overview, and solutions. arXiv preprint arXiv:2010.03954 2020.
Norbert Mitschke, Michael Heizmann, Klaus-Henning Noffz, and Ralf Wittmann. A fixed-point quantization technique for convolutional neural networks based weight scaling. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 3836-3840. IEEE, 2019.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling, Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1325-13344 2019.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysci Bondarenko, Mart van Baalen, and Tij- men Blankevoort. A white paper on neural network quantization. arXiv preprint arXiv:2/06..08295, 2021.
Nvidia. Nvidia models, 2021. pyt amp
URL httpss///ogg.uudaaacccc//tttttod
Sangyun Oh, Hyeonuk Sim, Sugil Lee, and Jongeun Lee. Automated log-scale quantization for low-cost deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 742-751, 2021.
Eunhyeok Park, Junwhan Ahn, and Sungjoo Yoo. Weighted-entropy-baaee quantization for deep neu- ral networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5456-5464 2017.
Eunhyeok Park, Sungjoo Yoo, and Peter Vajda. Value-aware quantization for training and inference of neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 580-595, 2018.
QCOM. Qualcomm @ hexagon tm dsp, 2019. URL http:::/deevellgee..ooa......oo/
wofttaawweeeewaaeeoeooooeeeeooee dsp-sckk
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-5555-4 542. Springer, 2016.
Oleg Sémery. Pytorchev library, 2021. URL httppps//bppp..orrr/ppp//t////ee
Steven W Smith et al. The scientist and engineer's guide to digital signal processing. 1997.
Shyam A Tailor, Javier Marques and Nicholas D Lane. quaret-quaat: Quantization
training for graph neural networks. arXiv preprint arXiv:2008.05000 2020.
Lizhe Tan and Jean Jiang. Digital signal processing: fundamentals and applications. Academic Press, 2018
Hidenori Tanaka, Daniel Kunin, Daniel L.K Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. arXiv preprint arXiv:2006.05466, 2020.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-awee aware automated quantization with mixed precision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8612-8620, 2019.
Peisong Wang. Qinghao Hu, Yifan Zhang, Chunjie Zhang, Yang Liu, and Jian Cheng. Two-step quantization for low-bit neural networks. In Proceedings of the IEEE Conference on computer vision and pattern recognition, pp. 4376-4384 2018
13

---

Published as a conference paper at ICLR 2022
Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint wxiv:2004.09602. 2020.
Linjie Yang and Qing Jin. Fracbits: Mixed precision quantization via fractional bit-widths. arXiv preprint 2221022007.02017, 1, 2020.
Zhaohui Yang. Yunhe Wang, Kai Han, Chunjing Xu, Chao Xu, Dacheng Tao, and Chang Xu. Searching for low-bit weights in quantized neural networks. arXiv preprint arXiv:2009.0880 08699, 2020.
Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang, Qijing Huang, Yida Wang, Michael Mahoney, et al. Hawq-v3: Dyadic neural network quantization. In International Conference on Machine Learning, 2021.
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast meeooryeconomii sparse training framework on the edge. Advances in Neural Information Processing Systems, 34, 2021.
Xishan Zhang, Shaoli Liu, Rui Zhang, Chang Liu, Di Huang, Shiyi Zhou, Jiaming Guo, Qi Guo, Zidong Du, Tian Zhi, et al. Fixed-point back- pack-propagatton training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2330-2338, 2020.
Kang Zhao, Sida Huang, Pan Pan, Yinghan Li, Yingya Zhang, Zhenyu Gu, and Yinghui Xu. Dis- tribution adaptive int8 quantization for training cnns. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, 202
Sijie Zhao, Tao Yue, and Xuemei Hu. Distribution-awan adaptive multi-bit quantization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 9281-9290, 2021b.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: Towards lossless cnns with low-precision weights. arXiv preprint arXiv:1702.03044, 2017.
Aojun Zhou, Anbang Yao, Kuan Wang, and Yurong Chen. Explicit loss-error-aware quantization for low-bit deep neural networks. In Proceedings of the IEEE conference on computer vision and pattem recognition, pp. 9426-9435 9435, 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 16X:016606.0660,, 2016.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint arXiv: 2016.
Feng Zhu, Ruihao Gong, Fengwei Yu, Xianglong Liu, Yanfci Wang, Zhelong Li, Xiuqi Yang, and Junjie Yan. Towards unified int8 training for convolutional neural network. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1969-1979, 2020.
14

---

Published as a conference paper at ICLR 2022
7 APPENDIX
7.1 MORE EXPERIMENTAL. DETAILS
More Details for Conventional Training. For conventional training method, we train the quantized model initialized with pre-trained full-precision one. The training of full pall-preciion and quantized models shares the same hyperparameters, including learning rate and its scheduler, weight decay, number of epochs, optimizer, and batch size. For ResNet18 and MobileNet VI, we use an initial learning rate of 0.05, and for MobileNet V2, it is 0.1. We find the value of learning rate, i.e., 0.1 and 0.05, does not have much impact on the final performance. Totally, 150 epochs of training are conducted, with cosine learning rate scheduler without restart. The warmup strategy is adopted with linear increasing (batchsize/ 256 x 0.05) (Goyal et al. 2017) during the first five epochs before cosine learning rate scheduler. The input image is randomly cropped to 224 224 and randomly flipped horizontally, and is kept as 8-bit unsigned fixed-point numbers with FL = 8 and without standardization. For R8sNet18 and MobileNet V1/V2, we use batch size of 2048 and run the experiments on 8 A100 GPUs The parameters are updated with SGD optimizer and Nesterov momentum with a momentum weight of 0.9 without damping. The original structure of MobileNet V2 uses ReLU6 as its activation. Since our unified PACT and the fixed-point quantization already has clipping operation, and can be equivalently formulated with ReLU6 by rescaling weight or activation, we eliminate ReLU6 in our implementation.
Discussion for Weight Decay. We set weight decay to x 10 and find the weight decay scheme is critical for good performance, especially for the quantized model. We analyze weight decay for different models as follows:
For ResNet18, we apply weight decay on all layers, including convolution, fully- dally-connected,, and BN layers.
For MobileNet V1, previous methods only apply weight decay on conventional convolution and fully-connected layers, but not on depthwise convolution and BN (Howard et al. 2017). We find this leads to the overfitting problem, making some early convolution layers have large weights, which is not friendly for quantization. We further observe that some channels of some depthwise convolution layers have all zero inputs, due to some channels of previous layer become all negative and ReLU is applied afterwards, making the running statistics of the corresponding channels in the following BN layer almost zero. This breaks the regularization effect of BN (Luo et al., 2018). Since each output channel only depends on one input channel for depthwise convolution layers, the weights connecting them become uncontrolled, and the effective weights become large, leading to an overfitting problem. Applying weight decay on the depthwise convolution and BN layers helps alleviate this problem, and the resulting effective weights become small.
For MobileNet V2, we find overfitting plays the role of reducing the validation error (although the training error is lower), and applying weight decay on depthwise convolution or BN weights impairs the training procedure. The underlying reason might be related the residual connecting structure of this model (note MobileNet V1 does not use residual connection).
In summary, we apply weight decay on all layers, including depthwise convolution and BN layers for ResNet18 and MobileNet V1, and do not apply weight decay on depthwise convolution and BN layers for MobileNet V2.
More Details for Tiny Fine-tuning, For tiny fine-tuning on full-precision models, we follow the same strategy proposed in Yao et al. (2021). Specifically, we use a constant learning rate of 10 -4 with 500 iterations of finettuning (or equivalently data ratio of around 0.05 with batch size of 128). Different from (Yao et al., (021). we find fixed BN is not helpful, and we allow it to update during the whole fine-tuning step. As mentioned Sec. 5 we apply grid search to determine the fractional lengths for both weight and input, as the training cost is very small and applying grid search does not introduce too much effort or training time. Also, since the original full-precision model uses the normalized input, also apply normalization on the images and quantize images with signed fixed- point numbers (and format determined with grid search) before being fed into the first convolution layer of the model.
15

---

Published as a conference paper at ICLR 2022
7.2 MORE DETAILS FOR STATISTICAL ANALYSIS
For the toy example in Fig. 3
we sample 10, 000 zero-mean Gaussian random variables with different
standard deviations, and apply ReLU activation for the rectified Gaussian variables with unsigned quantization. The variables are then quantized with fixed-point quantization given in 2 and ( respectively. We calculate the relative quantization eπror and plot against the standard deviation for each fixed-point format. Note that zero-mean is a reasonable simplifying assumption if we assume to neglect the impact of bias BN for analysis purposes.
7.3 DERIVATION FOR FIXED-POINT AND PACT RELATION
Here we derive the relationship between PACT and fixed-point quantization shown in 4 the PACT quantization in 33 can be formulated as follows for positive at
Specifically,
(xC
a
round
clip (z, 0,
(7a)
M
a
round
Mcclpp 0.
(7b)
M
M
(WT.
round
clip
I,
2WL
(7c)
2WL
a
2T. 1
2F
M
(2WL.
M
2WL - 2FL
round
2WL
clip
2FLa
20
,,,,, ,0,2
2
-
(7d)
For M = 2T, which is the typical setting for quantization, we have:
2 PACT(x) 2WT. - 2FT. round
21.
clip
2 x 2PL 0,, 2L
(8)
Comparing with the expression for fixed-point quantization
2b. we can immediately get
7.4 DOUBLE SIDE QUANTIZATION FOR WEIGHT AND MOBILENET V2
In (2). we only give the formula for fixed-point quantization of unsigned case. For weight and activation from some layer without following ReLU nonlinearity (such as some layers in MobileNet V2), signed quantization is necessary, and the expression similarly given as:
fix-quant(r) = 2FL round (clip (x 2F,, WT. + 1T. -1)),
(9)
where clip is the clipping function, and 0 < FL < WL-1 -
7.5 DERIVATION OF EFFECTIVE WEIGHT
Here we derive the equation of effective weights relating two adjacent layers in Sec. 4.3 Specifically, for Conv-BN-ReLL block with conventional PACT quantization using input clipping, quantization and dequantization, the general procedure can be described as
Nonlinear:
clip
(1 (1-1) 0, (,
(10a)
Input Quant (uint8):
round all)
M
2(11
(10b)
a ill
Input Dequant:
4
(10c)
M
(1)
Conv:
W (()
ij 4
(10d)
1=1
BN:
4
(1)
(1) V
(10e)
16

---

Published as a conference paper at ICLR 2022
11) (1) yi
(1) 2
3
(4) P
(10f)
where a: is the input before clipping, is the integer input after quantization, 7 is the full-precision input after dequantization, clip is the clipping function, the clipping-level, M 2 is the scaling for quantization, Wiy is weight from convolution layer, and 7 3, a. are weight, bias, running standard deviation, and running mean from BN layer, respectively, and and j are spatial indices. We first note that (10a), (10b) and (10c) can be combined as:
PACT(x (x-)) (111)
(11a)
1 11-1)
llix fix quant
(11b)
(1)
Tix
Tix 4
()) (
(11c)
where q is the fixed-point activation and we have used the relationship given by 4 and the definition in ( From this derive that:
4.
(1+1)
fix quant
1+1)
(2)
(12a)
Tlii
(1)
7000
fix-quant
(1+1)
(1)
M4
(1)
(300)
(n) u
(12b)
7ix
20
( (1)
fix fix_qqunt
fj +
(12c)
(++1
Tlix
j=l
(1)
fix-quant
(1+1)
(4)
W
Mix
(1) 11)
q3
(8
(1)
(12d)
7lix
(1)
= fix .uann
Vix
3
(12e)
(411
(411)
Thx
1/x
which is just (6)
7.6
PRIVATE FRACTIONAL LENGTHS ENABLING DIFFERENT CLIPPING-LEVELS
Here we analyze the effect of using private fractional lengths between sibling layers to indicate that this effectively enables private clipping-levels for them. In fact, the original PACT quantization step given as
(13a)
2FL a
(W -
2F r x 2 2,002.W. WL -1 -1)
2WT. 2FI. round
clip
(13b)
where we have omitted layer and spatial indices for simplification. Now if we private fractional lengths for sibling layers while require them to share the same clipping level, and use the master child's fractional length for calculating the effective weight in (6) denoting the fractional length of
the master layer as Fl,m, the above function becomes
2FL a
WW -1
* 2 2,,2,,,, 2WL - 1))
round clip
(14a)
2WL
2FL.
2FI.
(14b) where a' FI. From this that using private fractional lengths effectively enables different clipping-levels between sibling layers, and the cost is only some bit shifting.
F2FL-FL F
2FL
1 round
clip
(WWL_ -1 x 2FL ,,, 2WL - 1)) 2FL
2WL
2FL
17

---

Published as a conference paper at ICLR 2022
Fractional Length vs Layer (8-bit ResNet50)
Fractional Length
6
4
2
O
Layer
2
4
6
8
# Weight FL
# #Activation FL
Figure Fractional lengths of each layer for a well-trained fixed-point model for ResNet50.
Table 3; Analysis of the impact of the searching space for fractional length (ResNet50 ImageNet).
Method
Frac. Len. Range
BL
Top-1
Baseline (FP)
77.6
77.6
F8Net (ours)
6, 7,8
77.6
72.4
F8Net (ours)
77.6
77.6
7.7 MORE DISCUSSION OF THE OPTIMAL. FRACTIONAL LENGTH
Here we give some further discussion using standard deviation determine the optimal fractional length. The main reason is that standard deviation is a more robust statistics than others, such as dynamic range, and is an easily-estimated parameter for Gaussian distributed weights and pre- activations. Considering depth-wise convolution layers that contain much fewer weights and inputs, using robust statistics becomes essential as these layers might include weights or inputs with strange behavior, e.g., the pre-activation values of some channels become all negative with large magnitude. Therefore, the standard deviation is more suitable and robust than the dynamic range.
7.8 FRACTIONAL LENGTH FOR RESNET50
Here provide results of fractional lengths distribution in Fig. 6 the well-trained ResNet50 with 8-bit fixed-point numbers finetuned from the Baseline #2 in Table 2b As we can see, the optimal fractional lengths are layer-dependent and their distribution is highly different from those in MobileNet V2 (as shown Fig. 2b). Specifically, for MobileNet V2, some layers have vanishing weight fractional lengths and less than 4% of all layers have activation fractional length less than 4, while for ResNet50, more than 88% of all layers have an activation fractional length that is less or equal 4.
7.9 ANALYZING SEARCHING SPACE OF FRACTIONAL. LENGTHS
In the main paper, we adopt the largest possible searching space for the fractional lengths of 8-bit fixed-point. As shown in Fig. 2h many layers have fractional length less than 4, either for input or
18

---

Published as a conference paper at ICLR 2022
weight. Here we study whether it is possible to use only fractional lengths between 6 and 8. To this end, we finetune on ResNet50b using the Baseline #1. The results are listed in Table from which we find that restricting the fractional lengths between 6 to 8 significantly impacts the performance of the final quantized model, as the top-1 accuracy drops from 77.6% to 72.4%.
19


Reviews:
[{"review_id": "qcI9Ce8oxRc", "reviewer": "Reviewer_7TRc", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pro: \nThis is a very strong paper and I have not seen any paper of such quality in this space for some time. The analysis of relative error and normal distributions gives a strong empirical and theoretical basis for the approach used in this work and provides deep insight into what the optimal fixed-point data type in each layer would look like. This analysis alone will inform future research in this area. Usually, such data types would be learned. It is very creative to abandon this approach and instead start from the analysis. Later this approach is merged with learned quantization through unification with PACT.\n\nOverall, the results are outstanding and appear to be reliable given the extensive analysis of the approach.\n\nCon:\nSome details might not be accessible to readers that do not have an extensive background, especially pre-estimation of the fraction length and the sharing of the clipping thresholds for residual connections. It might be beneficial to trim the conclusion and add more context for these sections.\n\n\n==============================UPDATE===================================\n\nAll the other reviewers take a very hardware-oriented perspective. I feel like they are not seeing the broader picture of this work which goes much beyond its initial applications. Many insights of this work are directly applicable for mixed-precision hardware where most, but not all operations are done in some 8-bit data type. From my own experience, I know that most 8-bit training methods fail when scaled up to models with billions of parameters.  For that area, quantization precision is the most important unsolved problem. This is a very important problem since transformer models keep getting larger and larger and accelerating training through 8-bit methods can make it more feasible to train these models. This work is one of the only ones in this space that makes some headway by its excellent analysis and impressive results. Many of the insights of the paperw are immediately applicable to my research. \n\nOverall, I disagree with the perspective of the other reviewers and I still believe this is an outstanding paper. As such, I keep my score of 10.", "summary_of_the_paper": "The paper analyzes the relationship between relative quantization errors and fixed-point formats for zero-centered normal distributions and finds a linear model which fits the best exponent length of the fixed-point data type given a standard deviation. These insights are then unified with parameterized clipping activation (PACT) to normalize incoming floating point data into the desired fixed point range. To handle the network with sole 8-bit multiplications a forward pass in floating precision is used to compute batch norm statistics for the main 8-bit forward/backward pass. Additional adjustments are made between successive layers and residual layers which rely on reusing some statistics of the previous layer.\n", "summary_of_the_review": "The approach taken in this paper is very creative and the insights gained from this paper will inform research in this space for quite some time. As such, I strongly recommend acceptance of this work. I would be happy for this work to be selected as an oral presentation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "OuQS6LQLfi", "reviewer": "Reviewer_xQht", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1- The proposed numerical format is evaluated in variant of benchmarks which mean the proposed approach can be deployed and generalized for different models\n2- Combining the PACT and fixed point is interesting approach.\n\nWeaknesses:\n1- The correlation between standard division and fraction bit-width is interesting but it is obvious and in my opinion it is not required. The fraction length can be selected based on dynamic range and I think the reason behind the constant in the equation 1 is the correlation between standard division and dynamic range (DR=3ST). If we closely look at the figure 3, we find most of these standard divisions ( more than 1 ) is not useful for your case study, since dynamic range of most parameters is less than one for most of the layers. This small dynamic range of parameters explains why most of the layers parameters are using 6 to 8 bit fractions in figure 2. This brings a doubt on the motivation of why using the fixed point instead of INT8 approach?\n2- The state of approach like dyadic quantization [1] and linear quantization with bit shift [2] performs DNN inference with 8-bit quantization without 32-bit INT multiplication. How is your approach different compared to these studies?\n3- The author claims the performance of his approach is better than state of the art. However, the 32-bit baseline is different and the degradation accuracy should be reported in Tables 1 and 2. For instance, in ResNet18 result the degradation of accuracy in HAWQ-V3 is less than the proposed approach. \n4- The new approach needs to compare with pervious work for 4-bit quantization. It is difficult to understand the advantages of this approach in compared to pervious work in 8-bit? \n[1] Yao, Zhewei, et al. \"Hawq-v3: Dyadic neural network quantization.\" International Conference on Machine Learning. PMLR, 2021.\n[2] Langroudi, Hamed F., et al. \"Cheetah: Mixed low-precision hardware & software co-design framework for dnns on the edge.\" arXiv preprint arXiv:1908.02386 (2019).", "summary_of_the_paper": "The paper proposes a low-precision DNN inference models with 8-bit fixed point. To realize the number of fraction bits, the author uses the variance of DNN parameters and combines it with PACT approach in QAT. The new approach is evaluated in various neural networks such as MobileNet V1/V2 and ResNet18/50 on ImageNet for image classification and the result are mostly par with the state of the art approaches. ", "summary_of_the_review": "As a reviewer mentioned, the benefits and motivation of the proposed works is not obvious and the experimental result have not shown the advantages of this approach compared to state of the art works in 8-bit integer quantization. Therefore, I recommended this manuscript is marginally rejected. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "52WwCTCzidu", "reviewer": "Reviewer_jedd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n - The paper is well written and easy to understand. Technical details are clear.\n - The idea seems practical and straightforward to implement in real systems\n\nWeaknesses:\n - The paper claims that the quantized training flow uses only 8-bit fixed-point multiplications. However, calculating the fractional length requires the standard deviation, which needs to be done in float? The two-pass method for batch norm also uses float in the first pass. It's not clear if the technique is suitable on a specialized accelerator that performs only fixed-point multiplication. Instead the technique seems to target quantization-aware training (QAT) on GPUs for producing a quantized model. If this is the case, the results seem weak - the improvement over SOTA is very small, and the models tested are all small scale. The results are not enough to convince me that this is a significant contribution to QAT literature.\n - The novelty of the formulas to compute the fractional length seems weak. It's common in QAT to scale the tensor by dividing by the max value or dividing by some clip value. The scaled tensor then has range [-1, 1] and maps to a fixed-point representation with no integer portion. It's not clear what is gained by using stddev to estimate the fractional length instead. I would like to see some small experiments specifically comparing these ways to handle the dynamic range of a float tensor. In my mind they should achieve similar accuracy results.\n - It's not clear to me what combining PACT with quantization achieves. PACT is already meant to be used with quantization, and it's not clear what we gain by combining the two instead of doing PACT, followed by quantization. Again, some small experiments comparing the two would be useful.\n ", "summary_of_the_paper": "The authors propose a new quantization flow to train DNNs using only 8-bit fixed-point multiplications. They show that 8-bit fixed point can represent different exponent ranges based on fractional length, thus choosing the right fractional length is critical. They then empirically derive a formula to calculate the optimal fractional length for a tensor based on its standard deviation. The authors combine PACT (learnable clip threshold) with fixed-point quantization, and propose a two-pass method to handle batch norm. They can achieve a small accuracy improvement (<1%) when training from scratch or fine-tuning on ImageNet using a variety of small models (ResNet-18 and MobileNet variants), compared to other quantization-aware training methods.\n", "summary_of_the_review": "The authors propose a method to estimate the optimal fractional length of a fixed-point format given a float tensor, as well as other minor improvements to a quantization-aware training flow such as (1) combining PACT and quantization, (2) a two-pass method to handle batch norm. However, the novelty of the proposed methods is low, and it's not clear to me what improvements they offer over existing standards. The end-to-end accuracy improvement of the authors' technique is fairly small. As a result I rate the paper marginally below acceptance. If the authors can provide experiments that show specifically how each of their proposals improve over existing standard, then I would consider raising my score.\n\nEDIT: The authors have addressed most of my concerns and I will raise my score to a 6. I still think that the formulas to compute the fractional length are not novel. The idea of estimating the dynamic range of the tensor and using that to determine the quantization range is well known.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "0pqOSH93LZ3", "reviewer": "Reviewer_1v2J", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths:\n+ The authors consider a practical issue from the perspective of DNN hardware implementation. The problem is well-defined.\n+ The paper is easy to understand. The solution is presented clearly. \n\nWeakness:\n\n- I am not convinced that 8-bit fix-point format is cheaper to implement than the other numeric formats. Quantization has been studied very extensively by the ML community, and numerous numeric formats have been applied to facilitate the implementation of the DNN. (e.g., binary quantized DNN, Logarithm quantization [a1], block floating point [a2], etc). The authors should better motivate this fix-point format by providing some hardware evidence. Without this, it is hard to convince the reviewer about the usefulness of this work.\n- Equation 1 is based on the empirical approximation with Gaussian distribution, not the real DNN trace. Given the importance of this equation, authors should better justify the correctness of this empirical approximation with real DNN trace.\n- This work assumes the ReLU is used for nonlinear operation. How to handle other types of nonlinear operations (e.g., leaky ReLU)? \n- In the paper, the author mentioned that they use previously stored activation fractional length for pre-estimating the fractional length for the following layer. What does the previously stored activation mean? Is it the activation from the last training batches?\n- Observation 1 and 2 is obvious. It is nice to have some real empirical results to demonstrate this,  but I think they should be described very briefly.\n\n[a1] Oh, Sangyun, et al. \"Automated Log-Scale Quantization for Low-Cost Deep Neural Networks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[a2] Darvish Rouhani, Bita, et al. \"Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point.\" Advances in Neural Information Processing Systems 33 (2020).", "summary_of_the_paper": "This paper describes a novel quantization framework that involving only fix-point 8-bit multiplication for DNN execution. The paper first highlights the advantages of the fixed-point numeric format. The paper then conducts some statistical study and derive an empirical formula to relate the fraction length of the fix-point representation with the standard deviation of the value distribution. After that, the paper introduces a novel approach to determine the right format for each layer during the forward propagation of the training. The proposed solution, F8Net, has been evaluation on ImageNet using multiple DNN structures (e.g., MobileNet, ResNet). \n", "summary_of_the_review": "Overall, I think this paper lacks some motivation for the fix-point format. Given the fact that quantization has been studied extensively by the previous work, I am not convinced that 8-bit fix-point can achieve a better hardware efficiency than the previous work. Some hardware results may be helpful.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach for 8-bit fixed point training of NNs, based on a careful analysis of quantization error in fixed-point methods. They present convincing and thorough empirical results in addition to a detailed analysis providing insights about their method. Reviews for this paper were quite split. One reviewer was a strong advocate, asserting that the paper will have substantial impact in the area, and that the authors’ approach of minimizing quantization error for fixed-point training is of substantial practical interest. Other reviewers were concerned that the proposed method was not novel enough, and that the proposed approach was not practical enough to work in realistic hardware use cases. The authors provided substantial detailed responses addressing the majority of reviewers’ concerns, and after following the discussion in detail I agree with the reviewer advocating for the paper, that the paper presents a practical, novel approach with valuable insights for the field from their analysis and results. 

I indicated I am certain about this decision, but I would be ok with the paper being bumped down from oral to poster.
Decision: Accept (Oral)