End-to-end Quantized Training via Log-Barrier Extensions
Authors: Juncheng B Li
Shuhui Qu
Xinjian Li
Emma Strubell
Florian Metze
Venue:  (2021)
Keywords: Quantization
Constrained Optimization
Mu-law
8-bit training

Abstract: Quantization of neural network parameters and activations has emerged as a successful approach to reducing the model size and inference time on hardware that sup-ports native low-precision arithmetic.   Fully quantized training would facilitate further computational speed-ups as well as enable model training on embedded devices, a feature that would alleviate privacy concerns resulting from the transfer of sensitive data and models that is necessitated by off-device training.  Existing approaches to quantization-aware training (QAT) perform “fake” quantization in the forward pass in order to learn model parameters that will perform well when quantized, but rely on higher precision variables to avoid overflow in large matrix multiplications, which is unsuitable for training on fully low-precision (e.g. 8-bit)hardware. To enable fully end-to-end quantized training, we propose Log BarrierTail-bounded Quantization (LogBTQ). LogBTQ introduces a loss term, inspired by the log-barrier for constrained optimization, that enforces soft constraints on the range of values that model parameters can take on. By constraining and sparsifying model parameters, activations and inputs, our approach eliminates over-flow in practice, allowing for fully quantized 8-bit training of deep neural network models. We show that models trained using our approach achieve results competitive with state-of-the-art full-precision networks on the MNIST, CIFAR-10 andImageNet classification benchmarks.

Under review as a conference paper at ICLR 2021
END-TO-END QUANTIZED TRAINING BARRIER EXTENSIONS
VIA LOG-
Anonymous authors Paper under double-blind review
ABSTRACT
Quantization of neural network parameters and activations has emerged as a suc- cessful approach reducing model size and inference time on hardware that sup- ports native low-precision arithmetic. Fully quantized training would facilitate further computational speed-ups as well as enable model training on embedded devices, feature that would alleviate privacy concerns resulting from the transfer of sensitive data and models that is necessitated by off-device training. Existing approaches to quantization-awai the forward pass in order to learn model parameters that will perform well when quantized, but rely higher precision variables to avoid overflow in large matrix multiplications, which is unsuitable for training on fully low-precision (e.g. 8-bit) hardware. To enable fully end-to-end quantized training, we propose Log Barrier Tail-bounded Quantization (LogBTQ). LogBTQ introduces a loss term, inspired by the log-barrier for constrained optimization, that enforces soft constraints on the range of values that model parameters can take on. By constraining and spar- sifying model parameters, activations and inputs, our approach eliminates over- flow in practice, allowing for fully quantized 8-bit training of deep neural network models. We show that models trained using our approach achieve results compet- itive with ofate-of-the-art full-precision networks on the MNIST, CIFAR-10 and ImageNet classification benchmarks.
training (QAT) perform "fake" quantization in
INTRODUCTION
As state-of-the-art deep learning models for vision, language understanding and speech grow creasingly large and computationally burdensome (He et al. 2017, Devlin et al. 2018 Karita et al.) 2019), there is increasing antithetical demand, motivated by latency, security and privacy concerns, to perform training and inference in these models smaller devices at the edge rather than in server farms in the cloud. Model quantization has emerged as a promising approach to enable deployment of deep learning models edge devices that reduce energy, latency and storage requirements by performing floating- boint computation in low precision (less than 32 bits).
There two primary strategies for quantization: Post-training approaches quantize the parameters of a model trained in full precision post-hoe, and tend to suffer a heavy penalty on accuracy since their inference graph differs substantially from training (Jacob et al. (2018). Quantization-awa training (QAT) (Bhuwalka et al. 2020) combats this discrepancy by simulating quantization during training, so that model parameters are learned that will work well when inference is performed in low precision. In this work, we focus on the latter setting, suitable for fully quantized training on low-precision (e.g. 8-bit) devices.
Though QAT results in quantized models that perform largely on par with their non -oon-guatized coun- terparts, current ofate-of-theeart QAT methods (Wu et al. 2018 Wang al. 2018 Bhuwalka al. 2020) are not suitable for training on fully low-precision hardware because they employ fake quan- tization, meaning each operation executed using 32- or 16-bit floating point arithmetic, and its output is quantized to lower precision, e.g. int8. This results in two key incompatibilities with fully low-pre -re-rrcciionn training, and consequently deployment on real low-precision hardware. First, existing QAT approaches assume perfect accumulators used to compute matrix multiplies (the acc row in Table I must be higher precision than the values being multiplied (other bit-precision in Table T) This is avoid losing res-
in inner product operations, which means that the

---

Under review as a conference paper at ICLR 2021
quantization range
-2
2
log barrier constraintiu)
Figure 1: Left: Visualization of the log barrier constraint applied to network parameters quantized in range 1-2, 2. 2). See g3.3 for an approximated tail bound on possible overflow. Right: μ-law encoding vs. FP8(1-5-2) and FP8(1-4-3) for all possible values on interval [-2, 2]. p-law maintains higher precision at concentrated small values.
olution in low-precision additions, also known as swamping [Wang et al. 2018 Second, QAT commonly leverages dynamic quantization ranges per-layer, meaning the mapping between high- and low-precision values varies by layer, carefully tuned as a function of the network architecture, optimization dynamics and data during training. While this practice results in higher quantized in- ference accuracy, it is also challenge to low-precision training, since it is unclear how to tune those ranges when training on new data the absence of high-precision arithmetic. These incompati- bilities present substantial hurdle to quantized training in practice. For example, automotive electronics manufacturer may want to deploy a machine learning model on its 8-bit door lock or power window controller to adaptively fit the users' habits. In this scenario, existing approaches for quantized training would fail (Skkr et al. 2019).
In response, we propose new approach for fully quantized training of neural network models, in- spired by the barrier method from convex optimization (Boyd & Vandenberghe 2004). Log Barrier Tail- aillbounded Quantization (LogBTQ) utilizes log barrier extension loss (Kervadec et al.) 2019) to constrain the output of the network, encouraging all model parameters and activations to stay within the same predefined range. The log barrier function itself is smooth approximation of the indicator function, which is ideal for selecting the weights that are within the range of quantization (see Figure [ left). By fixing a single quantization range throughout the network the beginning of training, our approach both obviates the need for dynamic ranges, and the limits of the range are set so as to alleviate overflow in matrix multiply accumulations. We combine the log barrier extension loss with an L' regularization term (Hoffer et al. 2018) to further reduce the total mag- nitude of parameters and activations in the model. To allow for gradients, which tend form a peaky distribution near extremely small values (Zhou et al. [2066] Jain et al., (2020), to be quantized using same range as rest of network, we also adopt the nonlinear lalaww algorithm from audio applications (Deng & Doroslovacki] (006) to construct new MU8 codebook that better deals with "swamping" issues compared to the standard IEEE Float Standard. Experiments show that our ap- proach achieves competitive results compared state-of-art full- cell-preiision models the MNIST, CIFAR-10 10 and ImageNet classification benchmarks, despite our models being trained end-to-end using only bits of precision.
'Swamping: Accumulation of floating-point numhers, where the small magnitude value ignored cated) when it added to large magnitude sum.
?Overflowing: for the fixed-point accumulation where the accumulated value wraps around to the small value when it exceeds the largest value representable by the given accumulation precision.
2

---

Under review as a conference paper at ICLR 2021
Training Precision
Train/Test Prec.
Quantized Training Scheme
Quant. Range
W
dW
dx
FP32
Low
DoReFa-Net (Zhou al.
2016
32
6
32
Dynamic
55.9
46.1
WAGE (Wu et al. 2018)
8
8
32
Dynamic
51.6
DFP (Da el al. 2018)
16
16
16
16
32
Dynamic
75.70
75.77
MPT (Micikevicjus et al. 2018
16
16
16
16
32
Dynamic
75.92
76.04
Wang et al. (2018)
8
16
Dynamic
72.14
71.72
HFP8 (Sun et al. 2019
8
8
8
8/16
Dynamic
76.44
76.22
LogBTQ (ours)
8
8
Fixed
73.59
71.11
Table 1: Comparison of seduced-precistee training for top-1 accuracy (%) using ResNet-50 (Im-
ageNet). For works that did not evaluate on ResNet-50, we include AlexNet results (italicized). Dynamic indicates that quantization ranges vary by layer and must be learned or tuned; Fixed indi- cates single quantization range is fixed globally throughout the network.
BACKGROUND AND RELATED WORK
2.1 POST-TRAINING QUANTIZATION
There was been a recent surge of interest in quantization research. 2020 alone, there were a number of important developments in post-training quantization. Rusci et al. (2020); Jain et al.) (2020); Esser et al. (2000); Uhlich et al. (2020) proposed learning-basee approaches for determin- ing the quantization ranges of activation and weights at low precision. [Stock et al. (2020) advo- cates preserving the quality of the reconstruction of the network outputs rather than its weights. They all show excellent performance compared full-precision models after quantization. [Sakr & Shanbhag (2019) presented detailed analysis of reduced precision training for a feedforward net- work that accounts for both the forward and backward passes, demonstrating that precision can be greatly reduced throughout the network computations while largely preserving training quality. Our work share the same intuition of preferring small predetermined dynamic range (PDR) and small clipping rate However. Sakr & Shanbhag 2019 's approach requires the network first be trained to convergence at full 32 bit precision, which a significant limitation. In this paper, we focus on training rather than inference on low-pree wwwpprecision hardware, therefore, we do not assume access to full full-precision tegh-perfrmmin model as a starting point.
2.2
QUANTIZATION-AWARE TRAINING
Pioneering works in this domain Zhou et al., 2016 Courbariaux et al. 2015) looked at quantizing model weights, activations, gradients to lower precision to accelerate neural network training. The terminology quantization aware training (QAT) was first introduced by Jacob et al. (2018). QAT incoporates quantization error as noise during training and as part of the overall loss, which the optimization algorithm tries to minimize. Hence, the model learns parameters that are more robust to quantization, but QAT is not meant to be performed entirely in low precision, it aims to learn parameters that will work well for low-precision inference. More recently, several works further pursued the goal of enabling fully low-precision training (Wu et al.) 2018 Wang et al.) 2018 Das al. 2018 Sun al., 2019). As shown in Table [ most existing work employs fake quantization. resorting to higher precision values to compensate for the swamping issue, especially during gradient accumulation. Mixed preeisppeeision quantization (Das et al. 2018 Wang et al.) 2018 Zhang et al., 2020a), which quantizes neural network using multiple bit precisions across layers, still relies on higher-precision gradients to preserve model accuracy. This means it is difficult, if not impossible, to implement these approaches on low-bit (e.g. 8-bit) hardware.
Most similar to our work, [Sun al. (2019) claim is possible to do every step in low precision, but the quantization range for the layers in their work is very carefully chosen empirically, which presents great difficulty if we were to train models from scratch on low-precision hardware. Their method also requires copy of the quantization error (residual) in FP16(1-6-9) (hence 8/16 in Ta-
see Appendix B of Sakr & Shanbhag 2019 explaining PDR; refer their Criterion about clipping rate.
3

---

Under review as a conference paper at ICLR 2021
Output
Error
Err quant
Act Quant
8-6x
Act Quant
Conv quant
STE
quant
Conv
8-br
bit
Input
Input quant
Wt quant
Weight
Error
Grad quant
weight
Figure 2: Left: Diagram of the forward pass using LogBTQ quantization. Right: Gradient propaga- tion.
ble D. In addition to the 9-bit mantissa, the exponent bit in their floating point format would need to be manually modified to store the residual due its small value.
In this paper, propose new quantization scheme: log-barrier tail-bounded quantization (Log- BTQ) that can perform fully end-to-end low precision training, suitable for deployment on low- precision hardware. Our major contributions are the following:
1. We apply a log barrier extension loss to soft-threshold the values of network weights and activations to constrain all the values to be small. Our quantization scheme also enables global fixed-range quantization which together significantly alleviates the overflow issue caused by large numbers and dynamic range.
2. We add an L loss term to encourage sparsity and further reduce overflow.
3. We propose y-law quantization (MU8) instead INT8, FP8(I-4-3) or FP8(1-5-2) to con- struct more accurate codebook that better compensates for the peaky concentration of network parameters around small values.
3 LOG BARRIER TAIL-BOUNDED QUANTIZATION (LoGBTQ)
The overall diagram of our quantization scheme is shown in Figure 2 (left). Figure 2 (right) shows the backward pass, where we quantize everything at each layer and all operations including input x, weights w, activations eгrогs and gradients g (including the gradient accumulation step). We denote all these values as the set Z = [x, w, a, e, g) In this work, different from previous works (Sakr & Shanbhag/ 2019 Zhang al. 2020b) that used adaptive quantization range, adopt globally fixed quantization range for every element E Z, and set E 2. 2]. We do not need to adjust the range and precision during training in other quantization work that relies layer-wise dynamic ranges. This would greatly reduce the overhead for implementation on hardware.
3.1 CONSTRAINED FORMULATION
Let D = {I' !N. denote the labeled set of N training images, and f denote the neural network model. 0 here denotes all the parameters of the neural network including weights w. For task, such as image classification, we are usually solving such an optimization problem: min[(fo(T)) where 8
C is the loss function of our neural network training objective. In this work, we use the typical cross-entropy loss, and since we are interested in constraining the quantization threshold, we are effectively performing constrained optimization in such form:
minimize
L (fo(
(1)
subject to 10, S u, I,..., N.
4

---

Under review as a conference paper at ICLR 2021
With our desired barrier (perturbation). In practice, set 0.1 to ensure represent as much information as possible within our quantization range (Figure | left). This setting is also explained further in Section 3.
3.2 LOG-BARRIER EXTENSION FUNCTION
Theoretically, problem should be best solved by the log barrier method which is an interior point method that perfectly handles inequality constraints. (Tibshirani [219 [Boyd Vandenberghe 2004]: In phase I, we would perform Lagrangian-dual optimization to find the feasible points:
N
maximize minimize
LL (x, A) L(f((l)
Σ 1000n|
subject to
A 0, n n= 1,... N.
where À E RR xx the Lagrangian multiplier (dual variable). After we find feasible set of network
parameters, we can use the barrier method to solve equation (T) as an unconstrained problem:
N
minimize ((fa(T))
Σ
vh. - uu)
(3)
solving problem (3) is Phase II, where is the standard log-barrier function: t/. - 1o0g(( (10) As t approaches infinity, the approximation becomes closer to the indicator function. Also, for any value of any of the constraints is violated, the value of the barrier approaches infinity. However, huge limitation in applicability to practical problems such as ours is that the domain of Eq. (3) must be the set of feasible points. The canonical barrier method above is also prohibitively computationally expensive given there are millions of parameters in the network, and we need to alternate the training between primal and dual and do projected gradient ascent for the dual variable.
We are not particularly concerned with the weak duality gap lower-bound the optimal solution in this work, instead, we are interested in the property of the barrier method handle inequality constraints. Therefore, inspired by Kervadec al. (2019), we formulate quantization an uncon- strained loss to approximate the constrained optimization problem:
N
minimize 0
C(fo(I))
w. (l u)
(4)
where v. is the log-barrier extension, which is convex, continuous, and tvice-differentiabl::
; log( t2 - ( + otherwise
2 -
(2)
(5)
in our case, the input to the log-barrier extension is the same 2 we defined in the beginning of Section and t is the scaling parameter. It basically shares the same property with the standard log- barrier function, when t approaches +00, our log-barrier extension would approach a hard indicator: and +oo otherwise. But its domain is restricted the feasible points only.
H()) = 0 if This removes the demanding requirement for explicil Lagrangian optimization.
We doing approximated Lagrangian optimization with implicit dual variables. Our striet positive gradient of th will get higher when approaches violation and effectively push back into the feasible value set. Because our penalty does not serve as a strict barrier of the feasible set, there is possibility of overflow. However, our goal is achieve the practical goal of fully-quantized training on low-bit hardware, as long as the majority of values stay within a high confidence interval, the approach will work in practice, as we show in experimental results
Recall the scenario in Figure left).
3.3 TAIL BOUND OF DISTRIBUTION
In this section, we demonstrate that the probability of overflowing the quantization range can be controlled for a standard ResNet model. The ResNet model has L layers, each layer contains a CNN operation and ReLU activation. Consider at layer 1, for each output element O E R in the
5

---

Under review as a conference paper at ICLR 2021
layer, CNN operation with the kernel size l and channel size can be viewed as a rectified dot product between the input feature Reck with w Rcck"
= ReLUg Σ w,l
(6)
Suppose wi (O, ( and oi) is dependent on the layer and the parameter u we choose.
The second moment of any element 2(1+1) R from layer I + can be connected with the second moment of element =()) from layer as follows:
ch2
B[ (1+++))
2 (1) (l
2
In the He initialization, we could set dw control oi) in order reduce the chance of overflow. We choose oB to depend u which was defined in optimization problem (1): (
V
ck? to cancel the first two terms. In this work, we
Then, we simplify Eq.
Suppose the input feature to the first layer has second moments E(1) B[=ī), then the second moments layer can estimated as: Egy [ wl E(y-- Next, for each element 2(1) at layer its tail distribution outside the quantization region can be bounded by
Var(z()
P(]=0) 2) = P(z 2) P(Iz0
E| 2 E[ (2 E(
where the second inequality is established by Chebyshev's inequality, here, is guaranteed to be nonnegative since it's the output of the ReLU layer. We can further estimate the worst upper bound for the right hand side by considering the following optimization problem:
Var(z())
sup subject to
(2 - E(2()))
(E[z()))² + Var(z(n) = E())
As a()) is the rectified value, B[zu]] > One upper bound can be established as follows:
(2 2( Var(z()) < (2 E
E(D
sup Varl
(10)
[[((0) < / Eng and Var(=()) < Egy- Notice that the last term can be approximated to Eay when Feyy is small enough, and we can establish our probability bound of overflowing the quantization range:
because
(2 - FAD (E
E 4
P([an) > 2)
(11)
Finally, consider the average overflow probability
R from any layer:
1 E(y
E(D)
P[|2| > 2) Σ P(zo) > 2)
(12)
4L
4L(1 w)
Formula (12) derived because deep neural networks, usually large number allowing us to effectively ignore wt. Therefore, the overflow probability is determined by the input's second moments E, the layer size L and the barrier parameter u. In practice, we can adjust both u and L to control the overflowing tails. In this work, E(y) is set around 1.0 because the input features normalized, and L is around 50 in the ResNet-50. By choosing u 0.1, the number of overflowing parameters be controlled under 2.3%.

---

Under review as a conference paper at ICLR 2021
3.4 SPARSITY
In order to achieve the goal of practical implementation on 8-bit hardware, we aggressively fix the range of quantization to [-2,2] This leaves us with the task of mapping millions of parameters to this range. Therefore, we desire sparse solution, and naturally, as is pointed out by Hoffer et al. (0118), we use L' penalty |en| to encourage sparsity. Our unconstrained loss then becomes:
N
minimize C(fe(I)) + Σ ( - u) +
Een
(13)
n=1
where 7 > 0 and is tuning variable, and ahe is the log-barrier extension as proposed.
3.5 μ-LAW QUANTIZATION CODEBOOK (MU8)
Even after sparsifying the weight, we still are left will many parameters which are non-linear dis- tributed. Luckily, they should all be small enough by now thanks to log-barrier constraint. Uniform quantization (e.g. INT8) would cause huge information loss in this case. Inspired by y-law's (Deng & Doroslovacki] 2006) application in audio encoding, codebook accordingly, which could be implemented on 8-bit hardware directly. As is shown in Figure right,, we computed all possible values of FP8(1-4-3) or FP8(1-5-2 used by (un et al. 2211). and we can see our MU8 encoding are better at handling very small values e.g. FP8 (1-4- 3)'s smallest possible number is 1.9 x 10 FP8 (1-5-2) is 1.5 x 10 whereas our MU8 could handle 5 x 10 6 with the cost of getting sparser in larger numbers, which we have almost eliminated using log-barrier. Let's denote our the quantization function as Qus(X) which take x as input and output quantized value x'. First, following the range of f-llaw encoding, we set the input to Equation 15 2/2, since E [-2, 2] in this paper.
construct non-linear
F(x) sgn(x) In(1 + p|x=)
-15051
(14)
In + p)
where l tuning variable, the larger 14 is, the more non-linear becomes, and sym(x)) is the sign function. Then, we get the y [-1.1] the output of F(x). then perform Stochastic
Rounding step introduced by Gupta et al. (2015). c.g. y x (2 - 1) E [15, 16), if y = 15.5/(27 1), then Più 16/(2 1)) P 157(2" 1)) = 0.5; if y 15.1/(2 1), then P((ū
15/(27 1)) = 0.9, P(ù = 16/(2' 1)) = 0.1, as shown in the following equation.
Stochastic Mnndinggyy,LLL,T UL) = {Uz
w.p. 1 (y LL) w.p. (y LL)
(15)
where U indicates upper limit and LL indicates lower limit. At last, we get the quantized value a' via the inverse function of encoding function (15):
s' - F (v) = sgn(y) ((1 + 11)a1 - 1)
(16)
3.6 OTHER USEFUL TECHNIQUES
Straight-through Estimator (STE): We adopt the same STE as Zhou et al. (2016) which is critical to the convergence of our models. The gradient of the quantization steps in the quantization model based on the straight through estimator.
Chunked Updates:
We also use the same gradient accumulation strategy as Sakr et al. (2019);
Wang et al. (018)) During updates, since we have quantized the gradient and weight parame-
ters using Q us (X), the gradient will be lower bounded by the smallest possible value of the MU8 encoding(10 ), which is small enough to preserve model accuracy within the 8-bit constraint. However, directly adding this small gradient to a large number (weight) in our MU8 quantization would cause this small gradient to be rounded-off and thus lose information due to the growing
*FP8(1-4-3) (1---))sss largest value is [-240, 240], and FP8(1-5-2)'s 's's''s is [-57344, 57344], they are better handling long-tails than MU8, but not interested in large numbers in the long-tail.
7

---

Under review as a conference paper at ICLR 2021
27
Figure 3: Left: LogBTQ ResNet-50 weight distribution with and without log-barrier loss throughout the training process. Right: Example weight distribution in layer of LogBTQ ResNet-50.
intervals of MU8 encoding larger values. Performing chunked gradient updates, updating the weights only after steps gradient descent (in this work, we set k = 20), helps to accumulate the gradients to be large enough to be rounded up, aiding convergence.
EXPERIMENTAL RESULTS
Table. shows the performance of models trained with LogBTQ from scratch the MNIST, CIFAR- 10, and ImageNet datasets. Our performance on CIFAR-10 even higher than the FP32 baseline, and our results using ResNet-50 trained from scratch on ImageNet are competitive against both of the works in comparison. This is particularly impressive given that LogBTQ uses strietly 8-bits to store gradients whereas previous work resorted to FP16 or hybrid 8/16. Our accuracy loss of ResNet-50 on ImageNet is only 2.48% which for many use cases is worth the benefit of keeping everything in only 8 bits of precision. For practical purposes, 2-5% accuracy loss should be tolerable if calculations can be kept strictly in 8 bits.
MNIST R FP32 8-bit
CIFAR-10
ImageNet
Training Scheme
ResNet-18 FP32
ResNet-50 FP32
MobileNet FP32
8-bit
8-bit
8-bit
Wang et al. (2018)
92.77
92.21
72.
71.72
HFP8 (Sun et al. 2019
76.44
76.22
71.81
71.61
LogBTQ FP8(1 4-3)
99.9
98.1
94.08
92.31
73.59
54.32
71.68
50.19
LogBTQ FP8(1 1- (1-2-)
99.9
98.3
94.08
93.21
73.59
58.46
71.68
51.33
LogBTQ MU8(ours)
99.9
99.6
94.08
94.50
73.59
71.11
71.68
68.41
Table 2: Results of LogBTQ training compared with the two most relevant low-precision training schemes. Note that the previous work listed here not directly comparable to ours; as is shown in Table D Though these works perform the majority of computation in 8 bits, they still have the advantage of FP16 or Hybrid FP8/16 when accumulating gradients, and dynamic quantization range. First 2 rows numbers reported by Wang et al. (2018) and Sun et al. (2019).
DISCUSSION
Hou et al. (2019) also provides a bound for the quantized gradient of QAT models, giving the intu- ition that quantized gradients would slow down convergence, which we also observe in training.
For each arithmetic operation, we perform our Qxs()) quantization right after each operation to ensure the number still falls into 8 bits. For out of range numbers, clamp. We verify our assump-
8

---

Under review as a conference paper at ICLR 2021
tions in our experiments: Figure can see that they are nicely bounded by our bog-brriier to within [-1.5, 1.5]. Figure (right) shows the weight distribution is concentrating to smaller values during training.
(left) shows a layer's weight distribution during training. and
Since we only have 10 precision to handle the gradients, we are inevitably losing some accuracy compared with higher precision training schemes, FP16 can handle 10-$ precision and FP32 can handle 10 32 precision. As we can see in Table 2 LogBTQ can perform almost perfectly on easier tasks such as MNIST and CIFAR-I0, but degrades when we are training on the more challenging ImageNet dataset. Our Mu8 encoding performs better than 1-5-2 and 1-4-3 with our LogBTQ quantization scheme, showing p--law encoding can handle small fixed quantization range better than 1-5-2 or 1-4-3. As is pointed out by [Sheng et al. (2018), MobileNet architecture has some layers that are unfriendly for quantization, e.g. An outlier one channel could cause huge quantization loss for the entire model due to enlarged data range. We employed the same techniques proposed by [Sheng et al. (2018), though MobileNet benchmark is still not as good as ResNet-50 due to the agressive reduction of parameters.
Overall, still depends on the use case of the deep learning models to consider the trade-off between accuracy and precision to choose which quantization scheme to adopt. As far as we are aware, LogBTQ is the first full 8-bit quantization scheme with competitive performance on a large-scale dataset such ImageNet. This opens up the prospect of training qully-quantized models on low- precision hardware.
CONCLUSION
Motivated by the limitations of fake quantization, propose Log Barrier Tail-bounded Quantiza- tion (LogBTQ) which introduces a log-barrier extension loss term that enforces soft constraints on the range of values that model parameters can take every operation. Our approach eliminates overflow in practice, sparsifying the weights and using μ-law non-uniform quantization, allowing for fully quantized 8-bit training of deep neural network models. By constraining the neural net- work parameters driven by theoretical motivations, this work enables the possibility for the first time fulnt-quantized training on low precpiiiioon hardware.
REFERENCES
Pulkit Bhuwalka, Alan Chiao, Suharsh Sivakumar, Raziel Alvarez, Feng Liu, Lawrence Chan, Skirmantas Kligys, Yunlu Li, Khanh LeViet, Billy Lambert, Mark Daoust, Tim Davis, Sarah Sirajuddin, and François Chollet. Quantiza- tion aware training with tensorflow model optimization toolkit performance with accuracy, April 2020. URL
quantization- hrt.::/oe......aaaaaaaaaa.a.a.rr
html
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Matthieu Courbariaux, Yoshua Bengio, and Pieare David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pp. 3123-3131, 2015.
Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere, Dhiraj Kalamkar, Sasikanth Avancha, Kunal Banerjee, Srinivas Sridharan, Karthik Vaidyanathan, Bharat Kaul, Evangelos Georganas, et al. Mixed precision training of convolutional neural networks using integer operations. In International Conference on Learning Representations, 2018.
Hongyang Deng and Milos Doroslovacki. Proportionate adaptive algorithms for network echo can- 2006.
cellation. IEEE Transactions on Signal Processing,
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S Modha. Learned step size quantization. In International Conference on Learning Repre- sentations, 2020.

---

Under review as a conference paper at ICLR 2021
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning, pp. 1737-1746, 2015.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In Advances in Neural Information Processing Systems, pp. 2160-2170, 2018.
Lu Hou, Ruiliang Zhang, and James 'T Kwok. Analysis of quantized models. In International Conference on Learning Representations, 2019.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704-2713 2018.
efficient
Sambhav Jain, Albert Gural, Michael Wu, and Chris Dick. Trained quantization thresholds for accurate and efficient fixed-point inference of deep neural networks. In I. Dhillon, D. Pa- pailiopoulos, and V. Sze (eds.), Proceedings of Machine Learning and Systems, volume 2, pp. 112 128. 2020. URL Bttp::t:: httptp::://prcceed.n.g www.sml.....mmmmmmmmmmmmmm...m.m www.llys..rgg.ppeeeee wttp:/pppp/////////...// file/ 22c40000222041110000202lelll Satpeee--9b/1111--ppee...e.. pdf
Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xiaofei Wang, al. com- parative study on transformer vs rnn in speech applications. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 449-456. IEEE, 2019.
Hoel Kervadec, Jose Dolz, Jing Yuan, Christian Desrosiers, Eric Granger, and Ismail Ben Ayed. Constrained deep networks: Lagrangian optimization via log-barrier extensions, 2019.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Gareia, Boris Ginsburg. Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations, 2018.
Manuele Rusci, Alessandro Capotondi, and Luca Benini. Memory-driven mixed low precision quantization for enabling deep network inference on microcontrollers. In Dhillon, D. Papailiopoulos, and V. Sze (cds.), Proceedings of Machine Learning and Systems, volume 2, pp. 326-335. 2020. URL "https://github" www.coommo//o//e mumn// training- mi xed
Charbel Sakr and Naresh R Shanbhag. Per-tensor fixed-point quantization of the back-propagation algorithm. In 7th International Conference on Learning Representations, ICLR 2019. 2019.
Charbel Sakr, Naigang Wang. Chia- Yu Chen, Jungwook Choi, Ankur Agrawal, Naresh Shanbhag, and Kailash Gopalakrishnan. Accumulation bit-width scaling for ultra-low precision training of deep networks. International Conference on Learning Representations, 2019.
Tao Sheng, Chen Feng, Shaojie Zhuo, Xiaopeng Zhang, Liang Shen, and Mickey Aleksic. A quantization-frien friendly separable convolution for mobilenets. In 2018 1st Workshop Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2), pp- 14-18. IEEE, 2018
Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. And the bit goes down: Revisiting the quantization of neural networks. In Eighth International Conference on Learning Representations, 2020.
Xiao Sun, Jungwook Choi, Chia-Yo Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalak- shmi Viji Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In Advances in Neural Information Processing Systems, pp. 4909, 2019.
10

---

Under review as a conference paper at ICLR 2021
Ryan Tibshirani. Log barrier method, Sept 2019. URL https:/ www stat cmu edu. cyyaaa/ls//cooaxxot///r/ebb///// - wachood-ssrieddde
pdf
Stefan Uhlich, Lukas Mauch, Fabien Cardinaux, Kazuki Yoshiyama, Javier Alonso Garcia, Stephen Tiedemann, Thomas Kemp, and Akira Nakamura. Mixed precision dnns: All you need is good
parametrization. In International Conference on Learning Representations, 2020.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yo Yu Chen, and Kailash Gopalakrishnan. Train- ing deep neural networks with 8-bit floating point numbers. In Advances neural information processing systems, pp. 7675-7684, 2018.
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. In International Conference on Learning Representations, 2018.
Jingzhao Zhang, Tianxing Hc, Suvrit Sra, and Ali Jadbabaic. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Rep- resentations, 2020a.
Xishan Zhang, Shaoli Liu, Rui Zhang, Chang Liu, Di Huang, Shiyi Zhou, Jiaming Guo, Qi Guo, Zidong Du, Tian Zhi, and Yunji Chen. Fixed-point pack-propagation training. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020b.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: 001.600666660 2016.


Reviews:
[{"review_id": "vJG3eLE8lR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper introduced a log-barrier based regularization method to reduce the dynamic range of data types (activation, weight, error, gradient, and input) in neural networks. The authors claim that such regularization is important to avoid overflow or swamping in the accumulation of matrix multiplication. \n\nHowever, the reviewer is afraid to think that there are serious technical issues with this claim. Here are a few details regarding this concern.\n\n- The main claim of this paper seems to be that by reducing the dynamic range of the neural network data, they can avoid overflow. However, it does not seem to be (always) true; what matters more would be the resolution of information you need (or want) to keep, and the number of data elements you accumulate. E.g., assume that we use 8-bit -- even if the magnitude of data is constrained to be around 2^-3, if the resolution of info is 2^-8, there will be an overflow after 8 times of accumulation. Also, note that the magnitude of activation or weight of a layer is often not important due to scale-invariance of the normalization techniques (like batchnorm). Thus it is not clear if constraining the magnitude of data is always beneficial.  \n\n- The authors claim that previous reduced-precision training techniques (like [Sun et al., NeurIPS19]) have dynamic quantization range across the layers. This is NOT true, since the reduced-precision floating point format has the fixed dynamic range; e.g., [Sun et al., NeurIPS] employs (1-5-2) format for representing back-prop error, implying the fixed dynamic range of 2^(-15) -- 2^(16). In fact, one very compelling benefit of using the reduced-precision floating-point over the fixed-point format is that the floating-point format does not need scaling of the magnitude across the layers. \n\n- It is not clear if the authors' understanding of \"overflow\" or \"swamping\" is the same as what is reported in prior work (e.g., [Sun et al., NeurIPS19]).  First of all, overflow and swamping issues are different - the former issue is for the fixed-point accumulation where the accumulated value wraps around to the small value when it exceeds the largest value representable by the given accumulation precision. Whereas, the latter is for the accumulation of floating-point numbers, where the small magnitude value is ignored (or truncated) when it is added to the large magnitude sum. Thus, it is very confusing when the authors use these two different terms interchangeably in the text (e.g., the 3rd paragraph of Intro). \n\n- There are many confusing points in the analysis of the overflow in Sec 3. First of all, the authors seem to claim that Prob(z > 2) is the overflow condition... but why? I couldn't find the notion of bit-width in this analysis... but then how can we check the overflow?\n\n- The authors mention that they employ STE, which does not make much sense in case of the quantized training; what does it mean to take STE for the quantization of back-prop error?\n\n- It is not clear how the proposed method is evaluated. The computational graph in Fig 2 does not seem to include the accumulation quantization. Also, more detailed information should be provided about the following; how the MU8 format is implemented to use 8-bit hardware? how the proposed method is implemented in the deep learning framework?\n\n\nTo sum up, this paper seems to have several technical flaws, which should be carefully addressed. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "pn5QGsQNRn_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce a log-barrier extension loss term enforcing soft constraints on the range of values to enable fully end-to-end quantization-aware training. \n\nStrengths of the paper: \n\n- The paper addresses an important topic, because there are increasing concerns in performing fully end-to-end low precision training to deploy on low-precision hardware. \n- The method has a practical goal and could be interesting for practitioners \n\nWeaknesses of the paper: \n\n- Lack of positioning with respect to the SOTA quantization-aware training(QAT) and post-training quantization(PTQ) schemes, there are plenty of missing related literatures on both quantization schemes. Some statements in the background and related work could be wrong. For example, QAT also focuses on efficient inference as well as PTQ. The levels of practical applicability of a variety of quantization solutions have been introduced in DFQ(Nagel etal., 2019). Survey on the related work is not sufficient. \n- Having a benchmark would be interesting if it will include some SOTA methods and evaluates with them. The comparison targets are mostly out-of-date. It is lack of convincing evaluation results to support the proposed scheme.\n- Organizing the whole contents is ok but not good enough for the readers to easily follow and understand. \n\nDetailed comments: \n\n(1) The terminology on swamping might not be familiar with the ML community. Explaining the criticality of swamping problem is not good enough in the intro. You should provide how critical the problem is on the low-precision hardware with the other SOTA quantization schemes. For example, the probability of occuring swamping without applying the proposed scheme, etc.\n\n(2) Evaluation results provided in the paper are just for comparing accuracy. Accuracy loss is intrinsic in fully end-to-end low-precision training. The benefits of employing the proposed scheme would be beyond accuracy, say memory or energy-saving constraints for on-device training. Experimental evaluations to support the necessity and merits of the proposed scheme should be provided. \n\n(3) The quantization range is fixed in the proposed scheme. Is it a merit that the proposed scheme does not need to adjust the range and precision either per-layer or per-channel during training as in other SOTA methods?\n\n(4) Writing on the constrained optimization formulation is a bit verbose and not properly formulated. \n\n(5) Inducing the tail bound of distribution to demonstrate that the probability of swamping can be controlled, several assumptions and approximations have been applied for the worst-case upper bound. Are the assumptions reasonable to work in practice? For example, assuming that the weight distribution is Gaussian is too strong to be practical. \n\n(6) The paper has a conceptual overlap with other quantization approaches and some of the proposed scheme is not entirely novel resulting in a weak contribution. \n\n(7) In Table 2, the MobilNet has more severe degradation of accuracy than the ResNet on the low-precision(8-bit) setting. Could you explain why this happens?\n\nMinors: \n- Several typos: There was been in p.2, to soft threshold the range of in p.3, theta-i in eq.(3), some more in p.7", "extra_scores": {}, "extra_text": {}}, {"review_id": "a0S-2hRcDUb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is addressing an important and challenging problem of end-to-end training of deep nets in fixed-point, in this case, with 8-bit precision. A good solution to this problem can have a major impact on the deployability of deep nets on embedded hardware. The basic idea is to introduce an additional term (the log-barrier constraint) in the loss function to constrain the allowable range over which model parameters are allowed to take values. The authors use of mu-encoding to assign non-uniform quantization levels to minimize the quantization error. The main results are in Table 2 showing that the method eliminates overflow in practice and allows quantized networks to approach the accuracy of full-precision networks on the MNIST, CIFAR-10 and ImageNet.\n\nA few comments/questions:\n\n1) How was the quantization range ([-u,+u]) chosen? There will be a trade-off between the precision (8-b vs. 6-b) and the quantization range u. Is there any notion of optimality of the chosen value of u for 8-b quantization.\n2) How important is mu-law coding? How much accuracy is lost when uniform coding within the quantization range is applied? Please note, doing arithmetic on non-uniformly quantized variables is harder than with uniform coding, which has a bearing on hardware realizations. It will be good to quantify the accuracy loss in the absence of mu-law quantization.\n3) The results in Table 2 are sparse. It further indicates that the accuracy of MobileNet using the proposed method is quite significantly lower (66% vs. 72% for Sun) compared to previous works. MobileNet is designed for embedded IoT type hardware. 8-b or even 9-b quantized MobileNet results would put this work firmly in the domain of end-to-end fixed-point training of lightweight networks and hence make it more useful. These results will strengthen Table 2 significantly.\n4) Please comment on the choice of hyperparameters and initialization. Is this method robust to these choices?\n5) How does 8-b fixed-point compare with say MiniFloat-8 or equivalent? Such low-precision floating-point realizations are an attractive choice and a competitor to fixed-point.  \n6)The paper is missing comparisons with a couple of highly relevant papers on fixed-point training listed below:\n\n[1] Zhang et al., Fixed-point Back Propagation Training, CVPR 2020.\n[2] Sakr et al., Per Tensor Fixed-Point Quantization of the Back Propagation Algorithm, ICLR 2019.\n\nOverall a very nice paper but needs more work (indicated above) to strengthen the results.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1WBqk7FccDI", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a set of well crafted regularization techniques that improve the accuracy of QAT.\n\nThe overview of related works, particularly QAT is very well done. We clearly see what the present limitations are. I would like to point out that there have been works that have analytically determined suitable fixed quantization ranges in the context of backprop. The authors claim otherwise, I would suggest including works such as [1] in section 2.2, and revisit that claim.\n\nThe proposed method method is very nice, Lagrangian optimization techniques are used to ensure weights do not fall into the tails of the distribution. This reduces the probability of overflows. \n\nI wish to point out that the term 'swamping' does not describe an overflow phenomenon per se, it is rather an underflow that occurs when adding two very different floating-point numbers. Avoiding tail samples is definitely useful to reduce swamping, but some of the wording in section 3.3 and in the introduction probably ought to be revisited. It is not a very serious issue, but the clarity in that regard could be improved. Otherwise, the analysis in Section 3 is excellent.\n\nOne question about eq. (9): the tail distribution is defined as P(z>2) (subscripts aside for convenience). Shouldn't it be P(|z|>2)? Won't large negative numbers be problematic as well? In the grand scheme of things this won't affect the conclusions, but I think this correction (or at least a comment) should be made for extra rigor. \n\nThe experimental results are good. I wonder if the authors could spare a few sentence to discuss how they simulated accumulation quantization. This issue is not as straightforward as representation quantization that would apply to activations/weights/gradients. The cited papers that have looked at swamping previously (Wang & Sakr) do mention something about modifying the deep learning framework at the GEMM level. Is the same technique used?\n\nOne question about a statement made in the conclusion: Throughout the paper, a clipping level of 2 has been chose, but in the final discussion, it is stated that the weights are contrained to the range [-1.5,2.5]. Why the disparity? Is this an error or something expected? If so, can it be explained?\n\nMinor issue:\nIt is stated in section 3 that \\phi_t approaches the indicator function when t approaches infinity. How so? I get that z<0 -> \\phi_\\ifty = 0 but z>0 is not well defined. Maybe the definition of \\phi for z>0 can be explicitly stated.\n\n[1] Sakr, C., & Shanbhag, N. Per-tensor fixed-point quantization of the back-propagation algorithm. In 7th International Conference on Learning Representations, ICLR 2019.\n\n\n=======================================================================================================\nComments Post Rebuttal:\nI still find the technical contribution of this paper to be a good one. However, As stated in my original review, there were some clarity issues with the manuscript. While, I personally was able to follow along, the other reviewers are correct in their claim that the writing can be quite confusing (Reviewer 4 makes a strong case). To that end, I have decided to decrease my score to a 5 because I can no longer say the manuscript is ready for publication. Nonetheless, I urge the authors not to be disappointed. The presented work has many merits, and I am sure that with a thorough \"clean up\" of the text, this paper can be a great one!", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduced a log-barrier based regularization method to reduce the dynamic range of data types in neural networks. As pointed out by the reviewers, there are many technical issues. The authors agree with the reviewers in the rebuttal, though claimed that they are fixed in the revised version of the paper.

Experimental results are not convincing. It is not clear how the proposed method is evaluated. Accuracy of MobileNet using the proposed method is quite significantly lower compared to previous works. The work needs additional results/comparisons with other highly relevant papers on fixed-point training.

There are also many clarity issues that need to be fixed.
Decision: Reject