Is Integer Arithmetic Enough for Deep Learning Training?
Authors: Alireza Ghaffari
Marzieh S. Tahaei
Mohammadreza Tayaranian
Masoud Asgharian
Vahid Partovi Nia
Venue: NeurIPS 2022 Accept (2022)
Keywords: Integer Training
Accelerated Training
Integer-only SGD
Integer back-propagation

Abstract: The ever-increasing computational complexity of deep learning models makes their training and deployment difficult on various cloud and edge platforms. Replacing floating-point arithmetic with low-bit integer arithmetic is a promising approach to save energy, memory footprint, and latency of deep learning models. As such, quantization has attracted the attention of researchers in recent years. However, using integer numbers to form a fully functional integer training pipeline including forward pass, back-propagation, and stochastic gradient descent is not studied in detail. Our empirical and mathematical results reveal that integer arithmetic seems to be enough to train deep learning models. Unlike recent proposals, instead of quantization, we directly switch the number representation of computations. Our novel training method forms a fully integer training pipeline that does not change the trajectory of the loss and accuracy compared to floating-point, nor does it need any special hyper-parameter tuning, distribution adjustment, or gradient clipping. Our experimental results show that our proposed method is effective in a wide variety of tasks such as classification (including vision transformers), object detection, and semantic segmentation.

Is Integer Arithmetic Enough for Deep Learning Training?
Alireza Ghaffari'
Marzieh S. Tahaei'
Mohammadreza Tayaranian'
Masoud Asgharian
Vahid Partovi Nia?
Huawei Noah's 's Ark Lab, Montreal Research Center
Department of Mathematics and Statisties, McGill University
falireza. ghaffari, marzieh. tahaei, mohammadreza, [yyyaraanan)(hhaaaae com com vahid, wat.oorraaaaiaaaee......
masoud. asgharian20ccgill.
Abstract
The ever-increasing computational complexity of deep learning models makes their training and deployment difficult various cloud and edge platforms. Replacing floating-Pol point arithmetic with low-bit integer arithmetic is a promising approach to energy, memory footprint, and latency of deep learning models. As such, quantization has attracted the attention of researchers recent years. However, using integer numbers to form fully functional integer training pipeline including forward pass, back- pack-propagation, and stochastic gradient descent is not studied in detail. Our empirical and mathematical results reveal that integer arithmetic seems to be enough to train deep learning models. Unlike recent proposals, instead of quantization, we directly switch the number representation of computations. Our novel training method forms a fully integer training pipeline that does not change the trajectory of the loss and accuracy compared to floainn--piin, nor does it need any special hyper-parasd parameter tuning, distribution adjustment, or gradient clipping. Our experimental results show that our proposed method effective in a wide variety of tasks such as classification (including vision transformers), object detection, and semantic segmentation.
Introduction
Recently, deep learning models have evolved to deliver acceptable performance in many areas such as computer vision, natural language processing, and speech recognition. However, the number of parameters and computational complexity of most deep learning applications have increased significantly. This ever-increasing computational complexity makes the deployment of deep neural networks harder on edge devices. Furthermore, the energy required train a deep learning model increases proportionately with computational complexity. Thus, the training cost of these large and complex models also increases significantly the high-end cloud servers. Although, quantization techniques are commonly used to accelerate inference deep learning models, here we target a fully integer training pipeline (i.e. forward propagation, proppppppontionn and SGD). Unlike recent proposals in quantized back pack-propagation, we directly change the number representation of floating- point values. Furthermore, show empirically and theoretically that our proposed integer training methodology is effective in training deep learning models with integer-only arithmetic.
Despite the fact that accelerated training using integer back propagation seems intriguing, there are some challenges associated with integer gradient computation. For example, (i) gradients must be scaled correctly in order be adapted to the limited dynamic range of the integer number (e.g. int8: 128, 127).. (ii) The numerical error the gradient must be unbiased in order to preserve the convergence trajectory of the training algorithm. A small numerical error can be accumulated
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

through the course of training and change the convergence behaviour. (iii) The integer training must be distribution independent, i.c. the training method should not depend on the distribution of the gradients, weights, or training data.
We propose novel integer training method designed address aforementioned challenges (i), (ii) and (ii) simultaneously. Having a linear dynamic fixed-point mapping coupled with a non-linear inverse mapping the key to full integer training.
To this end, we make the following contributions:
A hardware-friendl] integer training method is proposed based on extracting the maximum floating-point exponent of the tensors as the scale. We propose linear fixed-point mapping of tensors, while the corresponding inverse mapping for floating point is non-linear. Our proposed method addresses all the previously mentioned challenges: (i) it computes the scales dynamically, and moreover the scale does not need to be adjusted in the course of training; (ii) it provides an unbiased estimation of the gradients and consequently, its conver- gence trajectory closely follows the floating- poinin version; (iii) our proposed representation mapping does not depend on the distribution of the training data, weights, or gradients.
We study the optimality gap of our proposed integer training algorithm and show it is analogous to its floating-point counterpart in the course of training (Theorem 1). Our analysis of stochastic gradient descent with our proposed fixed-point gradients shows the original Moating-poine optimality gap is only shifted by negligible amount (Remark 3).
Our proposed method effectively performs all operations required in modern neural networks using integer-only arithmetic. For instance, the computation of linear layer, convolutional layer, batch-norm and layer-norm, residual connections and also stochastic gradient descent (including gradients, momentum, weight decay, and weight update) are all performed in integer arithmetic. To the best of our knowledge, this is the first time that back- prooppropagttiin of a batch-norm, and the computation of stochastic gradient descent (SGD) is performed in integer arithmetic with negligible loss in the accuracy for large datasets such as ImageNet.
The rest of this paper structured as follows. Section 2 reviews some previous works in the field of quantized pack-propagation and quantifies the similarities and differences with our representation mapping method. Section 3 discusses our integer training methodology in detail. Theoretical aspects of our integer training method on SGD are studied in Section 4 Experimental results supporting our integer training methodology and convergence theory are presented in Section S
2 Related works
Banner et al. (1. proposed a bifurcated back pec-proopagttion method where the gradient of weights are in floating-poin point format and imput gradients in 8-bit fixed-point format. Moreover, they proposed a range-based batch-norm which is more tolerant to quantization noise. The main difference of our proposed algorithm with Banner et al. M is that our integer training method does not demand bifurcation. In other words, our gradients with respect to weights and inputs are represented and computed using int8 format. Furthermore, in our method, the back propppropagttion of batch-norm, as well as stochastic gradient descent (SGD) algorithm, are also represented and computed using integer values. Zhang et al. [2] have proposed a layer-wise does not affect the distribution of the data. In their method, the quantization error is measured and the quantization scale is adjusted over the course of training accordingly. Similarly, in Zhao et al. a distribution adaptive quantization method is proposed that takes into account the distribution of gradients in the channel dimension. In addition, they introduced gradient clipping strategy in order to normalize the magnitude of the gradients in the back- propagation. The scale of the quantization adapted to the distribution of the gradients and adjusted iteratively over the course of training. Unlike the distribution adaptive methods such as [2 B), our proposed integer training method does not depend on the distribution of training data, weights, and gradients. Thus, there is no need to adjust the scale iteratively in the course of training. In Zhu et al. [4] "direction sensitive gradient clipping method" is proposed to avoid inappropriate updates in the back-propagation. Moreover, they proposed learning rate scaling that tackles the problem of unbiased gradient error accumulation. In a similar work, Sakr and Shanbhag 5 proposed precision assignment methodology to improve the convergence of the quantized back-propagation. This precision assignment is based on some criterion
addapiiin quantization method that

---

on internal accumulator noise, quantization noise of backward and forward propagation, and gradient clipping. Unlike 4 15. our proposed method enjoys having the advantage of unbiased gradients. hence the convergence trajectory closely follows the floating-point counterpart. Thus, our method does not need any specifie gradient correction, gradient clipping strategy, and learning rate corrections as opposed to [ [ Jin et al. K proposed unified fixed-point and parameterized clipping activation to achieve high accuracy. Furthermore, they proposed a method that directly trains the fractional length (i.e. scale) of the fixed-point quantization. Additionally, they use double forward batch-norm fusion to determine the scaling factor of the batch-norm layer in the forward propagation, while the back-pree proopaaaaaatiion of batch- norm remains floating-poing In contrast, our proposed integer training method is capable of implementing integer variant of the batch-norm layer in forward propagation without double fusion opposed to K We also perform batch-norm's 's back-propagation in integer arithmetic which was ignored in the previous works. Note that back-propagation of batch-norm and layer-norm is sensitive quantization, as such, naive quantization leads to training divergence. Additionally, we perform stochastic gradient descent (SGD) computation including weight update, weight decay, and momentum in integer arithmetic with significant loss of accuracy. Wang et al. 7 explored an integer-arithmetic implementation of some preliminary CNN architectures such as AlexNet, however they experienced considerable accuracy drop for ImageNet dataset. Moreover, more complicated architectures such as ResNet and transformers are not considered. In our proposed method, we used dynamic fixed-point number format X as the primary number format for integer training. This specific fixed-poing number format (also known as block floating- format) has been used previously to quantize inference of deep learning models in Courbariaux al. [8] Drumond et al. (9) and Rouhani et al. 100
3 Methodology
Common quantization methods, which use division and clipping techniques (refer to Appendix A.6 are inefficient for propaaaaa...... Therefore, we decided to go one step deeper and change the number format directly. Here, we propose a hardware- friendll number representation mapping using the dynamic fixed-point number format where the scale is defined per tensor. In this approach, each tensor can be represented by its int8 version while it is multiplied by a shared scale, as opposed to other methods that allow multiple shared scales for different partitions of the tensor (see Rouhani et al. [10 Figure 4]). One shared scale per tensor makes the computations casier in the computing hardware (c.g. CPU or GPU). However, the training algorithm diverges if the representation mapping suggest two subtle changes; (i) we propose to perform fixed-point mapping of a tensors in a linear fashion while the inverse mapping is non-linear as explained in Sections 3.1 and 3.2 This is the key to success of algorithm since linear fixed-point mapping allows monotonic conversion of floating-poin format to fixed-point, while a non-linear inverse mapping allows preserving information. (ii) We suggest to use stochastic rounding in the back-propagation in a way that preserves the expected value of the tensors as well as their vital statistics, such the mean. Stochastic rounding in conjunction with representation mapping is crucial to keep the integer training loss trajectory close to its floating-poin counterpart.
is not exccuted properly. To tackle this problem,
3.1 Linear fixedppoint mapping
Our proposed integer training method is based on manipulating the floating point number format. This is a very simple and effective way of converting floating-point point values to fixed-point values as opposed to other commonly used methods that involve division operation. Our method essentially uses shift and round operations to convert floating-poii to fixed-point format, see Figure a
In this method, tensor comprising n floating point numbers (f, ..., fn) is converted to a fixed- point tensor. First, as shown in Figure [ sign, exponent, and mantissa of each floating- number are extracted using the unpack integer function. For instance, fi unpacked $1, f m, where (s, e, m.) is used to denote (sign, exponent, mantissa). We simply find the maximum element of e1, 62, ... Cn i.e. Cmax = max(e1, (e.(ee,e....... e,, to extract the shared scale of the tensor. Subsequently, the original mantissas are shifted to the right according to the difference of their exponent and Cmax- to get the individual mantissas. For instance, my is shifted to right by Emax - e, which is denoted by mi >> (emax - e1). By shifting mantissas to right, we intentionally push the small values to the sub-normal region, where the most significant bit of mantissa is not 1 in binary format i.e. (1)2, see Zuras et al. [T]] Pushing the mantissas to the sub-normal region is performed to align their exponents
3

---

Tensor
Layer Computation
Linear
Add
Integer Linear Layer (GEMM)
Unpack To Integer
24-bit
Shared
8-bit Scale
exponent
Shifted 24-bit Montissas
Reconstructing
Rounding Extracting
exponents
sign +
Non-linear Inverse Mapping
+
Sign
Exponent
Mantissa
Stochastic Rounding
Signed B-biit Mantissas
Floating point alignment (Using LZA unit)
+ Pack to float
Singed Mantissas
Send to Integer Layer
Float Tenaor
(a)
(b)
Figure 1: (a) Linear fixed-poine mapping, (b) Non-linear inverse mapping.
with Cmax and this unifies the scale for fixed-point values in a tensor. In the next step, to create int8 mantissas, the bit single floating- point mantissas (i.e. 23-bit mantissa implicit hidden bit) are further rounded to 7-bit mantissas to construct signed int8 values i. 7-bit unsigned integer and 1 sign bit. an example, us assume the shifted mantissa (0.0101100) 10010101011111000000000000000000 then it is going to be randomly rounded to either (0.010110); or (0.010111); based on a probability (Refer A. for details).
3.2 Non-linear inverse mapping
Our proposed fixed-point mapping method is performed by pushing mantissa values to the sub-normal region and rounding them stochastically. Hence, the inverse mapping must be performed using floating-por -point alignment module that normalizes the mantissas. A normalized floating- poing mantissa required start with the most significant bit (1)2. For instance, an alignment module converts 2127 x (0.0101)2 to 2127-2 x (1.0100)2. The alignment module is very well-known logic circuit that available in the commodity hardware using Leading Zero Anticipator (LZZ) block ([12]. Coupling linear fixed-point mapping with non-linear inverse mapping in this way keeps the number format close to floating-point Moreover, combining them with stochastic rounding, results in having unbiased fixed-point variant that forces the trajectory of the integer training closely follow the floating-pot ppont counterpart.
Figure [[b) shows the inverse mapping unit. This unit receives a tensor of mantissas and a single value shared exponent computed by an integer layer. Then tensor of exponents is reconstructed by repeating the value of the shared exponent. The size of this tensor is equal to the mantissa tensor's s size. Moreover, the mantissa tensor is rounded and its sign extracted. Note that the rounding is used here because the output of the previous layer might have some excessive mantissa bits; this normally happens in matrix multiplication and convolution operations. Then the sign, exponent, and mantissa tensors are sent to an alignment unit that shifts the mantissa and adjust the exponent in order to normalize the floating-por point number format [.]. Finally, the result is packed and carried out to the next layer.
3.3 Integer layer computations
When the fixed-point mapping of floating-poins values completed, the fixed-point values are used to perform the integer-only layer computations. For instance, let us consider a linear layer which consists of a General Matrix Multiplication (GEMM) operation, but, the idea can be generalized other types of layers. Figure 2 demonstrates the layer-wise integer computations of linear layer where the shared exponents and integer mantissas are treated separately. As shown in the Figure in order to perform an integer-only GEMM operation, we need to multiply scales (25maxi x 2F ). This multiplication performs an integer addition operation on the exponents (emaxl + Cmax2). Furthermore, the integer mantissas sent to integer GEMM module to compute the output mantissa tensor.

---

Float Tensor
Float Tensor
Unpack To Integer
Fixed-pee Mapping
Intoger
Sign
Expenent
Uuard lepoone
Layor Computation e.g. Linear Layer
Add
Integer Linear Layer (GEMM)
Mapping
Layer
Figure 2: A fully integer linear layer.
Also note that in our implementation, when the mantissa tensor is in int8 format, multiplication is in int16 format and accumulation is in int32 format.
3.4
Understanding the representation mapping
Here we provide an intuition of how our proposed representation mapping approach works. Let us denote as a tensor and Å as its fixed-point version in way that is introduced earlier. In addition, random variables A, and A, are zth element of those tensors. Also note that A, and A, are different representations of the same real number, but one is in floating-pon point format and the other is in dynamic fixed- fixed-point format. Thus,
relate A, and Å, with random eπror term ō, as A; A; 8; Since in training method we used stochastic rounding [3]. E ((A,) A,, or cquivalently ] = 0 (see Appendix II In other words, the fixed-point value is on the average equal the floating-point value. Note that here consider single precision Roating-point values as a surrogate of real values.
Linear and convolutional layers:
For these two types of layers, both forward and backward propagation computations are based on inner products. Thus, is easy to see that our proposed fixed-point inner product Čig = Ex Ast Baj is an unbiased estimator the floating-poim inner product
E E(C,,) -
E
Σ
Au Bus
E(A + (Bxr + +
Σ Bu = Cij-
(1)
Residual connections: A residual connection involves element-wise addition of two tensors. Each fixed-point element C.j = Asj + Bij is an unbiased estimator its floating-pon version
E{Cr,} E[ + Bij E[ + 65) + (Big + o Aij + Bi Cij-
(2)
Batch-norm: A batch-norm layer is defined as
Å 0
+
+
For this type of layer, it is important to compute signal statistics such as mean fi and variance 8² correctly integer arithmetic. Thus, the fixed-point mean á is also an unbiased estimator of the true mean μ
N D
A
N (A. + 5,) + n.
E (A) = E
Σ
(4)
N
N
Then the following derivation holds for fixed-point variance 6²
E((²) }- = E
N (Å,
p))
1
Σ (AA - 122 + +88
= 0² +
(5)
N
N
5

---

where o² is the true floating-poin variance of the batch and o3 is the variance of the noise introduced by the linear mapping to fixed-point. Note that the error variance og is rather small and can be integrated e in the denominator of equation
4 Theoretical analysis of SGD using the proposed method
The generic equation of weight update in the hthh iteration of SGD
#6++1 We + (gg( Ea);
(6)
where g(ws, {x) is the estimated gradients of random samples of the batch generated by the seed Ex. and ag is the learning rate. We make the following common assumption in the sequel.
Assumption 1 (Lipschitz-continutty). The loss function [(w) is continuously differentiable and its gradients ([w) satisfies the following inequality where L > is the Lipchitz constant
[(w) & ((w) + VZ(w) (w w) + 1
2 71|lc w1::
4 w. w E Rd.
(7)
Assumption 2. (i) L(ur)) is bounded. (ii) Estimated gradients g(we, Ex) is an unbiased estimator of the true gradients of the loss function
Vl(w)) Eg, (g(w,, Ex)] = [| ||E fg(w), Ex)
and (ii,a) there exist scalars M > 0 and My > 0 such that for all iterations of SGD
VeL (g(wk: [x]] < M + Myl[V[((>)]I)
Note that here we define
Ve, (g(wu [k)] = Es,, (llg(wk, ( 1x) - || (o(wk, Ex)
Also from Assumption 2. (ii) and (ii,,),, the second moment bound can be derived
Eeg (ww.llg, 11 M + Ma [V with Mc 1 My.
Effect of gradient variance on convergence: The quality of the estimated gradients g(wk: Ex) directly affects the convergence of the SGD. The effect of first and second moments of gradient are already studied on real numbers in the literature.
Lemma 1. Suppose Assumption is true, then we have
Egg (C(wx=1)) - [(wx)) < -11 2 an LMc)aa (V
a LM.
2
Proof: See Bottou et al.
14 Lemma 4.4).
Inequality @ shows the effect of gradient variance bounds, M and Mg. on each iterate of SGD, and shows the greater the variance, the more deterioration in the quality of SGD steps. The first term, -(1 - -i-aaaLLMM)aee |/ [(ww)) ([w)))] Il2 contributes to the decrease of the loss function while the second term, f L.M, prevents it. Upon choosing the correct gradient estimates, the right hand side of inequality 4 bounded by deterministic quantity, and asymptotically ensures sufficient descent of the loss L [(w). Note that the expectation Eg, is taken over random samples with seed &
4.1 Fixed-point gradient noise
When performing representation mapping in the back-propagation, the quality of the gradients deteriorate. Thus, there need to consider the effect of fixed-point mapping variance. Then, Assumption 2 (iii,a) should be modified accordingly.
Remark 1. Note that Assumption 2 (i), (ii) still hold after fixed-point mapping because of stochastic rounding, i.e. the fixed-point gradient remains an unbiased estimator of gradient (refer to Appendix A.I.
Assumption 2 bi When the gradients are in fixed-point format i.e. g(wk: Ex), there exist scalars M > 0. My 0, M 2 0 and Mt such that for all iterations of SGD
V Ee (0(w,,&& {x)) M + M + (Mv M3 V

---

If IM := M + M and My := My + Mÿ then Assumption 2 (iii takes the exact form of Assumption 2 (ii,a) i.. Vs, (g(uk, {x)) & M V However, here we separated M and M2 to emphasize the effect of fixed-point mapping on the true gradients.
Remark 2. If Assumption 2 (iii,b) holds true, inequality version
! can be transformed to its fixed- fixed-point
1 Eg, [[(wx+1)) [(wk) & -(1
2 a ja.L(M M2) ax v[(())] | + 1 2 + +
with Mc = MM-.
(10)
Inequality T0 shows the effect of added representation mapping variance with bounds, M4 and Ma- on each iterate of SGD. This observation shows that fixed-point mapping degrades the convergence of SGD unless its variance hounds relatively small, or controlled by the learning rate. As an A.aendix(A.2eee
example, refer to Appendix A.2 for analytical derivations of A and M9 for the back-propagation of linear layer involving fixed-point inner product.
4.2 Strongly convex and locally convex loss
Assumption 3 (Strong convexity). The loss function [(w) differentiable and strongly convex. We recall that strong convexity for differentiable functions is equivalent to the following inequality with some constant > 0
[(w)> > ((w) + VL(w) (w T)) 2 cll 3Il:
V w. Rd
(11)
strongly convex function has a unique minimum point at w. with the loss value La L(w.).
Theorem 1. Suppose Assumptions 1, 2()), 2(i)) (iii,), 3 are all true, then a SGD method running with fixed-point gradients i.e. (ww,, Ek) and a fixed learning rate 0 < ā & +.M)+Mgg satisfies the
following bound for its optimality gap with the minimum loss L. the thth iteration
aL(M + M1)
+ (1 - (c)(k-1) (A(w)) - L. aL(M + M*)
[([(wR) L.} &
2c
2c
t (L(M + MM)
(12)
2c
Proof. See Appendix
Remark 3. Note that when k + 0C, &LM is the original optimality gap (i.e. [[[(wx) c.} with 2c
real gradients, see Bottou al. 14 Theorem 4.6], and then, the optimality gap is also increased by &T.A5" due fixed-point representation. Here argue that by keeping the variance bound M9 relatively small via choosing the correct number format, we can theoretically achieve the original performance. On the other hand, optimality gap is related to which means smaller learning rates leads to smaller optimality gap.
Strongly convex loss is not realistic assumption in large deep learning models. However, local convexity is a more realistic assumption i.e. L is convex around the minimum point w.. Hence, inequality 12 still holds around the minimum point w, of locally convex loss. We essentially train our ImageNet ResNet18 classification in the same way: first we train the network with fixed learning rate until reaching certain optimality gap; then the learning rate is reduced to smaller fixed value in order to shrink the optimality gap.
Remark 4 (Local convexity).
Empirical evidence: Figure aa) demonstrates the locally convex loss landscape of ResNet 8 training on CIFARIO dataset using floating-pon point computation. Figure B(b shows the same loss landscape in the fixed-poin int8 format. We perturbed the weights around the in and y axes using Gaussian noise and evaluated the loss C in the axis to acquire Figures Fa and 3).. Comparing these two figures pronounces assumption of local convexity in both integer and floating -poitt tests. Figure c) shows a comparison of the loss trajectory for Seating-poin and integer training. The fixed-point gradients unbiased estimators of the true gradients, the trajectory of the integer training closely follows the trajectory of its floating-poit counterpart.
Remark 5. We implemented an integer weight update, hence, the computations of equation is also performed in integer arithmetic. shown in Appendix AeeendixA.41ee that integer weight update with stochastic rounding an unbiased estimator of the true weight update.
7

---

ResNet18 Training Loss CIFARIO
Iato Tranning
1.4
0.8
1.0
1.0
0.2
0.5
0.5
-1.
-05
0.5
0.0
-1.0
-0.5
00.5
0.0
1
50
100
150
200
0.5
Epochs
1.0
1.0
(b)
(c)
Figure 3: (a) Floating-poimt loss landscape, (b) Fixed-point int8 loss landscape c) Training loss trajectory comparison. Integer training setup: int8 linear layer, int8 convolutional layer and int8 batch-norm layer.
5 Experimental results
Image classification: Table reportss
reports the experimental results of our proposed integer training method on the conventional vision classification models. In this set of models, we used int8 linear layer, int8 convolutional layer, int8 batch-norm layer, and int16 SGD form a fully integer training pipeline. Note that all forward and back prok-propngttion computations of layers are performed in integer format. Moreover, we have chosen smaller vision models such as ResNet18 and MobileNetV2 knowing that smaller models are harder to train. Our experimental results show that our proposed integer training method as good as foating-point with negligible loss of accuracy (< 0.5% on ImageNet).
Remark 6. We compare our integer training results with official Pytorch floating-point training. For instance, in conventional vision models (Table we compare the results directly with the accuracy of torchvision models that are reported on the official Pytorch website [ We further emphasize that there is no change in typre-parameters of the training. The full set of paraaet-s of our experiments are reported in Appendix A..5
Vision transformer:
We validated the applicability of our proposed integer training on the original vision transformer model. notably 16-224 [13 We took the floating point checkpoint pre- trained on ImageNet21 K We used Huggingface 16 to fine-tune the model on CIFAR10 10. this experiment, used int8 linear layer, int8 matrix multiplication, int8 convolutional layer, and int8 layer-norm for our integer training pipeline. The result is reported in Table [
demonstrating negligible loss of accuracy (& 0.5%). Note that for this experiment, the computation of softmax in attention mechanism is in floating point.
Semantic segmentation: To validate our proposed integer training semantic segmentation task, we used DeepLab 1/V2 with ResNet- 101 as the backbone. We trained the models on PASCAL VOC-2012 [17] and MS COCO 10K [IS] For the PASCAL dataset, suggested by Chen al. [19, the models were initialized with MS COCO checkpoint and data augmentation was used during training. Likewise, for COCO dataset used PASCAL checkpoint for initialization. The batch-norm layers are frozen in our experiments as suggested in [1). We used linear and convolutional layers in int8 format. Table shows that the mean intersect of union (mIOU) of our method closely matches the floating-point baseline.
Object detection: We used Faster R-CNN 20 with ResNet50 its backbone and Single Shot Detector (SSD) 211 which relies on VGG-16 16 for its backbone. In these experiments, we used int8 linear and convolutional layers, while batch-norm layers are frozen. We integrated our integer training method with MMDetection 22) toolbox and performed our experiments on MS COCO 10K [8] PASCAL VOC-2007 23 VOC-2012[I) and Cityscapes [4] datasets. As shown in Table the mean average precision (mAP) of our proposed integer training method very close to floating-pon baseline.
RRefer to Models: http://hhhhhhth/////o//////////a html
RRefe to:
Refer to https://github.co http:/ttww....eeeeeaaee////p/paa

---

Table 1: Classification
Accuracy
int8 (top1 top5)
Pytorch baseline float (top1 top5)
Model
Dataset
Conventional Vision Models
ResNet18
CIFARI0
94.84 N/A
95.34- N/A
ResNet18
CIFAR
75.38 N/A
76.14- N/A
ResNNet18
IetgeNet
69.25
88.79
69.75 89.07
MobileNetV2
ImageNet
72.80 90.83
71.87 90.28
Vision Transformer
VTI-B (fine-tuning)
CIFARIO
98.80 N/A
99.10 N/A
Table 2: Semantic Segmentation
Table 3: Object Detection
mIOU
mAP
baseline float
baseline
Method
Dataset
int8
Method
Dataset
int8
float
DeepLabbVV
VOC
74.73
75.00
COCO
37.40
37.80
COCO
34.80
34.70
Faster R-CNN
VOC07+12
80.14
80.31
Cityscapes
39.50
40.00
DeepLab-V2
VOC
77.65
77.71
COCO
35.90
36.00
SSD
COCO
42.50
43.60
Table 4: Comparison with State of the Art
Method
Model
Dataset
Ours
3
MobileNetV2 ResNet18
ImageNet
72.8
70.5
71.9
71.2
72.6
ImageNet
69.3
70.2
69.7
71.1
DeepLab-Vi
VOC
74.7
69.9
Faster R-CNN
COCO
37.4
37.4
34.9
Table 5: Low-bit Integer Training
bit-width
Model
Dataset
int8
int7
int6
int5
int4
ResNet18
CIFARIO
94.8
94.7
94.47
88.5
Diverges
Comparison with state of the art: Table provides a comparison between our training method and state of the across different experiments. There are some important differences between our proposed method and other works: (i) our proposed integer training method uses fixed-poin batch- norm layer where both forward and back propagation computed using integer arithmetic, (ii) our proposed training method uses integer-only SGD, (iii) in our training method, no hyper-parameter changed while other methods have changed hyper- papammearsestrs or used gradient clipping.
Low-bit integer training: Table provvidss an ablation study how lowering integer bit-width can affect the training accuracy. Our experiments shows that training has significant drop of accuracy with int5 and diverges using int4 number formats.
6 Conclusion
We proposed hardware-friendly integer training method based on coupling a linear fixed- fxxed-point mapping with non-linear inverse mapping. This method performs the representation mapping directly on the floating-point number format. Furthermore, our method uses stochastic rounding and produces unbiased estimators of gradients in the back-propagation Thus, benefiting from unbiased

---

gradients and effective representation mapping, the training loss trajectory closely follows its floating- point version. Moreover, there is no need to tune hyper-parameters or perform gradient clipping to correct the inappropriate gradient and weight updates. Using our proposed technique, we designed the integer version of the most vital components of deep learning such as linear layer, convolutional layer, batch-norm, layer-norm, and stochastic gradient descent (SGD). Our experimental results show the effectiveness of the proposed method, where the accuracy loss is negligible a wide variety of the training tasks. Furthermore, in our classification tests all training components are in integer arithmetic. We theoretically studied the effect of our proposed method on SGD, and demonstrated why the optimality gap of convergence is shifted by negligible amount.
Acknowledgments and Disclosure of Funding
The authors would like to thank Richard Wu and Vanessa Courville for their constructive comments. The authors would like thank Sara Elkerdawy for initiating the object detection and semantic segmentation experiments during her internship at Huawei.
References
[1] Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. Advances in neural information processing systems, 31, 2018.
[2] Xishan Zhang, Shaoli Liu, Rui Zhang, Chang Liu, Huang. Shiyi Zhou, Jiaming Guo, Qi Guo, Zidong Du, Tian Zhi, et al. Fixed-point hack-propagation training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2330--338,, 2020.
[3] Kang Zhao, Sida Huang, Pan Pan, Yinghan Li, Yingya Zhang, Zhenyu Gu, and Yinghui Xu. Distribution adaptive int8 quantization for training cnns. In Proceedings of the Thirty- Fifth AAAI Conference on Artificial Intelligence, 2021.
[4] Feng Zhu, Ruihao Gong, Fengwei Yu, Xianglong Liu, Yanfei Wang. Zhelong Li, Xiuqi Vang, and Junjie Yan. Towards unified int8 training for convolutional neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1969-1979, 2020.
[5] Charbel Sakr and Naresh Shanbhag. Per-tensor fixed- skedtpointt quantization of the back back proppapapannionnn algorithm. arXiv preprint arXiv: 1812. 1732, 2018.
[6] Qing Jin, Jian Ren, Richard Zhuang, Sumant Hanumante, Zhengang Li, Zhiyu Chen, Yanzhi Wang. Kaiyuan Yang, and Sergey Tulyakov. F8net: Fixed-point 8-bit only multiplication for network quantization. arXiv preprint 2.XXv:2202.052399 2022.
[7] Maolin Wang, Seyedramin Rasoulinezhad, Philip HW Leong, and Hayden K-H So. Niti: Training integer neural networks using integer-only arithmetic. IEEE Transactions on Parallel and Distributed Systems, 33 (11):3249-2261, 2022.
[8] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Training deep neural networks with low precision multiplications. arxiv 2014. arXiv preprint arXiv:1412. 1412.7024 2014.
[9] Mario Drumond, Tao Lin, Martin Jaggi, and Babak Falsafi. Training dnns with hybrid block floating point. Advances in Neural Information Processing Systems, 31, 2018.
[10] Bita Rouhani, Daniel Lo. Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, et al. Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point. Advances in Neural Information Processing Systems, 33:10271-1000 2020.
[11] Dan Zuras, Mike Cowlishaw, Alex Aiken, Matthew Applegate, David Bailey, Steve Bass, Dileep Bhan- darkar, Mahesh Bhat, David Bindel, Sylvie Boldo, et al. leee standard for boating-point arithmetic. IEEE Std, 754(2008):1-00, 2008.
[12] Martin S Schmookler and Kevin J Nowka. Leading zero anticipation and detection-a comparison of methods. In Proceedings 15th IEEE Symposiun on Computer Arithmetic. 15 2001, pages 7-12. IEEE, 2001.
[13] Michael P Connolly, Nicholas J Higham, and Theo Mary. Stochastic rounding and its probabilistic backward error analysis. SIAM Journal Scientific Computing, 43(1):A566-AM) 2021.
10

---

[14] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60,2):223-31, 2018.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010. /929,, 2020.
[16] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault. Rémi Louf, Morgan Funtowicz. al. Huggingface's transformers: State-of-the art natural language processing. arXiv preprint 1rXiv2191..0377,, 2019.
[17] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://ww.pascall httpccccclllleee//////o22/000///
The
[18] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[19] Liang-Chieh Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2017.
[20] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[21] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yane Yang Fu, and Alexander Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pages 21-37. Springer, 2016.
[22] Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong. Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 1906. 071550 2019.
[23] M. Everingham, Van Gool, K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.paccak network.org/challlngeee hwtpa//lngnss//ooo/oooooooo///na
[24] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler. Rodrigo Benenson, Uwe Franke, Stefan Roth. and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3213-3223, 2016.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Theorem and Remark 3 in Section
4.2 reflecting the effect of representation mapping noise. Moreover. the accuracy deviation from baseline floating point training experiments reported Table [ Table
2 and Table
(c) Did you discuss any potential negative societal impacts of your work? [N/]] The work is purely methodological and related to number format.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
1f you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
4
(b) Did you include complete proofs all theoretical results? [Yes] See Appendix A.I Appendix
Appendix A.3) and Appendix A.4
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [No] The code is proprietary, however
we will provide them upon publication. We the experiments publicly available datasets.
11

---

(b) Did specify all the training details (e.g., data splits, hyperparameters. how they were chosen)? [Ys]] See Appendix A.S
(c) Did you report error bars (e.g.. with respect to the random seed after running experiments multiple times)? [No] It was costly train multiple times, just the experiments once. Also note the type of training experiments we reported are resilient to random initialization changes.
(d) Did you include total amount of compute and the type resources used (e.g., type GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.5
If you using existing assets (e.g., code, data, models) curating/reccasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're us-
ing/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you descrihe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [NA]
12


Reviews:
[{"review_id": "xnAnIWXVfb", "reviewer": "Reviewer_XTiL", "summary": "This paper proposes a fixed-point based model training pipeline for deep neural networks. It uses linear fixed-point mapping and non-linear inverse mapping to convert the numbers (weights, activations, gradients) during training to fixed-point and back. It conducted theoretical analysis on the noise introduced by the quantization process and the convergence speed with SGD. Experiments are done with several tasks: classification, detection and segmentation. Models include Resnet18, MobileNetV2, ViT-B, DeepLab and FasterRCNN.", "questions": "Other issues:\n-\tOn line 33: shouldn’t int8 range be [-127, 128]?\n-\tFigure 2. On the second block, the right side should be sign2, exponent 2 etc.\n-\tTable 2&3, the decimal places are not unified.\nOverall, my major concern of this work is the over-claim of \"int8\" pipeline where a lot of operations are not actually int8. And how is the used pipeline different from existing works. More clarifications need to be made. The \"int8\" bragging and boasting needs to be taken down.", "limitations": "Not sure.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n-\tThe proposed method achieves pretty impressive results. On ImageNet, the average performance drop for a int8 training pipeline is only around 0.5%. On semantic segmentation, performance drop is around  0.2% and on detection, drop at around 0.3 – 1%.\n-\tComprehensive analysis is provided on the impact of int8 training for the SGD convergence, gradient noise and loss landscapes. Although the analysis is rather simplified (e.g. only strong convexity is discussed on the loss landscape part), it still provides interesting insight to the process.\nWeaknesses:\n-\tThere are some existing work on using integer arithmetic for training deep models such as (NITI: Training Integer Neural Networks Using Integer-only Arithmetic. TPDS). There is no directly comparison with these methods. The NITI work also uses stochastic rounding which is similar to this work.\n-\tThe authors claim to be using “int8” integer operations for the computation. However, this is misleading since a lot of computation are not “int8”. For example, the multiplication of two int8 numbers/vectors will be using an int16 accumulator. The SGD part is also using int6 arithmetics (line 266). For ViT, the attention computation which contains softmax layers seems not to be using int8 operations (it is not described in the paper).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "n3huF8tJ04r", "reviewer": "Reviewer_FRGs", "summary": "The paper proposes a novel integer training method and shows that it is effective to train deep learning models with integer-only arithmetic. More specifically, the proposed method is designed to address and tackle the common challenges observed in integer gradient computation: how to guarantee that the gradients are properly scaled, the numerical error is  unbiased and the training is distribution independent.", "questions": "I have listed most of my concerns in previous section and there are several additional questions that might help improve the score.\n* Though the authors claim the pipeline is easy to implement and the method is hardware-friendly, it is not that obvious to me as I found the process could be quite complex in practice.\n* For Appendix A.2, the bound from equation (18) could become arbitrarily large as $M^q$ and $M^q_v$ are now dependent on the norm of $X$. I am not sure if the bound still makes sense in that case. Maybe there should be some additional assumptions required.", "limitations": "It seems there is no negative social impact of the work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n* The paper is well-written and the structure is clear.\n* Instead of doing quantization, the proposed method directly switches to the number representation of computations and forms a fully integer training pipeline, which is quite novel to me.\n* The paper provides a clean theoretical analysis and its motivation is clearly explained in the contribution section.\n* The empirical results look good. Compared with baseline, the integer arithmetic performs well.\n\n\nWeaknesses\n\n* Though the paper is targeted for the experts with quantization computation experience, it would be nice if the authors could elaborate the background and introduce more related work to help the readers to understand the general setup and goal of this area. For a machine learning conference lots of researchers are not that familiar with low-bit integer arithmetic. \n* The experiment setup is a little bit too simple. It seems the authors only compare their method with the baselines but not other state-of-art quantization or integer arithmetic training methods. The lack of comparison make the results much less convincible.\n* I believe part of the motivation to introduce the integer training pipeline is to reduce the memory complexity and speed up the training. However, I could not see any reports reflecting on these aspects.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WWxor9__uqT", "reviewer": "Reviewer_m2A4", "summary": "The paper provides mathematical analysis and experimental results to demonstrate an INT8 neural network that are trained with int8 arithmetic. The proposed method is hardware friendly and does not rely on training data and weights distribution.", "questions": "N/A", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The proposed method is interesting and the mathematical deduction is sound to me.\n2. The results of the compact models are comparable with the FP32 models.\n\nWeaknesses\n1. Please compare with other INT8 methods instead of only FP32 models.\n2. The title and abstract are misleading to me. There are also floating-point arithmetic in the model, so it is not integer arithmetic only.\n3. I suggest to replace cifar10 experiments with cifar100. Though, ImageNet experiments can demonstrate the method's effectiveness on large datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SbMYwRHVRU", "reviewer": "Reviewer_jC5f", "summary": "* The authors have proposed a new integer arithmetic for neural network training. What's different from the previous work is that instead of using the scale from the linear floating point to integer mapping they used the maximum floating-point exponent as the scale. The mapping between the integer and floating point values is thus dramatically different from the existing quantization approaches. \n* Previous approaches tried to keep both the forward propagation and the back propagation entirely in the int8 domain, but the training suffered from significant accuracy loss. With the new integer arithmetic proposed in this paper which also keeps the forward propagation and the back propagation entirely in the int8 domain, a few experiments have been performed on vision models demonstrating that the new approach sacrifices negligible accuracy.\n* Theoretical analysis has also been performed on the new approach.\n", "questions": "* Line 33 why the dynamic range for int8 is [-127, 126] instead of [-128, 127]? Please put some explanations somewhere in the paper.\n* When describing mantissa, please align with the IEEE standards. For example, for FP32 values, the bit-width of mantissa is 23 instead of 24. I understand that the authors are counting the leading hidden bit. But please describe it explicitly in the paper.\n* There is a lack of clearance in explaining the stochastic rounding of mantissas, i.e., how the the 24-bit single floating-point mantissas are further rounded to 7-bit mantissas. If it is possible, please provide a few concrete examples in the paper, just like the one in line 147. The stochastic rounding is a key for the proofs shown later in the paper. If $\\mathbb{E}\\\\{\\hat{A}_i\\\\} = A_i$ cannot be well justified by the stochastic rounding, most of the proofs in the paper will be invalid.\n* If shifting bits in Figure 1(a) is linear mapping, why the alignment module in Figure 1(b) which also shift bits is non-linear? The naming was a little bit confusing.\n* Please also show a table of comparing the proposed approach with the previous \"SOTA\"s in the paper. I believe this is beneficial for the paper because previous work does not always stay in the int8 domain for training.\n* Please also discuss about how even lower bit-width, such as 4-bit, will affect the training accuracy. Depending on the authors' implementation, it is possible to test it in the experiments as well.\n* In section 4.1, assumption 2 (iii, b), is it possible that $M \\geq 0$, $M^q \\leq 0$ and $M + M^q \\geq 0$? Same for $M_V$ and $M_V^q$. If it is possible, it might alter the remark 2 and 3. If it is not possible, please elaborate in the paper.\n* Line 235, with \"some\" constant $c > 0$.\n* The authors claimed that \"by keeping the variance bound $M^q$ relatively small, we can theoretically achieve the original performance\". How could we measure or ensure this for training different neural networks in practice? In line 53, the authors claimed that the optimality gap is only shifted by a negligible amount. I did not see the concrete amount and how the amount is measured.\n\n", "limitations": "* Although this work is targeted for neural network training, the inference arithmetic is slightly different from the existing popular symmetric quantization arithmetic, which makes most of the existing integer inference frameworks incompatible with the new approach. The author should discuss about the impact of the new approach to the existing inference infrastructures, since it will require new software to make it efficient.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The approach proposed is simple yet seems surprisingly effective in practice.\n* The authors were aware of the limitations of the assumptions for the proofs. The proofs are relatively speaking clean.\n\nWeaknesses\n* This is not necessarily a weakness. As I mentioned in the following discussions, there are a few places in the paper that can be further elaborated and clarified to make the paper even better.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept