Training Transformers with 4-bit Integers
Authors: Haocheng Xi
ChangHao Li
Jianfei Chen
Jun Zhu
Venue: NeurIPS 2023 poster (2023)
Keywords: neural network quantization
transformer
matrix multiplication
randomized numerical linear algebra
TL;DR: We propose an algorithm for training transformers with 4 bit integers by utilizing properties of matrix multiplications in transformers.

Abstract: Quantizing the activation, weight, and gradient to 4-bit is promising to accelerate neural network training. However, existing 4-bit training methods require custom numerical formats  which are not supported by contemporary hardware. In this work, we propose a training method for transformers with all matrix multiplications implemented with the INT4 arithmetic. 
Training with an ultra-low INT4 precision is challenging. To achieve this, we carefully analyze the specific structures of activation and gradients in transformers to propose dedicated quantizers for them. For forward propagation, we identify the challenge of outliers and propose a Hadamard quantizer to suppress the outliers. For backpropagation, we leverage the structural sparsity of gradients by proposing bit splitting and leverage score sampling techniques to quantize gradients accurately. Our algorithm achieves competitive accuracy on a wide range of tasks including natural language understanding, machine translation, and image classification. Unlike previous 4-bit training methods, our algorithm can be implemented on the current generation of GPUs. Our prototypical linear operator implementation is up to 2.2 times faster than the FP16 counterparts and speeds up the training by 17.8\% on average for sufficiently large models. Our code is available at https://github.com/xijiu9/Train\_Transformers\_with\_INT4.

Training Transformers with 4-bit Integers
Haocheng Xi Changhao Li', Jianfei Chen 1 and Jun Zhu' Dept. Comp. Sci. and Tech., Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosee Bosch Joint ML Center, Tsinghua University 2 Institute for Interdisciplinary Information Sciences, Tsinghua University {xihe20, c0 0mails. tsinghua, edu. en, d
edu, cn
Abstract
Quantizing the activation, weight, and gradient to 4-bit is promising to accelerate neural network training. However, existing 4-bit training methods require custom numerical formats which are supported by contemporary hardware. In this work, we propose a training method for transformers with all matrix multiplications implemented with the INT4 arithmetic. Training with ultra-low INT4 precision is challenging. To achieve this, we carefully analyze the specific structures of acti- vation and gradients in transformers to propose dedicated quantizers for them. For forward propagation, we identify the challenge of outliers and propose a Hadamard quantizer suppress the outliers. For backpropagation, we leverage the struc- tural sparsity of gradients by proposing bit splitting and leverage score sampling techniques quantize gradients accurately. Our algorithm achieves competitive ac- curacy wide range of tasks including natural language understanding, machine translation, and image classification. Unlike previous 4-bit training methods, our algorithm can be implemented on the current generation of GPUs. Our prototypical linear operator implementation is up to 2.2 times faster than the FP16 counterparts and speeds up the training by 1 17.8% on average for sufficiently large models. Our code is available at https::/githhbbb wttpo//oooooooioi/ociaaa Transformers,
1 Introduction
Training neural networks is computationally demanding. Training with low-precision arithmetic (a.k.a., fully quantized training or FQT) is promising to improve computational and memory efficiency. FQT methods add some quantizers and dequantizers the original full-precision computational graph, and replace expensive floating-point operations with cheap low-precision ones.
Research in FQT aims reduce the training numerical precision, without sacrificing much conver- gence speed or accuracy. The required numerical precision has been reduced from FP16 [33 to FPS [54 (6), INT32+INTS 3 and INT8+INTS 7 FP8 training implemented Nvidia's H100 GPU with Transformer Engine (35) achieving impressive speedup for the training of large-scale transformers. Recently, the training numerical precision has been pushed down to 4 bits. Sun et al. [47] successfully trained several modern networks with INT4 activation/weight and FP4 gradients; and Chmiel et al. S propose a custom 4-bit logarithmic numerical format to further improve the accuracy. However, these 4-bit training methods cannot be directly utilized for acceleration as they require custom numerical formats that are not supported on contemporary hardware.
There are significant optimization challenges to train neural networks at an extremely low 4-bit level. First, the non-differentiable quantizers in forward propagation make the loss landscape rugged, where gradient-baat based optimizers can easily stuck at local optima [3T]. Second, gradients are only computed
"This work was done during an internship in the Department of Computer Science and Technology, Tsinghua University
'Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

approximately in low- www-pecision. Such imprecise gradients slow down the training process and even cause the training to be unstable or diverge.
In this work, we propose a novel INT4 training algorithm for a class of popular neural networks, transformers 122. All the costly linear operations for training transformers can be written in matrix multiplication (MM) form. This MM form allows us to design more flexible quantizers, which better approximate FP32 matrix multiplications by utilizing specific structures of the activations, weights, and gradients in transformers. Our quantizers leverage advances in the field of randomized numerical linear algebra (RandNLA) [E
For forward propagation, we find that outliers in the activation are the main reason for accuracy degradation. To suppress outliers, we propose a Hadamard quantizer, which quantizes transformed version of the activation matrix. The transformation is a block diagonal Hadamard matrix, which spreads the information carried in outliers to its nearby entries of the matrix and thus reduces the numerical range of the outliers.
For backpropagation, we exploit the structural sparsity of activation gradients. We find that the gradients few tokens are extremely large. Meanwhile, the gradients for the rest majority of the tokens are very small, even smaller than the quantization residuals of larger gradients. Rather than computing these small gradients, it is better to save the computational resources for calculating the residuals of the larger gradients. To utilize such sparsity, we propose bit splitting, which splits the gradient each token into higher 4 bits and lower 4 bits. Then, we choose the most informative gradients by leverage score sampling, which is an importance sampling technique for RandNLA.
Combining quantization techniques for forward and backward propagation, we propose an algorithm that uses INT4 MMs for all linear operations in transformers. We evaluate our algorithm for training transformers on a wide variety of tasks, including natural language understanding, question answering. machine translation, and image classification. Our algorithm achieves competitive superior accu- racy compared with existing works on 4-bit training 47 g Moreover, our algorithm is compatible with contemporary hardware like GPUs, since it does not require custom numerical formats like FP4 or logarithm formats. Our prototypical quantization + INT4 MM operator implementation is up to 2.2 times faster than the FP16 MM baseline, and it speeds up the training by up to 35.1%.
Finally, we would like to point out that utilizing ultra-low 4-bit numerical formats for training neural networks is still an open problem, and the main purpose of this research is to study whether it is possible to design an INT4 training algorithm that can achieve reasonable accuracy on practical tasks. The current research state of INT4 training is not yet mature enough to provide significant speedup for most tasks a generic and plug-and-play manner. See Sec. 6 for further discussions.
2
Related Work
Fully Quantized Training Fully quantized training (FQT) 333 54
15
57 65 29 30
68
methods accelerate training by quantizing the activations, weights, and gradients low-precision, so linear and nonlinear operators during training can be implemented with low-precision arithmetic. Research on FQT design novel numerical formats and quantization algorithms that better approximate full-precision tensors. The current research frontier is 4-bit FQT. FQT is challenging due to the vast numerical range of the gradient and the optimization issues of training quantized networks from scratch. Due to these challenges, existing 4-bit FQT algorithms [47 g still have 1-22.5% accuracy drop on several tasks, and they cannot support contemporary hardware.
Other Efficient Training Methods Mixture-of-expert
43 improves the model capacity without
increasing the training budget. Structural dropout (1 17 exploits computationally efficient ways to regularize the model. Efficient attention (27 [0] reduces the quadratic time complexity for computing attention. Distributed training systems [2, 22 reduce training time by leveraging more computational
resources. Our work on reducing numerical precision is orthogonal with these directions.
3 Forward Propagation
Neural network training is an iterative optimization procedure with stochastic gradients computed by forward and backpropagation. We accelerate forward and back propagation with 4-bit integer

---

(INT4) arithmetic. We first describe the forward propagation of our training procedure. The forward propagation can be formulated as a composition of linear and non-linear (GeLU, normalization. softmax, etc.) operators. In our training procedure, we accelerate all the linear operators with INT4 arithmetic and leave all the eess-comuuaationnally-iien intteesiiveee non-linear operators in the 16-bit floating- point (FP16) format. All linear operations in transformers can be written in matrix multiplication (MM) form. For case of presentation, we consider the acceleration of the following simple matrix multiplication throughout this paper:
Z = XW where Z E R-NxC ,X E IN and W E CCDD
The most predominant use case of such MM is the fully-connected layer. Consider transformer with an input shape of (batch size S, sequence length T, dimensionality D). The fully-connected layer can be written as Eq. ! where X is the activation for N = ST tokens, and W is the weight matrix. For attention layers, batch matrix multiplications (BMMs) might be required. Our proposed techniques can be applied BMMs, and we leave the discussion of BMMs in Appendix. A.1
3.1 Learned Step Size Quantization
To accelerate training, the forward propagation must be computed with integer arithmetic. We leverage the learned step size quantizer (LSQ) [16] for this purpose. LSQ is a static quantization method whose quantization scale does not depend on the input, and is thus cheaper than dynamic quantization methods [23] which need to compute the quantization scale dynamically per iteration.
Given FP matrix X, LSQ quantizes X to integer with
inl,x X
[clamp(X/ss QN QP)1,
where x is learnable scalar parameter, clamp restricts its input to the range [-QN Qp). L-1 is rounding operation, and X/sx is computed elementwise. The resultant matrix takes values from {-QN, -Q + 1..... Qr}. Since we aim to perform INT4 MMs, we set QN Qp 7. The
integer matrix can be dequantized back to FP through float (intsx (X)) = sxintsx (X) x
With LSQ, Eq. can be computed approximately as Y = XW R 8xswinnt,x (X) intsw (W)T, where the INT4 MM intax (X) intew (W) can be implemented efficiently on hardware.
Remark: Quantization- training (QAT) 9 63 67
23 12
49 64
18
55
an inference acceleration technique which trains networks with quantizers inserted the forward propagation graph, so the trained network can perform efficiently during inference. QAT can compress tempting think that directly applying a quantizer for QAT to FQT can lead to similar low activation/weiggh bit- bit-width. However, even only quantizing the forward propagation for FQT is much more challenging than QAT because: (1) QAT requires converged full-precision model as initialization [16] and/or as teacher model for knowledge distillation 2 (2) QAT can adopt expensive multi-stage training pipelines without worrying about the convergence speed (32), while FQT algorithm must converge as fast as full- precision training algorithms to be useful; (3) QAT may approximate the discrete quantizer with continuous functions during training
activation/weiigts to extremely low precision (e.g. 1-2 bits).
[19] which cannot be implemented with integer arithmetic. Due
to these challenges, it is still an open problem to do FQT with 4-bit activations/weights.
3.2 Activation Outliers
Simply applying LSQ for FQT with 4-bit activation/weights leads accuracy degradation due to activation outliers [58]. As shown in Fig. [(a) activations have some outlier entries, which are much larger in magnitude than other entries. In this case, the step size x poses trade-off between quantization granularity and representable numerical range. If sx is large, we can represent the outliers well at the expense of representing most other entries in a very coarse manner. On the other hand, if sx is small, we have to truncate the entries outside the range [-QN 2P.x]] Unfortunately, the transformers tend to store information these outliers, and such truncation would seriously harm accuracy (see Sec. 5.2 for details). The outlier problem is particularly significant when the training task is to fine-tune a pre-trained model on some new downstream tasks, since the pre-train model contains more outliers (58) than random initialization.
3

---

duralty 10
100
0.0008
gadient nom
peroerlage
80
0,0066
0.0004
80.0002
18
0.0000
500
1000
1500
500 1000 1500
tokens
(5)
(8)
(0)
Figure 1: Histogram of activation of the linear-1-2 layer in a BERR-base-uncased model. (a) Original activation distribution; (b) tranmaaorrdra-sfommeddd activation distribution.
Figure 2: (a) The distribution of gradient norm along the token dimension. (b) The cumulative sum of the top X values percentage of the sum of all norms along the token dimension.
There exists some works handle activation outliers for post-training quantization (PTQ). Outlier Suppression [56] discover that LayerNorms amplify outliers, and propose Gamma Migration and Wisen Clipping solve this issue and achieves 6-bit BERT PTQ without too much degradation. SmoothQuant [58) migrates the quantization difficulty of activation outliers to weights and achieves 8-bit PTQ for large language models, such as OPT-175B Outlier Channel Splitting [66] duplicates channels containing outliers with small overhead on the size of the network. However, these methods mainly focus PTQ or QAT, and seldom successfully deal with ultra-low 4-bit training.
3.3 Hadamard Quantization
We propose a Hadamard quantizer (HQ) to solve the outlier problem. Its main idea is to quantize the matrices in another linear space which has fewer outliers.
The outliers activation matrices form a feature-wist wise structure [88. They are typically concentrated on few dimensions, i.e., only a few columns of X are significantly larger than others. Hadamard transform [48] is linear transformation, which can amortize the outliers into other entries. Specifi- cally, the Hadamard transform Hş is 2 x 2 matrix, where
Ho [1], H,
1/ H H H HH-I].
Hadamard matrices are orthogonal and symmetric: H. H - so H&H: = I,
Consider any coordinate row vector e E IB2 we have e, He 2 2/2 Vi, where 1
(#1, #1) is a 2 oi-dimensional vector with all of its values being 1 or 11. This demonstrates the extreme case when a single outlier dominates all the rest dimensions. transformation effectively turns the vector into a coantization-friennde all-one-vector. The practical effect of the Hadamard transform on suppressing activation outliers is demonstrated Fig. I(b)
this case, Hadamard
HQ uses block-diagonal transformation matrix H E R DDDD; H BlockDiag(H,,... H&), where D is a multiple of 2 To suppress outliers, we quantize transformed version of and W:
X = HXH) x sxint,x (XH) HJ, W (WW)H ss swintew (WH) H
Combining the quantized matrices, we get
Y T exswint,x (XH) Hintow (HWW)) = =xxsinttxx (XH) (HTWT).
where the inverse transformations cancel with each other, and the MM can be implemented as:
Procedure HQ-MM
1. Compute XH and H T in FP16.
Quantize the resultant matrices INT4 by LSQ.
3. Multiply the two INT4 matrices.
Dequantize the resultant INT32 matrix to FP16 by multiplying 8xaw.
For time complexity, Step takes ND + C)) FP16 multiply-accumultes (MACs); Step
"A vector which i-th dimension I, and all other dimensions are

---

and Step 4 takes O + C)) FPI6 MACs in total; and Step 3 takes O(NDC) INT4 MACs. Comparing with the plain LSQ Eq. 2 the amount FP16 MACs increases by 2 times, from O(N + C)) to (2 ND + C)). However, our HQ-MM is still much cheaper than an FP16 MM given 2 < D and 2 < C. The number & shows tradeoff between the ability to suppress outliers and computation complexity. Larger &o allows for amortizing the outlier within a larger horizon, at the cost of being more expensive. We propose depending on the outlier scale, as discussed in Appendix A.5 The typical value is k = 5, while the dimensionality C and D ranges from 768 to 4096.
adaptive algorithm to choose & for each activation
4
Backpropagation
We now consider accelerating the backpropagation of the linear layer with INT4 operations. The linear operator HQ-MM defined in Eq. (3 has four inputs: activation X, weight W, and step sizes 8x. sw. Given the output gradient Vy w.r.t. some loss function L, we need to compute the gradient of all four inputs. We discuss the computation of activation/weight gradients in this section, and left the discussion of step size gradients to Appendix denote the gradient in the following text.
A.3 For simplicity, we omit L and simply use Vy to
By the straight through estimator
|-]|
=
[5] and the chain rule, we have
Vw = =S (V olw H
Vx= o Vx VY
(4)
where define X int,y (XH). W intsw (WH), Ix = I(-QN S X/sx I Qp), and lw = I[-QN S W/sw S Qp). For computing the gradients, three types of matrix multiplications
required:
The element-wise multiplication o of a 0/1 matrix Ix (or liw) with another INT4 (or INT32) matrix. This operation has low time complexity.
2. The multiplication of an INT32 matrix with FP16 block-wise Hadamard matrix sw HI which also has low-time complexity, as discussed Sec. Sec.3.3
The multiplication of the FP16 gradient Vy with an INT4 matrix X or W, which we will accelerate by quantizing Vy INT4.
In the rest of this section, we will discuss quantization methods to compute the "type 3" MMs VTX and Vy W. We quantize Vy dynamically for each MM, while X and W have been already We start by discussing the structure of the gradient.
calculated in forward propagation in Section. g
4.1 Structural Sparsity of Gradients
We note that the gradient matrix Vy tends to be very sparse along the training process. Furthermore, the sparsity has a structure: few rows (i.e., tokens) of Vy have large entries, while most other rows are close to an all-zero vector. We illustrate this by plotting the histogram of per-row norm [I k..ll for all the rows in Fig.
Such a structural sparsity arises from the heavy overparameterization 62 modern neural networks. During almost the entire training process, the network operates in the overparameterried scheme [34] where it can fit most training data well, except for a few hard examples. Therefore, the (activation) gradient will be close structural sparsity quickly emerges after only few training epochs. For fine-tuning tasks, the gradient is always sparse during the whole training process.
for well-fitted data points. We find that for pretraining tasks, such
4.2 Bit Splitting and Leverage Score Sampling
Here, we discuss how to design gradient quantizers to accurately compute the MMs during backprop- agation by leveraging structural sparsity. The high-level idea that many rows of the gradient are so small that they have little impact on the parameter gradient, yet they waste abundant computation. On the other hand, the large rows cannot be accurately represented with INT4. We drop some small rows and use the saved computation to represent large rows more accurately.
5

---

First, we propose bit splitting (BS), which splits full- precision matrix as higher and lower 4 bits:
(5) floating-point scalars, and V Vy INT4 matrices representing the higher and lower 4 bits, respectively. BS can be implemented by first quantizing Vy to INT4 as Vy R st V and then quantize the residual to INT4 as Vy - -s- V INT8 representation of matrix, where V and V are the higher and lower 4 bits of the INT8 representation. Next, we discuss how to compute the weight and activation gradient.
= =
where 84
sy Vty. BS can be viewed as an
Weight Gradient As discussed earlier, weight gradient involves the matrix multiplication V X, where Vy E R NC and X is an N x D INT4 matrix. By Eq.
)
X
6
where define Vt - V 8 x T R2NXC and X
X X be 2N D INT4 matrix. Eq. (6) represents the product of an INT8 V and an INT4 W, and can be implemented by two
INT4 MMs V X and VY X. Such MM is rather accurate since Vy is represented with 8 bits.
However, comparing to a naive quantization of VY to INT4, BS doubles the amount of INT4 operations for MM. We propose leverage score sampling (LSSS) to cut the operations of Eq. (5) by half, to the same amount as the naive MM 84 V X. Noticing that the MM Eq. (6) can written as the sum of 2NV rank-1 I matrices:
X
Σ Vw.
where Vw. V
YY Due to the sparsity of Vy, the matrices Vw, differ in magnitude and small matrices can be discarded without having a big influence on the result.
Our proposed LSS assigns each Vw, probability ps [0, 1, I.... 2N, that satisfies E2N Ps = N. We define random masks m, ~ Bern(p;) and mask matrix M, and approximate it as
2N
M
where M diag
m2
M2N
which is an unbiased approximation since E
Mx
x
X
In expectation, there are only N nonzero m4s. Therefore, LSS reduces the cost of MM by half. For L.S to be accurate, we minimize its variance. We have: Proposition 4.1. (LSS variance for weight gradient)
2N
2N
Var
m,
1
Pr
where Var X = E
[|
X
EX|
EX
Pi
P
The coefficient G : |IV |||
ll is called the leverage score, which can be easily computed in low time complexity. When P: x Gi the variance attends its minimum due to Cauchy inequality:
2N Σ
2N
2N
2N
2N
c
P
where the equality holds when p; c, Intuitively, LSS approximate the MM Eq. 77) well with significantly lower computational cost when the leverage scores {c,) } are diverse, which is indeed the case as shown in Fig.
Define M to be the tep-left N x N submatrix of M and M to be the bottom-right one, we have
M = = V M + v M'X
Y

---

Table 1: Results on language model fine-tume tinettuning, transformer pretraining. and vision transformers fine-tuning and pretraining. Standard deviation is reported as subscript. FT refers to Fine-tuning, and PT refers to Pre-training. For WMT the result of 25.4 is result of Ultra-Low, not INT8.
BASISLINES
4-BIT TRAINING
DATASET
TRAIN TYPE
MODEL
METRIC NAME
FP
INTS
LSQ+LUQ
HQ+L5S
GLUE-DEV
FT
BERT-BASE
AVG
82.67o
81.450.1
75.236.t
80.811.3:
BERT-LARGE
AVG
84.57042
82.7
45.93
62.25u.as
SQUAD VI
FT
BERT-BASE
FI
38.326.30
S8.42a.n
85.70c.
87.60p.25
SQUAD V2
FT
BERT-BASE
FI
o6.04
75.630.0
71.02.
74.63p.18
ADVERSARIAL QA
FT
BERT-BASE
FI
40.00
40.17
31. 31.85g.30
38.70p.77
SWAG
FT
BERT-BASE
ACC
19
79.18039
70.79.
77.499.1s
CONLL
FT
BERT-BASE
AcC
93.386
93.186 84
87.63
91.90u.cs
BLEU
27.5
25. 25.40ULTAA Low)
27.17
WMT
PT
TRANSPORMER-BA
BACREBLEEU
26.5
25.57
CIFAR10 1D
FT
VYT-L/32
VII-B/32
Topl Acc
98. 88.77p
98.98
98.59u
98.76
97.76
98.38
98.36g.0s
98.47
CIPAR100
FT
VIT-L/32
93.07
13
90.96
92.2
90.97
63.63
$9.788u.u
91.13
VIT-B/32
81.88
80.42
77.25
79.18
IMAGENeeTX
FT
VIT-L/32
VIT-L/16
Topl
84.55
83.05
82.4
82.61
PT
DEIT-SMLLL
Topl Ace
73.1
70.95
69.96
69.18
which can be implemented by two INT4 MMs with sampled rows/columns. Putting every- thing together, we propose the following MM procedure to compute the weight gradient:
Procedure LSS-MM
Quantize Vy with BS to obtain Vir and V in INT4.
2. Compute the leverage score || | X in FP16.
Sample the masks {m;}.
Sample rows of Vy and X given the masks {m;).
Compute INT4 MMs vt M X and Vty M'X,
6. Dequantize and sum up the resultant INT32 matrices to obtain the FP16 result Vy Mx.
As M only has N non-zero elements in expectation, the two matrix multiplications in Step 5 take about 2NCD INT4 MACs, which aligns with the cost of the naive MM 8 V X. The overhead of all the other steps is O(NC + ND) in total.
Activation Gradient
Similar to the previous discussion, the gradient of input can be written
Vy W (st V
ts V W 8
V W + sy V W (i'v)) W
(8)
where we define V ist V ts V 22NxC and 1 [I I] to be N 2N INT4 matrix, I is a N x identity matrix. The original product can also be implemented by two INT4 MMs V W
and Vi W. But different from weight gradients, we now focus on f'v in Eq. (8 and do leverage score sampling on this MM. A detailed discussion can be found in Appendix [.2] and we only present the leverage score here. Similarly, we write the MM as the sum of 2N smaller multiplications:
2NV
2N
- N
I'vi
m;
EV VY..
Pi
2=1
where we define Vv: I + and associate the probability pt and Bernoulli mask my ~ Ber((p)
with the i multiplication. The leverage score for activation gradient is c, := V ell and the variance attains minimum when p, c, More details about the algorithm can be found at Appendix. A.3 On the implementation side, once the mask {m:) is known, we can decompose the MM Eq. 8 as two
INT4 MMs: (1'MV&) W = 84 M v W + M V W.
5 Experiments
We evaluate our INT4 training algorithm on a wide variety of tasks including language model fine-tuning, machine translation, and image classification. We implement our proposed HQ-MM and
7

---

LSS-MM algorithms with CUDA and cutlas and the implementation details can be found in Ap- pendix A
We replace all the floating- point linear operators with our INT4 implementation except simply using LSQ for embedding layers, and leaving the last classifier layer in full precision. We adopt default architectures, optimizers, schedulers, and hyper-parameters for all the evaluated models.
5.1 Converged Model Accuracy
We compare the accuracy of the converged model on various tasks in Table The compared methods include full-precision training (FP), INT8 training (I FP4 training 47 ("Ultra-low"), 4-bit logarithm quantization [8] with LSQ for activations and weights (LSQ+LUQ), and our algorithm which utilizes HQ for forward and LSS for backpropagation (HQ+LSS). Ultra-lww does not have public implementation, so only report its performance from its original paper on the machine translation task. Except for the large machine translation task and the task of large vision transformers, repeat each run by three times and report the standard deviation subscripts in tables. We do not include any kind of knowledge distillation or data augmentation.
Language model fine-tuning: We use the pretrained BERTT-ase--nncsee and BaRg- uncased [24] model, and evaluate the performance of our method on GLUE dev-set [53]. SQUAD [41), SQUADv2 [40), Adversarial QA 4 CoNLL-2003 42 and SWAG [61] datasets. We present the average result of the bert-base-uncasse dataset. The full results are listed in Appendix C.2 Compared with LSQ+LUQ. our method achieves 5.5% improvement of accuracy on average for the bert-base model and achieves > 25% improve- ment of accuracy on average for the bert-large model. We further show the result on the SQUAD, SQUAD 2.0, Adversarial QA, CoNLL-2003, and SWAG datasets. On all of the tasks, compared with LSQ+LUQ, our method achieves better performance. We improve by 1.8% and 3.6% on SQUAD and SQUAD 2.0 compared to LSQ+LUQ, respectively. On the more difficult Adversarial QA, we improve by 6.8% on F1 score. On SWAG we improve by 7% and on 20NL we improve by 4.2% accuracy.
and bert-large-uncased models on the GLUE
Machine translation: We also apply our method for pretraining. We train a Transformer- base [22 model on WMT 14 En-De dataset [6] for machine translation. Note that we reproduce this experiment with Fairseq's ' recipe i which reports the SacreBleu score (26.5 for FP) (37, while Ultra-low and LUQ report the more optimistic original BLEU score (27.5 for FP) 36). Our HQ+LSS has about 1.0% BLEU degradation, which is smaller than 2.1% of Ultra-low and higher than 0.3% reported in the LUQ paper. Nevertheless, HQ+LSS still performs comparably with existing methods for this pretraining task, and it supports contemporary hardware.
Image Classification: We load ViT checkpoints pretrained on ImageNet211 Ik [13, and fine-tune it on CIFAR-10, C10 (28). and ImageNet lk. We use ViT-B/32 and ViT-L/32 for CIFAR datasets and use ViT-B/32, V/T-L/32 and V for ImageNetlk. On CIFARIO we achieve < 0.5% accuracy degradation. while LSQ+LUQ has 1% degradation for VIT-B/32 and 0.6% degradation for ViT-L/32. On CIFAR100, INT8 already has ~ 1% accuracy degradation, which shows its difficulty. We improve by 1.1% accuracy for ViT-B/32 and 0.2% accuracy for ViT-L/32 compared with LSQ+LUQ On ImageNetlk, we improve by 2% accuracy for ViT-B/32, 2.6% accuracy for ViT-L/32 and 0.2% for VIT-L/32 compared with LSQ+LUQ We further test the effectiveness of our algorithm for pretraining a DeiT-Small model (51) on ImageNerI where HQ+LSS can still converge to similar accuracy level compared to LSQ+LUQ, while being more hardware friendly.
5.2 Ablation Study
Here, we conduct ablation studies to show the effectiveness of our forward and backward methods independently on the challenging CoLA dataset. To study the effectiveness of different quantizers for forward propagation, we leave backpropagation in FP16. The result is shown in Fig. 33a) We first validate the claim in Sec. 3.2 that outliers are the main cause of accuracy degradation in quantized forward propagation. We test an "outlier" method which maintains 1% largest activation entries in FP. The "outlier" method achieves good performance, which proves that outliers indeed the most
8

---

nestod
mwettdd UUG
LGG
Figure 4: Compar- ison of basic FP16 MM, HQ, and LSS
Figure 3: CoLAA performance under different methods using different bits. (a) Comparison of forward methods. (b) Comparison of backward methods.
Figure 5: SpeedUp of our INT4 train- ing algorithm compared with FP16 PyTorch AMP on (a) Bert-Large (b) Gpt2-base.
operators.
significant challenge of the transformer's ' forward quantization. The hardware- unffiidddy "outlier" method serves upper bound of methods to handle outliers. Our HQ outperforms LSQ by better handling the outliers and achieves comparable results to maintaining the outliers.
We also investigated whether more granular quantizers, such as per-token quantization per-channel quantization could used to quantify outliers, or whether existing methods like SmoothQuant (58) could be used for INT4 FQT. The results are listed in Appendix C.3] and we find that without HQ, none of these methods achieve good accuracy under 4-bit quantization, and the result HQ is not strongly affected when more granular quantization methods are applied.
For backpropagation, we compare simple minimax quantizer 3 LUQ g and our LSS, and leave forward propagation in FP16. The minimax quantizer divides the numerical range from the minimum
the maximum into equally large quantization bins. The result is shown Fig. [(b] While the bit-width is higher than 2. our LSS achieves results that are comparable and even slightly higher than L.Q Meanwhile, LSS is more hardware friendly requires only INT4 arithmetic.
5.3 Computational and Memory Efficiency
Finally, demonstrate the potential of method accelerate neural network training by evaluating our prototypical implementation discussed in Appendix [A.6] We emphasize that our implementation not fully optimized. For example, the backward computation requires an INT4 MM in the form of
Y AB, while cutlass only supports Y AB T_ so explicit transpose is required. We also do not fuse the linear operators with nonlinearities and normalizations. Therefore, the results cannot fully reflect the potential of INT4 training algorithms. fully optimized implementation requires heavy engineering, which exceeds the scope of our paper.
Operator Speed: We compare the throughput of our proposed HQ-MM (HQ), LSS for computing weight gradient (LSSWeight), LSS for computing activation gradient (LSSAct), and their average throughput (INT4) with baseline tensor-core FP16 GEMM implementation (FP16) provided by cutlass in Fig. on an Nvidia RTX 3090 GPU which has peak throughput at 142 FP16 TFLOPs and 568 INT4 TFLOPs. As the matrix size grows, the overhead of quantization diminishes and our INT4 operators can be up to 2.2 times faster compared with FP16 MM. We further analyze the quantization overhead for each operator in Appendix C.5
Training Throughput: We compare the training throughput of the FP16 PyTorch AMP and our INT4 training algorithm for training BERT 24 and GPT -sylee language models on system of 8 Nvidia A100 GPUs. We vary the hidden layer size, intermediate fully-connected layer size, and batch size, and plot the speedup of INT4 training in Fig. 5 Our INT4 training algorithm can achieve up to 35.1% 1%% speedup and an average of 15.8 % for BERT-style models and up to 26.5% speedup and an average 19.2 % for GPT-style models. The training time can be found Appendix C.4
We compare the inference speed of our algorithm with I-BERT 25 by compar- ing the speedup numbers reported in its original paper. Following the I-BERT paper, we compare the speedup of integer-based inference algorithms relative to FP32 baseline on an Nvidia T4 GPU on the BERT-base and BERT-large models, and test with sequence lengths of 128 and 256. While I-BERT only reported speedup numbers for batch sizes 1, 2, 4, 8, we test the speedup for batch sizes
Inference Speed:

---

Table The inference speed up compared with I-BERT. Since the open- sourced versions of I-BERT are based on fake quantization, we compare the inference speed of our algorithm with I-BERT by comparing the speedup numbers reported in its original paper with respect to FP32.
BATCH SIZF
MODEL
МETHОО
16
32
64
128
256
512
1024
I-BERT
2.42
3.36
3.39
3.31
BERT-BASE
OURS
0.17
0.25
0.46
0.87
1.22
2.07
3.61
3,57
4.5
4.54
4.92
I-BERT
3.2
4
3.98
3.81
BERT-LARGE
OURS
0.22
0.43
0.81
1.46
2.13
3.34
4.28
4.81
5.4
6.08
6.48
IERRT
3.11
2.96
2.94
3.15
OURS
0.25
0.54
0.95
1.53
1.94
3.2
3.76
4.15
4.15
4.14
256
BERT
3.19
3.51
3.37
3.4
BERT-LARGE
OURS
0.45
0.86
1.47
2.44
2.67
3.99
4.87
5.24
5.11
5.41
0OM
ranging from I to 1024 to better reflect the performance of throughput-orientee scenarios (such as cloud language model service provider).
In Table 2 we report the speed up result. While I-BERT's speedup numbers seems to be insensitive to the batch size and sequence length, our speedup increases with the batch size. I-BERT shows up to 3.98x speedup for smaller batch size, while our algorithm can achieve higher speedup for batch sizes higher than 64, and eventually gives speedup of 6.48x for sequence length of 128 and batch size of 1024.
For the BERT-base model, method shows inference speed improvement of 3.57-4.92 times compared to FP32 for batch sizes larger than 64. In comparison, I-BERT achieved an inference speed improvement of 2.42-3.39 times compared FP32 when the batch size small. For BERT-large, our method shows an inference speed improvement of 4.81-6.48 times compared to FP32 for batch sizes larger than 64. In comparison, I-BERT achieved inference speed improvement of 3. 3.20-4.00 times compared to FP32 when the batch size is small. Therefore, our algorithm can potentially achieve higher throughput than I-BERT.
6 Conclusions
We propose a hardware-friendly INT4 training method for transformers. By analyzing the properties of MMs transformers, we propose HQ and LSS methods to quantize activations and gradients while preserving accuracy. On several important tasks, our method performs comparably or better than existing INT4 methods. Our work can be potentially extended beyond transformers to other MM-only architectures, such as MiX-Mixer [50), graph neural networks (26). and recurrent neural We leave as future direction.
networks 20).
Broader Impacts: Our algorithm can improve efficiency and reduce the energy consumption of training neural networks, which helps reduce the carbon footprint caused by deep learning. However, our efficient training algorithm might also facilitate the development of large language models with safety concerns for human beings; and malicious Al applications such as fake content generation.
Limitations: Our INT4 training algorithm does not support convolutional layers. Moreover, our algorithm cannot yet work well for those extremely large models such OPT-175B. LSS utilizes gradient sparsity, and may not work well for certainty pretraining tasks. Finally, our algorithm may incur more memory accesses than FP16 training algorithms, which might affect the speedup.
Acknowledgements
The authors would like thank Weilin Zhao, Sixuan Ma, Xiaoxuan Liu, Ziteng Wang, Bingrui Li, Cheng Lu, and Zhiyuan Liu for valuable discussions and help on the training of large-scale language models. This work was supported by the National Key Research and Development Program of China (No. 2021ZD01 10502), NSFC Projects (Nos. 62061 136001, 62106123, 62076147, U19A2081, 61972224, 621 21106100, Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. J.Z is also supported by the XPlorer Prize.
10

---

References
[1] Menachem Adelman and Mark Silberstein. Faster neural network training with approximate tensor operations. arXiv preprint arXiv: 2018.
[2] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.1570. 2020.
[3] Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In Advances in Neural Information Processing Systems, pages 5145-5153, 2018.
[4] Max Bartolo, Alastair Roberts. Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. Beat the ai: Investigating adversarial human annotation for reading comprehension. Transactions of the Association.for for Computational Linguistics, 8:662-674 -666 2020.
[5] Yoshua Bengio, Nicholas Léonard. and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1308.3332, 2013.
[6] Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation, pages 12-58, 2014.
[7] Jianfei Chen, Yu Gai, Zhewei Yao, Michael W Mahoney, and Joseph E Gonzalez. A statistical framework for low-bitwidth training of deep neural networks. In Advances in neural information processing systems, 2020.
[8] Brian Chmiel, Ron Banner, Elad Hoffer, Hilla Ben Yaacov, and Daniel Soudry. Logarithmic un- biased quantization: Practical 4-bit training deep learning. arXiv preprint arXiv:2112.10769, 2021.
[9] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint
06085, 2018.
[10] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations, 2020.
[11] Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. arXiv preprini arXiv:
2019.
[12] Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. ICCV, 2019.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins 1xXi:20000011929 2020.
[14] Petros Drineas and Michael W Mahoney. Randnla: randomized numerical linear algebra. Communications of the ACM, (9))80000, 2016.
[15] Mario Drumond, LIN Tao, Martin Jaggi, and Babak Falsafi. Training dans with hybrid block floating point. In Advances in Neural Information Processing Systems, pages 453 463,,463,445 2018.
[16] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S Modha. Learned step size quantization. arXiv preprint 0rXiv:1902.08153 2019.
[17] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In International Conference Learning Representations, 2019.
11

---

[18] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware mini- mization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
[19] Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. Proceedings of the IEEE/CVF Interaational Conference on Computer Vision, pages 4852-4861, 2019.
[20] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation. 9(3):1735-1780, 1997.
[21] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinherger. Deep networks with stochastic depth. In European conference on computer vision, pages 646-661. Springer, 2016.
[22] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, Hy- oukJoong Lee, Jiquan Ngiam, Quoc Le. Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
[23] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmeticonnl inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2704-2713, 2018
[24] Jacob Devlin Ming-Wet Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186, 2019.
[25] Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-bert: Integer-only bert quantization. In International conference machine learning, pages 5506- 5518. PMLR, 2021.
[26] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint 1rXiv:1609.0223 2016.
[27] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2019.
[28] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009.
[29] Hamed F Langroudi. Zachariah Carmichael, and Dhireesha Kudithipudi. Deep learning training on the edge with low-precision posits. arXiv preprint arXiv: arXiv:1907. 13216, 2019.
[30] Hamed F Langroudi, Zachariah Carmichael, David Pastuch, and Dhireesha Kudithipudi. Chee- tah: Mixed low-precision hardware & software co-design framework for dnns on the edge. arXiv preprint 00t.::/9900022333 2019.
[31] Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, and Swang-Ting Cheng. How do adam and training strategies help bnns optimization. In International Conference on Machine Learning, pages 6936-6946 PMLR, 2021.
[32] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Swang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 143-159. Springer, 2020.
[33] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, al. Mixed precision training. In International Conference on Learning Representations, 2018.
[34] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Iyya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):12000) 2021.
12

---

[35] Nvidia. Transformer Engine. Online; accessed 23 January 2023.
2023.
[36] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[37] Matt Post. A call for clarity in reporting bleu scores. arXiv preprint 1rXiv:1804.08777 2018.
[38] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 18):9, 2019.
[39] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEE, 2020.
[40] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. arXiv preprint
2018.
[41] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: 05250, 2016.
[42] Erik F Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language- independent named entity recognition. arXiv preprint cs/0306050, 2003.
[43] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The aparsely-gater mixture-of-expprts layer. arXiv preprint arXiv: 100100670 2017.
[44] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. arXiv preprint arXiv: 05840, 2019.
[45] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8815-8821, 2020.
[46] Xiao Sun, Jungwook Choi, Chia- Yu Chen, Naigang Wang. Swagath Venkataramani, Vijayalak- shmi Viji Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In Advances in Neural Information Processing Systems, pages 4901-4918 10, 2019.
[47] Xiao Sun, Naigang Wang, Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi Viji Srinivasan, and Kailash Gopalakr- ishnan. Ultra-low precision 4-bit training of deep neural networks. In Advances in Neural Information Processing Systems, volume 33, 2020.
[48] James Joseph Sylvester. Lx. thoughts inverse orthogonal matrices, simultaneous signsuc- cessions, and tessellated pavements in two or more colours, with applications to newton's s rule, oraamenta tile-work and the theory of numbers. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 34(232):461-475, 1867.
[49] Hanlin Tang, Xipeng Zhang, Kai Liu, Jianchen Zhu, and Zhanhui Kang, Mkq-bert: Quantized bert with 4-bits weights and activations. arXiv preprint arXiv:2203. 1020001.1488 13483, 2022.
[50] Ilya Tolstikhin, Neil Houlsby. Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung. Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp- mixer: An all-mlp architecture for vision. Advances neural information processing systems, 14-24261-24272. 2021.
[51] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference machine learning, pages 10347-10357. PMLR, 2021.
13

---

[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[53] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1804.07700 2018.
[54] Naigang Wang, Jungwook Choi, Daniel Brand, Chia- Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. In Advances in Neural Information Processing Systems, pages 7675-7684, 2018.
[55] Zheng Wang, Juncheng B Li, Shuhui Qu, Florian Metze, and Emma Strubell. Squat: Sharpness- and quantization-aave -wwae training for bert. arXiv preprint arXiv:2110.07171, 2022.
[56] Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang. Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. arXiv preprint arXiv:2209.13325 2022.
[57] Shuang Wu, Guoqi Li. Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. In International Conference on Learning Representations. 2018.
[58] Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211. 10438, 2022.
[59] Yukuan Yang. Lei Deng, Shuang Wu, Tianyi Yan, Yuan Xie, and Guoqi Li. Training high- performance and large-scale deep neural networks with full 8-bit integers. Neural Networks, 125:70-82, 2020.
[60] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-Neurllie Edition (EMC2-NIPS), pages 36-39. IEEE, 2019.
[61] Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv: 05326, 2018
[62] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
[63] Dongqing Zhang, Jiaolong Yang. Dongqiangzi Ye, and Gang Hua. LQ-Nets: Learned quantiza- tion for highly accurate and compact deep neural networks. In The European Conference on Computer Vision (ECCV), September 2018
[64] Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. Ternarybert: Distillation-aware lot bit bert. arXiv preprint arXiv:2009. 12812, 2020.
[65] Xishan Zhang, Shaoli Liu, Rui Zhang, Chang Liu, Di Huang. Shiyi Zhou, Jiaming Guo, Yu Kang, Qi Guo, Zidong Du, et al. Adaptive precision training: Quantify back propagation in neural networks with fixed-point numbers. arXiv preprint arXiv: 19t::001100006 2019.
[66] Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural net- work quantization without retraining using outlier channel splitting. In International conference on machine learning, pages 7543-7552. PMLR, 2019.
[67] Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: Towards lossless cnns with low- lowpprccision weights. International Conference on Learning Representations, 2017.
[68] Feng Zhu, Ruihao Gong, Fengwei Yu, Xianglong Liu, Yanfei Wang, Zhelong Li, Xiuqi Yang, and Junjie Yan. Towards unified int8 training for convolutional neural network. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 1969-1979, 2020.
14

---

A Implementation Details
In this section, present works that need be done actually accelerate the training process on hardware.
A.1 BMM in Attention
In attention, there are batch matrix multiplications (BMMs) that need to be dealt with. We now show that our method for MMs can be extended to BMMs.
Consider the following BMM product:
T BMM(Q.
where we define T R BXNXP as
Q R RXNX Ke RXXPXM The Hadamard matrix is defined
Repeat B H = Repeat, y (BlockDiag(H)) (H....., H&)),
where H E BBBMXXM H E RR H E RR x2" In this case,
T BMM Q, H), BMM(K,MMKKK (H) H))
which verifies that our HQ can be applied to BMMs.
For backward, the gradient of weight and activation can be calculated by the straight-through estimator |x] = 1 and the chain rule:
VQ = SQ (BMM(VI. K) l1Q) H
VK = sKllK BMM Q)H BMM o VT, Q)HT,
where we define sQ ERB R being the batch step size, K ints (BMM(K, H)).
Q int (BMM(Q,H)) = I[ s Q/sQ S QP), and Ix = 1(-Qx S K/sk & Qp).
Similar to Sec. 4.2. we only focus on BMM I T: K) and VT, since we do leverage sampling on them.
For VM K), we define the sample probability p and sample the M in the same way as MMs. The matrix can be computed as BMM H), K.), where H is defined as CONCAT(H, HB). V and Ke follows the same definition Eq. 6nd the leverage score Ch.i = |IV To. | b.s. for 0 < SisSis 2M.
For VT, similarly, can be viewed as VT = BMM(1,, v where we define V CONCAT(I I]) E BBNNNNN So it can be computed as BMM H). v.). where H is defined as CO, HB), and the lever- age score is Cb,s 2 Vi,.ll for < b B,B 1800iC <is 2M, which verifies that our LSS can be applied to BMM.
CONCAT((s1) Vigii SL (.dl) E BBX2NXP it Sta, Vin $4. cc
V follows the definition of Eq.5
A.2 Computing Leverage Score
In the previous discussion, we find the optimal sample probability P; that can minimize the variance of the gradient. However, it is likely for the proportional p; is larger than one, which is invalid for the Bernoulli distribution. Accordingly, we propose an algorithm to solve this issue.
Define the probability array
2N P
P
N,
15

---

we first clamp the array to p! E [0, 1]. In this case, Σi= P S N, so we scale the which is smaller than make sure their sum again N. However, this will probably introduce some more elements larger than 1, so we cycle through the above operations until all the p. [0, 1.. This process will certainly stop, since if after the scaling operation, no element is larger than then we get valid distribution. Otherwise, the number larger than is reduced by at least one, thus the process will halt after at most ((N) times.
A.3
Learning Quantizer Parameters
In this section, we discuss the detail of how to calculate the gradient of activation and quantization
step size.
For gradient activation, the coefficient C; == | Y is the leverage score for activation gradient, and the variance achieves its minimum When p; c; by the Cauchy Inequality.
Putting everything together, we propose the following MM procedure compute activation gradient:
Procedure LSS-MM
Quantize Vy with BS obtain V and V in INT4.
2. Compute the leverage score V in FP16.
Sample the masks {m,)-
4. Sample rows of V given the masks {m:}.
5. Compute IM Vt and IM V by discard some of its rows.
6. Compute INT4 MMs IM V W and IM' I W.
7. Dequantize and sum up the resultant INT32 matrices to obtain the FP16 result i' w.
The two matrix multiplications in Step take about 2NCD INT4 MACs in expectation. For the quantization step sizes. Following the chain rule, we have
Vaw = ((sw) V (wws,), Vax - o (xx)),
where we define g(ww) 1/v U/vQ..w.. g(8x) 1/ W/pQN.x. Nw and Nx being the number of elements of weight and activation, 6x int,x (X) Ix (X/xx), and (sws intew (W) Iw o (W/) W/sw).
Notice that for computing Vay and Vax the most expensive MMs are V and Vy W, which already calculated through Eq. 7 and Eq. (8) during previous caleulations, it does require extra computation. The elementwise multiplication with fx(xx) and (s(sw) requires minor computation.
A.4 Cold Start Problem
There is a cold start problem. When the model is trained from scratch (i.e., from random initializa- tion), distributions of weights and activations can change rapidly the early stage of optimization. In this case, jointly optimizing the quantization step size and the weights would cause the training to be unstable. As remedy, do not learn the step size in the first few iterations, and use heuristic rule dynamically set the step size for each tensor X 2mean (X)/ D/p in each iteration. For the WMT experiment, we stop to re-initialize at the second epoch, since the BLEU score is relatively high the second epoch (about 21, while the final BLEU score after 30 epochs is 25.5). However, for the deit-small pretraining experiment, we re-initialize it at the end of the 10th epoch (we train for 90 epochs in total), since the end of the second epoch, the accuracy is not high, but at the end of the 10th epoch, the accuracy is relatively high.
A.5 Choose hadamard matrix size
For the hadamard matrix, let the hadamard matrix be H E DxD, H BlockDiag(HH,,... H&), where D is multiple of 2* We first define
X sxintex (XH) H
W swint,w (WH) H
16

---

where X and W can be viewed as an approximation of X and W. Then, we define the quantization eтror to (SE X) x MSE(W, W). We search for the optimal k that can minimize this quan- tization error. For fine-tuning tasks, once the hadamard matrix size has been calculated, we fix it through the training process. For the pre-training task, since the distribution shifts greatly as we train the model, we empirically define a time when we re- re-initialize the hadamard matrix size and the LSQ step size. Usually, we do this when the first 2 epochs finish.
A.6 GPU Implementation
In the previous discussion, we get to know HQ-MM and LSS-MM from an algorithm level, nevertheless it is not enough to actually implement it on hardware. In this section, we will delve deeper into hardware implementation details as well extra limitations.
HQ-MM can be divided into 5 parts: Hadamard matrix multiplication, Quantize, Data Pack, INT4 GEMM, and Dequantize.
For the Hadamard matrix multiplication process, since can be interpreted as half float matrix multiplication process where the two matrices involved in the operation are input/weight matrix and hadamard matrix, respectively, we implement Python, because Py PoToch MM uses CublassGemm and is more efficient then CutlassGemm.
In the quantize process, we quantize input/weight into INT4 data respectively, and also preserve a corresponding FP16 version for the LSQ Back Propagation process to use.
In the previous discussion, we assume the quantize part of HQ-MM is quantizing the resultant matrices to INT4, however, the smallest representation unit of data is INT8. As a result, we actually use INT8 data type to represent quantized data and pack two adjacent data into one data using (data[1] << (d((daa(())&11 (1)) in the data packing process, which means we use one INT8 data to represent two adjacent INT4 data. With both input matrices' data packed in this way, we then use cutlass tensor-core INT4 GEMM to do the matrix multiplication.
For the GEMM process, we choose Nvidia CutlassGemm because it's the most efficient open open-source operator library we can find, We use INT4 Tensor Core Gemm for our implementation and requires the two input matrices A&B to be RowMaoor and ColMajor, respectively. Since the default Pytorch tensor is RowMaaor, we have to use Transpos++Contigunn operations to make it ColMajor, which is very cime-consuming and needs further optimization in the future.
Finally, we dequantize the INT GEMM result back into FP16 output using dequantize kernel, which is the final output of the forward kernel.
As compared, LSS-MI is more complicated, and can be divided into 7 parts: Quantization of higher lower 4-bit, Leverage Score Calculating, Sampling, Data Pack, INT4 GEMM, Dequantize, and LSQ Back Propagation.
In the Quantize process, we fuse the quantize operation of higher 4-bit and lower 4-bit into a single kernel for acceleration. In the Leverage Score Calculating process, to calculate the score and scale up it in the final because integer arithmetic is far more efficient than float arithmetic.
the quantized INT8 data
In the sampling process, we sample out rows/columns given the previously calculated leverage score. Note that in Section. A.2 we repeat our proposed algorithm for several loops to sample out specific elements, which is effective but not efficient. According to experiments, however, we notice that simply selecting elements whose leverage score is bigger than 0 can also work well, even better than our proposed algorithm in some cases. So in real quantization implementation, we just sample out rows/ columns whose Euclidean norm is bigger than 0 to accelerate our training process.
Pack, Gemm, and Dequantize processes are as similar as before. It's worth noting that for Int4 Tensor Core Gemm, suppose two input matrices have shape M x K and K x N, K needs to be multiple 32 so that the Tensor core Gemm address can aligned. We do not need to consider this in the Forward Propagation process because the input data shape always satisfies. However, in the Back Propagation process, the matrix shape may not meet the requirement after sampling. As result, we need zero cero__paddin the sampled matrix so that K can be a multiple of 32.
17

---

Finally, we utilize the dequantized data to do the LSQ Back Propagation. We also fuse all operations into a single Cuda kernel for acceleration, and the metric remains.
Besides the component of HQ-MM and LSS-MM, there is still something that needs to be mentioned.
1. We omit the Quantization and Leverage Score Calculating process in LSSinput, and use the same value as LSSWeight to accelerate the training process.
2. For Element- Wise kernel, we set block size as 256, grid size as Reduction kernels like sum and min/max, we set block size as 32, grid size as RowNum, reducing elements each row to the first 32 elements. We find this setting be most efficient through experiments.
For
B Proofs.
In this section, we present the proofs of the leverage score.
B.1 Proof of Proposition.
Proposition B.1. (LSS variance,for for weight gradient)
2N
2N
Var
mi
Σ
Pi
VV
VY.
Proof.
2N
Var(Uw) = Var
m,
v
Z. x()
Pi
1=1
m: Zj. X!
Var
pi
k=
2
P:(1
Pi)
Var
(
D
V
Pi
1=1
j=1k=1
2N
C
D
ps
x
Zjs
1=1
j=
So that
Var(V
>
2N
1
1)
V
Zjs )(
T
) x
(9)
Pi
1=1
2N
1
1)
V
Z
X
112
(10)
pi
which proves.
B.2 Proof of Activation Leverage Score in Sec. 4.2
we divide the matrix multiplication into the sum of 2N smaller multiplications:
2N
2N
F
Σ
Σ
V
(11)
1=1
where we define VY. = = vf Yi
18

---

We assigns each Vy, a probability E [0, = 1,... 2N, that satisfies Σi= = N. We define random masks my Bern(p:), and define M = diag estimation:
m1 maN
pan ). and make an unbiased
pt
I'v I'
Vy-
P:
Define M to be the top-left N N submatrix of M and M be the bottom-right one, have
f'M = IM vt + + IM'
In this case, IM V and IM V both only have parts of its rows being non zero, and the rest rows are zeros since they are discarded. Then, when we multiply it by W, there are half of rows being zeros in IM v W and IM v W. So there's no need calculate them, and we successfully cut off half of the computation in this case.
Now focus the variance that
Proposition B.2. (LSS variance for activation gradient)
22
2N
Var
Σ
Pt
V 11.
Yi
Proof.
Var (Vx) Var
(m I. x
pi
i=1
2N
Var
m di V
Pi
2N
p:(1
pi)
Var
P
f=
2N
D
1- Pi
Pi
111
k=
2N
1)
(V))))
1=1
2N
1)
11
11
P:
2N
1)1l
i=1
In this way, the coefficient c; = V x
|l is the leverage score.
C Experiments.
In this section, present more details for experiments in Sec.
19

---

hadamard
quantize
pack
gemm
dequantize
leverage
sample
LSQ
HQ
LSSWeight
LSSAct
1.0
1.0
10%
6%
P+
2%
6.%
305
T4N
5%
5%
55
221
0.3
0.8
0.8
Tinle ratto
42%
1224
207
375
137
20N
274
0.6
0.6
0.6
0.4
2014
8%
25N
2754
0%
0.4
$359
15%
0.4
375
37%
0.2
0.2
1%
7%
9%
0.2
14h
179
17%
12%
1794
20%
17%
277
199
199
(4688 5120, 61441
0.0
(126, 6144 8133)
(8144 6144, 92261
Matrix size (M.N c1see0 5704, 10752)
(7198, 6456, 6764)
(8152, 7660 9728)
(4608, 5120, 6144)
0.0
(St20 6144, 8192)
(6144, 6164, 92288
(P1E8 6656, 8784)
ret92, 7080, 97241
(15360, 8704 10153)
(4808, 5120, 61441
0.0
(5120, 8144, 6103)
(6144, 6144 52181
(055. 7650 9738)
27681
cisee 9104, 19752
(7108) 6656,
Matrix size (M.NKK
Matrix size (M, N, k)
Figure 6: Time proportion for each part in HQ-MM and LSS-MM operator:
C.1 Experiments setup
For the GLUE, QA. SWAG, and CONLL tasks, we implement our algorithm based on https 7/github. com/
ttansformrssrsssssse For the machine translation task, we implement our For the ViT fine-tuning task, we implement our algorithm based on https: t/tp:///tttuubbb. com/ http:///wwwollll---ytooooo For the deit pretraining task, we implement our algorithm based on https: com/ facebookressarcchaeell
algorithm based on https://gitta
com hoom///fccbbborreerrraaaaaaaaaae
We employed NVIDIA GeForce RTX 3090 for running most of the experiments, while the NVIDIA A40 was utilized to evaluate the performance of BERT-Large and ViT-L. Furthermore, we conducted runtime measurements using the NVIDIA T4, 3090, and A100 GPUs.
C.2 GLUE results
In this section, we present the detailed result of fine-tuning the GLUE dataset on BERR-base-umcased and BERT-large-uncased.
On BERT-base, on STSB, SST2, QNLI, and QQP, HQ+LSS only has < 0.5% accuracy degradation. On the most challenging tasks CoLA and RTE, our accuracy degradation is much smaller compared to LSQ+LUQ. On QQP and MNLI, our method achieves < 1.3% degradation, while LSQ + LUQ has > 1.8% degradation. The trend is that the more difficult the task is, the more significant our advantage over LSQ+LUQ.
On BERT-large, the improvement is significant. On CoLA, QNLI, and MNLI, the accuracy im- provement compared with LSQ+LUQ > 30%. On other datasets like SST2 and QQP, the accuracy improvement is > 10%. On RTE the accuracy improvement is >> 5%, and on STSB and MRPC the improvement 3%.
We suspect that for those challenging tasks, there is more information stored in the outliers, which results in larger gap between our method and LSQ+LUQ
C.3 More Granular Quantization Methods
In this section, in Table 5 we show that the more granular quantization methods, such as per-token quantization and per-channel quantization, or smoothing techniques, such as SmoothQuant, do not work under the 4-bit FQT setting. Meanwhile, combining these methods with HQ will not bring significant improvement.
find that LSQ is beneficial for all of these more granular quantization methods under low-bit settings, which highlights the importance of LSQ. Meanwhile, we also notice that the smoothquant will even harm the result of LSQ when the bit-width low. Our explanation is that the motivation of LSQ is to Iearn trade-off between outliers and inlicrs, while smoothquant aims to sacrifice the
20

---

Table GLUE results on BERT-base-uncased and BaRgelarge uncased. FP refers to full precision training. INT8 refers to INT8 training, LSQ+ LUQ refers to learned step size quantization for forward and logarithmic unbiased quantization for backward, and HQ LSS refers to Hadamard quantization for forward and leverage score sampling for backward.
QUANTIZATION METHODS
MODEL
DATASET
FP
INT8
LSQ+LUQ
HQ+LSS
CoLA
56.89o.66
56.150.94
18.76 18.763.58
52.461.46
STSB
88. 78.14..73
87.050.38
84.31o.29
87.770.30
RTE
64.801.26
62.271.26
56.800.92
62.451.08
MRPC
88.61o.66
86.850.76
86.23 36.230.67
86.540.83
BERT-BASE
SST2
92.720.06
92.370.17
90.370..6
92.490.29
QNLI
91.52n.22
90.920.24
87.330.48
90.530.23
QQP
91.09o.11
90.570.05
89.266.03
89.800.05
MNLI
84.52n.22
84.10o.os
81.799.18
83.59o.12
MNLI-MM
84.68a.20
84.490.31
82.22g.33
83.750.28
CoLA 60.33o.4p
58.801.5.
0.00o.00
53.461.17
STSB
87.592.39
86.530.20
83.08..41
87.570.78
RTE
71.121.88
63.711.24
53.060.72
64.620.78
MRPC
91.06g.20
87.57
82.56p.59
87.620.51
BERT-LARGE
SST2
93.98p.17
93.750.63
83.94p.69
93.520.40
QNLI
92.266.05
92.290.29
63.1813.10
91.530.38
QQP
91.04g.63
90.71o.00
75.6212.44
90.770.02
MNLI 1..71
85.82o.08
33.421.38
85.86o.10
MNLI-MM
86.410.35
85.87o.14
33.541.35
85.82o.07
Table 4: Experiments GPT2-base and Bert-large. Total time spent for epoch 1-5 are reported.
Training METHODS
MODEL
(HIDDEN SIZE, INTERMIDIATE SIZE, BATCH SIZE)
FP16
HQ+LSS
SPEEDUP
(2560, 10240, 2048)
15,0945
18.9498
25.5%
(4096, 16384, 1280)
32.0168
30.5948
4.4%
(5120, 20480, 960)
47.4188
39.4828
16.7%
(7680, 30720, 600)
95.8325
67.2538
29.8%
BERT-LARGE
(8960, 35840, 480)
128.4418
83.3888
35.1%
(9600, 38400, 160)
161.1148
114.325s
29.0%
(12800, 51200, 100)
326.265s
255.9665
21.5%
(14400, 57600, 96)
409.291s
346.354s
15.3%
(2560, 10240. 1536)
17.253s
22.037s
27.7%
(4096, 16384, 960)
35,9375
35.694s
(5120, 20480. 768)
52.723s
46.548s
11.7%
GPT2-BASE
(7680. 30720. 260)
113.855s
92.548s
18.7%
(8960. 35840. 200)
150.680s
114.881s
23.8%
(9600, 38400. 180)
172.182s
126.540s
26.5%
(12800. 51200. 112)
320.757s
236.433s
26.3%
precision of inliers in order to exactly maintain the information of outliers. When the bitwidth high. this is not problem, since there are still enough bits to quantize the inliers. But when the bitwidth is low, such sacrifice will cause severe problems since the inlier information is discarded.
C.4 Large Language Model Operator Speed
In this section, we show that our hardware-friendly INT4 training method can really accelerate the training process on Large Language Models. We run distributed training on system of 8 A100 cards
21

---

70
hudamard
quantize
peck
geem
dequentize
leverage
semple
FP36
HB
BM14
50
0.8
20
10
og
wot rads
CSM cre
WIM
Matric NUO
Mae the ON
Narni IM 2000
Morro the AMAC
Figure 7: Real quantization performance Nvidia T4.
1adamand
quarlice
peck
dequaniee
IS0
330
300
230
200
150
100
50
a1330 loka
mo
mtte 4tr Ate
120
me
rite 40t 4io
unio
GLD ALAL
GPAN AEAE
OTEE asta
oots rots
Figure Real quantization performance on Nvidia A100.
Table Comparison of different quantization methods, quantize the activation and weight into the same bit from 2 to 8. Per-token refers to quantize activation per-token, while Per-channel refers to quantize weight per-channel.
Quantize Bits
quantization methods
8
Per-tensor
0
0
0
50.2
54.6
Per-token
0
0
0
0
31.4
52.8
56
Per-channel
0
0
0
0
51.9
56.7
smoothquant
0
0
0
49.4
57.7
Per-token + Per-channel smoothquant
0
0
0
0
40.7
55.7
56.7
LSQ Per-token + LSQ
0
9.16
24.2
37.3
39.6
45.3
51.4
15.3
27.8
31.6
42.9
54.4
Per-channel + LSQ
0
8
23.9
29.3
40
45.5
50.7
smoothquant LSQ
49.6
54.9
57
Per-token + Per-channel smoothquant LSQ
0
0
0
0
28.8
52.4
55.2
HQ
45.2
54.6
54.2
56.5
57.4
58.4
HQ + Per-token + Per-channel HQ + Per-token Per-channel smoothquant
0
48.4
54.1
54.9
55
56
56
46.6
54.9
55.9
55.8
56.5
and our implementation uses distributed data parallel training with zero-3, gradient checkpointing, and optimizer offloading.
We experimented with two architectures: BERT-Large and GPT2-base. We vary the network width and batch size make full utilization the GPU memory and show the end-to-end performance for fine-tuning these models the SuperGLUE RTE dataset in Table
22

---

C.5 More experiments on Operator Speed
Time proportion We examine the proportion of time for each part of computation in HQ-MM and LSS-MM operator in Fig. [ when the shapes of input matrices vary. In HQ, hadamard means multi- plying the input matrix with the Hadamard matrix, pack means packing imput data into INT4 data, gemm means the matrix multiplication of two INT4 matrices. In L.SS LeiSeight, quantize corresponds to the quantization of higher and lower 4-bit, leverage means computing leverage score, sample means sample out rows/columns given the leverage score, dequantize is the process of dequantizing INT data back into FP16 data, and LSQ the backpropagation process of LSQ method. In LSSAct, we ignore quantize and leverage process, using the same value LSSWeight for saving time, other processes share the same meaning with Weight. Note that our implementation not fully optimized, and optimizations like operator fusion can further improve the performance.
Operator Speed on more GPUs On an Nvidia RTX 3090 GPU with a Cuda capability of sm 86., we show the comparison of FP16 MM, HQ, and LSS operators in Section 5.3 well as time proportion in each operator in Figure. 6 We also adjust our hardware implementation and test its performance Nvidia T4 GPU and Nvidia A100 GPU, which have Cuda capability of sm_75 and sm sm_80, respectively. The result is shown Fig. and Fig. 8
23


Reviews:
[{"review_id": "F7ueKuvr7P", "reviewer": "Reviewer_jhrg", "summary": "This paper deals with 4-bit training (fine-tuning) of transformers. For forward propagation, a Hadamard quantizer is proposed to solve the problem of outliers. For backpropagation, the authors leverage the structural sparsity of gradients by proposing bit splitting and leverage score sampling\ntechniques to quantize gradients accurately. Experiments show up to 35.1% speedup for 4-bit training with ", "questions": "Please see the weaknesses.", "limitations": "NA", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed Hadamard quantizer is simple while useful for transformer quantization with outliers.\n2. The authors illustrate the structural sparsity of gradients and propose bit splitting (BS), which splits a full-precision matrix as higher and lower 4 bits. The similar idea is used in quantization for inference. This is the first to use it for training.\n3. The experiments are extensive. The training speed on GPUs are reported.", "weaknesses": "1. The proposed method introduce extra FP16 computations. The BS with selection needs data rearrangement, which could slow down the speed.\n2. The title is somewhat over-claim. The experiments are mainly about fine-tuning (FT). It seems that the proposed method works for fine-tuning (FT) of a converged model. For pre-training (PT), there is large accuracy loss. \n3. The speed-up of up to 35.1% is misleading. It is better to use the average speed-up in the abstract.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the acknowledgment of our contributions and the careful constructive comments.  Below we provide a point-to-point response to the reviewer's questions.\n\n**Q1**: The proposed method introduces extra FP16 computations. The BS with selection needs data rearrangement, which could slow down the speed\n\n**A1**: Thanks for the insightful comment. While it is true that the proposed method involves some additional FP16 computations to calculate the leverage score, it is important to note that the time complexity of additional FP16 computations is lower than that of the original matrix multiplication. \nFor the FP16 operation used in the Hadamard Quantizer, as discussed in Section 3.3, we show that the amount of FP16 computation is $O(2^kN(D + C))$, which is much cheaper than the original $O(NCD)$ matrix multiplication(where a typical value of $k=5$ is the group size, $N$ is the batch size, $D, C$ are a number of neurons). Therefore, the overhead is negligible as matrices grow large. In practice, the FP16 computation only takes 12% $\\sim$ 18% time in our HQ-MM operator, as shown in Fig. 6 in the appendix. \n\nIt is true that the data rearrangement needed for the BS with selection could slow down the speed. Again, the time complexity of the data rearrangement step is $O(NC + ND)$, while the time complexity of the matrix multiplication is $O(NCD)$. So our method can still be attractive when the matrices are large. As shown in Fig. 4, our INT4 operators can be up to 2.2x faster than FP16. \n\n\n**Q2**: The title is somewhat over-claim. The experiments are mainly about fine-tuning (FT). It seems that the proposed method works for fine-tuning (FT) of a converged model. For pre-training (PT), there is a large accuracy loss.\n\n**A2**: Thanks for your consideration. Our proposed technique can be applied to both fine-tuning and pre-training, and we run experiments on both fine-tuning and pre-training tasks. Even though there is still an accuracy loss on pre-training tasks, our result for pre-training is on par with previous 4-bit training methods for dedicated hardware [6, 7], whose titles are also training rather than fine-tuning.\n\nIn general, 4-bit training for transformers is still an extremely challenging task to study. We believe that our work makes a reasonable step towards both 4-bit fine-tuning and pre-training, despite there is still room to improve. \n\n\n**Q3**: The speed-up of up to 35.1\\% is misleading. It is better to use the average speed-up in the abstract.\n\n**A3**: Thanks for the constructive comment. We apologize for any confusion this may have caused. \nWe will use the average speed-up in the final version, as suggested by the reviewer. \n\nWe are grateful for the time and effort you have invested in evaluating our submission. We are more than willing to address any further questions or concerns you may have.\n\nThe references are listed in the global rebuttal section."}, {"review_id": "99wr58hPIW", "reviewer": "Reviewer_7XeV", "summary": "The research paper presents a framework for training transformer-based neural networks using 4-bit integers, offering enhanced computational and memory efficiency. \nThe proposed method includes a dedicated Hadamard quantizer for forward propagation, designed to suppress outliers which typically cause degradation of model accuracy. The Hadamard quantizer works by transforming the activation matrix, spreading outlier information across nearby entries in the matrix, and reducing the numerical range of outliers.\nIn backpropagation, the authors exploit the structural sparsity of activation gradients. They note that a few token gradients are extremely large, while most are much smaller, even below the quantization residuals of larger gradients. To make efficient use of computational resources, the authors propose a technique called bit splitting. This technique divides each token's gradient into two parts: the higher 4 bits and the lower 4 bits. They then identify the most informative gradients using a technique known as leverage score sampling, an importance sampling technique in the field of RandNLA.\nThe combination of these techniques results in an algorithm that employs 4-bit integer MMs for all linear operations in transformers. This algorithm has been evaluated and found to perform competitively across a wide range of tasks, including natural language understanding, question answering, machine translation, and image classification.", "questions": "The concept of using a Hadamard matrix to decompose matrix multiplication is a thoughtful and fitting approach in low-bit quantization scenarios, especially for extreme cases such as binary and ternary methods. The paper meticulously analyzes the time complexity of the Hadamard Quantizer Matrix Multiplication (HQ-MM) procedure in Section 3.3, and in Appendix A.5, it further elucidates on how to select an appropriate Hadamard matrix size. I am intrigued to learn more about the finer details of this procedure.\n\nGiven the significant shift in distribution during training, it is indeed crucial to reconfigure both the Hadamard matrix size and the Learned Step Size Quantization (LSQ) step size. This brings up a few questions: Is the re-initialization process executed only once, or is it a recurring process? Does the system stabilize after approximately two epochs? Furthermore, is X representative of the activation of a batch of input data? These details will enhance the understanding and application of the proposed methodology.\n\n", "limitations": "Implementing advanced quantization techniques such as the Hadamard quantizer and bit splitting may be technically challenging and potentially difficult to integrate into existing neural network training pipelines.\nThe method exhibits limitations in its scalability, as it is not equipped to support exceedingly large models like OPT-175B. This underscores a broader, unresolved issue in the realm of machine learning, where even the application of INT8 training to these substantial models remains an open-ended problem.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Unlike many other low-precision training methods, this proposed method does not require custom numerical formats and is compatible with contemporary hardware, such as GPUs. This opens the potential for widespread adoption and usability.\n- The proposed method demonstrated competitive or superior accuracy across a range of tasks, compared to existing 4-bit training methods. This indicates the practical effectiveness of the method in diverse applications.\n- The authors have developed specialized quantization methods for both forward and backpropagation. These techniques manage outliers and leverage the structural sparsity of activation gradients, maintaining model accuracy despite the ultra-low numerical precision.", "weaknesses": "My main concern is that the paper chose to compare the performance and computation complexity of their approach with FP16 rather than examining its efficiency against INT8 or other INT4 alternatives. This could potentially limit the depth and breadth of the performance analysis.\n\nThe key contributions of this research are primarily focused on the implementation of a Hadamard quantizer and bit splitting techniques. Both techniques leverage INT4 matrix multiplication (MM). However, the HQ-MM has the 4 steps procedure to leverage the efficiency of INT4 computation and the bit-splitting technique deploys two INT4 MMs as a replacement for one INT8 MM. This can potentially induce more overhead than simply utilizing INT8, particularly considering that INT4 does not straightforwardly translate to half the computational, memory access, or energy costs associated with INT8 computations.\n\nWhile the aim of utilizing INT4 computations is to increase efficiency, it's crucial to recognize that the benefits of moving from INT8 to INT4 are not always linear. According to a report by NVIDIA, INT4, in an optimal scenario involving Convolutional Neural Network (CNN) inference in image classification tasks, resulting in a throughput improvement of approximately 59% with a negligible accuracy loss (around 1%) on NVIDIA T4. Meanwhile, on TITAN RTX, the speedup was about 52%. These improvements, although significant, don't necessarily equate to halving the computational resources or doubling the performance, and this is also applied to the transformer-based models scenario of this paper.\n\nHence, a more thorough comparative study involving INT8 and other INT4 counterparts could potentially provide a more nuanced and comprehensive understanding of the computational advantages and trade-offs of the proposed techniques. Additionally, if there are any misinterpretations or inaccuracies in my understanding of the paper, I would appreciate it if you could provide corrections to ensure my perspective aligns accurately with the content and intentions of the research.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the acknowledgment of the potential and effectiveness of our work and the detailed constructive comments. Below we provide a point-to-point response to all comments.\n\n**W1**: compare the performance and computation complexity with FP16 rather than examining its efficiency against INT8 or other INT4 alternatives\n\n**A1**: There might be some misunderstandings. There are many INT8 and INT4 implementations for **inference**, but to the best of our knowledge, there are currently no open-source implementations of INT8 / INT4 **training** for transformers in the existing literature. \nMost low-precision training works simply do simulated quantization, without a real implementation to actually realize the speedup.\nThere exists some INT8 training work [1-5] that reports speedup, but none of them are open-source. \n\nTo the best of our knowledge, we are the first work that performs INT4 fully-quantized training. So there are no other INT4 alternatives. \n\n**W2**: speedup potential of INT4 comparing to INT8\n\n**A2**: We agree with the reviewer that the benefit of reducing numerical precision is not always linear, and there are still many implementation challenges to consider before INT4 training can be really used in commodity training systems. \n\nRegarding the overhead, indeed, INT4 MM can induce more overhead than simply utilizing INT8. However, as the additional operations of HQ-MM and LSS-MM have lower time complexity $O(2^k N (D+C))$ compared to $O(NCD)$ (where $k=5$ is the group size, $N$ is the batch size, $D, C$ are a number of neurons), the proportion of overhead decreases as the matrices grow large. So we believe that INT4 is ultimately more promising than INT8. \n\nWe would like to emphasize that the implementation is only a small fraction of our contribution, and our paper should not be evaluated mainly based on the implementation. INT4 training is an extremely difficult task, with challenges ranging from numerical format, optimization, model architecture, and software and hardware implementation. \nIt is not likely to solve all the challenges with a single research paper. \n\nAmong the challenges, the first question to answer is, does there exists an INT4 training **algorithm**, which can achieve good accuracy by utilizing mainly low-precision arithmetic? Our work gives an initial affirmative answer to this question. This is the main contribution of our research.  Actually, most research papers on low-precision training (e.g., [6, 7], [10, 11]) only do simulated quantization, without having an actual implementation. While we do provide an implementation, it is only a prototypical one without heavy optimizations, as we stated in the paper. The implementation optimization itself can be a worthy research, and we leave it as a future work. \n\n**Q3**: Finer details of Hadamard Quantizer\n\n**A3**: We search the Hadamard group size among the exponential of 2, from 1 to 1024. Before the actual training, we pass a \"probing batch\" to the model and use it to search for the optimal Hadamard group size and LSQ step size. For each of the linear layers, we search the group size by quantizing the activation and weight. We select the group size with the lowest $MSE(\\bar{X}, X) * MSE(\\bar{W}, W)$, which is described in Appendix A.5.\n\n**Q4**: Details about the re-initialization process.\n\n**A4**: For the re-initialization process, we usually repeatedly do this at the end of each epoch for the first few epochs. \nThen, the Hadamard group size is frozen in the rest of the training, and the LSQ step size is learned jointly with model parameters. \nWe observe that $\\frac{2mean(x)}{Q_p}$ (which is how we calculate the initial value of LSQ step size) changes dramatically during the first few epochs, then it stabilizes. \n\nA rule of thumb is, the system definitely stabilizes when the accuracy is relatively high.\nFor the WMT experiment, we stop to re-initialize at the second epoch, since the BLEU score is relatively high at the second epoch (about 21, while the final BLEU score after 30 epochs is 25.5). However, for the deit-small pretraining experiment, we re-initialize it at the end of the 10th epoch (we train it for 90 epochs in total), since at the end of the second epoch, the accuracy is not high, but at the end of the 10th epoch, the accuracy is relatively high.\nWe will revise our final version to include these details. \n\n**Q5**: Is X representative of the activation of a batch of input data?\n\n**A5**: We randomly choose the batch of input data for initialization. But we do not observe too much difference when we change the data we used for initialization. For the hadamard group size, possibly it will change slightly when we change the data used for initialization (from 64 to 32, for example), but this will not bring too much difference since it can still reduce the outlier magnitude. And for the LSQ step size, since it continually changes during the training, as long as the initial value is not very far away from the optimal step size, which data used for initialization will not affect the final result. We conducted each experiment with three different seeds, and their results are similar, further validating our viewpoint.\n\n**Q6**: Limitations in its scalability, as it is not equipped to support exceedingly large models like OPT-175B.\n\n**A6**: We appreciate the constructive criticism and insightful observations regarding the technical challenges and scalability of our method. We recognize that even INT8 training of large models like OPT-175B is an open problem (see [9] Appendix.C.3). So our work is actually a pioneer work on fully quantized training, since we focus on an even more aggressive INT4 setting.\n\nWe are grateful for the time and effort you have invested in evaluating our submission. We are more than willing to address any further questions or concerns you may have.\n\nThe references are listed in the global rebuttal section due to the rebuttal length restriction."}, {"review_id": "vkOr2V4wRh", "reviewer": "Reviewer_au4S", "summary": "The authors propose 4-bit training methods for Transformers. It first proposes a Hadamard quantizer with activation outliers issue being solved by Hadamard Transformation. Optimization for backpropagation is proposed by leveraging gradient sparsity with bit splitting and score sampling. The proposed 4-bit training methods achieve competitive accuracy on NLP, Machine translation, and Image classification.", "questions": "* It would be helpful if you could explain why the speed is not comparable with the baseline in Fig.5 when the hidden layer size is small.", "limitations": "* Conv cannot be supported\n* INT4 training on extremely large models remains to be an open question.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Achieves competitive or superior accuracy compared with existing works on 4-bit training.\n* Optimize backpropagation with hardware-efficient structural sparsity.\n* Hardware-friendly INT4 training method is runnable on GPUs with up to 2.2 times faster than the FP16 MM.", "weaknesses": "* The activation outlier handling in Transformers is not new, as in SmoothQuant. And the comparison is lacking in the paper. Are those techniques not able to be used in 4-bit training? And what is the efficiency comparison between those methods when handling outlier activations?\n* Could you elaborate more on why the proposed methods cannot be applied to Conv layers? What is the bottleneck?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments and feedback. Below we provide a point-to-point response to all comments.\n\n**Q1**: The comparison with SmoothQuant\n\n**A1**: There might be some misunderstandings. \nIt's important to note that SmoothQuant is primarily designed for **INT8 inference** and not designed for **INT4 training**. INT4 is significantly more challenging than INT8. Furthermore, SmoothQuant is a post-training quantization method, and the weight can be quantized in a pre-processing stage. On the other hand, in our scenario, all the tensors must be quantized on-the-fly in each iteration. Furthermore, it is unclear whether post-training quantization methods can produce stable gradients for training. Consequently, a direct comparison between our approach and SmoothQuant might not be appropriate. Nonetheless, we have conducted a thorough comparative analysis with SmoothQuant, and the results are shown in Appendix C.3 as well as Table 4.\n\nIn Appendix.C.3, we show that SmoothQuant always fails (0\\% accuracy) in the INT4 training setting. We also show that even if we combined SmoothQuant with the more granular quantization methods, such as per-token quantization and per-channel quantization, it still does not work under the 4-bit FQT setting. Meanwhile, combining these methods with HQ will not bring significant improvement.\n\nIntuitively, our HQ is stronger than SmoothQuant.\nOur explanation is that the motivation of LSQ is to learn a trade-off between outliers and inliers, while smoothquant aims to sacrifice the precision of inliers in order to exactly maintain the information of outliers. When the bitwidth is high, this is not a problem, since there are still enough bits to quantize the inliers. But when the bitwidth is low, such sacrifice will cause severe problems since the inlier information is discarded.\n\n**Q2**: why the proposed methods cannot be applied to Conv layers\n\n**A2**: Thank you for bringing up this concern. \nINT4 training is an extremely challenging open problem, we leverage the specific structure of matrix multiplications to make this possible. \n\nSpecifically, we leverage the sparsity property of transformer gradients. For transformers, the back propagation is multiplying a [batch size * sequence length, hidden size] gradient matrix with an activation / weight matrix. The gradient matrix has some almost-zero rows, so we can save computation by dropping these rows for matrix multiplication. However, Conv layers convolve the [batch size, height, width, channel] gradient map with another kernel. Even if the gradient map have spatial sparsity (almost zero for some locations), such sparsity cannot be easily leveraged to reduce the amount of computation. \n\n**Q3**: INT4 training on extremely large models remains to be an open question.\n\n**A3**: We appreciate the constructive criticism and insightful observations regarding the technical challenges and scalability of our method. We recognize that even INT8 training of large models like OPT-175B is an open problem (see [9] Appendix.C.3). So our work is actually a pioneer work on fully quantized training, since we focus on an even more aggressive INT4 setting.\n\nWe are grateful for the time and effort you have invested in evaluating our submission. We are more than willing to address any further questions or concerns you may have.\n\nThe references are listed in the global rebuttal section due to the rebuttal length restriction."}, {"review_id": "vdEKh6Mevp", "reviewer": "Reviewer_XvWB", "summary": "This paper presents a 4-bit training method mainly to speed up the training process. In the forward propagation, Hadamard quantizer is proposed to suppress the outliers whereas structure sparsity and bit splitting are used for quantization of gradients. It has been shown that the proposed training method can successfully train transformer models across different datasets with some performance degradation w.r.t. the baseline.", "questions": "In summary, the idea of Int4 training to accelerate the process is interesting. The authors have shown that Int4 training can obtain a comparable accuracy w.r.t. the baseline and Int8 with around ~1% to ~2% degradation, which is fine. However, the speedup amount is not provided for the benchmarks used for accuracy evaluation in Table 1. Given the above explanations, I have the following questions:\n1) What is the speedup for tasks listed in Table 1?\n2) How do you compare your method w.r.t. Int8 in terms of training speed? In other words, why should I select Int4 over Int8?\n3) How was the accuracy performance measured? Did you report the accuracy in Table 1 using quantized inference (Int4) or full-precision? \n4) Please provide more details for Table 3.", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-- The main strength of this paper is theoretical analysis of their proposed quantization method for both forward and backward steps.\n-- Use of different models and tasks for evaluation purposes is another strength of this paper. ", "weaknesses": "-- The main concern of mine is the limited speedup of this paper. The main goal of quantized training in this paper is to speed up the computations and not to reduce the memory. However, the authors have only shown up to 35% speedup w.r.t. the baseline (FP16). First, there is no comparison with Int8 training speed. It's not clear why I should choose Int4 over Int8 while I know the accuracy of Int8 is better than Int4. Second, the speedup amounts are only measured across specific configurations which are not the ones used for evaluation purposes in Table 1. Measuring the training time for the tasks listed in Table 1 can make the contribution of this paper clear.\n-- Fig. 4 and Fig. 5 are hard to read. Again, why not simply reporting the speedup for each task in Table 1.\n-- I have a hard time understanding Table 3. What does \"epoch 1-5\" mean? Which dataset was used for the measurement? What is the accuracy performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments and feedback. Below we provide a point-to-point response to all comments.\n\n**W1**: limited speedup of this paper\n\n**A1**: We would like to emphasize that our main contribution is the first INT4 training **algorithm**. The implementation is only a small fraction of our contribution, and our paper should not be evaluated mainly based on the implementation.\nAs we stated in the paper, our implementation is only a prototype, **it does not fully reflect the potential of our INT4 training algorithm**. Our implementation has much room to improve, such as operator fusion. However, the engineering efforts for a heavily optimized INT4 training system exceed the scope of a research paper on the first INT4 training algorithm. \n\nINT4 training is an extremely difficult task, with challenges ranging from numerical format, optimization, model architecture, and software and hardware implementation. \nIt is impractical to solve all the challenges with a single research paper. For example, there already exists INT8[10]/FP8[11] training algorithms in 2018, but the accleration potential of FP8 for training is realized 4 years later in Nvidia's H100 GPUs with Transformer Engine in 2022, and how to use INT8 to accelerate training is still an active area of research. \n\nAmong the challenges of INT4 training, the first question to answer is, does there exists an INT4 training **algorithm**, which can achieve good accuracy by utilizing mainly low-precision arithmetic? Our work gives an initial affirmative answer to this question. This is the main contribution of our research. Only when we have an algorithm, people can then conduct research to optimize the actual software and hardware implementation and optimize the algorithm to be more accurate and software/hardware friendly. Actually, most research papers on low-precision training (e.g., [6,7,10,11]) only do simulated quantization, without having an actual implementation. While we do provide an implementation, it is only a prototypical one without heavy optimizations, as we stated in the paper. The implementation optimization itself can be a worthy research, and we leave it as a future work. \n\n**W2**: No comparison with INT8 training speed\n\n**A2**: To the best of our knowledge, there are currently no open-source implementations of INT8 training for transformers in the existing literature. \nMost low-precision training works simply do simulated quantization, without a real implementation to actually realize the speedup.\nThere exists some INT8 training work [1-5] that reports speedup, but none of them are open-source. Particularly, [1-4] focus on INT8 training of CNN-based models, but none of them are open-source, [5] has open-sourced their INT8 training Python code without the actual hardware implementation. \n\n**Q3**: What is the speedup for tasks listed in Table 1?\n\n**A3**: Our implementation cannot yet achieve speedup for tasks listed in Table 1, due to their small model size. However, as argued before, our implementation still has much room for optimization, and implementation is not the main focus of this paper. \n\n**Q4**: How do you compare your method w.r.t. INT8 in terms of training speed? In other words, why should I select Int4 over INT8?\n\n**A4**: Our INT4 training algorithm is superior than an INT8 one regarding to time complexity. \nConsider multiplying a matrix of $N\\times C$ with another matrix of $C\\times D$. Our INT4 MM takes $2NCD$ int4 multiplication-and-additions (MACs), while an INT8 MM takes $2NCD$ int8 MACs. The amount of FP16 operations required by our INT4 MM is $O(2^k N (D+C))$, where a typical value of $k=5$ is the group size. As matrix sizes grow large, the proportion of overhead diminishes, and the speedup of INT4 over INT8 should approach 2x. So we believe that INT4 is ultimately more promising than INT8. The speedup potential might not be realized in this single paper, as discussed earlier.  \n\nAnother reason for favoring INT4 over INT8 lies in the model's inference speed. The model trained by our algorithm can perform INT4 inference, while the model trained by an INT8 algorithm can only use INT8 for inference. At inference time, only the fast HQ-MM operator is used. As reported in Table 4, this leads to a 2.44-3.50 speedup compared to FP16, which should also be faster than INT8. For the inference phase, we can further optimize the implementation by precomputing the process of Hadamard transformation and quantization of the weight and achieving about another 20\\% speed acceleration.\n\n**Q5**: How was the accuracy performance measured? Did you report the accuracy in Table 1 using quantized inference (Int4) or full precision?\n\n**A5**: In Table 1, the \"HQ+LSS\" accuracy is measured using INT4 quantized inference by applying the hadamard quantizer. In the training phase, the weight is quantized to INT4 using our Hadamard Quantizer, so it is suitable for INT4 inference. And in a real application, INT4 inference will bring acceleration. So we report the accuracy using INT4 quantized inference in Table 1.\n\n**Q6**: Please provide more details for Table 3.\n\n**A6**: In Table3, we run distributed training on a system of 8 A100 cards. Our implementation uses distributed data-parallel training with zero-3, gradient checkpointing, and optimizer offloading.\n\nWe experimented with two architectures: BERT-Large and GPT-2. We vary the network width and batch size to make full utilization of the GPU memory.\n\nBy comparing our implementation with the baseline fp16 model, we can see that the end-to-end performance for fine-tuning these models is more efficient in our case, with an up to 35\\% speedup when the parameters are appropriate.\n\nWe are grateful for the time and effort you have invested in evaluating our submission. We are more than willing to address any further questions or concerns you may have.\n\nThe references are listed in the global rebuttal section due to the rebuttal length restriction."}]

Author rebuttal:
Due to the length limit, we have included the references we utilized in the general rebuttal section.

[1] Zhu, Feng, et al. "Towards unified int8 training for convolutional neural network." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[2] Zhao, Kang, et al. "Distribution adaptive int8 quantization for training cnns." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 4. 2021.

[3] Zhang, Xishan, et al. "Fixed-point back-propagation training." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[4] Wang, Shuai, and Yi Kang. "Gradient distribution-aware INT8 training for neural networks." Neurocomputing 541 (2023): 126269.

[5] Zhou, Qihua, et al. "Octo:{INT8} training with loss-aware compensation and backward quantization for tiny on-device learning." 2021 USENIX Annual Technical Conference (USENIX ATC 21). 2021.

[6] Sun, Xiao, et al. "Ultra-low precision 4-bit training of deep neural networks." Advances in Neural Information Processing Systems 33 (2020): 1796-1807.

[7] Chmiel, Brian, et al. "Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats." The Eleventh International Conference on Learning Representations. 2022.

[8] Dettmers, Tim, et al. "Qlora: Efficient finetuning of quantized llms." arXiv preprint arXiv:2305.14314 (2023).

[9] Wortsman, Mitchell, et al. "Stable and low-precision training for large-scale vision-language models." arXiv preprint arXiv:2304.13013 (2023).

[10] Banner, Ron, et al. "Scalable methods for 8-bit training of neural networks." Advances in neural information processing systems 31 (2018).

[11] Wang, Naigang, et al. "Training deep neural networks with 8-bit floating point numbers." Advances in neural information processing systems 31 (2018).


Meta-review: This paper proposes methods to accelerate the training of transformer models by utilizing 4-bit integers and arithmetic. The proposed methods include a Hadamard quantizer for forward propagation to suppress the outliers. In the backward propagation, the paper employs a bit splitting scheme to exploit the sparsity of gradients. The proposed methods are evaluated on transformer models such as BERT and ViT, demonstrating competitive accuracy and speedup on GPUs.

Overall, the paper is well-written. The techniques introduced are innovative, substantiated by a theoretical analysis. The evaluation is reasonably thorough. The proposed algorithm does not necessitate specialized hardware, enabling its implementation on standard GPUs.

The main concern is on the evaluation of hardware performance. Given the current INT4 algorithms introduce noticeable overhead in order to recover the accuracy loss, it aligns with the reviewers’ perspective that a comparison with simple INT8 training is necessary to understand the advantages of the algorithms. It is understandable that a comprehensive demonstration of the speedup is complex and challenging; however, a high-level complexity analysis and the inference speed with I-BERT, as presented in the rebuttal, should be incorporated in the final version with detailed explanation. 

In summary, this paper is technically solid with certain reservations. The authors have, to some extent, addressed the concerns in the rebuttal phase. Consequently, it is recommended that the paper be considered for borderline acceptance.
Decision: Accept (poster)