Title: Training Transformers with 4-bit Integers
Venue: NeurIPS 2023
Accepted: yes
Keywords: neural network quantization

transformer

matrix multiplication

randomized numerical linear algebra
Abstract: Quantizing the activation, weight, and gradient to 4-bit is promising to accelerate neural network training. However, existing 4-bit training methods require custom numerical formats  which are not supported by contemporary hardware. In this work, we propose a training method for transformers with all matrix multiplications implemented with the INT4 arithmetic. 
Training with an ultra-low INT4 precision is challenging. To achieve this, we carefully analyze the specific structures of activation and gradients in transformers to propose dedicated quantizers for them. For forward propagation, we identify the challenge of outliers and propose a Hadamard quantizer to suppress the outliers. For backpropagation, we leverage the structural sparsity of gradients by proposing bit splitting and leverage score sampling techniques to quantize gradients accurately. Our algorithm achieves competitive accuracy on a wide range of tasks including natural language understanding, machine translation, and image classification. Unlike previous 4-bit training methods, our algorithm can be implemented on the current generation of GPUs. Our prototypical linear operator implementation is up to 2.2 times faster than the FP16 counterparts and speeds up the training by 17.8\% on average for sufficiently large models. Our code is available at https://github.com/xijiu9/Train\_Transformers\_with\_INT4.

Training Transformers with 4-bit Integers

Abstract

Quantizing the activation, weight, and gradient to 4-bit is promising to accelerate neural network training. However, existing 4-bit training methods require custom numerical formats which are not supported by contemporary hardware. In this work, we propose a training method for transformers with all matrix multiplications implemented with the INT4 arithmetic. Training with an ultra-low INT4 precision is challenging. To achieve this, we carefully analyze the specific structures of activation and gradients in transformers to propose dedicated quantizers for them. For forward propagation, we identify the challenge of outliers and propose a Hadamard quantizer to suppress the outliers. For backpropagation, we leverage the structural sparsity of gradients by proposing bit splitting and leverage score sampling techniques to quantize gradients accurately. Our algorithm achieves competitive accuracy on a wide range of tasks including natural language understanding, machine translation, and image classification. Unlike previous 4-bit training methods, our algorithm can be implemented on the current generation of GPUs. Our prototypical linear operator implementation is up to 2.2 times faster than the FP16 counterparts and speeds up the training by 17.8% on average for sufficiently large models. Our code is available at https://github.com/xijiu9/Train_Transformers_with_INT4.

1 Introduction

Training neural networks is computationally demanding. Training with low-precision arithmetic (a.k.a., fully quantized training or FQT) is promising to improve computational and memory efficiency. FQT methods add some quantizers and dequantizers in the original full-precision computational graph, and replace expensive floating-point operations with cheap low-precision ones.

Research in FQT aims to reduce the training numerical precision, without sacrificing much convergence speed or accuracy. The required numerical precision has been reduced from FP16 [33] to FP8 [54, 46], INT32+INT8 [3] and INT8+INT5 [7]. FP8 training is implemented in Nvidia's H100 GPU with Transformer Engine [35], achieving impressive speedup for the training of large-scale transformers. Recently, the training numerical precision has been pushed down to 4 bits. Sun et al. [47] successfully trained several modern networks with INT4 activation/weights and FP4 gradients; and Chmiel et al. [8] propose a custom 4-bit logarithmic numerical format to further improve the accuracy. However, these 4-bit training methods cannot be directly utilized for acceleration as they require custom numerical formats that are not supported on contemporary hardware.

There are significant optimization challenges to train neural networks at an extremely low 4-bit level. First, the non-differentiable quantizers in forward propagation make the loss landscape rugged, where gradient-based optimizers can easily stuck at local optima [31]. Second, gradients are only computed

approximately in low-precision. Such imprecise gradients slow down the training process and even cause the training to be unstable or diverge.

In this work, we propose a novel INT4 training algorithm for a class of popular neural networks, transformers [52]. All the costly linear operations for training transformers can be written in a matrix multiplication (MM) form. This MM form allows us to design more flexible quantizers, which better approximate FP32 matrix multiplications by utilizing specific structures of the activations, weights, and gradients in transformers. Our quantizers leverage advances in the field of randomized numerical linear algebra (RandNLA) [14].

For forward propagation, we find that outliers in the activation are the main reason for accuracy degradation. To suppress outliers, we propose a Hadamard quantizer , which quantizes a transformed version of the activation matrix. The transformation is a block diagonal Hadamard matrix, which spreads the information carried in outliers to its nearby entries of the matrix and thus reduces the numerical range of the outliers.

For backpropagation, we exploit the structural sparsity of activation gradients. We find that the gradients of a few tokens are extremely large. Meanwhile, the gradients for the rest majority of the tokens are very small, even smaller than the quantization residuals of larger gradients. Rather than computing these small gradients, it is better to save the computational resources for calculating the residuals of the larger gradients. To utilize such sparsity, we propose bit splitting , which splits the gradient of each token into higher 4 bits and lower 4 bits. Then, we choose the most informative gradients by leverage score sampling , which is an importance sampling technique for RandNLA.

Combining quantization techniques for forward and backward propagation, we propose an algorithm that uses INT4 MMs for all linear operations in transformers. We evaluate our algorithm for training transformers on a wide variety of tasks, including natural language understanding, question answering, machine translation, and image classification. Our algorithm achieves competitive or superior accuracy compared with existing works on 4-bit training [47, 8]. Moreover, our algorithm is compatible with contemporary hardware like GPUs, since it does not require custom numerical formats like FP4 or logarithm formats. Our prototypical quantization + INT4 MM operator implementation is up to 2.2 times faster than the FP16 MM baseline, and it speeds up the training by up to 35.1%.

Finally, we would like to point out that utilizing ultra-low 4-bit numerical formats for training neural networks is still an open problem, and the main purpose of this research is to study whether it is possible to design an INT4 training algorithm that can achieve reasonable accuracy on practical tasks. The current research state of INT4 training is not yet mature enough to provide significant speedup for most tasks in a generic and plug-and-play manner. See Sec. 6 for further discussions.

2 Related Work

Fully Quantized Training Fully quantized training (FQT) [33, 54, 46, 3, 15, 1, 57, 65, 29, 30, 59, 68] methods accelerate training by quantizing the activations, weights, and gradients to low-precision, so linear and nonlinear operators during training can be implemented with low-precision arithmetic. Research on FQT design novel numerical formats and quantization algorithms that better approximate full-precision tensors. The current research frontier is 4-bit FQT. FQT is challenging due to the vast numerical range of the gradient and the optimization issues of training quantized networks from scratch. Due to these challenges, existing 4-bit FQT algorithms [47, 8] still have ∼ 1-2.5% accuracy drop on several tasks, and they cannot support contemporary hardware.

Other Efficient Training Methods Mixture-of-experts [43] improves the model capacity without increasing the training budget. Structural dropout [21, 17] exploits computationally efficient ways to regularize the model. Efficient attention [27, 10] reduces the quadratic time complexity for computing attention. Distributed training systems [39, 22] reduce training time by leveraging more computational resources. Our work on reducing numerical precision is orthogonal with these directions.

3 Forward Propagation

Neural network training is an iterative optimization procedure with stochastic gradients computed by forward and backpropagation. We accelerate forward and back propagation with 4-bit integer

(INT4) arithmetic. We first describe the forward propagation of our training procedure. The forward propagation can be formulated as a composition of linear and non-linear (GeLU, normalization, softmax, etc.) operators. In our training procedure, we accelerate all the linear operators with INT4 arithmetic and leave all the less-computationally-intensive non-linear operators in the 16-bit floatingpoint (FP16) format. All linear operations in transformers can be written in a matrix multiplication (MM) form. For ease of presentation, we consider the acceleration of the following simple matrix multiplication throughout this paper:

The most predominant use case of such MM is the fully-connected layer. Consider a transformer with an input shape of (batch size S , sequence length T , dimensionality D ) . The fully-connected layer can be written as Eq. (1) where X is the activation for N = ST tokens, and W is the weight matrix. For attention layers, batch matrix multiplications (BMMs) might be required. Our proposed techniques can be applied to BMMs, and we leave the discussion of BMMs in Appendix. A.1.

3.1 Learned Step Size Quantization

To accelerate training, the forward propagation must be computed with integer arithmetic. We leverage the learned step size quantizer (LSQ) [16] for this purpose. LSQ is a static quantization method whose quantization scale does not depend on the input, and is thus cheaper than dynamic quantization methods [23], which need to compute the quantization scale dynamically per iteration.

Given a FP matrix X , LSQ quantizes X to integer with

where s X is a learnable scalar parameter, clamp restricts its input to the range [ -Q N , Q P ] , glyph[floorleft]·glyph[ceilingright] is a rounding operation, and X /s X is computed elementwise. The resultant matrix takes values from {-Q N , -Q N +1 , . . . , Q P } . Since we aim to perform INT4 MMs, we set Q N = Q P = 7 . The integer matrix can be dequantized back to FP through float ( int s X ( X )) = s X int s X ( X ) ≈ X .

With LSQ, Eq. (1) can be computed approximately as Y = XW glyph[latticetop] ≈ s X s W int s X ( X ) int s W ( W ) glyph[latticetop] , where the INT4 MM int s X ( X ) int s W ( W ) glyph[latticetop] can be implemented efficiently on hardware.

Remark: Quantization-aware training (QAT) [9, 63, 67, 23, 12, 11, 44, 60, 45, 49, 64, 2, 18, 55] is an inference acceleration technique which trains networks with quantizers inserted in the forward propagation graph, so the trained network can perform efficiently during inference. QAT can compress activation/weights to extremely low precision (e.g. 1-2 bits). It is tempting to think that directly applying a quantizer for QAT to FQT can lead to similar low activation/weights bit-width. However, even only quantizing the forward propagation for FQT is much more challenging than QAT because: (1) QAT requires a converged full-precision model as initialization [16] and/or as a teacher model for knowledge distillation [2]; (2) QAT can adopt expensive multi-stage training pipelines without worrying about the convergence speed [32], while FQT algorithm must converge as fast as fullprecision training algorithms to be useful; (3) QAT may approximate the discrete quantizer with continuous functions during training [19], which cannot be implemented with integer arithmetic. Due to these challenges, it is still an open problem to do FQT with 4-bit activations/weights.

3.2 Activation Outliers

Simply applying LSQ for FQT with 4-bit activation/weights leads to accuracy degradation due to activation outliers [58]. As shown in Fig. 1(a), activations have some outlier entries, which are much larger in magnitude than other entries. In this case, the step size s X poses a trade-off between quantization granularity and representable numerical range. If s X is large, we can represent the outliers well at the expense of representing most other entries in a very coarse manner. On the other hand, if s X is small, we have to truncate the entries outside the range [ -Q N s X , Q P s X ] . Unfortunately, the transformers tend to store information in these outliers, and such truncation would seriously harm accuracy (see Sec. 5.2 for details). The outlier problem is particularly significant when the training task is to fine-tune a pre-trained model on some new downstream tasks, since the pre-train model contains more outliers [58] than random initialization.

There exists some works to handle activation outliers for post-training quantization (PTQ). Outlier Suppression [56] discover that LayerNorms amplify outliers, and propose Gamma Migration and Token-Wise Clipping to solve this issue and achieves 6-bit BERT PTQ without too much degradation. SmoothQuant [58] migrates the quantization difficulty of activation outliers to weights and achieves 8-bit PTQ for large language models, such as OPT-175B. Outlier Channel Splitting [66] duplicates channels containing outliers with small overhead on the size of the network. However, these methods mainly focus on PTQ or QAT, and seldom successfully deal with ultra-low 4-bit training.

3.3 Hadamard Quantization

We propose a Hadamard quantizer (HQ) to solve the outlier problem. Its main idea is to quantize the matrices in another linear space which has fewer outliers.

The outliers in activation matrices form a feature-wise structure [58]. They are typically concentrated on a few dimensions, i.e., only a few columns of X are significantly larger than others. Hadamard transform [48] is a linear transformation, which can amortize the outliers into other entries. Specifically, the Hadamard transform H k is a 2 k × 2 k matrix, where

Hadamard matrices are orthogonal and symmetric: H k = H glyph[latticetop] k = H -1 k , so H k H k = I , ∀ k ≥ 0 . Consider any coordinate row vector * e glyph[latticetop] i ∈ R 2 k , we have e glyph[latticetop] i H k = 2 -k/ 2 1 2 k , ∀ i , where 1 2 k = ( ± 1 , ± 1 , . . . , ± 1) is a 2 k -dimensional vector with all of its values being 1 or -1 . This demonstrates the extreme case when a single outlier dominates all the rest dimensions. In this case, Hadamard transformation effectively turns the vector into a quantization-friendly all-one-vector. The practical effect of the Hadamard transform on suppressing activation outliers is demonstrated in Fig. 1(b).

HQuses a block-diagonal transformation matrix H ∈ R D × D : H = BlockDiag ( H k , . . . , H k ) , where D is a multiple of 2 k . To suppress outliers, we quantize a transformed version of X and W :

Combining the quantized matrices, we get

where the inverse transformations cancel with each other, and the MM can be implemented as:

Procedure HQ-MM

Compute XH and H glyph[latticetop] W glyph[latticetop] in FP16.

Quantize the resultant matrices to INT4 by LSQ.

Multiply the two INT4 matrices.

Dequantize the resultant INT32 matrix to FP16 by multiplying s X s W .

For time complexity, Step 1 takes O (2 k N ( D + C )) FP16 multiply-accumulates (MACs); Step 2

and Step 4 takes O ( N ( D + C )) FP16 MACs in total; and Step 3 takes O ( NDC ) INT4 MACs. Comparing with the plain LSQ Eq. (2), the amount of FP16 MACs increases by 2 k times, from O ( N ( D + C )) to O (2 k N ( D + C )) . However, our HQ-MM is still much cheaper than an FP16 MM given 2 k glyph[lessmuch] D and 2 k glyph[lessmuch] C . The number k shows a tradeoff between the ability to suppress outliers and computation complexity. Larger k allows for amortizing the outlier within a larger horizon, at the cost of being more expensive. We propose an adaptive algorithm to choose k for each activation depending on the outlier scale, as discussed in Appendix A.5. The typical value is k = 5 , while the dimensionality C and D ranges from 768 to 4096.

4 Backpropagation

We now consider accelerating the backpropagation of the linear layer with INT4 operations. The linear operator HQ-MM defined in Eq. (3) has four inputs: activation X , weight W , and step sizes s X , s W . Given the output gradient ∇ Y L w.r.t. some loss function L , we need to compute the gradient of all four inputs. We discuss the computation of activation/weight gradients in this section, and left the discussion of step size gradients to Appendix A.3. For simplicity, we omit L and simply use ∇ Y to denote the gradient in the following text.

By the straight-through estimator glyph[floorleft] x glyph[ceilingright] ′ = 1 [5] and the chain rule, we have

where we define ˆ X = int s X ( XH ) , ˆ W = int s W ( WH ) , I X = I ( -Q N ≤ X /s X ≤ Q P ) , and I W = I ( -Q N ≤ W /s W ≤ Q P ) . For computing the gradients, three types of matrix multiplications are required:

The element-wise multiplication ◦ of a 0 / 1 matrix I X (or I W ) with another INT4 (or INT32) matrix. This operation has low time complexity.

The multiplication of an INT32 matrix with an FP16 block-wise Hadamard matrix s W H glyph[latticetop] , which also has low-time complexity, as discussed in Sec. 3.3.

The multiplication of the FP16 gradient ∇ Y with an INT4 matrix ˆ X or ˆ W , which we will accelerate by quantizing ∇ Y to INT4.

In the rest of this section, we will discuss quantization methods to compute the 'type 3' MMs ∇ glyph[latticetop] Y ˆ X and ∇ Y ˆ W . We quantize ∇ Y dynamically for each MM, while ˆ X and ˆ W have been already calculated in forward propagation in Section. 3. We start by discussing the structure of the gradient.

4.1 Structural Sparsity of Gradients

We note that the gradient matrix ∇ Y tends to be very sparse along the training process. Furthermore, the sparsity has a structure: few rows (i.e., tokens) of ∇ Y have large entries, while most other rows are close to an all-zero vector. We illustrate this by plotting the histogram of per-row norm ‖ ( ∇ Y ) i, : ‖ for all the rows i in Fig. 2.

Such a structural sparsity arises from the heavy overparameterization [62] of modern neural networks. During almost the entire training process, the network operates in the overparameterized scheme [34], where it can fit most training data well, except for a few hard examples. Therefore, the (activation) gradient will be close to zero for well-fitted data points. We find that for pretraining tasks, such structural sparsity quickly emerges after only a few training epochs. For fine-tuning tasks, the gradient is always sparse during the whole training process.

4.2 Bit Splitting and Leverage Score Sampling

Here, we discuss how to design gradient quantizers to accurately compute the MMs during backpropagation by leveraging structural sparsity. The high-level idea is that many rows of the gradient are so small that they have little impact on the parameter gradient, yet they waste abundant computation. On the other hand, the large rows cannot be accurately represented with INT4. We drop some small rows and use the saved computation to represent large rows more accurately.

First, we propose bit splitting (BS), which splits a full-precision matrix as higher and lower 4 bits:

where s ↑ , s ↓ are two floating-point scalars, and ∇ ↑ Y , ∇ ↓ Y are INT4 matrices representing the higher and lower 4 bits, respectively. BS can be implemented by first quantizing ∇ Y to INT4 as ∇ Y ≈ s ↑ ∇ ↑ Y and then quantize the residual to INT4 as ∇ Y -s ↑ ∇ ↑ Y ≈ s ↓ ∇ ↓ Y . BS can be viewed as an INT8 representation of a matrix, where ∇ ↑ Y and ∇ ↓ Y are the higher and lower 4 bits of the INT8 representation. Next, we discuss how to compute the weight and activation gradient.

Weight Gradient As discussed earlier, weight gradient involves the matrix multiplication ∇ glyph[latticetop] Y ˆ X , where ∇ Y ∈ R N × C and ˆ X is an N × D INT4 matrix. By Eq. (5):

where we define ∇ glyph[arrowbothv] Y = [ s ↑ ∇ ↑ Y ; s ↓ ∇ ↓ Y ] glyph[latticetop] ∈ R 2 N × C and ˆ X glyph[arrowbothv] = [ ˆ X ; ˆ X ] to be a 2 N × D INT4 matrix. Eq. (6) represents the product of an INT8 ∇ glyph[latticetop] Y and an INT4 ˆ W , and can be implemented by two INT4 MMs ∇ ↑ Y glyph[latticetop] ˆ X and ∇ ↓ Y glyph[latticetop] ˆ X . Such MM is rather accurate since ∇ Y is represented with 8 bits.

However, comparing to a naïve quantization of ∇ Y to INT4, BS doubles the amount of INT4 operations for MM. We propose leverage score sampling (LSS) to cut the operations of Eq. (5) by half, to the same amount as the naïve MM s ↑ ∇ ↑ Y ˆ X . Noticing that the MM Eq. (6) can be written as the sum of 2 N rank-1 matrices:

where ∇ W i = ∇ glyph[arrowbothv] Y glyph[latticetop] : ,i X glyph[arrowbothv] i . Due to the sparsity of ∇ Y , the matrices ∇ W i differ in magnitude and small matrices can be discarded without having a big influence on the result.

Our proposed LSS assigns each ∇ W i a probability p i ∈ [0 , 1] , i = 1 , · · · , 2 N , that satisfies ∑ 2 N i =1 p i = N . We define random masks m i ∼ Bern ( p i ) and mask matrix ˜ M , and approximate it as

which is an unbiased approximation since E [ ∇ glyph[arrowbothv] Y glyph[latticetop] ˜ MX glyph[arrowbothv] ] = ∇ glyph[arrowbothv] Y glyph[latticetop] E [ ˜ M ] X glyph[arrowbothv] = ∇ glyph[arrowbothv] Y glyph[latticetop] X glyph[arrowbothv] .

In expectation, there are only N nonzero m i s. Therefore, LSS reduces the cost of MM by half. For LSS to be accurate, we minimize its variance. We have:

Proposition 4.1. (LSS variance for weight gradient)

The coefficient c i := ‖∇ glyph[arrowbothv] Y i, : ‖‖ X glyph[arrowbothv] i, : ‖ is called the leverage score , which can be easily computed in low time complexity. When p i ∝ c i , the variance attends its minimum due to Cauchy inequality:

where the equality holds when p i ∝ c i . Intuitively, LSS can approximate the MM Eq. (7) well with significantly lower computational cost when the leverage scores { c i } are diverse, which is indeed the case as shown in Fig. 2.

Define M ↑ to be the top-left N × N submatrix of ˜ M and M ↓ to be the bottom-right one, we have

which can be implemented by two INT4 MMs with sampled rows/columns. Putting everything together, we propose the following MM procedure to compute the weight gradient:

Procedure LSS-MM

Quantize ∇ Y with BS to obtain ∇ ↑ Y and ∇ ↓ Y in INT4.

Compute the leverage score ‖∇ glyph[arrowbothv] Y i, : ‖‖ X glyph[arrowbothv] i, : ‖ in FP16.

Sample the masks { m i } .

Sample rows of ∇ Y and ˆ X given the masks { m i } .

Compute INT4 MMs ∇ ↑ Y glyph[latticetop] ˜ M ↑ ˆ X and ∇ ↓ Y glyph[latticetop] ˜ M ↓ ˆ X ,

Dequantize and sum up the resultant INT32 matrices to obtain the FP16 result ∇ glyph[arrowbothv] Y glyph[latticetop] ˜ MX glyph[arrowbothv] .

As ˜ M only has N non-zero elements in expectation, the two matrix multiplications in Step 5 take about 2 NCD INT4 MACs, which aligns with the cost of the naïve MM s ↑ ∇ ↑ Y ˆ X . The overhead of all the other steps is O ( NC + ND ) in total.

Activation Gradient Similar to the previous discussion, the gradient of input can be written as

where we define ∇ glyph[arrowbothv] Y = [ s ↑ ∇ ↑ Y ; s ↓ ∇ ↓ Y ] ∈ R 2 N × C and ˆ I glyph[arrowbothv] = [ I I ] to be a N × 2 N INT4 matrix, I is a N × N identity matrix. The original product can also be implemented by two INT4 MMs ∇ ↑ Y ˆ W

and ∇ ↓ Y ˆ W . But different from weight gradients, we now focus on ˆ I glyph[arrowbothv] ∇ glyph[arrowbothv] Y in Eq. (8) and do leverage score sampling on this MM. A detailed discussion can be found in Appendix B.2, and we only present the leverage score here. Similarly, we write the MM as the sum of 2 N smaller multiplications:

where we define ∇ Y i = ˆ I glyph[arrowbothv] : ,i ∇ glyph[arrowbothv] Y i and associate the probability p i and Bernoulli mask m i ∼ Bern ( p i ) with the i multiplication. The leverage score for activation gradient is c i := ‖∇ glyph[arrowbothv] Y i ‖ , and the variance attains minimum when p i ∝ c i . More details about the algorithm can be found at Appendix. A.3 On the implementation side, once the mask { m i } is known, we can decompose the MM Eq. (8) as two INT4 MMs: ( ˆ I glyph[arrowbothv] ˜ M ∇ glyph[arrowbothv] Y ) ˆ W = s ↑ ˜ M ↑ ∇ ↑ Y ˆ W + s ↓ ˜ M ↓ ∇ ↓ Y ˆ W .

5 Experiments

We evaluate our INT4 training algorithm on a wide variety of tasks including language model fine-tuning, machine translation, and image classification. We implement our proposed HQ-MM and

LSS-MM algorithms with CUDA and cutlass † , and the implementation details can be found in Appendix A. We replace all the floating-point linear operators with our INT4 implementation except simply using LSQ for embedding layers, and leaving the last classifier layer in full precision. We adopt default architectures, optimizers, schedulers, and hyper-parameters for all the evaluated models.

5.1 Converged Model Accuracy

We compare the accuracy of the converged model on various tasks in Table 1. The compared methods include full-precision training (FP), INT8 training [3](INT8), FP4 training [47] ('Ultra-low'), 4-bit logarithm quantization [8] with LSQ for activations and weights (LSQ+LUQ), and our algorithm which utilizes HQ for forward and LSS for backpropagation (HQ+LSS). Ultra-low does not have a public implementation, so we only report its performance from its original paper on the machine translation task. Except for the large machine translation task and the task of large vision transformers, we repeat each run by three times and report the standard deviation as subscripts in tables. We do not include any kind of knowledge distillation or data augmentation.

Language model fine-tuning: We use the pretrained BERT-base-uncased and BERT-largeuncased [24] model, and evaluate the performance of our method on GLUE dev-set [53], SQUAD [41], SQUADv2 [40], Adversarial QA [4], CoNLL-2003 [42] and SWAG [61] datasets. We present the average result of the bert-base-uncased and bert-large-uncased models on the GLUE dataset. The full results are listed in Appendix C.2. Compared with LSQ+LUQ, our method achieves 5 . 5% improvement of accuracy on average for the bert-base model and achieves > 25% improvement of accuracy on average for the bert-large model. We further show the result on the SQUAD, SQUAD 2.0, Adversarial QA, CoNLL-2003, and SWAG datasets. On all of the tasks, compared with LSQ+LUQ, our method achieves better performance. We improve by 1 . 8% and 3 . 6% on SQUAD and SQUAD 2.0 compared to LSQ+LUQ, respectively. On the more difficult Adversarial QA, we improve by 6 . 8% on F1 score. On SWAG we improve by 6 . 7% and on CoNLL-2003 we improve by 4 . 2% accuracy.

Machine translation: We also apply our method for pretraining. We train a Transformer-base [52] model on WMT 14 En-De dataset [6] for machine translation. Note that we reproduce this experiment with Fairseq's recipe ‡ , which reports the SacreBleu score (26.5 for FP) [37], while Ultra-low and LUQ report the more optimistic original BLEU score (27.5 for FP) [36]. Our HQ+LSS has about 1 . 0% BLEU degradation, which is smaller than 2 . 1% of Ultra-low and higher than 0 . 3% reported in the LUQ paper. Nevertheless, HQ+LSS still performs comparably with existing methods for this pretraining task, and it supports contemporary hardware.

Image Classification: We load ViT checkpoints pretrained on ImageNet21k [13], and fine-tune it on CIFAR-10, CIFAR-100 [28], and ImageNet1k. We use ViT-B/32 and ViT-L/32 for CIFAR datasets and use ViT-B/32, ViT-L/32 and ViT-L/16 for ImageNet1k. On CIFAR10 we achieve < 0 . 5% accuracy degradation, while LSQ+LUQ has 1% degradation for ViT-B/32 and 0 . 6% degradation for ViT-L/32. On CIFAR100, INT8 already has ∼ 1% accuracy degradation, which shows its difficulty. We improve by 1 . 1% accuracy for ViT-B/32 and 0 . 2% accuracy for ViT-L/32 compared with LSQ+LUQ. On ImageNet1k, we improve by 2% accuracy for ViT-B/32, 2 . 6% accuracy for ViT-L/32 and 0 . 2% for ViT-L/32 compared with LSQ+LUQ. We further test the effectiveness of our algorithm for pretraining a DeiT-Small model [51] on ImageNet1K, where HQ+LSS can still converge to similar accuracy level compared to LSQ+LUQ, while being more hardware friendly.

5.2 Ablation Study

Here, we conduct ablation studies to show the effectiveness of our forward and backward methods independently on the challenging CoLA dataset. To study the effectiveness of different quantizers for forward propagation, we leave backpropagation in FP16. The result is shown in Fig. 3(a). We first validate the claim in Sec. 3.2 that outliers are the main cause of accuracy degradation in quantized forward propagation. We test an 'outlier' method which maintains 1% largest activation entries in FP. The 'outlier' method achieves good performance, which proves that outliers are indeed the most

significant challenge of the transformer's forward quantization. The hardware-unfriendly 'outlier' method serves as an upper bound of methods to handle outliers. Our HQ outperforms LSQ by better handling the outliers and achieves comparable results to maintaining the outliers.

We also investigated whether more granular quantizers, such as per-token quantization or per-channel quantization could be used to quantify outliers, or whether existing methods like SmoothQuant [58] could be used for INT4 FQT. The results are listed in Appendix C.3, and we find that without HQ, none of these methods achieve good accuracy under 4-bit quantization, and the result of HQ is not strongly affected when more granular quantization methods are applied.

For backpropagation, we compare a simple minimax quantizer [3], LUQ [8] and our LSS, and leave forward propagation in FP16. The minimax quantizer divides the numerical range from the minimum to the maximum into equally large quantization bins. The result is shown in Fig. 3(b). While the bit-width is higher than 2, our LSS achieves results that are comparable and even slightly higher than LUQ. Meanwhile, LSS is more hardware friendly as it requires only INT4 arithmetic.

5.3 Computational and Memory Efficiency

Finally, we demonstrate the potential of our method to accelerate neural network training by evaluating our prototypical implementation discussed in Appendix A.6. We emphasize that our implementation is not fully optimized. For example, the backward computation requires an INT4 MM in the form of Y = AB , while cutlass only supports Y = AB glyph[latticetop] , so explicit transpose is required. We also do not fuse the linear operators with nonlinearities and normalizations. Therefore, the results cannot fully reflect the potential of INT4 training algorithms. A fully optimized implementation requires heavy engineering, which exceeds the scope of our paper.

Operator Speed: We compare the throughput of our proposed HQ-MM (HQ), LSS for computing weight gradient (LSSWeight), LSS for computing activation gradient (LSSAct), and their average throughput (INT4) with a baseline tensor-core FP16 GEMM implementation (FP16) provided by cutlass in Fig. 4 on an Nvidia RTX 3090 GPU which has a peak throughput at 142 FP16 TFLOPs and 568 INT4 TFLOPs. As the matrix size grows, the overhead of quantization diminishes and our INT4 operators can be up to 2.2 times faster compared with FP16 MM. We further analyze the quantization overhead for each operator in Appendix C.5.

Training Throughput: We compare the training throughput of the FP16 PyTorch AMP and our INT4 training algorithm for training BERT [24] and GPT [38]-style language models on a system of 8 Nvidia A100 GPUs. We vary the hidden layer size, intermediate fully-connected layer size, and batch size, and plot the speedup of INT4 training in Fig. 5. Our INT4 training algorithm can achieve up to 35.1% speedup and an average of 15.8 % for BERT-style models and up to 26.5% speedup and an average of 19.2 % for GPT-style models. The training time can be found in Appendix C.4.

Inference Speed: We compare the inference speed of our algorithm with I-BERT [25] by comparing the speedup numbers reported in its original paper. Following the I-BERT paper, we compare the speedup of integer-based inference algorithms relative to a FP32 baseline on an Nvidia T4 GPU on the BERT-base and BERT-large models, and test with sequence lengths of 128 and 256. While I-BERT only reported speedup numbers for batch sizes 1, 2, 4, 8, we test the speedup for batch sizes

ranging from 1 to 1024 to better reflect the performance of throughput-oriented scenarios (such as a cloud language model service provider).

In Table 2 we report the speed up result. While I-BERT's speedup numbers seems to be insensitive to the batch size and sequence length, our speedup increases with the batch size. I-BERT shows up to 3.98x speedup for smaller batch size, while our algorithm can achieve higher speedup for batch sizes higher than 64, and eventually gives a speedup of 6.48x for a sequence length of 128 and a batch size of 1024.

For the BERT-base model, our method shows an inference speed improvement of 3.57-4.92 times compared to FP32 for batch sizes larger than 64. In comparison, I-BERT achieved an inference speed improvement of 2.42-3.39 times compared to FP32 when the batch size was small. For BERT-large, our method shows an inference speed improvement of 4.81-6.48 times compared to FP32 for batch sizes larger than 64. In comparison, I-BERT achieved an inference speed improvement of 3.20-4.00 times compared to FP32 when the batch size is small. Therefore, our algorithm can potentially achieve higher throughput than I-BERT.

6 Conclusions

We propose a hardware-friendly INT4 training method for transformers. By analyzing the properties of MMs in transformers, we propose HQ and LSS methods to quantize activations and gradients while preserving accuracy. On several important tasks, our method performs comparably or better than existing INT4 methods. Our work can be potentially extended beyond transformers to other MM-only architectures, such as MLP-Mixer [50], graph neural networks [26], and recurrent neural networks [20]. We leave it as a future direction.

Broader Impacts: Our algorithm can improve efficiency and reduce the energy consumption of training neural networks, which helps reduce the carbon footprint caused by deep learning. However, our efficient training algorithm might also facilitate the development of large language models with safety concerns for human beings; and malicious AI applications such as fake content generation.

Limitations: Our INT4 training algorithm does not support convolutional layers. Moreover, our algorithm cannot yet work well for those extremely large models such as OPT-175B. LSS utilizes gradient sparsity, and may not work well for certainty pretraining tasks. Finally, our algorithm may incur more memory accesses than FP16 training algorithms, which might affect the speedup.

References

Menachem Adelman and Mark Silberstein. Faster neural network training with approximate tensor operations. arXiv preprint arXiv:1805.08079 , 2018.

Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.15701 , 2020.

Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In Advances in Neural Information Processing Systems , pages 5145-5153, 2018.

Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. Beat the ai: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics , 8:662-678, 2020.

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013.

Ondˇ rej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation , pages 12-58, 2014.

Jianfei Chen, Yu Gai, Zhewei Yao, Michael W Mahoney, and Joseph E Gonzalez. A statistical framework for low-bitwidth training of deep neural networks. In Advances in neural information processing systems , 2020.

Brian Chmiel, Ron Banner, Elad Hoffer, Hilla Ben Yaacov, and Daniel Soudry. Logarithmic unbiased quantization: Practical 4-bit training in deep learning. arXiv preprint arXiv:2112.10769 , 2021.

Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085 , 2018.

Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations , 2020.

Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. arXiv preprint arXiv:1911.03852 , 2019.

Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. ICCV , 2019.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Petros Drineas and Michael W Mahoney. Randnla: randomized numerical linear algebra. Communications of the ACM , 59(6):80-90, 2016.

Mario Drumond, LIN Tao, Martin Jaggi, and Babak Falsafi. Training dnns with hybrid block floating point. In Advances in Neural Information Processing Systems , pages 453-463, 2018.

Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. Learned step size quantization. arXiv preprint arXiv:1902.08153 , 2019.

Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations , 2019.

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412 , 2020.

Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4852-4861, 2019.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision , pages 646-661. Springer, 2016.

Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems , 32, 2019.

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2704-2713, 2018.

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT , pages 4171-4186, 2019.

Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-bert: Integer-only bert quantization. In International conference on machine learning , pages 55065518. PMLR, 2021.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations , 2019.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report , 2009.

Hamed F Langroudi, Zachariah Carmichael, and Dhireesha Kudithipudi. Deep learning training on the edge with low-precision posits. arXiv preprint arXiv:1907.13216 , 2019.

Hamed F Langroudi, Zachariah Carmichael, David Pastuch, and Dhireesha Kudithipudi. Cheetah: Mixed low-precision hardware & software co-design framework for dnns on the edge. arXiv preprint arXiv:1908.02386 , 2019.

Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, and Kwang-Ting Cheng. How do adam and training strategies help bnns optimization. In International Conference on Machine Learning , pages 6936-6946. PMLR, 2021.

Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16 , pages 143-159. Springer, 2020.

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations , 2018.

Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment , 2021(12):124003, 2021.

Nvidia. Transformer Engine. https://github.com/NVIDIA/TransformerEngine , 2023. Online; accessed 23 January 2023.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311-318, 2002.

Matt Post. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771 , 2018.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9, 2019.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1-16. IEEE, 2020.

Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822 , 2018.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 , 2016.

Erik F Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Languageindependent named entity recognition. arXiv preprint cs/0306050 , 2003.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 , 2017.

Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. arXiv preprint arXiv:1909.05840 , 2019.

Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 8815-8821, 2020.

Xiao Sun, Jungwook Choi, Chia-Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalakshmi Viji Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In Advances in Neural Information Processing Systems , pages 4901-4910, 2019.

Xiao Sun, Naigang Wang, Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi Viji Srinivasan, and Kailash Gopalakrishnan. Ultra-low precision 4-bit training of deep neural networks. In Advances in Neural Information Processing Systems , volume 33, 2020.

James Joseph Sylvester. Lx. thoughts on inverse orthogonal matrices, simultaneous signsuccessions, and tessellated pavements in two or more colours, with applications to newton's rule, ornamental tile-work, and the theory of numbers. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science , 34(232):461-475, 1867.

Hanlin Tang, Xipeng Zhang, Kai Liu, Jianchen Zhu, and Zhanhui Kang. Mkq-bert: Quantized bert with 4-bits weights and activations. arXiv preprint arXiv:2203.13483 , 2022.

Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlpmixer: An all-mlp architecture for vision. Advances in neural information processing systems , 34:24261-24272, 2021.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning , pages 10347-10357. PMLR, 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 , 2018.

Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. In Advances in Neural Information Processing Systems , pages 7675-7684, 2018.

Zheng Wang, Juncheng B Li, Shuhui Qu, Florian Metze, and Emma Strubell. Squat: Sharpnessand quantization-aware training for bert. arXiv preprint arXiv:2210.07171 , 2022.

Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. arXiv preprint arXiv:2209.13325 , 2022.

Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. In International Conference on Learning Representations , 2018.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438 , 2022.

Yukuan Yang, Lei Deng, Shuang Wu, Tianyi Yan, Yuan Xie, and Guoqi Li. Training highperformance and large-scale deep neural networks with full 8-bit integers. Neural Networks , 125:70-82, 2020.

Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS) , pages 36-39. IEEE, 2019.

Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.05326 , 2018.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107115, 2021.

Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. LQ-Nets: Learned quantization for highly accurate and compact deep neural networks. In The European Conference on Computer Vision (ECCV) , September 2018.

Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. Ternarybert: Distillation-aware ultra-low bit bert. arXiv preprint arXiv:2009.12812 , 2020.

Xishan Zhang, Shaoli Liu, Rui Zhang, Chang Liu, Di Huang, Shiyi Zhou, Jiaming Guo, Yu Kang, Qi Guo, Zidong Du, et al. Adaptive precision training: Quantify back propagation in neural networks with fixed-point numbers. arXiv preprint arXiv:1911.00361 , 2019.

Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. In International conference on machine learning , pages 7543-7552. PMLR, 2019.

Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: Towards lossless cnns with low-precision weights. International Conference on Learning Representations , 2017.

Feng Zhu, Ruihao Gong, Fengwei Yu, Xianglong Liu, Yanfei Wang, Zhelong Li, Xiuqi Yang, and Junjie Yan. Towards unified int8 training for convolutional neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1969-1979, 2020.

A Implementation Details

In this section, we present some works that need to be done to actually accelerate the training process on hardware.

A.1 BMMin Attention

In attention, there are batch matrix multiplications (BMMs) that need to be dealt with. We now show that our method for MMs can be extended to BMMs.

Consider the following BMM product:

where we define T ∈ R B × N × P , Q ∈ R B × N × M , K ∈ R B × P × M . The Hadamard matrix is defined as :

where ˆ H ∈ R B × M × M , H ∈ R M × M , H k ∈ R 2 k × 2 k . In this case,

which verifies that our HQ can be applied to BMMs.

For backward, the gradient of weight and activation can be calculated by the straight-through estimator glyph[floorleft] x glyph[ceilingright] ′ = 1 and the chain rule:

where we define s Q ∈ R B , s k ∈ R B being the batch step size, ˆ K = int s K ( BMM ( K , ˆ H ) ) ˆ Q = int s Q ( BMM ( Q , ˆ H ) ) , I Q = I ( -Q N ≤ Q /s Q ≤ Q P ) , and I K = I ( -Q N ≤ K /s K ≤ Q P ) .

,

Similar to Sec. 4.2, we only focus on BMM ( ∇ glyph[latticetop] T , ˆ K ) and ∇ T , since we do leverage sampling on them.

For BMM ( ∇ glyph[latticetop] T , ˆ K ) , we define the sample probability p i and sample the ˜ M in the same way as MMs. The matrix can be computed as BMM ( BMM ( ∇ glyph[arrowbothv] T glyph[latticetop] , ˆ ˜ H ) , ˆ K glyph[arrowbothv] ) , where ˆ ˜ H is defined as CONCAT ( ˜ H 1 , · · · , ˜ H B ) , ∇ glyph[arrowbothv] T glyph[latticetop] and ˆ K glyph[arrowbothv] follows the same definition of Eq. 6and the leverage score is c b,i := ‖∇ glyph[arrowbothv] T b,i, : ‖‖ K glyph[arrowbothv] b,i, : ‖ for 0 ≤ b ≤ B, 0 ≤ i ≤ 2 M.

For ∇ T , similarly, can be viewed as ∇ T = BMM ( ˆ I glyph[arrowbothv] , ∇ glyph[arrowbothv] T ) , where we define ∇ glyph[arrowbothv] Y = CONCAT ([ s ↑ b ∇ ↑ T b ; s ↓ b ∇ ↓ T b ]) ∈ R B × 2 N × P , ˆ I glyph[arrowbothv] = CONCAT ([ I I ]) ∈ R B × N × 2 N , s ↑ b , ∇ ↑ T b , s ↓ b , ∇ ↓ T b follows the definition of Eq.5. So it can be computed as BMM ( BMM ( ˆ I glyph[arrowbothv] , ˆ ˜ H ) , ∇ glyph[arrowbothv] T ) , where ˆ ˜ H is defined as CONCAT ( ˜ H 1 , · · · , ˜ H B ) , and the leverage score is c b,i := ‖∇ glyph[arrowbothv] T b,i, : ‖ for 0 ≤ b ≤ B, 0 ≤ i ≤ 2 M, which verifies that our LSS can be applied to BMM.

A.2 Computing Leverage Score

In the previous discussion, we find the optimal sample probability p i that can minimize the variance of the gradient. However, it is likely for the proportional p i is larger than one, which is invalid for the Bernoulli distribution. Accordingly, we propose an algorithm to solve this issue.

Define the probability array as

we first clamp the array to p 1 i ∈ [0 , 1] . In this case, ∑ 2 N i =1 p 1 i ≤ N , so we scale the p i which is smaller than 1 to make sure their sum is again N . However, this will probably introduce some more elements larger than 1, so we cycle through the above operations until all the p i ∈ [0 , 1] . This process will certainly stop, since if after the scaling operation, no element is larger than 1, then we get a valid distribution. Otherwise, the number larger than 1 is reduced by at least one, thus the process will halt after at most O ( N ) times.

A.3 Learning Quantizer Parameters

In this section, we discuss the detail of how to calculate the gradient of activation and quantization step size.

For gradient of activation, the coefficient c i := ‖∇ glyph[arrowbothv] Y i ‖ is the leverage score for activation gradient, and the variance achieves its minimum When p i ∝ c i by the Cauchy Inequality.

Putting everything together, we propose the following MM procedure to compute activation gradient:

Procedure LSS-MM

Quantize ∇ Y with BS to obtain ∇ ↑ Y and ∇ ↓ Y in INT4.

Compute the leverage score ‖∇ glyph[arrowbothv] Y i ‖ in FP16.

Sample the masks { m i } .

Sample rows of ∇ Y given the masks { m i } .

Compute I ˜ M ↑ ∇ ↑ Y and I ˜ M ↓ ∇ ↓ Y by discard some of its rows.

Compute INT4 MMs I ˜ M ↑ ∇ ↑ Y ˆ W and I ˜ M ↓ ∇ ↓ Y ˆ W .

Dequantize and sum up the resultant INT32 matrices to obtain the FP16 result ˆ I glyph[arrowbothv] ∇ glyph[arrowbothv] Y ˆ W .

The two matrix multiplications in Step 5 take about 2 NCD INT4 MACs in expectation.

For the quantization step sizes. Following the chain rule, we have

where we define g ( s W ) = 1 / √ Q p N W , g ( s X ) = 1 / √ Q p N X , N W and N X being the number of elements of weight and activation, δ X ( s X ) = int s X ( X ) -I X ◦ ( X /s X ) , and δ W ( s W ) = int s W ( W ) -I W ◦ ( W /s W ) .

Notice that for computing ∇ s W and ∇ s X , the most expensive MMs are ∇ glyph[latticetop] Y ˆ X and ∇ Y ˆ W , which are already calculated through Eq. (7) and Eq. (8) during previous calculations, so it does not require extra computation. The elementwise multiplication with δ X ( s X ) and δ W ( s W ) requires minor computation.

A.4 Cold Start Problem

There is a cold start problem . When the model is trained from scratch (i.e., from a random initialization), distributions of weights and activations can change rapidly in the early stage of optimization. In this case, jointly optimizing the quantization step size and the weights would cause the training to be unstable. As a remedy, we do not learn the step size in the first few iterations, and use a heuristic rule to dynamically set the step size for each tensor X to 2 mean ( X ) / √ Q p in each iteration. For the WMTexperiment, we stop to re-initialize at the second epoch, since the BLEU score is relatively high at the second epoch (about 21, while the final BLEU score after 30 epochs is 25.5). However, for the deit-small pretraining experiment, we re-initialize it at the end of the 10th epoch (we train it for 90 epochs in total), since at the end of the second epoch, the accuracy is not high, but at the end of the 10th epoch, the accuracy is relatively high.

A.5 Choose hadamard matrix size

For the hadamard matrix, let the hadamard matrix to be H ∈ R D × D : H = BlockDiag ( H k , . . . , H k ) , where D is a multiple of 2 k . We first define

where ¯ X and ¯ W can be viewed as an approximation of X and W . Then, we define the quantization error to be MSE ( ¯ X , X ) × MSE ( ¯ W , W ) . We search for the optimal k that can minimize this quantization error. For fine-tuning tasks, once the hadamard matrix size has been calculated, we fix it through the training process. For the pre-training task, since the distribution shifts greatly as we train the model, we empirically define a time when we re-initialize the hadamard matrix size and the LSQ step size. Usually, we do this when the first 2 epochs finish.

A.6 GPU Implementation

In the previous discussion, we get to know HQ-MM and LSS-MM from an algorithm level, nevertheless it is not enough to actually implement it on hardware. In this section, we will delve deeper into hardware implementation details as well as extra limitations.

HQ-MM can be divided into 5 parts: Hadamard matrix multiplication, Quantize, Data Pack, INT4 GEMM, and Dequantize.

For the Hadamard matrix multiplication process, since it can be interpreted as a half float matrix multiplication process where the two matrices involved in the operation are input/weight matrix and hadamard matrix, respectively, we implement it in Python, because PyTorch MM uses CublassGemm and is more efficient then CutlassGemm.

In the quantize process, we quantize input/weight into INT4 data respectively, and also preserve a corresponding FP16 version for the LSQ Back Propagation process to use.

In the previous discussion, we assume the quantize part of HQ-MM is quantizing the resultant matrices to INT4, however, the smallest representation unit of data is INT8. As a result, we actually use INT8 data type to represent quantized data and pack two adjacent data into one data using ( data [1] << 4) | ( data [0]&15) in the data packing process, which means we use one INT8 data to represent two adjacent INT4 data. With both input matrices' data packed in this way, we then use cutlass tensor-core INT4 GEMM to do the matrix multiplication.

For the GEMM process, we choose Nvidia CutlassGemm because it's the most efficient open-source operator library we can find. We use INT4 Tensor Core Gemm for our implementation and it requires the two input matrices A&B to be RowMajor and ColMajor, respectively. Since the default Pytorch tensor is RowMajor, we have to use Transpose+Contiguous operations to make it ColMajor, which is very time-consuming and needs further optimization in the future.

Finally, we dequantize the INT GEMM result back into FP16 output using a dequantize kernel, which is the final output of the forward kernel.

As compared, LSS-MM is more complicated, and can be divided into 7 parts: Quantization of higher lower 4-bit, Leverage Score Calculating, Sampling, Data Pack, INT4 GEMM, Dequantize, and LSQ Back Propagation.

In the Quantize process, we fuse the quantize operation of higher 4-bit and lower 4-bit into a single kernel for acceleration. In the Leverage Score Calculating process, we use the quantized INT8 data to calculate the score and scale up it in the final because integer arithmetic is far more efficient than float arithmetic.

In the sampling process, we sample out rows/columns given the previously calculated leverage score. Note that in Section. A.2, we repeat our proposed algorithm for several loops to sample out specific elements, which is effective but not efficient. According to experiments, however, we notice that simply selecting elements whose leverage score is bigger than 0 can also work well, even better than our proposed algorithm in some cases. So in real quantization implementation, we just sample out rows/ columns whose Euclidean norm is bigger than 0 to accelerate our training process.

Pack, Gemm, and Dequantize processes are as similar as before. It's worth noting that for Int4 Tensor Core Gemm, suppose two input matrices have shape M × K and K × N , K needs to be a multiple of 32 so that the Tensor core Gemm address can be aligned. We do not need to consider this in the Forward Propagation process because the input data shape always satisfies. However, in the Back Propagation process, the matrix shape may not meet the requirement after sampling. As a result, we need zero_padding the sampled matrix so that K can be a multiple of 32.

Finally, we utilize the dequantized data to do the LSQ Back Propagation. We also fuse all operations into a single Cuda kernel for acceleration, and the metric remains.

Besides the component of HQ-MM and LSS-MM , there is still something that needs to be mentioned.

We omit the Quantization and Leverage Score Calculating process in LSSinput, and use the same value as LSSWeight to accelerate the training process.

For Element-Wise kernel, we set block size as 256, grid size as input.numel()/256. For Reduction kernels like sum and min/max, we set block size as 32, grid size as RowNum, reducing elements in each row to the first 32 elements. We find this setting to be most efficient through experiments.

B Proofs.

In this section, we present the proofs of the leverage score.

B.1 Proof of Proposition. 4.1

Proposition B.1. (LSS variance for weight gradient)

Proof.

So that which proves.

B.2 Proof of Activation Leverage Score in Sec. 4.2

we divide the matrix multiplication into the sum of 2 N smaller multiplications:

where we define ˆ ∇ Y i = ˆ I glyph[arrowbothv] : ,i ∇ glyph[arrowbothv] Y i .

We assigns each ∇ Y i a probability p i ∈ [0 , 1] , i = 1 , · · · , 2 N , that satisfies ∑ 2 N i =1 p i = N . We define random masks m i ∼ Bern ( p i ) , and define ˜ M = diag ( m 1 p 1 , . . . , m 2 N p 2 N ) , and make an unbiased estimation:

Define M ↑ to be the top-left N × N submatrix of M and M ↓ to be the bottom-right one, we have

In this case, I ˜ M ↑ ∇ ↑ Y and I ˜ M ↓ ∇ ↓ Y both only have parts of its rows being non zero, and the rest rows are zeros since they are discarded. Then, when we multiply it by ˆ W , there are half of rows being zeros in I ˜ M ↑ ∇ ↑ Y ˆ W and I ˜ M ↓ ∇ ↓ Y ˆ W . So there's no need to calculate them, and we successfully cut off half of the computation in this case.

Now focus on the variance that

Proposition B.2. (LSS variance for activation gradient)

Proof.

In this way, the coefficient c i := ‖∇ glyph[arrowbothv] Y i ‖ is the leverage score .

C Experiments.

In this section, we present more details for experiments in Sec. 5.

C.1 Experiments setup

For the GLUE, QA, SWAG, and CONLL tasks, we implement our algorithm based on https: //github.com/huggingface/transformers . For the machine translation task, we implement our algorithm based on https://github.com/facebookresearch/fairseq . For the ViT fine-tuning task, we implement our algorithm based on https://github.com/jeonsworld/ViT-pytorch . For the deit pretraining task, we implement our algorithm based on https://github.com/ facebookresearch/deit .

We employed NVIDIA GeForce RTX 3090 for running most of the experiments, while the NVIDIA A40 was utilized to evaluate the performance of BERT-Large and ViT-L. Furthermore, we conducted runtime measurements using the NVIDIA T4, 3090, and A100 GPUs.

C.2 GLUE results

In this section, we present the detailed result of fine-tuning the GLUE dataset on BERT-base-uncased and BERT-large-uncased.

On BERT-base, on STSB, SST2, QNLI, and QQP, HQ+LSS only has < 0 . 5% accuracy degradation. On the most challenging tasks CoLA and RTE, our accuracy degradation is much smaller compared to LSQ+LUQ. On QQP and MNLI, our method achieves < 1 . 3% degradation, while LSQ + LUQ has ≥ 1 . 8% degradation. The trend is that the more difficult the task is, the more significant our advantage over LSQ+LUQ.

On BERT-large, the improvement is significant. On CoLA, QNLI, and MNLI, the accuracy improvement compared with LSQ+LUQ > 30% . On other datasets like SST2 and QQP, the accuracy improvement is > 10% . On RTE the accuracy improvement is > 5% , and on STSB and MRPC the improvement is > 3% .

We suspect that for those challenging tasks, there is more information stored in the outliers, which results in a larger gap between our method and LSQ+LUQ.

C.3 More Granular Quantization Methods

In this section, in Table 5, we show that the more granular quantization methods, such as per-token quantization and per-channel quantization, or smoothing techniques, such as SmoothQuant, do not work under the 4-bit FQT setting. Meanwhile, combining these methods with HQ will not bring significant improvement.

We find that LSQ is beneficial for all of these more granular quantization methods under low-bit settings, which highlights the importance of LSQ. Meanwhile, we also notice that the smoothquant will even harm the result of LSQ when the bit-width is low. Our explanation is that the motivation of LSQ is to learn a trade-off between outliers and inliers, while smoothquant aims to sacrifice the

precision of inliers in order to exactly maintain the information of outliers. When the bitwidth is high, this is not a problem, since there are still enough bits to quantize the inliers. But when the bitwidth is low, such sacrifice will cause severe problems since the inlier information is discarded.

C.4 Large Language Model Operator Speed

In this section, we show that our hardware-friendly INT4 training method can really accelerate the training process on Large Language Models. We run distributed training on a system of 8 A100 cards

(4608, 5120, 6144)

(5120, 6144, 8192)

(6144, 6144, 9216)

(7168, 6656, 8704)

(8192, 7680, 9728)

(15360, 8704, 10752)

Matrix size (M,N,K)

0

10

20

30

40

50

60

70

Tflops

FP16

HQ

LSSWeight

LSSAct

(4608, 5120, 6144)

(5120, 6144, 8192)

(6144, 6144, 9216)

(7168, 6656, 8704)

(8192, 7680, 9728)

(15360, 8704, 10752)

0

50

100

150

200

250

300

350

400

Tflops

FP16

HQ

LSSWeight

LSSAct

Matrix size (M,N,K)

and our implementation uses distributed data parallel training with zero-3, gradient checkpointing, and optimizer offloading.

We experimented with two architectures: BERT-Large and GPT2-base. We vary the network width and batch size to make full utilization of the GPU memory and show the end-to-end performance for fine-tuning these models on the SuperGLUE RTE dataset in Table 4.

C.5 More experiments on Operator Speed

Time proportion We examine the proportion of time for each part of computation in HQ-MM and LSS-MM operator in Fig. 6 when the shapes of input matrices vary. In HQ, hadamard means multiplying the input matrix with the Hadamard matrix, pack means packing input data into INT4 data, gemm means the matrix multiplication of two INT4 matrices. In LSSWeight, quantize corresponds to the quantization of higher and lower 4-bit, leverage means computing leverage score, sample means sample out rows/columns given the leverage score, dequantize is the process of dequantizing INT data back into FP16 data, and LSQ is the backpropagation process of LSQ method. In LSSAct, we ignore quantize and leverage process, using the same value as LSSWeight for saving time, other processes share the same meaning with LSSWeight. Note that our implementation is not fully optimized, and optimizations like operator fusion can further improve the performance.

Operator Speed on more GPUs On an Nvidia RTX 3090 GPU with a Cuda capability of sm_86., we show the comparison of FP16 MM, HQ, and LSS operators in Section 5.3 as well as time proportion in each operator in Figure. 6. We also adjust our hardware implementation and test its performance on Nvidia T4 GPU and Nvidia A100 GPU, which have Cuda capability of sm_75 and sm_80 , respectively. The result is shown in Fig. 7 and Fig. 8.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a 4-bit training method mainly to speed up the training process. In the forward propagation, Hadamard quantizer is proposed to suppress the outliers whereas structure sparsity and bit splitting are used for quantization of gradients. It has been shown that the proposed training method can successfully train transformer models across different datasets with some performance degradation w.r.t. the baseline.\n\nSTRENGTHS:\n-- The main strength of this paper is theoretical analysis of their proposed quantization method for both forward and backward steps.\n-- Use of different models and tasks for evaluation purposes is another strength of this paper.\n\nWEAKNESSES:\n-- The main concern of mine is the limited speedup of this paper. The main goal of quantized training in this paper is to speed up the computations and not to reduce the memory. However, the authors have only shown up to 35% speedup w.r.t. the baseline (FP16). First, there is no comparison with Int8 training speed. It's not clear why I should choose Int4 over Int8 while I know the accuracy of Int8 is better than Int4. Second, the speedup amounts are only measured across specific configurations which are not the ones used for evaluation purposes in Table 1. Measuring the training time for the tasks listed in Table 1 can make the contribution of this paper clear.\n-- Fig. 4 and Fig. 5 are hard to read. Again, why not simply reporting the speedup for each task in Table 1.\n-- I have a hard time understanding Table 3. What does \"epoch 1-5\" mean? Which dataset was used for the measurement? What is the accuracy performance?\n\nQUESTIONS:\nIn summary, the idea of Int4 training to accelerate the process is interesting. The authors have shown that Int4 training can obtain a comparable accuracy w.r.t. the baseline and Int8 with around ~1% to ~2% degradation, which is fine. However, the speedup amount is not provided for the benchmarks used for accuracy evaluation in Table 1. Given the above explanations, I have the following questions:\n1) What is the speedup for tasks listed in Table 1?\n2) How do you compare your method w.r.t. Int8 in terms of training speed? In other words, why should I select Int4 over Int8?\n3) How was the accuracy performance measured? Did you report the accuracy in Table 1 using quantized inference (Int4) or full-precision? \n4) Please provide more details for Table 3.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors propose 4-bit training methods for Transformers. It first proposes a Hadamard quantizer with activation outliers issue being solved by Hadamard Transformation. Optimization for backpropagation is proposed by leveraging gradient sparsity with bit splitting and score sampling. The proposed 4-bit training methods achieve competitive accuracy on NLP, Machine translation, and Image classification.

STRENGTHS:
* Achieves competitive or superior accuracy compared with existing works on 4-bit training.
* Optimize backpropagation with hardware-efficient structural sparsity.
* Hardware-friendly INT4 training method is runnable on GPUs with up to 2.2 times faster than the FP16 MM.

WEAKNESSES:
* The activation outlier handling in Transformers is not new, as in SmoothQuant. And the comparison is lacking in the paper. Are those techniques not able to be used in 4-bit training? And what is the efficiency comparison between those methods when handling outlier activations?
* Could you elaborate more on why the proposed methods cannot be applied to Conv layers? What is the bottleneck?

QUESTIONS:
* It would be helpful if you could explain why the speed is not comparable with the baseline in Fig.5 when the hidden layer size is small.

LIMITATIONS:
* Conv cannot be supported
* INT4 training on extremely large models remains to be an open question.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe research paper presents a framework for training transformer-based neural networks using 4-bit integers, offering enhanced computational and memory efficiency. \nThe proposed method includes a dedicated Hadamard quantizer for forward propagation, designed to suppress outliers which typically cause degradation of model accuracy. The Hadamard quantizer works by transforming the activation matrix, spreading outlier information across nearby entries in the matrix, and reducing the numerical range of outliers.\nIn backpropagation, the authors exploit the structural sparsity of activation gradients. They note that a few token gradients are extremely large, while most are much smaller, even below the quantization residuals of larger gradients. To make efficient use of computational resources, the authors propose a technique called bit splitting. This technique divides each token's gradient into two parts: the higher 4 bits and the lower 4 bits. They then identify the most informative gradients using a technique known as leverage score sampling, an importance sampling technique in the field of RandNLA.\nThe combination of these techniques results in an algorithm that employs 4-bit integer MMs for all linear operations in transformers. This algorithm has been evaluated and found to perform competitively across a wide range of tasks, including natural language understanding, question answering, machine translation, and image classification.\n\nSTRENGTHS:\n- Unlike many other low-precision training methods, this proposed method does not require custom numerical formats and is compatible with contemporary hardware, such as GPUs. This opens the potential for widespread adoption and usability.\n- The proposed method demonstrated competitive or superior accuracy across a range of tasks, compared to existing 4-bit training methods. This indicates the practical effectiveness of the method in diverse applications.\n- The authors have developed specialized quantization methods for both forward and backpropagation. These techniques manage outliers and leverage the structural sparsity of activation gradients, maintaining model accuracy despite the ultra-low numerical precision.\n\nWEAKNESSES:\nMy main concern is that the paper chose to compare the performance and computation complexity of their approach with FP16 rather than examining its efficiency against INT8 or other INT4 alternatives. This could potentially limit the depth and breadth of the performance analysis.\n\nThe key contributions of this research are primarily focused on the implementation of a Hadamard quantizer and bit splitting techniques. Both techniques leverage INT4 matrix multiplication (MM). However, the HQ-MM has the 4 steps procedure to leverage the efficiency of INT4 computation and the bit-splitting technique deploys two INT4 MMs as a replacement for one INT8 MM. This can potentially induce more overhead than simply utilizing INT8, particularly considering that INT4 does not straightforwardly translate to half the computational, memory access, or energy costs associated with INT8 computations.\n\nWhile the aim of utilizing INT4 computations is to increase efficiency, it's crucial to recognize that the benefits of moving from INT8 to INT4 are not always linear. According to a report by NVIDIA, INT4, in an optimal scenario involving Convolutional Neural Network (CNN) inference in image classification tasks, resulting in a throughput improvement of approximately 59% with a negligible accuracy loss (around 1%) on NVIDIA T4. Meanwhile, on TITAN RTX, the speedup was about 52%. These improvements, although significant, don't necessarily equate to halving the computational resources or doubling the performance, and this is also applied to the transformer-based models scenario of this paper.\n\nHence, a more thorough comparative study involving INT8 and other INT4 counterparts could potentially provide a more nuanced and comprehensive understanding of the computational advantages and trade-offs of the proposed techniques. Additionally, if there are any misinterpretations or inaccuracies in my understanding of the paper, I would appreciate it if you could provide corrections to ensure my perspective aligns accurately with the content and intentions of the research.\n\nQUESTIONS:\nThe concept of using a Hadamard matrix to decompose matrix multiplication is a thoughtful and fitting approach in low-bit quantization scenarios, especially for extreme cases such as binary and ternary methods. The paper meticulously analyzes the time complexity of the Hadamard Quantizer Matrix Multiplication (HQ-MM) procedure in Section 3.3, and in Appendix A.5, it further elucidates on how to select an appropriate Hadamard matrix size. I am intrigued to learn more about the finer details of this procedure.\n\nGiven the significant shift in distribution during training, it is indeed crucial to reconfigure both the Hadamard matrix size and the Learned Step Size Quantization (LSQ) step size. This brings up a few questions: Is the re-initialization process executed only once, or is it a recurring process? Does the system stabilize after approximately two epochs? Furthermore, is X representative of the activation of a batch of input data? These details will enhance the understanding and application of the proposed methodology.\n\nLIMITATIONS:\nImplementing advanced quantization techniques such as the Hadamard quantizer and bit splitting may be technically challenging and potentially difficult to integrate into existing neural network training pipelines.\nThe method exhibits limitations in its scalability, as it is not equipped to support exceedingly large models like OPT-175B. This underscores a broader, unresolved issue in the realm of machine learning, where even the application of INT8 training to these substantial models remains an open-ended problem.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper deals with 4-bit training (fine-tuning) of transformers. For forward propagation, a Hadamard quantizer is proposed to solve the problem of outliers. For backpropagation, the authors leverage the structural sparsity of gradients by proposing bit splitting and leverage score sampling
techniques to quantize gradients accurately. Experiments show up to 35.1% speedup for 4-bit training with

STRENGTHS:
1. The proposed Hadamard quantizer is simple while useful for transformer quantization with outliers.
2. The authors illustrate the structural sparsity of gradients and propose bit splitting (BS), which splits a full-precision matrix as higher and lower 4 bits. The similar idea is used in quantization for inference. This is the first to use it for training.
3. The experiments are extensive. The training speed on GPUs are reported.

WEAKNESSES:
1. The proposed method introduce extra FP16 computations. The BS with selection needs data rearrangement, which could slow down the speed.
2. The title is somewhat over-claim. The experiments are mainly about fine-tuning (FT). It seems that the proposed method works for fine-tuning (FT) of a converged model. For pre-training (PT), there is large accuracy loss. 
3. The speed-up of up to 35.1% is misleading. It is better to use the average speed-up in the abstract.

QUESTIONS:
Please see the weaknesses.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07