SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For  $\nabla_{\mathbf{T}}$ , similarly, can be viewed as  $\nabla_{\mathbf{T}} = \mathrm{BMM}(\hat{\mathbf{I}}^{\uparrow}, \nabla_{\mathbf{T}}^{\downarrow})$ , where we define  $\nabla_{\mathbf{Y}}^{\uparrow} = \mathrm{CONCAT}([s_{\uparrow b}\nabla_{\mathbf{T}b}^{\uparrow}; s_{\downarrow b}\nabla_{\mathbf{T}b}^{\downarrow}]) \in \mathbb{R}^{B \times 2N \times P}$ ,  $\hat{\mathbf{I}}^{\uparrow} = \mathrm{CONCAT}([\mathbf{I} \quad \mathbf{I}]) \in \mathbb{R}^{B \times N \times 2N}$ ,  $s_{\uparrow b}, \nabla_{\mathbf{T}b}^{\uparrow}, s_{\downarrow b}, \nabla_{\mathbf{T}b}^{\downarrow}$  follows the definition of Eq.5. So it can be computed as  $\mathrm{BMM}(\mathrm{BMM}(\hat{\mathbf{I}}^{\uparrow}, \hat{\mathbf{H}}), \nabla_{\mathbf{T}}^{\downarrow})$ , where  $\hat{\mathbf{H}}$  is defined as  $\mathrm{CONCAT}(\tilde{\mathbf{H}}_1, \dots, \tilde{\mathbf{H}}_B)$ , and the leverage score is  $c_{b,i} := \| \nabla_{\mathbf{T}b,i,:}^{\uparrow} \|$  for  $0 \leq b \leq B, 0 \leq i \leq 2M$ , which verifies that our LSS can be applied to BMM.
A.2 Computing Leverage Score
In the previous discussion, we find the optimal sample probability  $p_i$  that can minimize the variance of the gradient. However, it is likely for the proportional  $p_i$  is larger than one, which is invalid for the Bernoulli distribution. Accordingly, we propose an algorithm to solve this issue.
Define the probability array as
$$
P = [ p _ {1} ^ {0}, \dots , p _ {2 N} ^ {0} ], \sum_ {i = 1} ^ {2 N} p _ {i} ^ {0} = N,
$$
we first clamp the array to  $p_i^1 \in [0, 1]$ . In this case,  $\sum_{i=1}^{2N} p_i^1 \leq N$ , so we scale the  $p_i$  which is smaller than 1 to make sure their sum is again  $N$ . However, this will probably introduce some more elements larger than 1, so we cycle through the above operations until all the  $p_i \in [0, 1]$ . This process will certainly stop, since if after the scaling operation, no element is larger than 1, then we get a valid distribution. Otherwise, the number larger than 1 is reduced by at least one, thus the process will halt after at most  $O(N)$  times.
A.3 Learning Quantizer Parameters
In this section, we discuss the detail of how to calculate the gradient of activation and quantization step size.
For gradient of activation, the coefficient  $c_{i} \coloneqq \|\nabla_{\mathbf{Y}^{i}}^{\ddagger}\|$  is the leverage score for activation gradient, and the variance achieves its minimum. When  $p_{i} \propto c_{i}$  by the Cauchy Inequality.
Putting everything together, we propose the following MM procedure to compute activation gradient:
1. Quantize  $\nabla_{\mathbf{Y}}$  with BS to obtain  $\nabla_{\mathbf{Y}}^{\uparrow}$  and  $\nabla_{\mathbf{Y}}^{\downarrow}$  in INT4.
2. Compute the leverage score  $\| \nabla_{\mathbf{Y}i}^{\ddagger}\|$  in FP16.
3. Sample the masks  $\{m_i\}$ .
4. Sample rows of  $\nabla_{\mathbf{Y}}$  given the masks  $\{m_i\}$ .
5. Compute  $\mathbf{IM}^{\uparrow}\nabla_{\mathbf{Y}}^{\uparrow}$  and  $\mathbf{IM}^{\downarrow}\nabla_{\mathbf{Y}}^{\downarrow}$  by discard some of its rows.
6. Compute INT4 MMs  $\tilde{\mathbf{M}}^{\uparrow}\nabla_{\mathbf{Y}}^{\uparrow}\hat{\mathbf{W}}$  and  $\tilde{\mathbf{M}}^{\downarrow}\nabla_{\mathbf{Y}}^{\downarrow}\hat{\mathbf{W}}$
7. Dequantize and sum up the resultant INT32 matrices to obtain the FP16 result  $\hat{\mathbf{I}}^{\updownarrow}\nabla_{\mathbf{Y}}^{\updownarrow}\hat{\mathbf{W}}$
The two matrix multiplications in Step 5 take about  $2NCD$  INT4 MACs in expectation.
For the quantization step sizes. Following the chain rule, we have
$$
\nabla_ {s _ {W}} = g (s _ {W}) \nabla_ {\mathbf {Y}} ^ {\top} \hat {\mathbf {X}} \circ \delta_ {\mathbf {W}} (s _ {W}), \nabla_ {s _ {X}} = g (s _ {X}) \nabla_ {\mathbf {Y}} \hat {\mathbf {W}} \circ \delta_ {\mathbf {X}} (s _ {X}),
$$
where we define  $g(s_W) = 1 / \sqrt{Q_pN_W}$ ,  $g(s_X) = 1 / \sqrt{Q_pN_X}$ ,  $N_W$  and  $N_X$  being the number of elements of weight and activation,  $\delta_{\mathbf{X}}(s_X) = \mathrm{int}_{s_X}(\mathbf{X}) - \mathbb{I}_X \circ (\mathbf{X} / s_X)$ , and  $\delta_{\mathbf{W}}(s_W) = \mathrm{int}_{s_W}(\mathbf{W}) - \mathbb{I}_W \circ (\mathbf{W} / s_W)$ .
Notice that for computing  $\nabla_{s_W}$  and  $\nabla_{s_X}$ , the most expensive MMs are  $\nabla_{\mathbf{Y}}^{\top}\hat{\mathbf{X}}$  and  $\nabla_{\mathbf{Y}}\hat{\mathbf{W}}$ , which are already calculated through Eq. (7) and Eq. (8) during previous calculations, so it does not require extra computation. The elementwise multiplication with  $\delta_{\mathbf{X}}(s_X)$  and  $\delta_{\mathbf{W}}(s_W)$  requires minor computation.
A.4 Cold Start Problem
There is a cold start problem. When the model is trained from scratch (i.e., from a random initialization), distributions of weights and activations can change rapidly in the early stage of optimization. In this case, jointly optimizing the quantization step size and the weights would cause the training to be unstable. As a remedy, we do not learn the step size in the first few iterations, and use a heuristic rule to dynamically set the step size for each tensor  $\mathbf{X}$  to  $2\mathrm{mean}(\mathbf{X}) / \sqrt{Q_p}$  in each iteration. For the WMT experiment, we stop to re-initialize at the second epoch, since the BLEU score is relatively high at the second epoch (about 21, while the final BLEU score after 30 epochs is 25.5). However, for the deit-small pretraining experiment, we re-initialize it at the end of the 10th epoch (we train it for 90 epochs in total), since at the end of the second epoch, the accuracy is not high, but at the end of the 10th epoch, the accuracy is relatively high.
A.5 Choose hadamard matrix size
For the hadamard matrix, let the hadamard matrix to be  $\mathbf{H} \in \mathbb{R}^{D \times D}$ :  $\mathbf{H} = \mathrm{BlockDiag}(\mathbf{H}_k, \ldots, \mathbf{H}_k)$ , where  $D$  is a multiple of  $2^k$ . We first define
$$
\bar {\mathbf {X}} _ {k} = s _ {X} \operatorname {i n t} _ {s _ {X}} (\mathbf {X H}) \mathbf {H} ^ {\top}, \quad \bar {\mathbf {W}} = s _ {W} \operatorname {i n t} _ {s _ {W}} (\mathbf {W H}) \mathbf {H} ^ {\top},
$$
where  $\bar{\mathbf{X}}$  and  $\bar{\mathbf{W}}$  can be viewed as an approximation of  $\mathbf{X}$  and  $\mathbf{W}$ . Then, we define the quantization error to be  $\mathrm{MSE}(\bar{\mathbf{X}},\mathbf{X})\times \mathrm{MSE}(\bar{\mathbf{W}},\mathbf{W})$ . We search for the optimal  $k$  that can minimize this quantization error. For fine-tuning tasks, once the hadamard matrix size has been calculated, we fix it through the training process. For the pre-training task, since the distribution shifts greatly as we train the model, we empirically define a time when we re-initialize the hadamard matrix size and the LSQ step size. Usually, we do this when the first 2 epochs finish.
A.6 GPU Implementation
In the previous discussion, we get to know HQ-MM and LSS-MM from an algorithm level, nevertheless it is not enough to actually implement it on hardware. In this section, we will delve deeper into hardware implementation details as well as extra limitations.
HQ-MM can be divided into 5 parts: Hadamard matrix multiplication, Quantize, Data Pack, INT4 GEMM, and Dequantize.
For the Hadamard matrix multiplication process, since it can be interpreted as a half float matrix multiplication process where the two matrices involved in the operation are input/weight matrix and hadamard matrix, respectively, we implement it in Python, because PyTorch MM uses CublassGemm and is more efficient then CutlassGemm.
In the quantize process, we quantize input/weight into INT4 data respectively, and also preserve a corresponding FP16 version for the LSQ Back Propagation process to use.
In the previous discussion, we assume the quantize part of HQ-MM is quantizing the resultant matrices to INT4, however, the smallest representation unit of data is INT8. As a result, we actually use INT8 data type to represent quantized data and pack two adjacent data into one data using  $(data[1] << 4) | (data[0] & 15)$  in the data packing process, which means we use one INT8 data to represent two adjacent INT4 data. With both input matrices' data packed in this way, we then use cutlass tensor-core INT4 GEMM to do the matrix multiplication.
For the GEMM process, we choose Nvidia CutlassGemm because it's the most efficient open-source operator library we can find. We use INT4 Tensor Core Gemm for our implementation and it requires the two input matrices A&B to be RowMajor and ColMajor, respectively. Since the default Pytorch tensor is RowMajor, we have to use Transpose+Contiguous operations to make it ColMajor, which is very time-consuming and needs further optimization in the future.
Finally, we dequantize the INT GEMM result back into FP16 output using a dequantize kernel, which is the final output of the forward kernel.
As compared, LSS-MM is more complicated, and can be divided into 7 parts: Quantization of higher lower 4-bit, Leverage Score Calculating, Sampling, Data Pack, INT4 GEMM, Dequantize, and LSQ Back Propagation.
In the Quantize process, we fuse the quantize operation of higher 4-bit and lower 4-bit into a single kernel for acceleration. In the Leverage Score Calculating process, we use the quantized INT8 data to calculate the score and scale up it in the final because integer arithmetic is far more efficient than float arithmetic.
In the sampling process, we sample out rows/columns given the previously calculated leverage score. Note that in Section A.2, we repeat our proposed algorithm for several loops to sample out specific elements, which is effective but not efficient. According to experiments, however, we notice that simply selecting elements whose leverage score is bigger than 0 can also work well, even better than our proposed algorithm in some cases. So in real quantization implementation, we just sample out rows/columns whose Euclidean norm is bigger than 0 to accelerate our training process.
Pack, Gemm, and Dequantize processes are as similar as before. It's worth noting that for Int4 Tensor Core Gemm, suppose two input matrices have shape  $M \times K$  and  $K \times N$ ,  $K$  needs to be a multiple of 32 so that the Tensor core Gemm address can be aligned. We do not need to consider this in the Forward Propagation process because the input data shape always satisfies. However, in the Back Propagation process, the matrix shape may not meet the requirement after sampling. As a result, we need zero-padding the sampled matrix so that  $K$  can be a multiple of 32.
Finally, we utilize the dequantized data to do the LSQ Back Propagation. We also fuse all operations into a single Cuda kernel for acceleration, and the metric remains.
Besides the component of HQ-MM and LSS-MM, there is still something that needs to be mentioned.
1. We omit the Quantization and Leverage Score Calculating process in LSSinput, and use the same value as LSSWeight to accelerate the training process.
2. For Element-Wise kernel, we set block size as 256, grid size as input.Numel(/256. For Reduction kernels like sum and min/max, we set block size as 32, grid size as RowNum, reducing elements in each row to the first 32 elements. We find this setting to be most efficient through experiments.
B Proofs.
In this section, we present the proofs of the leverage score.
B.1 Proof of Proposition. 4.1
Proposition B.1. (LSS variance for weight gradient)
$$
\mathrm {V a r} \left[ \sum_ {i = 1} ^ {2 N} \frac {m _ {i}}{p _ {i}} \nabla_ {\mathbf {Y} _ {; ; i}} ^ {\updownarrow} \mathbf {\Sigma} ^ {\top} \mathbf {X} _ {i} ^ {\updownarrow} \right] = \sum_ {i = 1} ^ {2 N} \frac {1 - p _ {i}}{p _ {i}} \| \nabla_ {\mathbf {Y} _ {i ; ;}} ^ {\updownarrow} \| ^ {2} \| \mathbf {X} _ {i; ;} ^ {\updownarrow} \| ^ {2}.
$$
$$
\begin{array}{l} V a r (\nabla_ {\mathbf {W}}) = V a r \left(\sum_ {i = 1} ^ {2 N} \frac {1}{p _ {i}} \left(m _ {i} \nabla_ {\mathbf {Z} _ {:, i}} ^ {\updownarrow} ^ {\top} \mathbf {X} _ {i} ^ {\updownarrow}\right)\right) \\ = V a r \left(\sum_ {i = 1} ^ {2 N} \frac {1}{p _ {i}} \left(\sum_ {j = 1} ^ {C} \sum_ {k = 1} ^ {D} m _ {i} \nabla_ {\mathbf {Z} _ {j, i}} ^ {\updownarrow} ^ {\top} \mathbf {X} _ {i, k} ^ {\updownarrow}\right)\right) \\ = \sum_ {i = 1} ^ {2 N} \frac {p _ {i} (1 - p _ {i})}{p _ {i} ^ {2}} V a r \left(\left(\sum_ {j = 1} ^ {C} \sum_ {k = 1} ^ {D} \nabla_ {\mathbf {Z} _ {j, i}} ^ {\updownarrow} ^ {\top} \mathbf {X} _ {i, k} ^ {\updownarrow}\right)\right) \\ = \sum_ {i = 1} ^ {2 N} \frac {1 - p _ {i}}{p _ {i}} (\sum_ {j = 1} ^ {C} \sum_ {k = 1} ^ {D} \nabla_ {\mathbf {Z} _ {j, i}} ^ {\updownarrow^ {\top}} ^ {2} \mathbf {X} _ {i, k} ^ {\updownarrow^ {\top}} ^ {2}). \\ \end{array}
$$
$$
\begin{array}{l} V a r \left(\nabla_ {\mathbf {W}}\right) = \sum_ {i = 1} ^ {2 N} \left(\frac {1}{p _ {i}} - 1\right) \left(\sum_ {j = 1} ^ {C} \nabla_ {\mathbf {Z} _ {j, i}} ^ {\ddagger} ^ {\top} ^ {2}\right) \left(\sum_ {k = 1} ^ {D} \mathbf {X} _ {i, k} ^ {\ddagger} ^ {2}\right) (9) \\ = \sum_ {i = 1} ^ {2 N} \left(\frac {1}{p _ {i}} - 1\right) \| \nabla_ {\mathbf {Z} _ {:, i}} ^ {\updownarrow^ {\top}} \| ^ {2} \| \mathbf {X} _ {i,:} ^ {\updownarrow} \| ^ {2}, (10) \\ \end{array}
$$
which proves.
B.2 Proof of Activation Leverage Score in Sec. 4.2
we divide the matrix multiplication into the sum of  $2N$  smaller multiplications:
$$
\hat {\mathbf {I}} ^ {\updownarrow} \nabla_ {\mathbf {Y}} ^ {\updownarrow} = \sum_ {i = 1} ^ {2 N} \hat {\mathbf {I}} _ {:, i} ^ {\updownarrow} \nabla_ {\mathbf {Y} i} ^ {\updownarrow} = \sum_ {i = 1} ^ {2 N} \hat {\nabla} _ {\mathbf {Y} _ {i}}, \tag {11}
$$
where we define  $\hat{\nabla}_{\mathbf{Y}_i} = \hat{\mathbf{I}}_{:,i}^\uparrow \nabla_{\mathbf{Y}^i}^\uparrow$
We assign each  $\nabla_{\mathbf{Y}_i}$  a probability  $p_i \in [0,1], i = 1,\dots,2N$ , that satisfies  $\sum_{i=1}^{2N} p_i = N$ . We define random masks  $m_i \sim \mathrm{Bern}(p_i)$ , and define  $\tilde{\mathbf{M}} = \mathrm{diag}\left(\frac{m_1}{p_1},\ldots,\frac{m_{2N}}{p_{2N}}\right)$ , and make an unbiased estimation:
$$
\hat {\mathbf {I}} ^ {\updownarrow} \nabla_ {\mathbf {Y}} ^ {\updownarrow} \approx \hat {\mathbf {I}} ^ {\updownarrow} \tilde {\mathbf {M}} \nabla_ {\mathbf {Y}} ^ {\updownarrow} = \sum_ {i = 1} ^ {2 N} \frac {m _ {i}}{p _ {i}} \nabla_ {\mathbf {Y}} ^ {\updownarrow} i.
$$
Define  $\mathbf{M}^{\dagger}$  to be the top-left  $N\times N$  submatrix of  $\mathbf{M}$  and  $\mathbf{M}^{\downarrow}$  to be the bottom-right one, we have
$$
\hat {\mathbf {I}} ^ {\updownarrow} \tilde {\mathbf {M}} \nabla_ {\mathbf {Y}} ^ {\updownarrow} = s _ {\uparrow} \mathbf {I} \tilde {\mathbf {M}} ^ {\uparrow} \nabla_ {\mathbf {Y}} ^ {\uparrow} + s _ {\downarrow} \mathbf {I} \tilde {\mathbf {M}} ^ {\downarrow} \nabla_ {\mathbf {Y}} ^ {\downarrow},
$$
In this case,  $\mathbf{IM}^{\uparrow}\nabla_{\mathbf{Y}}^{\uparrow}$  and  $\mathbf{IM}^{\downarrow}\nabla_{\mathbf{Y}}^{\downarrow}$  both only have parts of its rows being non zero, and the rest rows are zeros since they are discarded. Then, when we multiply it by  $\hat{\mathbf{W}}$ , there are half of rows being zeros in  $\mathbf{IM}^{\uparrow}\nabla_{\mathbf{Y}}^{\uparrow}\hat{\mathbf{W}}$  and  $\mathbf{IM}^{\downarrow}\nabla_{\mathbf{Y}}^{\downarrow}\hat{\mathbf{W}}$ . So there's no need to calculate them, and we successfully cut off half of the computation in this case.
Now focus on the variance that
Proposition B.2. (LSS variance for activation gradient)
$$
\mathrm {V a r} \left[ \sum_ {i = 1} ^ {2 N} \hat {\mathbf {I}} _ {:, i} ^ {\updownarrow} \nabla_ {\mathbf {Y} ^ {i}} ^ {\updownarrow} \right] = \sum_ {i = 1} ^ {2 N} \frac {1 - p _ {i}}{p _ {i}} \| \nabla_ {\mathbf {Y} ^ {i}} ^ {\updownarrow} \| ^ {2}.
$$
$$
\begin{array}{l} V a r \left(\nabla_ {\mathbf {X}}\right) = V a r \left(\sum_ {i = 1} ^ {2 N} \frac {1}{p _ {i}} \left(m _ {i} \hat {\mathbf {I}} _ {:, i} ^ {\updownarrow} \mathbf {X} _ {i} ^ {\updownarrow}\right)\right) \\ = V a r \left(\sum_ {i = 1} ^ {2 N} \frac {1}{p _ {i}} \left(\sum_ {j = 1} ^ {C} \sum_ {k = 1} ^ {D} m _ {i} \hat {\mathbf {I}} _ {j, i} ^ {\uparrow} \nabla_ {\mathbf {Y} ^ {i}, k}\right)\right) \\ = \sum_ {i = 1} ^ {2 N} \frac {p _ {i} (1 - p _ {i})}{p _ {i} ^ {2}} V a r \left(\left(\sum_ {j = 1} ^ {C} \sum_ {k = 1} ^ {D} \hat {\mathbf {I}} _ {j, i} ^ {\updownarrow} \nabla_ {\mathbf {Y} ^ {i}, k}\right)\right) \\ = \sum_ {i = 1} ^ {2 N} \frac {1 - p _ {i}}{p _ {i}} \left(\sum_ {j = 1} ^ {C} \sum_ {k = 1} ^ {D} \left(\hat {\mathbf {I}} _ {j, i} ^ {\updownarrow}\right) ^ {2} \left(\nabla_ {\mathbf {Y}} ^ {\updownarrow} i, k\right) ^ {2}\right) \\ = \sum_ {i = 1} ^ {2 N} (\frac {1}{p _ {i}} - 1) \big (\sum_ {j = 1} ^ {C} (\hat {\mathbf {I}} _ {j, i} ^ {\updownarrow}) ^ {2} \big) (\sum_ {k = 1} ^ {D} (\nabla_ {\mathbf {Y} _ {i, k}} ^ {\updownarrow}) ^ {2} \big) \\ = \sum_ {i = 1} ^ {2 N} \left(\frac {1}{p _ {i}} - 1\right) \| \hat {\mathbf {I}} _ {:, i} ^ {\updownarrow} \| ^ {2} \| \nabla_ {\mathbf {Y} _ {i}} ^ {\updownarrow} \| ^ {2} \\ = \sum_ {i = 1} ^ {2 N} \left(\frac {1}{p _ {i}} - 1\right) \| \nabla_ {\mathbf {Y} _ {i}} ^ {\updownarrow} \| ^ {2}. \\ \end{array}
$$
In this way, the coefficient  $c_{i} \coloneqq \|\nabla_{\mathbf{Y}^{i}}^{\uparrow}\|$  is the leverage score.
C Experiments.
In this section, we present more details for experiments in Sec. 5.
Figure 6: Time proportion for each part in HQ-MM and LSS-MM operator.
C.1 Experiments setup
For the GLUE, QA, SWAG, and CONLL tasks, we implement our algorithm based on https://github.com/huggingface/transformers. For the machine translation task, we implement our algorithm based on https://github.com/facebookresearch/fairseq. For the ViT fine-tuning task, we implement our algorithm based on https://github.com/jeonsworld/ViT-pytorch. For the deit pretraining task, we implement our algorithm based on https://github.com/facebookresearch/deit.
We employed NVIDIA GeForce RTX 3090 for running most of the experiments, while the NVIDIA A40 was utilized to evaluate the performance of BERT-Large and ViT-L. Furthermore, we conducted runtime measurements using the NVIDIA T4, 3090, and A100 GPUs.
C.2 GLUE results
In this section, we present the detailed result of fine-tuning the GLUE dataset on BERT-base-uncased and BERT-large-uncased.
On BERT-base, on STSB, SST2, QNLI, and QQP, HQ+LSS only has  $< 0.5\%$  accuracy degradation. On the most challenging tasks CoLA and RTE, our accuracy degradation is much smaller compared to LSQ+LUQ. On QQP and MNLI, our method achieves  $< 1.3\%$  degradation, while LSQ + LUQ has  $\geq 1.8\%$  degradation. The trend is that the more difficult the task is, the more significant our advantage over LSQ+LUQ.
On BERT-large, the improvement is significant. On CoLA, QNLI, and MNLI, the accuracy improvement compared with  $\mathrm{LSQ + LUQ > 30\%}$ . On other datasets like SST2 and QQP, the accuracy improvement is  $>10\%$ . On RTE the accuracy improvement is  $>5\%$ , and on STSB and MRPC the improvement is  $>3\%$ .
We suspect that for those challenging tasks, there is more information stored in the outliers, which results in a larger gap between our method and LSQ+LUQ.
C.3 More Granular Quantization Methods
In this section, in Table 5, we show that the more granular quantization methods, such as per-token quantization and per-channel quantization, or smoothing techniques, such as SmoothQuant, do not work under the 4-bit FQT setting. Meanwhile, combining these methods with HQ will not bring significant improvement.
We find that LSQ is beneficial for all of these more granular quantization methods under low-bit settings, which highlights the importance of LSQ. Meanwhile, we also notice that the smoothquant will even harm the result of LSQ when the bit-width is low. Our explanation is that the motivation of LSQ is to learn a trade-off between outliers and inliers, while smoothquant aims to sacrifice the
Table 3: GLUE results on BERT-base-uncased and BERT-large uncased. FP refers to full precision training, INT8 refers to INT8 training, LSQ + LUQ refers to learned step size quantization for forward and logarithmic unbiased quantization for backward, and HQ + LSS refers to Hadamard quantization for forward and leverage score sampling for backward.
MODEL	DATASET	QUANTIZATION METHODS
FP	INT8	LSQ+LUQ	HQ+LSS
BERT-BASE	CoLA	56.890.64	56.150.94	18.763.58	52.461.46
STSB	88.140.73	87.050.38	84.310.29	87.770.30
RTE	64.801.26	62.271.26	56.800.92	62.451.08
MRPC	88.610.66	86.850.76	86.230.67	86.540.83
SST2	92.720.06	92.370.17	90.370.46	92.490.29
QNLI	91.520.22	90.920.24	87.330.48	90.530.23
QQP	91.090.11	90.570.05	89.260.03	89.800.05
MNLI	84.520.22	84.100.08	81.790.18	83.590.12
MNLI-MM	84.680.20	84.490.31	82.220.33	83.750.28
BERT-LARGE	CoLA	60.330.49	58.801.52	0.000.00	53.461.17
STSB	87.592.39	86.530.20	83.080.41	87.570.78
RTE	71.121.80	63.711.26	53.060.72	64.620.78
MRPC	91.060.28	87.571.47	82.560.59	87.620.51
SST2	93.980.17	93.750.63	83.940.69	93.520.40
QNLI	92.260.05	92.290.29	63.1813.10	91.530.38
QQP	91.040.63	90.710.00	75.6212.44	90.770.02
MNLI	86.710.19	85.820.08	33.421.38	85.860.10
MNLI-MM	86.410.35	85.870.14	33.541.55	85.820.07
Table 4: Experiments on GPT2-base and Bert-large. Total time spent for epoch 1-5 are reported.
MODEL	(HIDDEN_SIZE, INTERMEDIATE_SIZE, BATCH_SIZE)	TRAINING METHODS	SPEEDUP
FP16	HQ+LSS
BERT-LARGE	(2560, 10240, 2048)	15.094s	18.949s	-25.5%
(4096, 16384, 1280)	32.016s	30.594s	4.4%
(5120, 20480, 960)	47.418s	39.482s	16.7%
(7680, 30720, 600)	95.832s	67.253s	29.8%
(8960, 35840, 480)	128.441s	83.388s	35.1%
(9600, 38400, 160)	161.114s	114.325s	29.0%
(12800, 51200, 100)	326.265s	255.966s	21.5%
(14400, 57600, 96)	409.291s	346.354s	15.3%
GPT2-BASE	(2560, 10240, 1536)	17.253s	22.037s	-27.7%
(4096, 16384, 960)	35.937s	35.694s	~
(5120, 20480, 768)	52.723s	46.548s	11.7%
(7680, 30720, 260)	113.855s	92.548s	18.7%
(8960, 35840, 200)	150.680s	114.881s	23.8%
(9600, 38400, 180)	172.182s	126.540s	26.5%
(12800, 51200, 112)	320.757s	236.433s	26.3%
precision of inliers in order to exactly maintain the information of outliers. When the bitwidth is high, this is not a problem, since there are still enough bits to quantize the inliers. But when the bitwidth is low, such sacrifice will cause severe problems since the inlier information is discarded.
C.4 Large Language Model Operator Speed
In this section, we show that our hardware-friendly INT4 training method can really accelerate the training process on Large Language Models. We run distributed training on a system of 8 A100 cards
Figure 7: Real quantization performance on Nvidia T4.
Figure 8: Real quantization performance on Nvidia A100.
Table 5: Comparison of different quantization methods, quantize the activation and weight into the same bit-width from 2 to 8. Per-token refers to quantize activation per-token, while Per-channel refers to quantize weight per-channel.
quantization methods	Quantize Bits
2	3	4	5	6	7	8
Per-tensor	0	0	0	0	0	50.2	54.6
Per-token	0	0	0	0	31.4	52.8	56
Per-channel	0	0	0	0	0	51.9	56.7
smoothquant	0	0	0	0	0	49.4	57.7
Per-token + Per-channel + smoothquant	0	0	0	0	40.7	55.7	56.7
LSQ	0	9.16	24.2	37.3	39.6	45.3	51.4
Per-token + LSQ	0	15.3	27.8	31.6	42.9	46	54.4
Per-channel + LSQ	0	8	23.9	29.3	40	45.5	50.7
smoothquant + LSQ	0	0	0	0	49.6	54.9	57
Per-token + Per-channel + smoothquant + LSQ	0	0	0	0	28.8	52.4	55.2
HQ	0	45.2	54.6	54.2	56.5	57.4	58.4
HQ + Per-token + Per-channel	0	48.4	54.1	54.9	55	56	56
HQ + Per-token + Per-channel + smoothquant	0	0	46.6	54.9	55.9	55.8	56.5
and our implementation uses distributed data parallel training with zero-3, gradient checkpointing, and optimizer offloading.
We experimented with two architectures: BERT-Large and GPT2-base. We vary the network width and batch size to make full utilization of the GPU memory and show the end-to-end performance for fine-tuning these models on the SuperGLUE RTE dataset in Table 4.
C.5 More experiments on Operator Speed