SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haocheng Xi $^{2*}$ , Changhao Li $^{1}$ , Jianfei Chen $^{1\dagger}$ , and Jun Zhu $^{1}$
$^{2}$ Institute for Interdisciplinary Information Sciences, Tsinghua University {xihc20,lichangh20}@ mails.tsinghua.edu.cn, {jianfeic,dcszj}@tsinghua.edu.cn
Quantizing the activation, weight, and gradient to 4-bit is promising to accelerate neural network training. However, existing 4-bit training methods require custom numerical formats which are not supported by contemporary hardware. In this work, we propose a training method for transformers with all matrix multiplications implemented with the INT4 arithmetic. Training with an ultra-low INT4 precision is challenging. To achieve this, we carefully analyze the specific structures of activation and gradients in transformers to propose dedicated quantizers for them. For forward propagation, we identify the challenge of outliers and propose a Hadamard quantizer to suppress the outliers. For backpropagation, we leverage the structural sparsity of gradients by proposing bit splitting and leverage score sampling techniques to quantize gradients accurately. Our algorithm achieves competitive accuracy on a wide range of tasks including natural language understanding, machine translation, and image classification. Unlike previous 4-bit training methods, our algorithm can be implemented on the current generation of GPUs. Our prototypical linear operator implementation is up to 2.2 times faster than the FP16 counterparts and speeds up the training by  $17.8\%$  on average for sufficiently large models. Our code is available at https://github.com/xijiu9/Train_Transformers_with_INT4.
Training neural networks is computationally demanding. Training with low-precision arithmetic (a.k.a., fully quantized training or FQT) is promising to improve computational and memory efficiency. FQT methods add some quantizers and dequantizers in the original full-precision computational graph, and replace expensive floating-point operations with cheap low-precision ones.
Research in FQT aims to reduce the training numerical precision, without sacrificing much convergence speed or accuracy. The required numerical precision has been reduced from FP16 [33] to FP8 [54, 46], INT32+INT8 [3] and INT8+INT5 [7]. FP8 training is implemented in Nvidia's H100 GPU with Transformer Engine [35], achieving impressive speedup for the training of large-scale transformers. Recently, the training numerical precision has been pushed down to 4 bits. Sun et al. [47] successfully trained several modern networks with INT4 activation/weights and FP4 gradients; and Chmiel et al. [8] propose a custom 4-bit logarithmic numerical format to further improve the accuracy. However, these 4-bit training methods cannot be directly utilized for acceleration as they require custom numerical formats that are not supported on contemporary hardware.
There are significant optimization challenges to train neural networks at an extremely low 4-bit level. First, the non-differentiable quantizers in forward propagation make the loss landscape rugged, where gradient-based optimizers can easily stuck at local optima [31]. Second, gradients are only computed
*This work was done during an internship in the Department of Computer Science and Technology, Tsinghua University
approximately in low-precision. Such imprecise gradients slow down the training process and even cause the training to be unstable or diverge.
In this work, we propose a novel INT4 training algorithm for a class of popular neural networks, transformers [52]. All the costly linear operations for training transformers can be written in a matrix multiplication (MM) form. This MM form allows us to design more flexible quantizers, which better approximate FP32 matrix multiplications by utilizing specific structures of the activations, weights, and gradients in transformers. Our quantizers leverage advances in the field of randomized numerical linear algebra (RandNLA) [14].
For forward propagation, we find that outliers in the activation are the main reason for accuracy degradation. To suppress outliers, we propose a Hadamard quantizer, which quantizes a transformed version of the activation matrix. The transformation is a block diagonal Hadamard matrix, which spreads the information carried in outliers to its nearby entries of the matrix and thus reduces the numerical range of the outliers.
For backpropagation, we exploit the structural sparsity of activation gradients. We find that the gradients of a few tokens are extremely large. Meanwhile, the gradients for the rest majority of the tokens are very small, even smaller than the quantization residuals of larger gradients. Rather than computing these small gradients, it is better to save the computational resources for calculating the residuals of the larger gradients. To utilize such sparsity, we propose bit splitting, which splits the gradient of each token into higher 4 bits and lower 4 bits. Then, we choose the most informative gradients by leverage score sampling, which is an importance sampling technique for RandNLA.
Combining quantization techniques for forward and backward propagation, we propose an algorithm that uses INT4 MMs for all linear operations in transformers. We evaluate our algorithm for training transformers on a wide variety of tasks, including natural language understanding, question answering, machine translation, and image classification. Our algorithm achieves competitive or superior accuracy compared with existing works on 4-bit training [47, 8]. Moreover, our algorithm is compatible with contemporary hardware like GPUs, since it does not require custom numerical formats like FP4 or logarithm formats. Our prototypical quantization + INT4 MM operator implementation is up to 2.2 times faster than the FP16 MM baseline, and it speeds up the training by up to  $35.1\%$ .
Finally, we would like to point out that utilizing ultra-low 4-bit numerical formats for training neural networks is still an open problem, and the main purpose of this research is to study whether it is possible to design an INT4 training algorithm that can achieve reasonable accuracy on practical tasks. The current research state of INT4 training is not yet mature enough to provide significant speedup for most tasks in a generic and plug-and-play manner. See Sec. 6 for further discussions.
Fully Quantized Training Fully quantized training (FQT) [33, 54, 46, 3, 15, 1, 57, 65, 29, 30, 59, 68] methods accelerate training by quantizing the activations, weights, and gradients to low-precision, so linear and nonlinear operators during training can be implemented with low-precision arithmetic. Research on FQT design novel numerical formats and quantization algorithms that better approximate full-precision tensors. The current research frontier is 4-bit FQT. FQT is challenging due to the vast numerical range of the gradient and the optimization issues of training quantized networks from scratch. Due to these challenges, existing 4-bit FQT algorithms [47, 8] still have  $\sim 1 - 2.5\%$  accuracy drop on several tasks, and they cannot support contemporary hardware.
Other Efficient Training Methods Mixture-of-experts [43] improves the model capacity without increasing the training budget. Structural dropout [21, 17] exploits computationally efficient ways to regularize the model. Efficient attention [27, 10] reduces the quadratic time complexity for computing attention. Distributed training systems [39, 22] reduce training time by leveraging more computational resources. Our work on reducing numerical precision is orthogonal with these directions.
3 Forward Propagation
Neural network training is an iterative optimization procedure with stochastic gradients computed by forward and backpropagation. We accelerate forward and back propagation with 4-bit integer
(INT4) arithmetic. We first describe the forward propagation of our training procedure. The forward propagation can be formulated as a composition of linear and non-linear (GeLU, normalization, softmax, etc.) operators. In our training procedure, we accelerate all the linear operators with INT4 arithmetic and leave all the less-computationally-intensive non-linear operators in the 16-bit floating-point (FP16) format. All linear operations in transformers can be written in a matrix multiplication (MM) form. For ease of presentation, we consider the acceleration of the following simple matrix multiplication throughout this paper:
$$
\mathbf {Z} = \mathbf {X} \mathbf {W} ^ {\top}, \text {w h e r e} \mathbf {Z} \in \mathbb {R} ^ {N \times C}, \mathbf {X} \in \mathbb {R} ^ {N \times D} \text {a n d} \mathbf {W} \in \mathbb {R} ^ {C \times D}. \tag {1}
$$
The most predominant use case of such MM is the fully-connected layer. Consider a transformer with an input shape of (batch size  $S$ , sequence length  $T$ , dimensionality  $D$ ). The fully-connected layer can be written as Eq. (1) where  $\mathbf{X}$  is the activation for  $N = ST$  tokens, and  $\mathbf{W}$  is the weight matrix. For attention layers, batch matrix multiplications (BMMs) might be required. Our proposed techniques can be applied to BMMs, and we leave the discussion of BMMs in Appendix. A.1.
3.1 Learned Step Size Quantization
To accelerate training, the forward propagation must be computed with integer arithmetic. We leverage the learned step size quantizer (LSQ) [16] for this purpose. LSQ is a static quantization method whose quantization scale does not depend on the input, and is thus cheaper than dynamic quantization methods [23], which need to compute the quantization scale dynamically per iteration.
Given a FP matrix  $\mathbf{X}$ , LSQ quantizes  $\mathbf{X}$  to integer with
$$
\left. \operatorname {i n t} _ {s _ {X}} (\mathbf {X}) := \left\lfloor \operatorname {c l a m p} \left(\mathbf {X} / s _ {X}, - Q _ {N}, Q _ {P}\right) \right\rfloor , \right. \tag {2}
$$
where  $s_X$  is a learnable scalar parameter, clamp restricts its input to the range  $[-Q_N, Q_P]$ ,  $[\cdot]$  is a rounding operation, and  $\mathbf{X} / s_X$  is computed elementwise. The resultant matrix takes values from  $\{-Q_N, -Q_N + 1, \dots, Q_P\}$ . Since we aim to perform INT4 MMs, we set  $Q_N = Q_P = 7$ . The integer matrix can be dequantized back to FP through float  $(\mathrm{int}_{s_X}(\mathbf{X})) = s_X \mathrm{int}_{s_X}(\mathbf{X}) \approx \mathbf{X}$ .
With LSQ, Eq. (1) can be computed approximately as  $\mathbf{Y} = \mathbf{X}\mathbf{W}^{\top} \approx s_{X}s_{W}\mathrm{int}_{s_{X}}(\mathbf{X})\mathrm{int}_{s_{W}}(\mathbf{W})^{\top}$ , where the INT4 MM  $\mathrm{int}_{s_X}(\mathbf{X})\mathrm{int}_{s_W}(\mathbf{W})^\top$  can be implemented efficiently on hardware.
Remark: Quantization-aware training (QAT) [9, 63, 67, 23, 12, 11, 44, 60, 45, 49, 64, 2, 18, 55] is an inference acceleration technique which trains networks with quantizers inserted in the forward propagation graph, so the trained network can perform efficiently during inference. QAT can compress activation/weights to extremely low precision (e.g. 1-2 bits). It is tempting to think that directly applying a quantizer for QAT to FQT can lead to similar low activation/weights bit-width. However, even only quantizing the forward propagation for FQT is much more challenging than QAT because: (1) QAT requires a converged full-precision model as initialization [16] and/or as a teacher model for knowledge distillation [2]; (2) QAT can adopt expensive multi-stage training pipelines without worrying about the convergence speed [32], while FQT algorithm must converge as fast as full-precision training algorithms to be useful; (3) QAT may approximate the discrete quantizer with continuous functions during training [19], which cannot be implemented with integer arithmetic. Due to these challenges, it is still an open problem to do FQT with 4-bit activations/weights.
3.2 Activation Outliers
Simply applying LSQ for FQT with 4-bit activation/weights leads to accuracy degradation due to activation outliers [58]. As shown in Fig. 1(a), activations have some outlier entries, which are much larger in magnitude than other entries. In this case, the step size  $s_X$  poses a trade-off between quantization granularity and representable numerical range. If  $s_X$  is large, we can represent the outliers well at the expense of representing most other entries in a very coarse manner. On the other hand, if  $s_X$  is small, we have to truncate the entries outside the range  $[-Q_Ns_X, Q_Ps_X]$ . Unfortunately, the transformers tend to store information in these outliers, and such truncation would seriously harm accuracy (see Sec. 5.2 for details). The outlier problem is particularly significant when the training task is to fine-tune a pre-trained model on some new downstream tasks, since the pre-train model contains more outliers [58] than random initialization.
Figure 1: Histogram of activation of the linear-1-2 layer in a BERT-base-uncased model. (a) Original activation distribution; (b) Hadamard-transformed activation distribution.
Figure 2: (a) The distribution of gradient norm along the token dimension. (b) The cumulative sum of the top X values as a percentage of the sum of all norms along the token dimension.
There exists some works to handle activation outliers for post-training quantization (PTQ). Outlier Suppression [56] discover that LayerNorms amplify outliers, and propose Gamma Migration and Token-Wise Clipping to solve this issue and achieves 6-bit BERT PTQ without too much degradation. SmoothQuant [58] migrates the quantization difficulty of activation outliers to weights and achieves 8-bit PTQ for large language models, such as OPT-175B. Outlier Channel Splitting [66] duplicates channels containing outliers with small overhead on the size of the network. However, these methods mainly focus on PTQ or QAT, and seldom successfully deal with ultra-low 4-bit training.
3.3 Hadamard Quantization
We propose a Hadamard quantizer (HQ) to solve the outlier problem. Its main idea is to quantize the matrices in another linear space which has fewer outliers.
The outliers in activation matrices form a feature-wise structure [58]. They are typically concentrated on a few dimensions, i.e., only a few columns of  $\mathbf{X}$  are significantly larger than others. Hadamard transform [48] is a linear transformation, which can amortize the outliers into other entries. Specifically, the Hadamard transform  $\mathbf{H}_k$  is a  $2^k\times 2^k$  matrix, where
$$
\mathbf {H} _ {0} = \left[ 1 \right], \quad \mathbf {H} _ {k} = \frac {1}{\sqrt {2}} \left[ \begin{array}{l l l} \mathbf {H} _ {k - 1} & \mathbf {H} _ {k - 1}; \mathbf {H} _ {k - 1} & - \mathbf {H} _ {k - 1} \end{array} \right].
$$
Hadamard matrices are orthogonal and symmetric:  $\mathbf{H}_k = \mathbf{H}_k^\top = \mathbf{H}_k^{-1}$ , so  $\mathbf{H}_k\mathbf{H}_k = \mathbf{I}, \forall k \geq 0$ . Consider any coordinate row vector*  $\mathbf{e}_i^\top \in \mathbb{R}^{2^k}$ , we have  $\mathbf{e}_i^\top \mathbf{H}_k = 2^{-k/2}\mathbf{1}_{2^k}, \forall i$ , where  $\mathbf{1}_{2^k} = (\pm 1, \pm 1, \ldots, \pm 1)$  is a  $2^k$ -dimensional vector with all of its values being 1 or  $-1$ . This demonstrates the extreme case when a single outlier dominates all the rest dimensions. In this case, Hadamard transformation effectively turns the vector into a quantization-friendly all-one-vector. The practical effect of the Hadamard transform on suppressing activation outliers is demonstrated in Fig. 1(b).
HQ uses a block-diagonal transformation matrix  $\mathbf{H} \in \mathbb{R}^{D \times D}$ :  $\mathbf{H} = \mathrm{BlockDiag}(\mathbf{H}_k, \ldots, \mathbf{H}_k)$ , where  $D$  is a multiple of  $2^k$ . To suppress outliers, we quantize a transformed version of  $\mathbf{X}$  and  $\mathbf{W}$ :
$$
\mathbf {X} = (\mathbf {X H}) \mathbf {H} ^ {\top} \approx s _ {X} \operatorname {i n t} _ {s _ {X}} (\mathbf {X H}) \mathbf {H} ^ {\top}, \quad \mathbf {W} = (\mathbf {W H}) \mathbf {H} ^ {\top} \approx s _ {W} \operatorname {i n t} _ {s _ {W}} (\mathbf {W H}) \mathbf {H} ^ {\top}.
$$
Combining the quantized matrices, we get
$$
\mathbf {Y} = \mathbf {X} \mathbf {W} ^ {\top} \approx s _ {X} s _ {W} \operatorname {i n t} _ {s _ {X}} (\mathbf {X H}) \mathbf {H} ^ {\top} \mathbf {H} \operatorname {i n t} _ {s _ {W}} \left(\mathbf {H} ^ {\top} \mathbf {W} ^ {\top}\right) = s _ {X} s _ {W} \operatorname {i n t} _ {s _ {X}} (\mathbf {X H}) \operatorname {i n t} _ {s _ {W}} \left(\mathbf {H} ^ {\top} \mathbf {W} ^ {\top}\right), \tag {3}
$$
where the inverse transformations cancel with each other, and the MM can be implemented as:
Procedure HQ-MM
1. Compute  $\mathbf{X}\mathbf{H}$  and  $\mathbf{H}^{\top}\mathbf{W}^{\top}$  in FP16.
2. Quantize the resultant matrices to INT4 by LSQ.
3. Multiply the two INT4 matrices.
4. Dequantize the resultant INT32 matrix to FP16 by multiplying  $s_X s_W$ .
For time complexity, Step 1 takes  $O(2^k N(D + C))$  FP16 multiply-accumulates (MACs); Step 2
* A vector which  $i$ -th dimension is 1, and all other dimensions are 0.
and Step 4 takes  $O(N(D + C))$  FP16 MACs in total; and Step 3 takes  $O(NDC)$  INT4 MACs. Comparing with the plain LSQ Eq. (2), the amount of FP16 MACs increases by  $2^{k}$  times, from  $O(N(D + C))$  to  $O(2^{k}N(D + C))$ . However, our HQ-MM is still much cheaper than an FP16 MM given  $2^{k} \ll D$  and  $2^{k} \ll C$ . The number  $k$  shows a tradeoff between the ability to suppress outliers and computation complexity. Larger  $k$  allows for amortizing the outlier within a larger horizon, at the cost of being more expensive. We propose an adaptive algorithm to choose  $k$  for each activation depending on the outlier scale, as discussed in Appendix A.5. The typical value is  $k = 5$ , while the dimensionality  $C$  and  $D$  ranges from 768 to 4096.
4 Backpropagation
We now consider accelerating the backpropagation of the linear layer with INT4 operations. The linear operator HQ-MM defined in Eq. (3) has four inputs: activation  $\mathbf{X}$ , weight  $\mathbf{W}$ , and step sizes  $s_X$ ,  $s_W$ . Given the output gradient  $\nabla_{\mathbf{Y}}\mathcal{L}$  w.r.t. some loss function  $\mathcal{L}$ , we need to compute the gradient of all four inputs. We discuss the computation of activation/weight gradients in this section, and left the discussion of step size gradients to Appendix A.3. For simplicity, we omit  $\mathcal{L}$  and simply use  $\nabla_{\mathbf{Y}}$  to denote the gradient in the following text.
By the straight-through estimator  $\lfloor x\rfloor^{\prime} = 1$  [5] and the chain rule, we have
$$
\nabla_ {\mathbf {W}} = s _ {X} \left(\nabla_ {\mathbf {Y}} ^ {\top} \hat {\mathbf {X}} \circ \mathbb {I} _ {W}\right) \mathbf {H} ^ {\top}, \quad \nabla_ {\mathbf {X}} = s _ {W} \mathbb {I} _ {X} \circ \nabla_ {\mathbf {Y}} \hat {\mathbf {W}} \mathbf {H} ^ {\top}, \tag {4}
$$
where we define  $\hat{\mathbf{X}} = \mathrm{int}_{s_X}(\mathbf{XH})$ ,  $\hat{\mathbf{W}} = \mathrm{int}_{s_W}(\mathbf{WH})$ ,  $\mathbb{I}_X = \mathbb{I}(-Q_N \leq \mathbf{X} / s_X \leq Q_P)$ , and  $\mathbb{I}_W = \mathbb{I}(-Q_N \leq \mathbf{W} / s_W \leq Q_P)$ . For computing the gradients, three types of matrix multiplications are required:
1. The element-wise multiplication  $\circ$  of a  $0/1$  matrix  $\mathbb{I}_X$  (or  $\mathbb{I}_W$ ) with another INT4 (or INT32) matrix. This operation has low time complexity.
2. The multiplication of an INT32 matrix with an FP16 block-wise Hadamard matrix  $s_W \mathbf{H}^\top$ , which also has low-time complexity, as discussed in Sec. 3.3.
3. The multiplication of the FP16 gradient  $\nabla_{\mathbf{Y}}$  with an INT4 matrix  $\hat{\mathbf{X}}$  or  $\hat{\mathbf{W}}$ , which we will accelerate by quantizing  $\nabla_{\mathbf{Y}}$  to INT4.
In the rest of this section, we will discuss quantization methods to compute the "type 3" MMs  $\nabla_{\mathbf{Y}}^{\top}\hat{\mathbf{X}}$  and  $\nabla_{\mathbf{Y}}\hat{\mathbf{W}}$ . We quantize  $\nabla_{\mathbf{Y}}$  dynamically for each MM, while  $\hat{\mathbf{X}}$  and  $\hat{\mathbf{W}}$  have been already calculated in forward propagation in Section. 3. We start by discussing the structure of the gradient.
4.1 Structural Sparsity of Gradients
We note that the gradient matrix  $\nabla_{\mathbf{Y}}$  tends to be very sparse along the training process. Furthermore, the sparsity has a structure: few rows (i.e., tokens) of  $\nabla_{\mathbf{Y}}$  have large entries, while most other rows are close to an all-zero vector. We illustrate this by plotting the histogram of per-row norm  $\| (\nabla_{\mathbf{Y}})_{i,:}\|$  for all the rows  $i$  in Fig. 2.
Such a structural sparsity arises from the heavy overparameterization [62] of modern neural networks. During almost the entire training process, the network operates in the overparameterized scheme [34], where it can fit most training data well, except for a few hard examples. Therefore, the (activation) gradient will be close to zero for well-fitted data points. We find that for pretraining tasks, such structural sparsity quickly emerges after only a few training epochs. For fine-tuning tasks, the gradient is always sparse during the whole training process.
4.2 Bit Splitting and Leverage Score Sampling
Here, we discuss how to design gradient quantizers to accurately compute the MMs during backpropagation by leveraging structural sparsity. The high-level idea is that many rows of the gradient are so small that they have little impact on the parameter gradient, yet they waste abundant computation. On the other hand, the large rows cannot be accurately represented with INT4. We drop some small rows and use the saved computation to represent large rows more accurately.
First, we propose bit splitting (BS), which splits a full-precision matrix as higher and lower 4 bits:
$$
\nabla_ {\mathbf {Y}} \approx s _ {\uparrow} \nabla_ {\mathbf {Y}} ^ {\uparrow} + s _ {\downarrow} \nabla_ {\mathbf {Y}} ^ {\downarrow}, \tag {5}
$$
where  $s_{\uparrow}, s_{\downarrow}$  are two floating-point scalars, and  $\nabla_{\mathbf{Y}}^{\uparrow}, \nabla_{\mathbf{Y}}^{\downarrow}$  are INT4 matrices representing the higher and lower 4 bits, respectively. BS can be implemented by first quantizing  $\nabla_{\mathbf{Y}}$  to INT4 as  $\nabla_{\mathbf{Y}} \approx s_{\uparrow} \nabla_{\mathbf{Y}}^{\uparrow}$  and then quantize the residual to INT4 as  $\nabla_{\mathbf{Y}} - s_{\uparrow} \nabla_{\mathbf{Y}}^{\uparrow} \approx s_{\downarrow} \nabla_{\mathbf{Y}}^{\downarrow}$ . BS can be viewed as an INT8 representation of a matrix, where  $\nabla_{\mathbf{Y}}^{\uparrow}$  and  $\nabla_{\mathbf{Y}}^{\downarrow}$  are the higher and lower 4 bits of the INT8 representation. Next, we discuss how to compute the weight and activation gradient.
Weight Gradient As discussed earlier, weight gradient involves the matrix multiplication  $\nabla_{\mathbf{Y}}^{\top}\hat{\mathbf{X}}$  where  $\nabla_{\mathbf{Y}}\in \mathbf{R}^{N\times C}$  and  $\hat{\mathbf{X}}$  is an  $N\times D$  INT4 matrix. By Eq. (5):
$$
\nabla_ {\mathbf {Y}} ^ {\top} \hat {\mathbf {X}} \approx \left(s _ {\uparrow} \nabla_ {\mathbf {Y}} ^ {\uparrow} ^ {\top} + s _ {\downarrow} \nabla_ {\mathbf {Y}} ^ {\downarrow} ^ {\top}\right) \hat {\mathbf {X}} = \nabla_ {\mathbf {Y}} ^ {\uparrow} ^ {\top} \mathbf {X} ^ {\uparrow}, \tag {6}
$$
where we define  $\nabla_{\mathbf{Y}}^{\uparrow} = [s_{\uparrow}\nabla_{\mathbf{Y}}^{\uparrow};s_{\downarrow}\nabla_{\mathbf{Y}}^{\downarrow}]^{\top}\in \mathbb{R}^{2N\times C}$  and  $\hat{\mathbf{X}}^{\uparrow} = [\hat{\mathbf{X}};\hat{\mathbf{X}}]$  to be a  $2N\times D$  INT4 matrix. Eq. (6) represents the product of an INT8  $\nabla_{\mathbf{Y}}^{\top}$  and an INT4  $\hat{\mathbf{W}}$ , and can be implemented by two INT4 MMs  $\nabla_{\mathbf{Y}}^{\uparrow}\hat{\mathbf{X}}$  and  $\nabla_{\mathbf{Y}}^{\downarrow}\hat{\mathbf{X}}$ . Such MM is rather accurate since  $\nabla_{\mathbf{Y}}$  is represented with 8 bits.
However, comparing to a naive quantization of  $\nabla_{\mathbf{Y}}$  to INT4, BS doubles the amount of INT4 operations for MM. We propose leverage score sampling (LSS) to cut the operations of Eq. (5) by half, to the same amount as the naive MM  $s_{\uparrow}\nabla_{\mathbf{Y}}^{\uparrow}\hat{\mathbf{X}}$ . Noticing that the MM Eq. (6) can be written as the sum of  $2N$  rank-1 matrices: