SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jerry Chee
jerrychee@cs.cornell.edu
Yaohui Cai
yc2632@cornell.edu
Volodymyr Kuleshov
Cornell University eshov@cornell.edu
Cornell University cdesa@cs.cornell.edu
This work studies post-training parameter quantization in large language models (LLMs). We introduce quantization with incoherence processing (QuIP), a new method based on the insight that quantization benefits from incoherent weight and Hessian matrices, i.e., from the weights being even in magnitude and the directions in which it is important to round them accurately being unaligned with the coordinate axes. QuIP consists of two steps: (1) an adaptive rounding procedure minimizing a quadratic proxy objective; (2) efficient pre- and post-processing that ensures weight and Hessian incoherence via multiplication by random orthogonal matrices. We complement QuIP with the first theoretical analysis for an LLM-scale quantization algorithm, and show that our theory also applies to an existing method, OPTQ. Empirically, we find that our incoherence preprocessing improves several existing quantization algorithms and yields the first LLM quantization methods that produce viable results using only two bits per weight. Our code can be found at https://github.com/Cornell-RelaxML/QuIP.
Large language models (LLMs) have enabled advances in text generation, few-shot learning, reasoning, protein sequence modeling, and other tasks [2, 30, 35]. The massive size of these models—often reaching into hundreds of billions of parameters—requires sophisticated deployment methods and motivates research into efficient inference algorithms.
This work studies the post-training quantization of LLM parameters as a way to improve their runtime efficiency [4, 8, 22, 31, 33, 34]. Our key insight is that quantization can be most effective when weight and proxy Hessian matrices are incoherent—that the weights themselves are even in magnitude, and the directions in which it is important to have good rounding accuracy are not too large in any one coordinate. Intuitively, incoherence can be thought of as a principled form of outlier reduction, which makes it easier to adaptively round the weights to a finite set of compressed values. We use this intuition to develop theoretically sound two-bit quantization algorithms that scale to LLM-sized models.
Specifically, we introduce quantization with incoherence processing (QuIP), a new method motivated by the above insight. QuIP consists of two steps: (1) an adaptive rounding [20] procedure, which minimizes a quadratic proxy objective  $\ell (\hat{W}) = \mathrm{tr}((\hat{W} -W)H(\hat{W} -W)^T)$  of the error between the original weights  $W$  and the quantized weights  $\hat{W}$  using an estimate of the Hessian  $H$ ; (2) efficient pre- and post-processing that ensures that the weight and Hessian matrices are incoherent by multiplying
them by a Kronecker product of random orthogonal matrices. We denote "incoherence processing" as both the pre- and post- processing steps of our procedure. Incoherence processing can be viewed as a form of outlier suppression across the weights and the activation space.
We complement our method with a theoretical analysis—the first for a quantization algorithm that scales to LLM-sized models—which analyzes the role of incoherence and shows that our quantization procedure is optimal within a general class of rounding methods. Interestingly, we find that QuIP without incoherence processing yields a more efficient implementation of an earlier algorithm, OPTQ [8]; our paper thus also provides the first theoretical analysis for that method.
Empirically, we find that incoherence processing greatly improves the quantization of large models, especially at higher compression rates, and yields the first LLM quantization method that produces viable results using only two bits per weight. For large LLM sizes ( $>2\mathrm{B}$  parameters), we observe small gaps between 2-bit and 4-bit compression that further decrease with model size, hinting at the feasibility of accurate 2-bit inference in LLMs.
Contributions. In summary, this paper makes the following contributions: (1) we propose QuIP, a quantization method based on the insight that model parameters should ideally be incoherent; (2) we provide a theoretical analysis for a broad class of adaptive rounding methods that encompass QuIP and OPTQ; (3) we demonstrate that QuIP makes two-bit LLM compression viable for the first time.
Adaptive rounding. Nagel et al. [20] are the first to motivate the "adaptive rounding" proxy objective (Eq. (1)) in a principled way. There are many quantization methods which quantize by optimizing this proxy objective [5, 6, 9, 12, 14, 20, 32]. Many require further retraining which can be expensive, and are not evaluated on the current largest open LLMs (OPT [35], BLOOM [30]). Lybrand and Saab [15] propose a greedy per-neuron quantization procedure that is similar to ours, except they do not consider arbitrary linear functions of the error correction. Their work bounds the proxy objective, albeit on the first layer only.
Post training quantization in large models. There is a growing body of work on PTQ in LLMs such as OPT and BLOOM. The size of these models make it difficult to apply previously developed methods. The majority of these methods make quantization easier by somehow reducing the range of weights or activations, but still use nearest rounding. SmoothQuant [31] rescales between activations and weights to remove outliers from the activations and make quantization overall easier. ZeroQuant [33] proposes a per-layer knowledge distillation method. LLM.int8() [4] decompose matrix multiplications into a majority of 8 bit and a minority of 16 bit operations. LUT-GEMM [22] designs kernels to accelerate quantized matrix multiplications. RPTQ [34] reorders activations and quantizes them in groups, reducing the impact of range differences between channels.
OPTQ (Formerly known as GPTQ). OPTQ [8] is based on OBQ [7], and proposes a novel rounding method that can work on the largest OPT and BLOOM models. The method works iteratively over the weight columns in a fixed order: (1) quantize with nearest rounding and compute the error, (2) update the remaining weights with a scaled error, and (3) repeat.
Other quantization methods. There are other quantization procedures which do not round based on the proxy objective of [20], or are not designed for the largest language models [10, 11, 13, 19, 28, 29].
3 Quantization With Incoherence Processing: Adaptive Rounding Step
This section introduces quantization with incoherence processing (QuIP), a new method consisting of: (1) an adaptive rounding step; (2) efficient pre- and post-processing that ensures weight and Hessian incoherence. We define and analyze step (1) in this section; the next section focuses on step (2).
Following existing state-of-the-art post-training quantization methods, we round weights per-layer by minimizing the "adaptive rounding" proxy objective, as in Nagel et al. [20],
$$
\ell (\hat {W}) = \mathbf {E} _ {x} \left[ \left\| (\hat {W} - W) x \right\| ^ {2} \right] = \operatorname {t r} \left((\hat {W} - W) H (\hat {W} - W) ^ {T}\right). \tag {1}
$$
Here,  $W \in \mathbb{R}^{m \times n}$  is the original weight matrix for a given linear layer,  $\hat{W} \in \mathbb{R}^{m \times n}$  are the quantized weights,  $x \in \mathbb{R}^n$  is an input vector drawn uniformly at random from a calibration set, and  $H$  is the
second moment matrix of these vectors, interpreted as a proxy Hessian. Crucially, this formulation lets the quantization be run in parallel across neurons, which is tractable for large language models [8]. For simplicity, we will focus in this section on rounding to the integers; subsequent sections will extend the analysis to finite grids.
3.1 LDLQ: An Optimal Adaptive Rounding Method
Our strategy is to define a family of adaptive rounding methods for optimizing objective (1) and then define LDLQ, the optimal method within that class. Our defined methods iteratively perform the following update for  $k = 1,2,\ldots ,n$ :
$$
\hat {W} _ {k} = \mathcal {Q} (W _ {k} + (W _ {1: (k - 1)} - \hat {W} _ {1: (k - 1)}) a _ {k}),
$$
where  $W_{k}$  denotes the  $k$ -th column,  $W_{1:(k-1)}$  denotes the first  $k-1$  columns, the subroutine  $\mathcal{Q}$  denotes either nearest rounding or standard unbiased rounding to the integers (which rounds up or down such that  $\mathbf{E}[\mathcal{Q}(z)] = z$ ), and  $a_{k} \in \mathbb{R}^{k-1}$  is some sequence of vectors. This scheme rounds columns one at a time; at each step, it adds a "correction" term that is a linear function of the residual from the rounding we have done so far. The final  $\hat{W}$  satisfies the following matrix equation:
$$
\hat {W} = \mathcal {Q} (W + (W - \hat {W}) U), \tag {2}
$$
where  $U$  is a strictly upper-triangular matrix whose columns are the vectors  $a_{k}$  and  $\mathcal{Q}$  acts elementwise. Because  $U$  is upper-triangular,  $\hat{W}_k$  only depends on  $\hat{W}_{1:(k - 1)}$ .
If we let  $\eta = \mathcal{Q}(W + (W - \hat{W})U) - (W + (W - \hat{W})U)$  denote the quantization error of  $\mathcal{Q}$ , we find that  $\hat{W} - W = \eta (U + I)^{-1}$  and we can rewrite objective (1) as
$$
\mathrm {t r} ((\hat {W} - W) H (\hat {W} - W) ^ {T}) = \mathrm {t r} (\eta (U + I) ^ {- 1} H (U + I) ^ {- T} \eta^ {T}). \qquad (3)
$$
The LDLQ Method How should we specify  $U$ , the linear feedback from the quantization error of preceding columns in (2)? Equation 3 provides an answer. If we choose  $U \gets \dot{U}$  such that the LDL decomposition of  $H$  is
$$
H = (\dot {U} + I) D (\dot {U} + I) ^ {T}, \tag {4}
$$
where  $D$  is a (non-negative) diagonal matrix and  $\dot{U}$  is upper unit triangular, then the terms  $(U + I)$  in Eq. (3) cancel. We denote as LDLQ the rounding procedure in Eq. (2) with  $U\gets \dot{U}$  as the LDL assignment from Eq. (4). We will now see that the LDL assignment of  $U$  is in fact optimal.
3.2 Deriving the Optimality of the LDLQ Adaptive Rounding Procedure
In order to reason about optimality, we consider weights which are worst and average-case for the proxy loss. Let  $\mathcal{A}$  denote a rounding method, and let  $\mathcal{A}(W,H)$  be the resulting quantized weights. Define the worst-case  $(\mathcal{L}_{\mathrm{worst}})$  and average  $(\mathcal{L}_{\mathrm{avg}})$  proxy losses with respect to the input weights as
$$
\mathcal {L} _ {\text {w o r s t}} (\mathcal {A}, H) = \sup  _ {W \in \mathbb {R} ^ {m \times n}} \mathbf {E} \left[ \operatorname {t r} \left((\mathcal {A} (W, H) - W) H (\mathcal {A} (W, H) - W) ^ {T}\right) \right] \tag {5}
$$
$$
\mathcal {L} _ {\mathrm {a v g}} (\mathcal {A}, H) = \mathbf {E} _ {W \sim \operatorname {U n i f} [ 0, 1 ] ^ {m \times n}} \left[ \operatorname {t r} \left((\mathcal {A} (W, H) - W) H (\mathcal {A} (W, H) - W) ^ {T}\right) \right]. \tag {6}
$$
Theorem 1. LDLQ is worst and average-case optimal amongst rounding methods which specify the linear feedback  $U$  as a function of  $H$  (not of  $W$ ), and when rounding to the integers. That is, for all rounding methods  $\mathcal{A}$  in the class described by Eq. (2), for all positive semi-definite  $H$ , and for  $\mathcal{Q}$  as either nearest or stochastic rounding,
$$
\frac {m}{4} \operatorname {t r} (D) = \mathcal {L} _ {\mathrm {w o r s t}} (\mathsf {L D L Q}, H) \leq \mathcal {L} _ {\mathrm {w o r s t}} (\mathcal {A}, H) a n d \frac {m}{c} \operatorname {t r} (D) = \mathcal {L} _ {\mathrm {a v g}} (\mathsf {L D L Q}, H) \leq \mathcal {L} _ {\mathrm {a v g}} (\mathcal {A}, H),
$$
where  $D$  is the matrix from the LDL decomposition of  $H$ , and  $c = 12$  for nearest,  $c = 6$  for stochastic.
Remarks. The number of rows being quantized is  $m$ , and each quantization method operates across the  $n$  entries of each row. For all rounding methods described by Eq. (2), and for all positive semi-definite  $H$ ,  $\mathcal{Q}$  as nearest rounding achieves the same worst-case proxy loss as stochastic rounding, but achieves better average proxy loss.
Figure 1:  $\operatorname{eig}(H)$  from OPT-2.7b.
Figure 2: Max  $|W_{ij}|$  before and after incoherence processing on OPT-2.7b.
Figure 3: Max  $|Q_{ij}|$  before and after incoherence processing, with  $Q$  the eigenvectors of  $H$  on OPT-2.7b.
Moving beyond a generic algorithm  $\mathcal{A}$  within our framework, we consider the common baselines of nearest and stochastic rounding. These methods are represented within our framework by choosing the appropriate  $\mathcal{Q}$  subroutine, and setting all entries of the linear feedback to zero.
For these baseline methods, their optimality gap to LDLQ is governed by  $\operatorname{tr}(D)$  vs.  $\operatorname{tr}(H)$ . For any non-diagonal  $\tilde{H} \succeq 0$ , LDLQ achieves strictly lower worst and average-case proxy loss because  $\operatorname{tr}(D) < \operatorname{tr}(\tilde{H})$ . Let  $\mathcal{B} = \{\text{Near}, \text{Stoch}\}$ . Then,  $\mathcal{L}_{\text{worst}}(\text{LDLQ}, \tilde{H}) < \mathcal{L}_{\text{worst}}(\text{Stoch}, \tilde{H})$  and  $\mathcal{L}_{\text{avg}}(\text{LDLQ}, \tilde{H}) < \mathcal{L}_{\text{avg}}(\mathcal{B}, \tilde{H})$ . Across OPT models  $125\text{m}$  to  $2.7\text{b}$ ,  $\operatorname{tr}(D) / \operatorname{tr}(H) \leq 0.65$  empirically verifying that the gap is not insignificant. See Supplement C for full details.
3.3 Incoherence: Optimality with a Spectral Bound
Theorem 1 gives exact expressions for the proxy loss, albeit with  $\operatorname{tr}(D)$ , which can be difficult to reason about. In Figure 1, we empirically observe that  $H$  is approximately low-rank: we visualize the spectrum of several randomly chosen  $H$  from OPT-2.7b, and observe that the spectrum decays rapidly. In fact, across all layers of OPT-125m to 2.7b models, a vast majority of  $H$  matrices have fewer than a quarter of eigenvalues  $>1\%$  of the max eigenvalue; see Supplement C for full details. Given this observation about the low rank of  $H$ , can we bound the behavior of LDLQ, and thus  $\operatorname{tr}(D)$ , using the spectrum of  $H$ ?
We do this building on a variant of the incoherence assumption that is specialized to our case [3, 24].
Definition 1. We say a symmetric Hessian matrix  $H \in \mathbb{R}^{n \times n}$  is  $\mu$ -incoherent if it has an eigendecomposition  $H = Q \Lambda Q^T$  such that for all  $i$  and  $j$ ,  $|Q_{ij}| = |e_i^T Q e_j| \leq \mu / \sqrt{n}$ . By extension, we say a weight matrix  $W \in \mathbb{R}^{m \times n}$  is  $\mu$ -incoherent if all  $i$  and  $j$ ,  $|W_{ij}| = |e_i^T W e_j| \leq \mu \|W\|_F / \sqrt{mn}$ .
Note that "most"  $n \times n$  matrices are incoherent with  $\mu = \mathcal{O}(\sqrt{\log n}) = \tilde{\mathcal{O}}(1)$  because a random orthogonal matrix has entries with squared-magnitudes that concentrate around their mean of  $1/n$ . Incoherence in  $W$  can be viewed as a form of outlier reduction: a small bound on the magnitude of its entries means that we do not need to scale it as much to make it fit in the finite range of representable low-precision numbers. Figures 2 and 3 plot the max absolute weight and hessian eigenvector entries before and after our incoherence processing, on all layers in OPT-2.7b. A line with slope=1 is drawn for reference. We see that  $W$  and  $H$  are more incoherent after our incoherence processing is applied. Making  $H$  incoherent is less intuitive, but its utility is motivated by the following lemma.
Lemma 2. Let  $H \in \mathbb{R}^{n \times n}$  be a  $\mu$ -incoherent positive semi-definite symmetric matrix and let  $H = (\dot{U} + I)D(\dot{U} + I)^T$  be its LDL Cholesky decomposition, where  $\dot{U}$  is a strictly upper triangular matrix and  $D$  is a (non-negative) diagonal matrix. Then,
$$
\operatorname {t r} (D) \leq \frac {\mu^ {2}}{n} \operatorname {t r} \left(H ^ {1 / 2}\right) ^ {2}.
$$
To the best of our knowledge, this is a novel result using incoherence to obtain a bound on  $\operatorname{tr}(D)$  that depends only on the spectrum of  $H$ . To help interpret this result, we derive explicit proxy losses for plain nearest and stochastic rounding, which we will then compare to what LDLQ gets via Lemma 2.
Algorithm 1 QuIP - Incoherence Pre-Processing
Require:  $b\in \mathbb{N}$ $H\in \mathbb{R}^{n\times n}$  SPD, original  $W\in \mathbb{R}^{m\times n}$ $\rho \in \mathbb{R}_+$ $\alpha \in [0,1]$  1: seeded sample random two-factor orthogonal matrices  $U\in \mathbb{R}^{m\times m}$  and  $V\in \mathbb{R}^{n\times n}$  2:  $H = H + \alpha *\mathrm{mean}(\mathrm{diag}(H))I$  ▷ from OPTQ 3:  $\tilde{D}\gets \sqrt[4]{\mathrm{diag}(H) / \mathrm{diag}(W^T W)}$  ▷  $\sqrt[4]{-}$  applies element-wise 4:  $W\leftarrow W\tilde{D};H\leftarrow \tilde{D}^{-1}H\tilde{D}^{-1}$  ▷ diagonal rescaling 5:  $W\leftarrow UWW^{T};H\leftarrow VHVT$  ▷ incoherence 6:  $s\gets \rho \| W\| _F / \sqrt{mn};W\gets \frac{1}{2} (\frac{1}{s} W + 1)$  ▷ reduced quantization range due to incoherency 7:  $W\gets \mathrm{clamp}(W*(2^b -1),0,2^b -1)$  ▷ rescale  $W$  to lie within  $[0,2^b -1]$  8: return  $W,H,s,\tilde{D}$
Algorithm 2 QuIP - Incoherence Post-Processing
Require:  $b\in \mathbb{N}$ $H\in \mathbb{R}^{n\times n}$  SPD, quantized  $W\in [0,2^b -1]^{m\times n}$ $s\in \mathbb{R}\& \tilde{D}\in \mathbb{R}^{n\times n}$  (Alg 1) 1: seeded sample random two-factor orthogonal matrices  $U\in \mathbb{R}^{m\times m}$  and  $V\in \mathbb{R}^{n\times n}$  2:  $W\gets s*\big((W / (2^b -1))\ast 2 - 1\big)$  3:  $W\leftarrow U^{T}W V$  .  $H\gets V^T HV$  4: return  $W\gets W\tilde{D}^{-1}$  1  $\triangleright$  revert incoherence.  $\triangleright$  revert diagonal rescaling
Lemma 3. Let  $H$  be symmetric positive definite. In the worst case stochastic rounding achieves  $\mathcal{L}_{\mathrm{worst}}(\mathrm{Stoch},H) = (m / 4)\operatorname {tr}(H)$ . In the average case nearest and stochastic rounding achieve  $\mathcal{L}_{\mathrm{avg}}(\{\mathrm{Near},\mathrm{Stoch}\} ,H) = (m / c)\operatorname {tr}(H)$ , where  $c = 12$  for nearest, and  $c = 6$  for stochastic.
To interpret this result, consider  $H$  rank-  $k$  with  $\mu^2 k < n$ . By Cauchy-Schwarz,  $\mathrm{tr}(H^{1/2})^2 \leq k$  and  $H$  is a combining Lemma 2 with the LDLQ proxy losses of Theorem 1 and comparing with Lemma 3,
$$
\mathcal {L} _ {\text {w o r s t}} (\mathrm {L D L Q}, H) \leq \frac {m \mu^ {2}}{4 n} \operatorname {t r} \left(H ^ {1 / 2}\right) ^ {2} \leq \frac {m \mu^ {2} k}{4 n} \operatorname {t r} (H) \leq \frac {m}{4} \operatorname {t r} (H) = \mathcal {L} _ {\text {w o r s t}} (\mathrm {S t o c h}, H)
$$
$$
\mathcal {L} _ {\mathrm {a v g}} (\mathsf {L D L Q}, H) \leq \frac {m \mu^ {2}}{c n} \operatorname {t r} \left(H ^ {1 / 2}\right) ^ {2} \leq \frac {m \mu^ {2} k}{c n} \operatorname {t r} (H) \leq \frac {m}{c} \operatorname {t r} (H) = \mathcal {L} _ {\mathrm {a v g}} (\mathcal {B}, H),
$$
where  $\mathcal{B} \in \{\text{Near}, \text{Stoch}\}$ , and  $c$  is as given in Theorem 1. This shows that for sufficiently low-rank  $H$ , LDLQ is asymptotically better than plain nearest and stochastic rounding by a factor of  $\mu^2 k / n$ .
Without incoherence: no improvement with a spectral bound. By assuming incoherence, we were able to show LDLQ gets an asymptotically better bound in terms of just the spectrum of  $H$ . We might ask: was the incoherence assumption necessary to get this result? The following theorem answers this question in the affirmative by showing that without incoherence, the best spectral bound for LDLQ cannot differentiate it from the nearest and stochastic rounding baselines.
Theorem 4. Consider all  $\tilde{H}$  with the same spectrum as  $H$ . For any positive semi-definite  $H$ , the following holds. On the worst-case loss LDLQ achieves the same error as stochastic rounding,
$$
\sup _ {\tilde {H} \text {s . t . e i g} (\tilde {H}) = \text {e i g} (H)} \mathcal {L} _ {\text {w o r s t}} (\mathsf {L D L Q}, \tilde {H}) = \mathcal {L} _ {\text {w o r s t}} (\mathsf {S t o c h}, H) = \frac {m}{4} \operatorname {t r} \left(H\right).
$$
On the average-case loss LDLQ achieves the same error as the corresponding rounding routine. Let  $\mathcal{B} = \{\text{Near}, \text{Stoch}\}$  and  $c = 12$  for nearest,  $c = 6$  for stochastic.
$$
\sup_{\tilde{H} s.t.eig(\tilde{H}) = \operatorname {eig}(H)}\mathcal{L}_{\operatorname {avg}}(\mathsf{LDLQ}^{*},\tilde{H}) = \mathcal{L}_{\operatorname {avg}}(\mathcal{B},H) = \frac{m}{c}\operatorname {tr}\left(H\right).
$$
Note that the worst case for comparing LDLQ against these baselines occurs when  $H$  is diagonal, see Theorem 1 and Lemma 3. Assuming incoherence as we do is a natural way to exclude such cases.
4 Quantization With Incoherence Processing: Incoherence Processing Step
Next, we leverage the above incoherence analysis to introduce incoherence processing, the second step of the QuIP algorithm. Our strategy will be to pre-process weight and Hessian matrices to ensure
the favorable incoherence properties outlined above. One straightforward way to make a symmetric matrix incoherent is to conjugate it by a uniform random orthogonal matrix: this will result in each of its eigenvectors being a random unit vector, whose entries will concentrate around magnitude  $n^{-1/2}$ .
Specifically, let  $U \in \mathbb{R}^{m \times m}$  and  $V \in \mathbb{R}^{n \times n}$  be two random orthogonal matrices. (Let's temporarily ignore how these matrices are generated, or how we would efficiently perform inference.) We ensure the weight and Hessian are incoherent with high probability through random orthogonal multiplications  $\tilde{H} \gets VHV^T$  and  $\tilde{W} \gets UWV^T$ . Importantly, this transformation preserves the proxy quadratic form since  $\mathrm{tr}(\tilde{W}\tilde{H}\tilde{W}^T) = \mathrm{tr}((UWV^T)(VHV^T)(VW^TU^T)) = \mathrm{tr}(WHW^T)$ .
4.1 Incoherence via Efficient Orthogonal Multiplication
If all we wanted to do was to store or transmit the weights of the quantized neural network, the above procedure would introduce no overhead, since we can generate a random orthogonal matrix from a seed—making it essentially free to store. However, for running inference on a DNN, we need to multiply by the weight matrix  $W$ , and here the need to manifest and multiply by  $n \times n$  random orthogonal matrices  $U, V$  would be prohibitive.
To handle this, we propose to instead use a distribution over random orthogonal matrices for which multiplication is fast. Let  $n = pq$  be a factorization of  $n$  (where  $p \approx q \approx \sqrt{n}$ ), and set  $U = U_L \otimes U_R$  where  $U_L$  is sampled uniformly from the  $p \times p$  orthogonal matrices and  $U_R$  is sampled uniformly from the  $q \times q$  orthogonal matrices. Multiplication of a vector  $x \in \mathbb{R}^n$  by the matrix  $U$  can be accomplished by reshaping to a  $p \times q$  matrix, multiplying on the left by  $U_L$  and the right by  $U_R^T$ , and then reshaping back: this takes  $O(n(p + q)) = o(n^2)$  operations. Using more than two factors in this way is also possible, but using two suffices to make this preprocessing asymptotically non-dominant.
Lemma 5. Let  $H$  be a positive semi-definite matrix on  $\mathbb{R}^{n \times n}$  and  $W$  a matrix on  $\mathbb{R}^{m \times n}$ , and suppose that  $m = p_1 \cdot p_2 \cdots p_k$  and  $n = q_1 \cdot q_2 \cdots q_k$ . Let  $U_1, U_2, \ldots, U_k, V_1, V_2, \ldots, V_k$  be independent random orthogonal matrices on  $\mathbb{R}^{p_i \times p_i}$  and  $\mathbb{R}^{q_i \times q_i}$  respectively. Set  $U$  as the Kronecker product  $U = U_1 \otimes U_2 \otimes \dots \otimes U_k$  and  $V$  as  $V = V_1 \otimes V_2 \otimes \dots \otimes V_k$ . Then  $VH^T$  is  $\mu_H$ -incoherent with probability at least  $1 - \delta$ , and  $UWV^T$  is  $\mu_W$ -incoherent with probability at least  $1 - \delta$ , where
$$
\mu_ {H} = A ^ {k / 2} \log \left(\frac {C k n ^ {2}}{\delta}\right) ^ {k / 2} = \tilde {\mathcal {O}} (1) a n d \mu_ {W} = A ^ {k} \log \left(\frac {2 C k m n}{\delta}\right) ^ {k} = \tilde {\mathcal {O}} (1)
$$
for some global constants  $A$  and  $C$  independent of  $n$  and  $k$ .