SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 15: Average perplexity difference (i.e. unbiased - biased) for LDLQ/OPTQ on WikiText2, PTB, and C4. That is, we can run LDLQ with the  $\mathcal{Q}$  subroutine as stochastic rounding, instead of nearest. The average difference is positive, meaning that unbiased rounding performs worse than biased (i.e. nearest) across OPT models  $125\mathrm{m}$  to 2.7b. Note the magnitude of the gap increases at lower bits.
Model	WBits	Incoherence Processing (ours)	Baseline Processing
Wiki	PTB	C4	Wiki	PTB	C4
OPT-1.3b	4	16.54	22.12	17.58	15.43	21.92	16.80
3	18.27	23.96	18.66	20.45	28.86	20.68
2	38.13	51.78	31.09	6,438.75	6,099.27	2,057.71
OPT-350m	4	23.19	32.55	23.48	23.71	33.73	24.29
3	25.54	36.74	25.52	33.01	45.15	30.09
2	286.71	367.26	144.08	8,006.22	7,445.70	2,317.18
OPT-125m	4	32.04	44.56	29.08	32.59	41.95	28.67
3	40.66	51.90	32.91	50.73	74.14	41.04
2	1,649.83	240.86	136.55	3,714.11	4,703.76	1,848.72
Table 16: Quantizing OPT models using Algorithm 5 evaluated on WikiText2, PTB, and C4. At 2 bits and incoherence processing, we see improvements over LDLQ and LDLQ-RG on OPT-125m and OPT-350m, but diminishing improvements on OPT-1.3b. Due to Algorithm 5's relatively equivalent performance relative to QuIP at OPT-1.3b, and due to this algorithm's increased computational cost, we decide not to user it.
Table 16 shows results from using Algorithm 5 to quantize OPT models  $125\mathrm{m}$  to  $1.3\mathrm{b}$ , with incoherence processing and baseline processing. At 2 bits and incoherence processing, we observe modest improvements over QuIP in terms of perplexity on OPT models  $125\mathrm{m}$  and  $350\mathrm{m}$ . However, at the
larger OPT-1.3b QuIP beats Algorithm 5 on 2/3 language generation tasks. In addition, Algorithm 5 is computationally more work to run. Therefore we decide not to use it.
Another observation: in practice, we don't seem to encounter constructions of  $W$  and  $H$  that are bad for LDLQ/OPTQ. Therefore this "clamping" issue seems to not be an issue in practice, especially as model size increases.
D Proofs for Section 3 (Quantization With Incoherence Processing: Adaptive Rounding Step)
Subsection 3.2 (Deriving the Optimality of the LDLQ Adaptive Rounding Procedure)
Proof. Let  $X$  be the strictly upper triangular matrix associated with the rounding procedure  $\mathcal{A}$  such that  $U \gets X$  in Eq. (2). Let  $B \equiv (X + I)^{-1}(\dot{U} + I)$  where  $\dot{U}$  is from the LDL decomposition of  $H$  in Eq. (4). The proxy loss is then,
$$
\begin{array}{l} \operatorname {t r} \left((\mathcal {A} (W, H) - W) H (\mathcal {A} (W, H)) ^ {T}\right) \stackrel {(3), (4)} {=} \operatorname {t r} \left(\eta (X + I) ^ {- 1} (\dot {U} + I) D (\dot {U} + I) ^ {T} (X + I) ^ {- T} \eta^ {T}\right) \\ = \operatorname {t r} \left(\eta B D B ^ {T} \eta^ {T}\right). \tag {9} \\ \end{array}
$$
With the LDL assignment of  $U$ , we further have that,
$$
\operatorname {t r} \left(\eta B D B ^ {T} \eta^ {T}\right) = \operatorname {t r} \left(\eta D \eta^ {T}\right). \tag {10}
$$
First, consider the worst-case loss,  $\mathcal{L}_{\mathrm{worst}}$ . The goal is to construct a particularly bad case where the entries of  $\tilde{W}$  are  $1/2 \pm \epsilon$ , and thus when rounding to the integers we will always have error  $1/2$ . Construct a weight matrix  $\tilde{W} \in \mathbb{R}^{m \times n}$  such that each entry satisfies,
$$
\tilde {W} _ {i j} = \left\{ \begin{array}{l l} 0. 5 - \epsilon & w. p. 1 / 2 \\ 0. 5 + \epsilon & w. p. 1 / 2 \end{array} \right. \Rightarrow \eta_ {i j} = \left\{ \begin{array}{l l} + 0. 5 & w. p. 1 / 2 \\ - 0. 5 & w. p. 1 / 2 \end{array} \right.,
$$
and the quantization errors  $\eta \in \mathbb{R}^{m\times n}$  are for each entry  $\{+1 / 2, - 1 / 2\}$  with equal probability. For this particular  $\tilde{W}$ ,  $\mathcal{A}$  achieves proxy loss  $\mathcal{L}_{\mathrm{worst}}(\mathcal{A},H)\stackrel {(9)}{=}\mathbf{E}\left[\operatorname {tr}\left(\eta BDB^T\eta^T\right)\right] = \frac{m}{4}\operatorname {tr}\left(BDB^T\right)$ , with  $\mathcal{Q}$  as either nearest or stochastic rounding. It follows from the supremum in the definition of  $\mathcal{L}_{\mathrm{worst}}$  in Eq. (5) that,  $\mathcal{L}_{\mathrm{worst}}(\mathcal{A},H)\geq \frac{m}{4}\operatorname {tr}\left(BDB^T\right)$ . For the LDL assignment of  $U$ , the worst case expected quantization error rounding to the integers is  $1 / 2$ . Therefore,  $\mathcal{L}_{\mathrm{worst}}(\mathsf{LDLQ},H)\stackrel{(10)}{\underset{m}{=}}\operatorname {tr}(D)$ , again for  $\mathcal{Q}$  as either nearest or stochastic rounding.  $B$  must be a unit triangular matrix since it is the product of unit triangular matrices. Therefore  $\operatorname {tr}\left(BDB^T\right)$  is minimized when  $B = I$ , and
$$
\mathcal {L} _ {\text {w o r s t}} (\mathrm {L D L Q}, H) \leq \mathcal {L} _ {\text {w o r s t}} (\mathcal {A}, H).
$$
Next, consider the average loss,  $\mathcal{L}_{\mathrm{avg}}$ , where  $W \sim Unif[0,1]^{m \times n}$ . For  $\mathcal{Q}$  as nearest rounding, the entries of the quantization error  $\eta$  are  $Unif[-\frac{1}{2}, \frac{1}{2}]$ , because each entry is independent and uniformly distributed. It follows that for any entry of  $\eta$ ,  $\mathbf{E}\left[\eta_{ij}^2\right] = \int_{-1/2}^{1/2} x^2 dx = \frac{1}{12}$ . Therefore,  $\mathcal{L}_{\mathrm{avg}}(\mathcal{A}, H) \stackrel{(9)}{=} \mathbf{E}_{W \sim Unif[0,1]^{m \times n}}\left[\operatorname{tr}\left(\eta BDB^T\eta^T\right)\right] = \frac{m}{12}\operatorname{tr}\left(BDB^T\right)$ . For  $\mathcal{Q}$  as stochastic rounding, the entries of the quantization error  $\eta$  are  $Unif[-1,1]$ . It follows that for any entry of  $\eta$ ,  $\mathbf{E}\left[\eta_{ij}^2\right] = \int_0^1 x(1 - x)dx = \frac{1}{6}$ . Note that for stochastic rounding, the quantization error will be  $x$  with probability  $(1 - |x|)$ . Therefore,  $\mathcal{L}_{\mathrm{avg}}(\mathcal{A}, H) = \frac{m}{6}\operatorname{tr}\left(BDB^T\right)$ . Based on these same calculations of  $\mathbf{E}\left[\eta_{ij}^2\right]$ ,
we have that  $\mathcal{L}_{\mathrm{avg}}(LDL,H)\stackrel {(9)}{=}\frac{m}{12}\operatorname {tr}(D)$  with  $\mathcal{Q}$  as nearest, and  $= \frac{m}{6}\operatorname {tr}(D)$  with  $\mathcal{Q}$  as stochastic rounding. By the same reasoning on the minimization of  $\operatorname {tr}\left(BDB^T\right)$
$$
\mathcal {L} _ {\mathrm {a v g}} (\mathsf {L D L Q}, H) \leq \mathcal {L} _ {\mathrm {a v g}} (\mathcal {A}, H).
$$
Subsection 3.3 (Incoherence: Optimality with a Spectral Bound)
Definition 1. We say a symmetric Hessian matrix  $H \in \mathbb{R}^{n \times n}$  is  $\mu$ -incoherent if it has an eigendecomposition  $H = Q\Lambda Q^T$  such that for all  $i$  and  $j$ ,  $|Q_{ij}| = |e_i^T Q e_j| \leq \mu / \sqrt{n}$ . By extension, we say a weight matrix  $W \in \mathbb{R}^{m \times n}$  is  $\mu$ -incoherent if all  $i$  and  $j$ ,  $|W_{ij}| = |e_i^T W e_j| \leq \mu \|W\|_F / \sqrt{mn}$ .
Lemma 8. Let  $H \in \mathbb{R}^{n \times n}$  be a positive semi-definite symmetric matrix, and let  $a_1, \ldots, a_n$  be a sequence of vectors in  $\mathbb{R}^n$ . Consider the recurrence given by  $\Sigma_0 = 0 \in \mathbb{R}^{n \times n}$  and from  $k = 0$  to  $n - 1$
$$
\Sigma_ {k + 1} = \left(I - e _ {k} a _ {k} ^ {T}\right) \Sigma_ {k} \left(I - a _ {k} e _ {k} ^ {T}\right) + e _ {k} e _ {k} ^ {T}.
$$
Let  $\ell(a_1, \ldots, a_n) = \operatorname{tr}(H\Sigma_n)$ . Then if  $H = LDL^T$  is the LDL decomposition of  $H$ , a global minimum of  $\ell$  occurs when  $a_k$  is the  $k$ th column of  $L$ , and at this minimum,  $\ell = \operatorname{tr}(D)$ .
Proof. First observe that at step  $k$ ,  $\Sigma_{k}$  will be 0 in all entries  $(\Sigma_{k})_{ij}$  if  $\min(i,j) \geq k$ . This means that changing the last  $n - k$  entries of  $a_{k}$  does not change  $\Sigma$  (or  $\ell$ ) at all. Without loss of generality, set those entries of  $a_{k}$  to 0. If  $A$  is the matrix whose  $k$ th row is  $a_{k}$ , this is equivalent to saying that  $A$  is strictly lower triangular.
Next, let  $\eta$  be a random Gaussian sampled from  $\mathcal{N}(0,I)$ , and consider the recurrence given by  $x_0 = 0\in \mathbb{R}^n$  and
$$
x _ {k + 1} = x _ {k} - e _ {k} a _ {k} ^ {T} x _ {k} + e _ {k} e _ {k} ^ {T} \eta .
$$
It's straightforward to see that  $\Sigma_{k} = \mathbf{E}\left[x_{k}x_{k}^{T}\right]$ . But it's also easy to see that the step- $k$  update only modifies/assigns the  $k$ th entry of  $x$ , and does so based only on earlier entries of  $x$ . Since  $e_k^T x_k = 0$ , and no later step assigns the  $k$ -or-lower entries of  $x$ ,
$$
e _ {k} ^ {T} x _ {n} = e _ {k} ^ {T} x _ {k + 1} = 0 - a _ {k} ^ {T} x _ {k} + e _ {k} ^ {T} \eta = - a _ {k} ^ {T} x _ {n} + e _ {k} ^ {T} \eta ,
$$
which in vector form yields
$$
(I + A) x _ {n} = \eta .
$$
In particular, this immediately implies that
$$
\Sigma_ {n} = (I + A) ^ {- 1} (I + A) ^ {- T}
$$
$$
\ell = \operatorname {t r} \left(H \Sigma_ {n}\right) = \operatorname {t r} \left(\left(I + A\right) ^ {- T} H \left(I + A\right) ^ {- 1}\right) = \operatorname {t r} \left(B ^ {- T} H B ^ {- 1}\right).
$$
where  $B = I + A$ . Differentiating with respect to  $B$  in strictly lower triangular direction  $\Delta$  (the only direction in which we have degress of freedom, since the diagonal of  $B$  must be unit) yields
$$
- 2 \operatorname {t r} \left(B ^ {- T} H B ^ {- 1} \Delta B ^ {- 1}\right).
$$
It's not hard to see that if  $H = LDL^T$  is the LDL decomposition of  $H$ , and  $B^T = L$ , that the gradient is
$$
- 2 \operatorname {t r} \left(D \Delta B ^ {- 1}\right) = - 2 \operatorname {t r} \left(\Delta B ^ {- 1} D\right) = - 2 \langle \Delta^ {T}, B ^ {- 1} D \rangle .
$$
Since  $\Delta^T$  is strictly upper triangular, but  $B^{-1}D$  must be lower triangular, this is 0 so we have a minimum. The uniqueness of this minimum (up to assignments of the lower-triangular elements of  $A$  or  $B$ , which have no effect on  $\ell$ ) also immediately follows from the recurrence relation. This implies the minimum is global. This is what we wanted to show.
Proof. By continuity of  $\operatorname{tr}(D)$  and  $\operatorname{tr}\left(H^{1/2}\right)$ , it suffices to prove the lemma for positive definite  $H$ . First, the closure of positive definite symmetric matrices is the set of positive semi-definite symmetric matrices. Second, consider the set of  $H$  that are positive definite and satisfy  $\frac{\mu^2}{n} \operatorname{tr}\left(H^{1/2}\right)^2 - \operatorname{tr}(D) \geq 0$ , i.e. are non-negative. The closure of this set (i.e.  $H \succeq 0$ ) must also satisfy that the inequality is non-negative.
Let  $H = Q\Lambda Q^T$  be the eigendecomposition of  $H$ . First, observe that by incoherence,
$$
e _ {k} ^ {T} H ^ {1 / 2} e _ {k} = \sum_ {i = 1} ^ {n} \lambda_ {i} ^ {1 / 2} (e _ {i} ^ {T} Q e _ {k}) ^ {2} \leq \frac {\mu^ {2}}{n} \sum_ {i = 1} ^ {n} \lambda_ {i} ^ {1 / 2} = \frac {\mu^ {2}}{n} \operatorname {t r} \left(H ^ {1 / 2}\right).
$$
$$
\alpha = \frac {\mu^ {2}}{n} \operatorname {t r} \left(H ^ {1 / 2}\right),
$$
and consider the recurrence from Lemma 8 with
$$
a _ {k} = \frac {H ^ {1 / 2} e _ {k}}{\alpha}
$$
$$
\Sigma_ {k + 1} = \left(I - \alpha^ {- 1} e _ {k} e _ {k} ^ {T} H ^ {1 / 2}\right) \Sigma_ {k} \left(I - \alpha^ {- 1} H ^ {1 / 2} e _ {k} e _ {k} ^ {T}\right) + e _ {k} e _ {k} ^ {T}.
$$
Suppose by way of induction that for some scalar the covariance  $\Sigma_k \preceq \alpha H^{-1/2}$ . For the base case, this obviously holds since  $\Sigma_0 = 0$ . At step  $k$ ,
$$
\begin{array}{l} \Sigma_ {k + 1} \preceq \left(I - \alpha^ {- 1} e _ {k} e _ {k} ^ {T} H ^ {1 / 2}\right) \alpha H ^ {- 1 / 2} \left(I - \alpha^ {- 1} H ^ {1 / 2} e _ {k} e _ {k} ^ {T}\right) + e _ {k} e _ {k} ^ {T} \\ = \alpha H ^ {- 1 / 2} - 2 e _ {k} e _ {k} ^ {T} + \alpha^ {- 1} e _ {k} e _ {k} ^ {T} H ^ {1 / 2} e _ {k} e _ {k} ^ {T} + e _ {k} e _ {k} ^ {T} \\ \succ \alpha H ^ {- 1 / 2}. \\ \end{array}
$$
Note that with this assignment,
$$
a _ {k} ^ {T} \Sigma_ {k} a _ {k} \leq (\alpha^ {- 1} e _ {k} ^ {T} H ^ {1 / 2}) (\alpha H ^ {- 1 / 2}) (\alpha^ {- 1} H ^ {1 / 2} e _ {k}) = \alpha^ {- 1} e _ {k} ^ {T} H ^ {1 / 2} e _ {k} \leq 1.
$$
So, by induction it follows that
$$
\Sigma_ {n} \preceq \frac {\mu^ {2}}{n} \operatorname {t r} \left(H ^ {1 / 2}\right) \cdot H ^ {- 1 / 2},
$$
$$
\operatorname {t r} \left(H \Sigma_ {n}\right) \leq \frac {\mu^ {2}}{n} \operatorname {t r} \left(H ^ {1 / 2}\right) \operatorname {t r} \left(H \cdot H ^ {- 1 / 2}\right) = \frac {\mu^ {2}}{n} \operatorname {t r} \left(H ^ {1 / 2}\right) ^ {2}.
$$
But from Lemma 8, we know that  $\operatorname{tr}(D)$  is the global minimum of  $\operatorname{tr}\left(H\Sigma_{n}\right)$  for any assignment of  $a_{k}$ . This immediately gives us the desired result.
Proof. For nearest and stochastic rounding, set the linear feedback  $U$  in Eq. (2) to be zero. Stochastic rounding achieves worst-case loss,
$$
\mathcal {L} _ {\text {w o r s t}} (\text {S t o c h}, H) \stackrel {(3)} {=} \sup  _ {W \in \mathbb {R} ^ {m \times n}} \mathbf {E} [ \operatorname {t r} (\eta H \eta^ {T}) ] = \frac {m}{4} \operatorname {t r} (H). \tag {11}
$$
For the average-case proxy loss, recall the computations of  $\mathbf{E}\left[\eta_{ij}^{2}\right]$  from the proof of Theorem 1.
$$
\mathcal {L} _ {\text {a v g}} (\text {N e a r}, H) \stackrel {(3)} {=} \mathbf {E} _ {W \sim U n i f [ 0, 1 ] ^ {m \times n}} \left[ \operatorname {t r} \left(\eta H \eta^ {T}\right) \right] = \frac {m}{1 2} \operatorname {t r} (H) \tag {12}
$$
$$
\mathcal {L} _ {\text {a v g}} (\text {S t o c h}, H) \stackrel {(3)} {=} \mathbf {E} _ {W \sim U n i f [ 0, 1 ] ^ {m \times n}} \left[ \operatorname {t r} \left(\eta H \eta^ {T}\right) \right] = \frac {m}{6} \operatorname {t r} (H). \tag {13}
$$
Without incoherence: no improvement with a spectral bound
$$
\sup_{\tilde{H} s.t.eig(\tilde{H}) = \operatorname {eig}(H)}\mathcal{L}_{\text{worst}}(\mathsf{LDLQ},\tilde{H}) = \mathcal{L}_{\text{worst}}(\mathsf{Stoch},H) = \frac{m}{4}\operatorname {tr}\left(H\right).
$$
Proof. See Lemma 3 for calculations on the proxy loss for nearest and stochastic rounding.
For LDLQ, we will derive lower and upper bounds on  $\sup_{\tilde{H} s.t.\mathrm{eig}(\tilde{H}) = \mathrm{eig}(H)}\mathcal{L}_{\mathrm{worst}}(\mathsf{LDLQ},\tilde{H})$  and  $\sup_{\tilde{H} s.t.\mathrm{eig}(\tilde{H}) = \mathrm{eig}(H)}\mathcal{L}_{\mathrm{avg}}(\mathsf{LDLQ},\tilde{H})$ , and show they are equal. To construct a lower bound, consider  $\tilde{H} = I\Lambda I$  where  $\Lambda$  are the eigenvalues of  $H$ . This decomposition is also the LDL decomposition of  $\tilde{H}$ , rewritten as  $\tilde{H} = (U + I)D(U + I)^{-1}$ . It follows that  $\operatorname {tr}(D) = \operatorname {tr}\left(\tilde{H}\right)$  for this  $\tilde{H}$ . Combine this result with the worst and average-case losses calculated in the proof of Theorem 1. For the worst-case loss from the proof of Theorem  $1,\geq \frac{m}{4}\operatorname {tr}(H)$ . The lower bound for the average-case loss is  $\geq \frac{m}{12}\operatorname {tr}(H)$  for  $\mathcal{Q}$  as nearest, and  $\geq \frac{m}{6}\operatorname {tr}(H)$  for  $\mathcal{Q}$  as stochastic. Now upper bounds are derived using the preceding calculations in Eq. (11)-(13), and using the worst and average-case optimality of LDLQ proven in Theorem 1. The lower and upper bounds are tight, proving our result.
E Proofs for Section 4 (Quantization With Incoherence Processing: Incoherence Processing Step)
Subsection 4.1 (Incoherence via Efficient Orthogonal Multiplication)
Lemma 9 (Theorem 2.4 from Lalley [2]). There exist constants  $C$  and  $A$  independent of  $n$  such that for any function  $F$  from the unit sphere in  $n$  dimensions to  $\mathbb{R}$  that is 1-Lipschitz relative to the Riemannian metric on the sphere,
$$
\mathbf {P} _ {x \sim \mathcal {S} _ {n}} \left(F (x) - \mathbf {E} _ {x \sim \mathcal {S} _ {n}} [ F (x) ] \geq t\right) \leq C \exp \left(- \frac {n t ^ {2}}{A}\right)
$$
Lemma 10. Let  $B \in \mathbb{R}^{m \times n}$  be a matrix, and let  $x$  be a random vector uniformly distributed on the unit sphere in  $\mathbb{R}^n$ . Then there exist global constants  $A > 0$  and  $C > 0$  independent of  $m$  and  $n$  such that
$$
\mathbf {P} \left(\| B x \| ^ {2} \geq \frac {A \| B \| _ {F} ^ {2}}{n} \log \left(\frac {C}{\delta}\right)\right) \leq \delta ,
$$
$$
F (x) = \frac {\| B x \|}{\| B \| _ {F}}.
$$
$$
\nabla F (x) = \frac {B ^ {T} B x}{\| B x \| \cdot \| B \| _ {F}},
$$
$$
\| \nabla F (x) \| \leq 1.
$$
Also observe that for  $x$  drawn uniformly from the sphere in  $n$  dimensions,
$$
\mathbf {E} \left[ F (x) \right] \leq \sqrt {\mathbf {E} \left[ F (x) ^ {2} \right]} = \frac {1}{\| B \| _ {F}} \cdot \sqrt {\mathbf {E} \left[ \| B x \| ^ {2} \right]} = \frac {1}{\sqrt {n}}.
$$
So, applying Lemma 9,
$$
\mathbf {P} \left(\frac {\| B x \|}{\| B \| _ {F}} - \frac {1}{\sqrt {n}} \geq t\right) \leq C \exp \left(- \frac {n t ^ {2}}{A}\right).
$$
If we let  $\delta$  be
$$
\delta = C \exp \left(- \frac {n t ^ {2}}{A}\right),
$$
$$
\frac {A}{n} \log \left(\frac {C}{\delta}\right) = t ^ {2}
$$
Trivially, then, for some modified global constants  $A^{\prime}$  and  $C^\prime$
$$
\frac {A ^ {\prime}}{n} \log \left(\frac {C ^ {\prime}}{\delta}\right) = \left(t + \frac {1}{\sqrt {n}}\right) ^ {2}
$$
$$
\mathbf {P} \left(\frac {\| B x \| ^ {2}}{\| B \| _ {F} ^ {2}} \geq \frac {A ^ {\prime}}{n} \log \left(\frac {C ^ {\prime}}{\delta}\right)\right) \leq \delta ,
$$
$$
\mathbf {P} \left(\| B x \| ^ {2} \geq \frac {A ^ {\prime} \| B \| _ {F} ^ {2}}{n} \log \left(\frac {C ^ {\prime}}{\delta}\right)\right) \leq \delta ,
$$
This is what we wanted to prove.
Proof. First we will prove what we want to prove about  $H$ ; then we will prove what we want to prove about  $W$ . Let  $Q$  be a matrix of eigenvectors of  $H$ . Observe that since  $Q$  is an orthogonal matrix (by the spectral theorem, because  $H$  is symmetric),  $Qe_j$  is a unit vector, i.e.  $\| Qe_j \| = 1$ . Call  $Qe_j = y$ . Also observe that
$$
e _ {i} ^ {T} \left(U _ {1} \otimes U _ {2} \otimes \dots \otimes U _ {k}\right) = \left(\left(e _ {i _ {1}} ^ {T} U _ {1}\right) \otimes \left(e _ {i _ {2}} ^ {T} U _ {2}\right) \otimes \dots \otimes \left(e _ {i _ {k}} ^ {T} U _ {k}\right)\right)
$$
for some indices  $i_j$ . Call  $e_{i_j}^T U_j = x_j^T$ , and observe that the  $x_j$  are all independent unit random vectors. So,
$$
\left(\left(U _ {1} \otimes U _ {2} \otimes \dots \otimes U _ {k}\right) Q\right) _ {i j} = \left(x _ {1} \otimes x _ {2} \otimes \dots \otimes x _ {k}\right) ^ {T} y
$$
for random unit vectors  $x_{1},\ldots ,x_{k}$  and unit vector  $y$ . We can easily bound this with  $k$  applications of Lemma 10 and a union bound, yielding
$$
\mathbf {P} \left(\left(\left(x _ {1} \otimes x _ {2} \otimes \dots \otimes x _ {k}\right) ^ {T} y\right) ^ {2} \geq \frac {A ^ {k}}{n} \log \left(\frac {C}{\delta}\right) ^ {k}\right) \leq k \delta ,
$$
Setting  $\delta \mapsto \frac{\delta}{kn^2}$  yields
$$
\mathbf {P} \left(\left(\left(x _ {1} \otimes x _ {2} \otimes \dots \otimes x _ {k}\right) ^ {T} y\right) ^ {2} \geq \frac {A ^ {k}}{n} \log \left(\frac {C k n ^ {2}}{\delta}\right) ^ {k}\right) \leq \frac {\delta}{n ^ {2}},
$$
and unioning over all the entries of the large orthogonal matrix,
$$
\mathbf {P} \left(\max  _ {i, j} \left| \left(\left(U _ {1} \otimes U _ {2} \otimes \dots \otimes U _ {k}\right) Q\right) _ {i j} \right| \geq \sqrt {\frac {A ^ {k}}{n} \log \left(\frac {C k n ^ {2}}{\delta}\right) ^ {k}}\right) \leq \delta .
$$
Next, for  $W$ , observe that if we flatten  $W$ , then  $W / \|W\|_F$  is a unit vector. Then any entry of the resulting matrix can be written as
$$
\left(x _ {1} \otimes x _ {2} \otimes \dots \otimes x _ {k}\right) ^ {T} W \left(y _ {1} \otimes y _ {2} \otimes \dots \otimes y _ {k}\right)
$$
where  $x_{1}, \ldots, x_{k}$  and  $y_{1}, \ldots, y_{k}$  are  $k$  independent random unit vectors. We can easily bound this with  $2k$  applications of Lemma 10 and a union bound, yielding
$$
\mathbf {P} \left(\left(\left(x _ {1} \otimes x _ {2} \otimes \dots \otimes x _ {k}\right) ^ {T} W \left(y _ {1} \otimes y _ {2} \otimes \dots \otimes y _ {k}\right)\right) ^ {2} \geq \frac {A ^ {2 k}}{m n} \log \left(\frac {C}{\delta}\right) ^ {2 k}\right) \leq 2 k \delta ,
$$
Setting  $\delta \mapsto \frac{\delta}{2kmn}$  yields
$$
\mathbf {P} \left(\left(\left(x _ {1} \otimes x _ {2} \otimes \dots \otimes x _ {k}\right) ^ {T} W \left(y _ {1} \otimes x _ {2} \otimes \dots \otimes y _ {k}\right)\right) ^ {2} \geq \frac {A ^ {2 k}}{m n} \log \left(\frac {2 C k m n}{\delta}\right) ^ {2 k}\right) \leq \frac {\delta}{m n},
$$
and unioning over all the  $mn$  entries of the large orthogonal matrix,
$$
\mathbf {P} \left(\max  _ {i, j} \left| e _ {i} ^ {T} \left(U _ {1} \otimes U _ {2} \otimes \dots U _ {k}\right) W \left(V _ {1} \otimes V _ {2} \otimes \dots \otimes V _ {k}\right) e _ {j} \right| \geq \sqrt {\frac {A ^ {2 k}}{m n} \log \left(\frac {2 C k m n}{\delta}\right) ^ {2 k}}\right) \leq \delta .
$$
This is what we wanted to show.
F Proofs for Section 5 (Extensions and Further Analyses)
Subsection 5.1 (OPTQ is a Special Case of LDLQ)
Theorem 6. OTPQ [1] falls within the class of adaptive rounding procedures with linear feedback as described by Eq. (2), and is equivalent to LDLQ in Section 3.
Proof. OPTQ works in the following way. After OPTQ has quantized the first  $t - 1$  components of the row vector  $w$ , it minimizes the proxy loss over the remaining  $n - t + 1$  elements, keeping the first  $t - 1$  elements fixed. It then quantizes the  $t$ th element using nearest rounding to the grid and clamping. It then proceeds to the next column. If we let  $\Delta = \hat{w} - w$ , this proxy loss that it minimizes can be written in block form as
$$
\ell = \Delta_ {1: (t - 1)} H _ {1: (t - 1), 1: (t - 1)} \Delta_ {1: (t - 1)} ^ {T} + 2 \Delta_ {1: (t - 1)} H _ {1: (t - 1), t: n} + \Delta_ {t: n} H _ {t: n, t: n} \Delta_ {t: n} ^ {T}
$$
and its minimum over  $\Delta_{t:n}$  will occur when
$$
0 = \Delta_ {1: (t - 1)} H _ {1: (t - 1), t: n} + \Delta_ {t: n} H _ {t: n, t: n},
$$
$$
\Delta_ {t: n} = - \Delta_ {1: (t - 1)} H _ {1: (t - 1), t: n} \left(H _ {t: n, t: n}\right) ^ {- 1}.
$$
Now, suppose that  $H = \tilde{U}D\tilde{U}^T$  is the LDL decomposition of  $H$ , where  $\tilde{U}$  is unit upper triangular and  $D$  is diagonal. Since  $\tilde{U}$  is upper triangular,
$$
H _ {t: n, t: n} = \tilde {U} _ {t: n, t: n} D _ {t: n, t: n} \tilde {U} _ {t: n, t: n} ^ {T}.
$$
$$
H _ {1: (t - 1), t: n} = \tilde {U} _ {1: (t - 1), t: n} D _ {t: n, t: n} \tilde {U} _ {t: n, t: n} ^ {T}.
$$
$$
\Delta_ {t: n} = - \Delta_ {1: (t - 1)} \tilde {U} _ {1: (t - 1), t: n} \left(\tilde {U} _ {t: n, t: n}\right) ^ {- 1}.
$$
Now, the only part of the value of  $\Delta_{t:n}$  which matters is the first entry, since this is the one that's going to be used to make the next quantization decision. But since  $\tilde{U}_{t:n,t:n}$  is unit upper triangular and so is its inverse,  $\left(\tilde{U}_{t:n,t:n}\right)^{-1}e_t = e_t$ , and so
$$
\Delta_ {t} = \Delta_ {t: n} e _ {1} = - \Delta_ {1: (t - 1)} \tilde {U} _ {1: (t - 1), t: n} e _ {t} = - \Delta_ {1: (t - 1)} \tilde {U} _ {1: (t - 1), t} = - \Delta (\tilde {U} - I) e _ {t}.
$$
Finally, we quantize the  $t$ -th weight as
$$
\hat {w} _ {t} = \mathcal {Q} (w _ {t} - (\hat {W} - W) (\tilde {U} - I) e _ {t}).
$$
This update is equivalent to our adaptive rounding with linear feedback procedure in Eq. (2), with  $U$  assigned from the LDL decomposition of  $H$ .
Subsection 5.2 (A Bound for Rounding to a Finite Grid)
Algorithm 5 presents a quantization procedure which theoretically address OPTQ's clamping issue, by incorporating a restriction of  $|\hat{W}_{ij} - W_{ij}|$  into objective (7). Note that for simplicity, here we present the explicit case where only two factors are used in each Kronecker product of orthogonal matrices; however, the proof should generalize to any number of factors.