SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Choice of dynamic range for quantizers  $\mathrm{Q}$  and  $\mathrm{Q}'$ : Using Lemma B.3, the max-norm of the input to the first quantizer can be upper bounded as  $\| \widetilde{\mathbf{U}}_k\|_{\max} \leq \| \widetilde{\mathbf{U}}_k\|_2 = \sigma_1$ , and that of the second quantizer is  $\| \mathbf{V}_k\|_{\max} \leq \| \mathbf{V}_k\|_2 = 1$ . Therefore, choosing the dynamic ranges of  $\mathrm{Q}$  and  $\mathrm{Q}'$  to be  $\sigma_1$  and 1 respectively, would ensure that they remain unsaturated, and (105) can be rewritten as
$$
\mathbb {E} \left\| \mathrm {Q} \left(\widetilde {\mathbf {U}} _ {k}\right) \mathrm {Q} ^ {\prime} \left(\mathbf {V} _ {k} ^ {\top}\right) - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \leq \| \mathbf {A} _ {k} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} + \frac {n k \sigma_ {1} ^ {2}}{\left(2 ^ {\mathrm {B}} - 1\right) ^ {2}} + \frac {d k \sigma_ {1} ^ {2}}{\left(2 ^ {\mathrm {B} ^ {\prime}} - 1\right) ^ {2}} + k \frac {n \Delta^ {2}}{4} \frac {d \Delta^ {\prime 2}}{4}. \tag {106}
$$
Choice of bit-budgets B and  $\mathbf{B}^{\prime}$ : For a given  $\epsilon > 0$ , suppose we choose B and  $\mathbf{B}^{\prime}$  so that
$$
\mathrm {B} \geq \log_ {2} \left(\sigma_ {1} \sqrt {\frac {3 n k}{\epsilon}} + 1\right) \quad \text {a n d ,} \quad \mathrm {B} ^ {\prime} \geq \log_ {2} \left(\sigma_ {1} \sqrt {\frac {3 d k}{\epsilon}} + 1\right).
$$
With such a choice, the second and third terms in (106) would not exceed  $\epsilon$ . Furthermore, the last term of (106) will be
$$
k \frac {n \Delta^ {2}}{4} \frac {d \Delta^ {\prime 2}}{4} \leq d n k \frac {\epsilon}{3 n k} \frac {\epsilon}{3 d k \sigma_ {1} ^ {2}} \leq \frac {\epsilon}{3} \quad \text {w h e n e v e r} \epsilon <   3 k \sigma_ {1} ^ {2}.
$$
Prop. H.1 states that when  $n \approx d$ , in order to guarantee an  $\epsilon$ -additive error approximation with respect to the best rank-  $k$  approximation, we require (ignoring constant multiplicative factors inside the  $\log_2(\cdot)$ ), a total budget (ref. to Tab. 1) of
$$
\frac {n}{n + d} \mathrm {B} + \frac {d}{n + d} \mathrm {B} ^ {\prime} \approx \frac {1}{2} \left(\mathrm {B} + \mathrm {B} ^ {\prime}\right) = \frac {1}{2} \log_ {2} \left(\frac {3 k \sigma_ {1} ^ {2}}{\epsilon} \sqrt {n d}\right) \quad \text {b i t s}. \tag {107}
$$
I LPLR-SVD algorithm: Approximation error analysis
LPLR-SVD is a simpler variant of our LPLR algorithm in which instead of approximating the range space of  $\mathbf{A}$  using  $\mathbf{AS}$ , we compute the full-SVD of  $\mathbf{A} \in \mathbb{R}^{n \times d}$  as  $\mathbf{A} = \mathbf{U} \boldsymbol{\Sigma} \mathbf{V}^{\top}$ , where  $\mathbf{U} \in \mathbb{R}^{n \times n}$ ,  $\boldsymbol{\Sigma} \in \mathbb{R}^{n \times d}$ , and  $\mathbf{V} \in \mathbb{R}^{d \times d}$ . The pseudocode of LPLR-SVD is provided in Alg. 3.
We first compute the SVD,  $\mathbf{A} = \mathbf{U}\pmb{\Sigma}\mathbf{V}^{\top}$ , followed by quantizing the singular vectors corresponding to the top- $k$  singular values, scaled by the singular values. We denote  $\widetilde{\mathbf{U}} = \mathbf{U}\pmb{\Sigma}$ , and the sub-matrix formed by the first  $k$  columns as  $\widetilde{\mathbf{U}}_k$ . We quantize this basis to get  $\mathbf{L} = \mathrm{Q}(\widetilde{\mathbf{U}}_k)$  as the first low-rank factor. Subsequently, we project the columns of  $\mathbf{A}$  onto the subspace spanned by this quantized basis, i.e., we solve the optimization problem
$$
\min  _ {\mathbf {W} \in \mathbb {R} ^ {k \times d}} \left\| \mathrm {Q} \left(\widetilde {\mathbf {U}} _ {k}\right) \mathbf {W} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \tag {108}
$$
The solution of this is given by  $\mathbf{W}^{*} = \mathrm{Q}(\widetilde{\mathbf{U}}_{k})^{\dagger}\mathbf{A}$ . The second low rank factor is given by quantizing it again as  $\mathbf{R} = \mathrm{Q}'\left(\mathrm{Q}(\widetilde{\mathbf{U}}_k)^\dagger \mathbf{A}\right)$ .
The following result provides an upper bound on the approximation error of LPLR-SVD. The proof of this result is similar to Thm. 3.2 without the complications arising from Gaussian concentration.
Theorem I.1. (LPLR-SVD approximation error (formal)) Let our matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  have singular values  $\sigma_1 \geq \ldots \sigma_k \geq \ldots$ , and bounded row norms  $\left\| \mathbf{a}^{(i)} \right\| \leq R$ . Let  $\kappa(\mathbf{A}_k) = \sigma_1 / \sigma_k$  be the condition number of the  $\mathbf{A}_k$ , i.e., the best rank- $k$  approximation of  $\mathbf{A}$ . Consider some sufficiently small  $\epsilon$  that satisfies  $0 < \epsilon < 4k\sigma_1^2$ , and suppose the dynamic ranges of quantizers  $\mathrm{Q}$  and  $\mathrm{Q}'$  are set to be  $\sigma_1$  and
1 Compute SVD and get  $\mathbf{A} = \mathbf{U}\pmb{\Sigma}\mathbf{V}^{\top}$
2 Extract the top-  $k$  left to get an approximate basis for the range space of  $\mathbf{A}$  as  $\widetilde{\mathbf{U}}_k = (\mathbf{U}\pmb {\Sigma})_k$
3 Quantize and get an approximate basis  $\mathrm{Q}(\widetilde{\mathbf{U}}_k)$
4 Find  $\mathbf{W}^{*} = \arg \min_{\mathbf{W}}\left\| \mathrm{Q}(\widetilde{\mathbf{U}}_{k})\mathbf{W} - \mathbf{A}\right\|_{\mathrm{F}}^{2}$
5 Quantize  $\mathbf{W}^*$  using quantizer  $\mathrm{Q}'$  to get  $\mathrm{Q}'(\mathbf{W}^*)$
6 return Low-rank and low-precision approximation LR where  $\mathbf{L} = \mathrm{Q}(\widetilde{\mathbf{U}}_k)$ ,  $\mathbf{R} = \mathrm{Q}'(\mathbf{W}^*)$ .
$2\kappa(\mathbf{A}_k)$  respectively. Furthermore, suppose the bit-budgets of the quantizers satisfy
$$
\mathrm {B} \geq \max  \left\{\mathrm {B} _ {1}, \mathrm {B} _ {2} \right\} \quad a n d, \quad \mathrm {B} ^ {\prime} \geq \log_ {2} \left(4 \sigma_ {1} \kappa (\mathbf {A} _ {k}) \sqrt {\frac {d k}{\epsilon}} + 1\right),
$$
$$
\mathrm {B} _ {1} = \log_ {2} \left(2 \sigma_ {1} \sqrt {\frac {n k}{\epsilon}} + 1\right),
$$
$$
\mathrm {B} _ {2} = \log_ {2} \left(\frac {4 C \kappa (\mathbf {A} _ {k})}{\log 2} \left(\sqrt {n} + \sqrt {k} + \sqrt {\log \left(\frac {8 n \mathrm {R} ^ {2}}{\epsilon}\right)}\right) + 1\right).
$$
Then, the low-precision and low-rank factorization returned by Alg. 3 satisfies
$$
\mathbb {E} \left\| \mathbf {L R} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \leq \left\| \mathbf {A} _ {k} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} + \epsilon .
$$
Proof. Since the solution of  $\mathbf{W}^{*} = \arg \min_{\mathbf{W}}\left\| \mathrm{Q}(\widetilde{\mathbf{U}}_{k})\mathbf{W} - \mathbf{A}\right\|_{\mathrm{F}}^{2}$  is available in closed form as  $\mathbf{W}^{*} = \mathrm{Q}(\widetilde{\mathbf{U}}_{k})^{\dagger}\mathbf{A}$ , we have the low rank factorization as  $\mathbf{A} \approx \mathrm{Q}(\widetilde{\mathbf{U}}_{k})\mathrm{Q}'$  ( $\mathrm{Q}(\widetilde{\mathbf{U}}_{k})^{\dagger}\mathbf{A}$ ). Let us denote the quantization error matrices as  $\mathbf{E}' = \mathrm{Q}'$  ( $\mathrm{Q}(\widetilde{\mathbf{U}}_{k})^{\dagger}\mathbf{A}$ )- $\mathrm{Q}(\widetilde{\mathbf{U}}_{k})^{\dagger}\mathbf{A}$ , and  $\mathbf{E} = \mathrm{Q}(\widetilde{\mathbf{U}}_{k}) - \widetilde{\mathbf{U}}_{k}$ . Then, the approximation error is
$$
\begin{array}{l} \mathbb {E} \left. \left\| Q (\widetilde {\mathbf {U}} _ {k}) Q ^ {\prime} (Q (\widetilde {\mathbf {U}} _ {k}) ^ {\dagger} \mathbf {A}) - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} = \mathbb {E} \left\| Q (\widetilde {\mathbf {U}} _ {k}) (Q (\widetilde {\mathbf {U}} _ {k}) ^ {\dagger} \mathbf {A} + \mathbf {E} ^ {\prime}) - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \right. \\ = \underbrace {\mathbb {E} \left\| Q \left(\widetilde {\mathbf {U}} _ {k}\right) Q \left(\widetilde {\mathbf {U}} _ {k}\right) ^ {\dagger} \mathbf {A} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2}} _ {\mathrm {T} _ {1}} + \underbrace {\mathbb {E} \left\| Q \left(\widetilde {\mathbf {U}} _ {k}\right) \mathbf {E} ^ {\prime} \right\| _ {\mathrm {F}} ^ {2}} _ {\mathrm {T} _ {2}}, \tag {109} \\ \end{array}
$$
where the last equality follows the because the quantizer  $\mathrm{Q}'$  is unbiased. Here, the  $\mathbb{E}$  is over the stochasticity of the quantizers. This decomposition is similar to (69), and can be treated similarly. The term  $\mathrm{T}_1$  consists of the low rank approximation error and the error from the first quantizer, whereas the term  $\mathrm{T}_2$  consists of error from the second quantizer. We follow steps similar to the proof of Lemma G.1 which are detailed below.
Analyzing term  $\mathrm{T}_{1}$ : Using similar reasoning as (71), this term can be written as
$$
\mathbb {E} \left\| Q (\widetilde {\mathbf {U}} _ {k}) Q (\widetilde {\mathbf {U}} _ {k}) ^ {\dagger} \mathbf {A} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \leq \mathbb {E} \left\| Q (\widetilde {\mathbf {U}} _ {k}) \widetilde {\mathbf {U}} _ {k} ^ {\dagger} \mathbf {A} _ {k} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} = \mathbb {E} \left\| Q (\widetilde {\mathbf {U}} _ {k}) \mathbf {V} _ {k} ^ {\top} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \tag {110}
$$
This is the same as (100) in the analysis of direct-SVD quant., and hence can be upper bounded as
$$
\mathbb {E} \left\| \mathrm {Q} \left(\widetilde {\mathbf {U}} _ {k}\right) \mathrm {Q} \left(\widetilde {\mathbf {U}} _ {k}\right) ^ {\dagger} \mathbf {A} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \leq \| \mathbf {A} _ {k} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} + n k \frac {\Delta^ {2}}{4}. \tag {111}
$$
Algorithm 3: LPLR-SVD: LPLR factorization via Singular Value Decomposition
Analyzing term  $\mathrm{T}_2$ : This term is again the same as (101) in the analysis of direct-SVD quant., and can be upper bounded as
$$
\mathbb {E} \left\| \mathrm {Q} \left(\widetilde {\mathbf {U}} _ {k}\right) \mathbf {E} ^ {\prime} \right\| _ {\mathrm {F}} ^ {2} \leq d k \sigma_ {1} ^ {2} \frac {\Delta^ {\prime 2}}{4} + k \frac {n \Delta^ {2}}{4} \frac {d \Delta^ {\prime 2}}{4}. \tag {112}
$$
Using (111) and (112), we get:
$$
\mathbb {E} \left\| Q (\widetilde {\mathbf {U}} _ {k}) Q ^ {\prime} \left(Q (\widetilde {\mathbf {U}} _ {k}) ^ {\dagger} \mathbf {A}\right) - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \leq \| \mathbf {A} _ {k} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} + \underbrace {n k \frac {\Delta^ {2}}{4}} _ {\mathrm {T} _ {3}} + \underbrace {d k \sigma_ {1} ^ {2} \frac {\Delta^ {\prime 2}}{4}} _ {\mathrm {T} _ {4}} + \underbrace {k \frac {n \Delta^ {2}}{4} \frac {d \Delta^ {\prime 2}}{4}} _ {\mathrm {T} _ {5}}. \tag {113}
$$
Choice of dynamic range for quantizers Q: Since the input to the quantizer is  $\widetilde{\mathbf{U}}_k$ , we need an upper bound on  $\| \widetilde{\mathbf{U}}_k \|_{\max}$ . The  $i^{\text{th}}$  column of  $\widetilde{\mathbf{U}}_k$  is  $(\widetilde{\mathbf{U}}_k)_i = \sigma_i \mathbf{u}_i$ , where  $1 \leq i \leq k$ . From Lemma B.3,
$$
\left\| \widetilde {\mathbf {U}} _ {k} \right\| _ {\max } \leq \left\| \widetilde {\mathbf {U}} _ {k} \right\| _ {2} = \sigma_ {1}. \tag {114}
$$
Note that this upper bound is tight in the worst-case sense, i.e., if we consider all matrices with bounded spectral norm, we have  $\sup_{\mathbf{X}:\mathbf{X} = \mathbf{U}\Sigma \mathbf{V}^{\top},\| \mathbf{X}\|_{2}\leq r}\| \widetilde{\mathbf{U}}_{k}\|_{\max} = r$ . Hence, the dynamic range of quantizer  $Q$  is set as  $\sigma_{1}$ .
Choice of dynamic range for quantizers  $\mathbf{Q}'$ : On the other hand, the input to quantizer  $\mathbf{Q}'$  is  $\mathrm{Q}(\widetilde{\mathbf{U}}_k)^\dagger \mathbf{A}$ . We adopt an approach similar to the proof of Lemma E.2 to upper bound the max-norm of  $\mathrm{Q}(\widetilde{\mathbf{U}}_k)^\dagger \mathbf{A}$  as follows:
$$
\left\| \mathrm {Q} (\widetilde {\mathbf {U}} _ {k}) ^ {\dagger} \mathbf {A} \right\| _ {\max } \stackrel {\text {(i)}} {\leq} \left\| \mathrm {Q} (\widetilde {\mathbf {U}} _ {k}) ^ {\dagger} \mathbf {A} \right\| _ {2} \stackrel {\text {(i i)}} {\leq} \left\| \mathrm {Q} (\widetilde {\mathbf {U}} _ {k}) ^ {\dagger} \right\| _ {2} \sigma_ {1}, \tag {115}
$$
where (i) and (ii) follow from Lemmas B.3 and B.4 respectively. We upper bound  $\left\| \mathrm{Q}(\widetilde{\mathbf{U}}_k)^\dagger \right\|_2$  as:
$$
\begin{array}{l} \left\| \mathrm {Q} (\widetilde {\mathbf {U}} _ {k}) ^ {\dagger} \right\| _ {2} \stackrel {\mathrm {(i)}} {\leq} \left(\sigma_ {\min} \left(\mathrm {Q} (\widetilde {\mathbf {U}} _ {k})\right)\right) ^ {- 1} = \left(\sigma_ {\min} \left(\widetilde {\mathbf {U}} _ {k} + \mathbf {E}\right)\right) ^ {- 1} \\ \stackrel {\text {(i i)}} {\leq} \left(\sigma_ {\min } \left(\widetilde {\mathbf {U}} _ {k}\right) - \| \mathbf {E} \| _ {2}\right) ^ {- 1} = \left(\sigma_ {k} - \| \mathbf {E} \| _ {2}\right) ^ {- 1}. \tag {116} \\ \end{array}
$$
Here, (i) follows because the singular values of  $\mathrm{Q}(\widetilde{\mathbf{U}}_k)^\dagger$  are inverses of the singular values of  $\mathrm{Q}(\widetilde{\mathbf{U}}_k)$ , and (ii) follows from Lemma B.5.
We are now left with upper bounding  $\| \mathbf{E} \|_2$ . Since a choice of  $\sigma_1$  for the dynamic range of  $\mathrm{Q}$  ensures that quantizer  $\mathrm{Q}$  remains unsaturated, the entries of  $\mathbf{E}$  are bounded as:
$$
\left| E _ {i j} \right| \leq \Delta = \frac {2 \sigma_ {1}}{2 ^ {\mathrm {B}} - 1} \quad \text {f o r a l l} i \in [ n ] \text {a n d} j \in [ k ]. \tag {117}
$$
$$
\left\| \mathbf {E} \right\| _ {\psi_ {2}} \leq \frac {2 \sigma_ {1}}{\left(2 ^ {\mathrm {B}} - 1\right) \log 2}. \tag {118}
$$
$$
\left\| \mathbf {E} \right\| _ {2} \leq \frac {2 C \sigma_ {1}}{\left(2 ^ {\mathrm {B}} - 1\right) \log 2} \left(\sqrt {n} + \sqrt {k} + t\right) \text {w i t h p r o b a b i l i t y e x c e e d i n g} 1 - 2 e ^ {- t ^ {2}}. \tag {119}
$$
Setting  $t = \sqrt{\log\left(\frac{8nR^2}{\epsilon}\right)}$  we get:
$$
\left\| \mathbf {E} \right\| _ {2} \leq \frac {2 C \sigma_ {1}}{(2 ^ {\mathrm {B}} - 1) \log 2} \left(\sqrt {n} + \sqrt {k} + \sqrt {\log \left(\frac {8 n \mathrm {R} ^ {2}}{\epsilon}\right)}\right) \text {w i t h p r o b a b i l i t y e x c e e d i n g} 1 - \frac {\epsilon}{4 n \mathrm {R} ^ {2}}. \tag {120}
$$
Let us choose our bit-budget  $\mathrm{B}$  of quantizer  $\mathrm{Q}$  to be such that  $\| \mathbf{E}\| _2\leq \frac{\sigma_k}{2}$ , i.e.,
$$
\mathrm {B} \geq \log_ {2} \left(\frac {4 C \kappa (\mathbf {A} _ {k})}{\log 2} \left(\sqrt {n} + \sqrt {k} + \sqrt {\log \left(\frac {8 n \mathrm {R} ^ {2}}{\epsilon}\right)}\right) + 1\right), \tag {121}
$$
where  $\kappa (\mathbf{A}_k) = \sigma_1 / \sigma_k$  is the condition number of the best rank-  $k$  approximation of  $\mathbf{A}$ . Then, using (116), we have:
$$
\left\| \mathrm {Q} \left(\widetilde {\mathbf {U}} _ {k}\right) ^ {\dagger} \mathbf {A} \right\| _ {\max } \leq 2 \kappa (\mathbf {A} _ {k}) \quad \text {w i t h p r o b a b i l i t y e x c e e d i n g} 1 - \frac {\epsilon}{4 n R ^ {2}}. \tag {122}
$$
So, if we choose the dynamic range of quantized  $\mathrm{Q}'$  to be  $2\kappa (\mathbf{A}_k)$ , it will remain unsaturated with probability exceeding  $1 - \frac{\epsilon}{4nR^2}$ , provided that the bit-budget of first quantizer satisfies B satisfies (121).
Choice of bit-budgets B and  $\mathbf{B}^{\prime}$  : Referring to (113), the term  $\mathrm{T}_3$  is,
$$
n k \frac {\Delta^ {2}}{4} = n k \frac {\sigma_ {1} ^ {2}}{\left(2 ^ {\mathrm {B}} - 1\right) ^ {2}} \leq \frac {\epsilon}{4} \quad \text {i f B} \geq \log_ {2} \left(2 \sigma_ {1} \sqrt {\frac {n k}{\epsilon}} + 1\right). \tag {123}
$$
The term  $\mathrm{T}_4$  is:
$$
d k \sigma_ {1} ^ {2} \frac {\Delta^ {\prime 2}}{4} = d k \sigma_ {1} ^ {2} \frac {4 \kappa (\mathbf {A} _ {k}) ^ {2}}{\left(2 ^ {\mathrm {B} ^ {\prime}} - 1\right) ^ {2}} \leq \frac {\epsilon}{4} \quad \text {i f B} ^ {\prime} \geq \log_ {2} \left(4 \sigma_ {1} \kappa (\mathbf {A} _ {k}) \sqrt {\frac {d k}{\epsilon}} + 1\right). \tag {124}
$$
With the above choices of  $\mathrm{B}$  and  $\mathrm{B}^{\prime}$ , the term  $\mathrm{T}_{5}$  is:
$$
n d k \frac {\Delta^ {2}}{4} \frac {\Delta^ {\prime 2}}{4} \leq n d k \frac {\epsilon}{4 n k} \frac {\epsilon}{4 d k \sigma_ {1} ^ {2}} \leq \frac {\epsilon}{4} \quad \text {i f} \quad \epsilon \leq 4 k \sigma_ {1} ^ {2}. \tag {125}
$$
Tying it all together: Similar to what is done in the proof of Thm. G.2, for the purpose of analysis, we assume that Alg. 3 returns  $\mathbf{L} = \mathbf{0}$  and  $\mathbf{R} = \mathbf{0}$  if quantizer  $\mathrm{Q}'$  gets saturated. In practical implementation, it can easily be checked if either quantizer Q or  $\mathrm{Q}'$  gets saturated or not, and the algorithm can be repeated again with a new realization of the stochastic quantizer Q. Since the choice of dynamic ranges for Q ensures that it remains unsaturated with a high probability, "reasonably few" realizations would suffice to get at least one good realization in which Q is unsaturated.
However, in what follows, we assume that if quantizer  $\mathrm{Q}$  gets saturated, then the algorithm returns  $\mathbf{0}$  as an estimate of  $\mathbf{A}$ , resulting in a Frobenius norm error of  $\| \mathbf{A}\|_{\mathrm{F}}$ . Since this happens with a very small probability, we show that its contribution to the expected Frobenius norm error of Alg. 3 is small as well. Let us denote the event that quantizer  $\mathrm{Q}$  is unsaturated as  $\mathcal{E}$ . Then, the expected approximation error can be written as
$$
\begin{array}{l} \mathbb {E} \left\| \mathbf {L R} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} = \mathbb {E} \left[ \left\| \mathrm {Q} (\widetilde {\mathbf {U}} _ {k}) \mathrm {Q} ^ {\prime} \left(\mathrm {Q} (\widetilde {\mathbf {U}} _ {k}) ^ {\dagger} \mathbf {A}\right) - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \mathbb {1} _ {\varepsilon} \right] + \mathbb {E} \left[ \| \mathbf {A} \| _ {\mathrm {F}} ^ {2} \mathbb {1} _ {\varepsilon^ {C}} \right] \\ \stackrel {(i)} {\leq} \mathbb {E} \left[ \left\| Q (\widetilde {\mathbf {U}} _ {k}) Q ^ {\prime} (Q (\widetilde {\mathbf {U}} _ {k}) ^ {\dagger} \mathbf {A}) - \mathbf {A} \right\| _ {F} ^ {2} \mathbb {1} _ {\mathcal {E}} \right] + n R ^ {2} \Pr \left(\mathbb {1} _ {\mathcal {E} C}\right) \\ \stackrel {(i i)} {\leq} \| \mathbf {A} _ {k} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} + n k \frac {\Delta^ {2}}{4} + d k \sigma_ {1} ^ {2} \frac {\Delta^ {\prime 2}}{4} + k \frac {n \Delta^ {2}}{4} \frac {d \Delta^ {\prime 2}}{4} + n \mathrm {R} ^ {2} \Pr \left(\mathbb {1} _ {\varepsilon C}\right), \tag {126} \\ \end{array}
$$
where inequality (i) follows from Asm. 3.1 and the fact that the expectation of indicator function of an event is the probability of the event, and (ii) follows from (113). Using appropriate choices of B and  $\mathrm{B^{\prime}}$  and small enough  $\epsilon$ , from (122), (123), (124), and (125), we have
$$
\mathbb {E} \| \mathbf {L R} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} \leq \| \mathbf {A} _ {k} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} + \frac {\epsilon}{4} + \frac {\epsilon}{4} + \frac {\epsilon}{4} + n \mathrm {R} ^ {2} \frac {\epsilon}{4 n \mathrm {R} ^ {2}} = \| \mathbf {A} _ {k} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} + \epsilon . \tag {127}
$$
Note that from (123) and (121), the bit-budget of the quantizers must satisfy:
$$
\mathrm {B} \geq \max  \left\{\mathrm {B} _ {1}, \mathrm {B} _ {2} \right\}, \quad \text {a n d} \mathrm {B} ^ {\prime} \geq \log_ {2} \left(4 \sigma_ {1} \kappa (\mathbf {A} _ {k}) \sqrt {\frac {d k}{\epsilon}} + 1\right) \tag {128}
$$
$$
\mathrm {B} _ {1} = \log_ {2} \left(2 \sigma_ {1} \sqrt {\frac {n k}{\epsilon}} + 1\right), \tag {129}
$$
$$
\mathrm {B} _ {2} = \log_ {2} \left(\frac {4 C \kappa (\mathbf {A} _ {k})}{\log 2} \left(\sqrt {n} + \sqrt {k} + \sqrt {\log \left(\frac {8 n \mathrm {R} ^ {2}}{\epsilon}\right)}\right) + 1\right). \tag {130}
$$
I.1 Informal version of Thm. I.1
Consider  $n \approx d$ . From Thm. I.1, we can get a (simplified) asymptotic dependence of the bit-budgets B and B'. In what follows, we ignore the constant multiplicative factors inside  $\log_2(\cdot)$ . Comparing the expressions for B₁ and B₂, we have
$$
\mathrm {B} \geq \log_ {2} \left(\sigma_ {1} \sqrt {\frac {n k}{\epsilon}}\right), \tag {131}
$$
$$
\mathrm {B} ^ {\prime} \geq \log_ {2} \left(\sigma_ {1} \kappa (\mathbf {A} _ {k}) \sqrt {\frac {d k}{\epsilon}}\right). \tag {132}
$$
This gives a total bit-budget of
$$
\frac {1}{2} \log_ {2} \left(\sigma_ {1} ^ {2} \kappa (\mathbf {A} _ {k}) \frac {k}{\epsilon} \sqrt {n d}\right) \quad \text {b i t s}. \tag {133}
$$
J Additional numerical simulations
J.1 Image Compression
In addition to Fig. 1 in the main paper, we also provide additional image compression experiments on images of different dimensions in Figs. 6, 5, 7, and 8. The values of B and  $\mathbf{B}^{\prime}$  are provided in the respective captions of these figures. For every image, the sketch size/target rank  $m$  and the bit-budgets B and  $\mathrm{B^{\prime}}$  are chosen so as to ensure parity with a naive quantizer with bit budget  $\mathrm{B_{nq}}$ . In other words, for an  $n\times d$  dimensional image,  $\mathrm{B}\cdot nm + \mathrm{B}^{\prime}\cdot md = \mathrm{B_{nq}}\cdot nd$ . The values of  $n$  and  $d$  are also provided in the corresponding captions.
For image compression, we also perform a normalize and shift operation of our DSVD/LPLR/LSVD output in order to adjust the scale and bias of the output image, as described next.
Normalize and shift: Let us denote the factorization obtained from executing DSVD/LPLR/LSVD on the input matrix  $\mathbf{X} \in \mathbb{R}^{n \times d}$  by  $\mathbf{Y} = \mathbf{L}\mathbf{R}$ . Then, we can improve the estimate provided by this factorization by considering  $\alpha \mathbf{Y} + \beta \mathbf{I}_{n \times d}$ . The overhead in storing these two additional scalars in full-precision is negligible when  $n$  and  $d$  are large. The optimal values of  $\alpha$  and  $\beta$  can be found as:
$$
\left(\alpha^ {*}, \beta^ {*}\right) = \underset {\alpha , \beta} {\arg \min } \| \alpha \mathbf {Y} + \beta \mathbf {I} - \mathbf {X} \| _ {\mathrm {F}} ^ {2}. \tag {134}
$$
Closed-form expressions for  $(\alpha^{*},\beta^{*})$  can be obtained by equating the derivatives of  $\| \alpha \mathbf{Y} + \beta \mathbf{I} - \mathbf{X}\|_{\mathrm{F}}^2$  to zero, and solving the resulting linear equations. Doing this algebra yields,
$$
\alpha^ {*} = \left(\| \mathbf {Y} \| _ {\mathrm {F}} ^ {2} - \frac {\langle \mathbf {Y} , \mathbf {I} \rangle^ {2}}{n d}\right) ^ {- 1} \left(\langle \mathbf {X}, \mathbf {Y} \rangle - \frac {\langle \mathbf {X} , \mathbf {I} \rangle \langle \mathbf {Y} , \mathbf {I} \rangle}{n d}\right),
$$
$$
\text {a n d}, \quad \beta^ {*} = \frac {\langle \mathbf {X} , \mathbf {I} \rangle}{n d} - \frac {\langle \mathbf {Y} , \mathbf {I} \rangle}{n d} \alpha^ {*}. \tag {135}
$$
J.2 Wall-clock time for compressing image embeddings
We now report the wall-clock time for computing the low-rank quantized factors via LPLR, LSVD, and DSVD for embedding compression in Tab. 10. This reinforces the validity of our theoretical assertions concerning the computational advantage of LPLR when compared to its alternative methods.
J.3 High-resolution figures for Llama weight compression
We now provide higher resolutions figures of Figs. 2a and 2b as Figs. 3 and 4, respectively. The code to reproduce these numerical simulations is available at our github repository https://github.com/pilancilab/matrix-compressor.
Figure 3: Comparison of LPLR and LPLR-SVD on LlaMa weight matrices with  $\mathrm{B} = \mathrm{B}' = 8$  bits,  $\mathrm{B}_{\mathrm{sq}} = 4$  bits, ordered by the original sequence of layers on the "Layer" - axis. We observe consistently better Frobenius norm error using LPLR and LPLR-SVD, with the exception of specific layers which lend themselves to naive quantization.
Figure 4: Comparison of LPLR and LPLR-SVD on LlaMa weight matrices with  $\mathrm{B} = \mathrm{B}' = \mathrm{B}_{\mathrm{sq}} = 4$  bits, ordered by the original sequence of layers on the "Layer" - axis. LPLR and LPLR-SVD demonstrate equivalent performance on a uniform compression budget of 4 bits, while outperforming naive quant. on almost all layers.
Table 10: Comparison of wall-clock time for computing low rank quantized factors via LPLR, LSVD and DSVD. Each image embedding forms a row of the input matrix, with low rank factors computed for each class of the input dataset. Bit-budgets used are  $\mathrm{B} = {\mathrm{B}}^{\prime } = 8,{\mathrm{\;B}}_{\mathrm{{nq}}} = 1$  . We report mean and standard deviation.