SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ \left(\mathrm {Q} (\mathbf {x} ^ {\top} \mathbf {s}) - \mathbf {x} ^ {\top} \mathbf {s}\right) _ {2} ^ {2} \right] \leq \frac {t ^ {2}}{(2 ^ {\mathrm {B}} - 1) ^ {2}} + \left(\frac {\mathrm {R} ^ {2}}{m} + t ^ {2}\right) \operatorname {e r f c} \left(\frac {t \sqrt {m}}{\mathrm {R} \sqrt {2}}\right) - \frac {\mathrm {R} t \sqrt {2}}{\sqrt {\pi m}} e ^ {- \frac {m t ^ {2}}{2 \mathrm {R} ^ {2}}} \\ \stackrel {(i)} {\leq} \frac {t ^ {2}}{\left(2 ^ {\mathrm {B}} - 1\right) ^ {2}} + \left(\frac {\mathrm {R} ^ {2}}{m} + t ^ {2}\right) \frac {\mathrm {R} \sqrt {2}}{t \sqrt {\pi m}} e ^ {- \frac {m t ^ {2}}{2 \mathrm {R} ^ {2}}} - \frac {\mathrm {R} t \sqrt {2}}{\sqrt {\pi m}} e ^ {- \frac {m t ^ {2}}{2 \mathrm {R} ^ {2}}} \\ = \frac {t ^ {2}}{\left(2 ^ {\mathrm {B}} - 1\right) ^ {2}} + \frac {\mathrm {R} ^ {3} \sqrt {2}}{m ^ {3 / 2} t \sqrt {\pi}} e ^ {- \frac {m t ^ {2}}{2 \mathrm {R} ^ {2}}}. \tag {26} \\ \end{array}
$$
Here, (i) follows from the upper bound  $\operatorname{erfc}(z) \leq \frac{e^{-z^2}}{\sqrt{\pi}z}$  from Karagiannidis and Lioumpas [29].
Quantization error of a vector Gaussian sketch: We now consider for any  $t \geq 0$ , the expected quantization error for  $\mathbf{x} \in \mathbb{R}^d$  with  $\| \mathbf{x} \|_2 \leq \mathrm{R}$  and  $\mathbf{S} \in \mathbb{R}^{m \times d}$  with  $S_{ij} \sim \mathcal{N}\left(0, \frac{1}{m}\right)$ . Each row of  $\mathbf{S}$  now independently plays the role of  $s$  in (26). Using (26), the vector quantization error is simply  $m$  times the scalar quantization error, and is now given by,
$$
\mathbb {E} \| Q (\mathbf {S x}) - \mathbf {S x} \| _ {2} ^ {2} \leq \frac {m t ^ {2}}{\left(2 ^ {\mathrm {B}} - 1\right) ^ {2}} + \frac {\mathrm {R} ^ {3} \sqrt {2}}{t \sqrt {\pi m}} e ^ {- \frac {m t ^ {2}}{2 \mathrm {R} ^ {2}}}. \tag {27}
$$
Choice of dynamic range  $t$ : Setting  $t = \frac{\mathrm{R}}{\sqrt{m}}$  in (27) yields,
$$
\mathbb {E} \| \mathrm {Q} (\mathbf {S x}) - \mathbf {S x} \| _ {2} ^ {2} \leq \frac {\mathrm {R} ^ {2}}{\left(2 ^ {\mathrm {B}} - 1\right) ^ {2}} + \frac {\mathrm {R} ^ {2} \sqrt {2}}{\sqrt {\pi e}}. \tag {28}
$$
Note that since  $\mathbb{E}[\mathbf{S}^\top \mathbf{S}]$  is an identity matrix, an estimate  $\widehat{\mathbf{x}}$  of  $\mathbf{x}$  can be recovered from  $\mathrm{Q}(\mathbf{S}\mathbf{x})$  as  $\mathbf{S}^\top \mathbf{Q}(\mathbf{S}\mathbf{x})$ . We can use the O(1) bound on  $\| \mathrm{Q}(\mathbf{S}\mathbf{x}) - \mathbf{S}\mathbf{x}\|^2$  to get a bound on  $\| \mathbf{S}^\top \mathbf{Q}(\mathbf{S}\mathbf{x}) - \mathbf{x}\|^2$  as follows:
$$
\begin{array}{l} \| \mathbf {S} ^ {\top} Q (\mathbf {S x}) - \mathbf {x} \| ^ {2} \leq \| Q (\mathbf {S x}) - \mathbf {S x} \| ^ {2} + \| \mathbf {S} ^ {\top} Q (\mathbf {S x}) \| ^ {2} - \| Q (\mathbf {S x}) \| ^ {2} + \| \mathbf {x} \| ^ {2} - \| \mathbf {S x} \| ^ {2} \\ \leq \left\| Q (\mathbf {S} \mathbf {x}) - \mathbf {S} \mathbf {x} \right\| ^ {2} + \left\| \mathbf {S} ^ {\top} Q (\mathbf {S} \mathbf {x}) \right\| ^ {2} + \left\| \mathbf {x} \right\| ^ {2} \\ \leq \left\| Q (\mathbf {S} \mathbf {x}) - \mathbf {S} \mathbf {x} \right\| ^ {2} + R ^ {2} \left(\sigma_ {\max } ^ {2} (\mathbf {S}) + 1\right) \tag {29} \\ \end{array}
$$
From properties of random Gaussian matrices, we know that  $\sigma_{\max}^2 (\mathbf{S})\leq \frac{d}{m}$  with high probability. Hence, the error  $\| \mathbf{S}^\top \mathbf{Q}(\mathbf{S}\mathbf{x}) - \mathbf{x}\| ^2$  only depends on the aspect ratio  $d / m$ , and not the dimension  $d$  directly. Although the reconstruction error  $\| \mathbf{S}^{\top}\mathbf{Q}(\mathbf{S}\mathbf{x}) - x\|^{2}$  scales as  $d / m$ , it does not necessarily increase with  $d$  if we choose the sketch size  $m$  to be proportional to  $d$ , i.e.,  $m = O(d)$ .
Despite this, Gaussian embeddings are not used practically for vector quantization because  $\mathbf{S}$  is dense matrix and computing  $\mathbf{Sx}$  entails a complexity of  $\mathrm{O}(d^2)$ . The entries of  $\mathbf{S}$  themselves are floating point numbers that have to be stored in full precision, hence this defeats the whole purpose of quantizing  $\mathbf{x}$  using fewer bits. However, this is not an issue for matrix compression because in LPLR, we do not explicitly compute the  $\mathbf{S}^\top \mathbf{Q}(\mathbf{SA})$  anywhere. In other words, the effects of  $\mathbf{S}$  in the first low-rank factor is nullified by the second low-rank factor. Unlike vector quantization, the corresponding sketch size  $m$  for LPLR only needs to be the same order as the inherent rank  $k$ , which can be much smaller than  $\min\{n, d\}$ , i.e., the dimensions of the matrix being compressed.
E Dynamic range of quantizers
We now derive high probability upper bounds on the maximum magnitude of the input to uniform quantizers  $\mathrm{Q}$  and  $\mathrm{Q}'$ . Choosing these upper bounds to be the dynamic range of the uniform scalar quantizers will ensure that the quantizer remains unsaturated with a high probability, which is our desired regime of operation.
E.1 Quantization for the first low-rank factor
We first look at the choice of dynamic range for the quantizer  $\mathbf{Q}$ , which is used to obtain the first low-rank factor. The input to the quantizer is  $\mathbf{A}\mathbf{S} \in \mathbb{R}^{n\times m}$  where  $\mathbf{A} \in \mathbb{R}^{n\times d}$  and  $\mathbf{S} \in \mathbb{R}^{d\times m}$ , and the entries of  $\mathbf{S}$  are i.i.d. as  $S_{ij} \sim \mathcal{N}\left(0,\frac{1}{m}\right)$ . Lemma E.1 below gives a high probability upper bound on the max norm of  $\mathbf{A}\mathbf{S}$ . This probability is computed over the randomness in the construction of  $\mathbf{S}$ .
Lemma E.1. (Max norm of AS) Given matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  with bounded row norms, i.e.,  $\left\| \mathbf{a}^{(i)} \right\| \leq \mathrm{R}$ , and  $\mathbf{S} \in \mathbb{R}^{d \times m}$  with entries distributed as  $S_{ij} \stackrel{i.i.d.}{\sim} \mathcal{N}\left(0, \frac{1}{m}\right)$ , with probability exceeding  $1 - \frac{\epsilon}{8n\mathrm{R}^2}$ , the max norm of AS satisfies,
$$
\left\| \mathbf {A} \mathbf {S} \right\| _ {\max } \leq \mathrm {R} \sqrt {\frac {2 \log \left(\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon}\right)}{m}}. \tag {30}
$$
Proof. Since  $(\mathbf{A}\mathbf{S})_{ij} = \mathbf{s}_j^\top \mathbf{a}^{(i)}$ , where  $\mathbf{a}^{(i)} \in \mathbb{R}^d$  is the  $i^{\text{th}}$  row of  $\mathbf{A}$  and  $\mathbf{s}_j \in \mathbb{R}^d$  is the  $j^{\text{th}}$  column of  $\mathbf{S}$ , we have  $(\mathbf{A}\mathbf{S})_{ij} \sim \mathcal{N}\left(0, \frac{\|\mathbf{a}_i\|_2^2}{m}\right)$ . Using Lemma B.8 and an application of union bound gives,
$$
\Pr \left(\left| (\mathbf {A S}) _ {i j} \right| \geq t\right) \leq 2 e ^ {- \frac {m t ^ {2}}{2 \left\| \mathbf {a} ^ {(i)} \right\| _ {2} ^ {2}}} \leq 2 e ^ {- \frac {m t ^ {2}}{2 R ^ {2}}}. \tag {31}
$$
A subsequent application of union bound over all the entries of AS yields,
$$
\Pr \left(\| \mathbf {A S} \| _ {\max } \geq t\right) \leq 2 n m e ^ {- \frac {m t ^ {2}}{2 R ^ {2}}}. \tag {32}
$$
Setting  $t = \mathrm{R}\sqrt{\frac{2\log\left(\frac{16\mathrm{R}^2n^2m}{\epsilon}\right)}{m}}$  in the above completes the proof.
E.2 Quantization for the second low-rank factor
We now obtain a high-probability upper bound on the max norm of  $\mathrm{Q}(\mathbf{AS})^{\dagger}\mathbf{A}$ , which is the input to the second quantizer  $\mathbf{Q}'$ . Let us define  $\mathcal{Q}$  to be the event that the quantizer  $\mathbf{Q}$  does not saturate. From Lemma E.1,  $\mathcal{Q}$  occurs with a sufficiently high probability. An appropriate choice of dynamic range for  $\mathbf{Q}'$  will ensure that conditioned on the event that  $\mathcal{Q}$  occurs, the quantizer  $\mathbf{Q}'$  also does not saturate with a high probability. The following lemma states this formally.
Lemma E.2. (Max norm of  $\mathrm{Q}(\mathbf{AS})^\dagger \mathbf{A}$ ) Let our input matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  have non-zero singular values  $\sigma_1, \ldots, \sigma_k, \sigma_{k+1}, \ldots, \sigma_r$ , where  $r = \mathrm{rank}(\mathbf{A})$ , and bounded row norms  $\|\mathbf{a}^{(i)}\| \leq \mathrm{R}$ . Let  $\kappa(\mathbf{A}) = \sigma_1 / \sigma_r$  and  $\kappa(\mathbf{A}_k) = \sigma_1 / \sigma_k$  respectively be the condition numbers of  $\mathbf{A}$  and the best rank- $k$  approximation of  $\mathbf{A}$ , and for some small  $\epsilon > 0$ , let us denote
$$
t = \sqrt {\frac {2 \log \left(\frac {3 2 n R ^ {2}}{\epsilon}\right)}{m}}, \quad \kappa = \min  \left\{\kappa (\mathbf {A}), \frac {\kappa (\mathbf {A} _ {k})}{1 - \frac {\sigma_ {k + 1}}{\sigma_ {k}} \left(\frac {\sqrt {\gamma} + 1 + t}{\sqrt {\gamma} - 1 - t}\right)} \right\}, \tag {33}
$$
where  $\gamma = d / m$  is the aspect ratio of the sketching matrix  $\mathbf{S} \in \mathbb{R}^{d \times m}$  with  $S_{ij} \stackrel{i, i, d}{\sim} \mathcal{N}\left(0, \frac{1}{m}\right)$ . Furthermore, suppose the dynamic range of the quantizer  $\mathrm{Q}$  is set to  $\mathrm{R}\sqrt{\frac{2\log\left(\frac{16\mathrm{R}^2n^2m}{\epsilon}\right)}{m}}$  as dictated by Lemma E.1, and suppose for some absolute constant  $C$ , the bit-budget B satisfies,
$$
\mathrm {B} \geq \log_ {2} \left(\frac {4 C \mathrm {R}}{\max  \left\{\sigma_ {r} , \sigma_ {k} - \sigma_ {k + 1} \left(\frac {\sqrt {\gamma} + 1 + t}{\sqrt {\gamma} - 1 - t}\right) \right\} \log 2} \left(\frac {\sqrt {\gamma} + 1 + t / \sqrt {2}}{\sqrt {\gamma} - 1 - t}\right) \sqrt {2 \log \left(\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon}\right)} + 1\right). \tag {34}
$$
Then, we have,
$$
\left\| \mathrm {Q} (\mathbf {A S}) ^ {\dagger} \mathbf {A} \right\| _ {\max } \leq \frac {2 \kappa}{\sqrt {\gamma} - 1 - t} \text {w i t h p r o b a b i l i t y e x c e e d i n g} 1 - \frac {\epsilon}{4 n R ^ {2}}. \tag {35}
$$
Proof. We have the following chain of inequalities:
$$
\left\| Q (\mathbf {A S}) ^ {\dagger} \mathbf {A} \right\| _ {\max } \stackrel {(i)} {\leq} \left\| Q (\mathbf {A S}) ^ {\dagger} \mathbf {A} \right\| _ {2} \stackrel {(i i)} {\leq} \left\| Q (\mathbf {A S}) ^ {\dagger} \right\| _ {2} \sigma_ {1} (\mathbf {A}), \tag {36}
$$
where (i) and (ii) follow from Lemmas B.3 and B.4 respectively. We now need to upper bound  $\left\| \mathbf{Q}(\mathbf{A}\mathbf{S})^{\dagger}\right\|_{2}$ , which is done as follows:
$$
\left\| \mathrm {Q} (\mathbf {A S}) ^ {\dagger} \right\| _ {2} \stackrel {\text {(i)}} {=} \left( \right.\sigma_ {\min } \left(\mathrm {Q} (\mathbf {A S}))\right) ^ {- 1} = \left(\sigma_ {\min } \left(\mathbf {A S} + \mathbf {E}\right)\right) ^ {- 1} \stackrel {\text {(i i)}} {\leq} \left(\sigma_ {\min } (\mathbf {A S}) - \| \mathbf {E} \| _ {2}\right) ^ {- 1} \tag {37}
$$
Here,  $\mathbf{E} = \mathrm{Q}(\mathbf{A}\mathbf{S}) - \mathbf{A}\mathbf{S} \in \mathbb{R}^{n \times m}$  is the quantization error matrix from Q, (i) follows because the singular values of  $\mathrm{Q}(\mathbf{A}\mathbf{S})^{\dagger}$  are inverses of the singular values of  $\mathrm{Q}(\mathbf{A}\mathbf{S})$  (assuming  $\mathrm{Q}(\mathbf{A}\mathbf{S})$  is invertible), and (ii) follows from Lemma B.5.
Lower bounding  $\sigma_{\mathrm{min}}(\mathbf{AS})$ : It now suffices to derive the a lower bound on  $\sigma_{\mathrm{min}}(\mathbf{AS})$ , which we do next. We derive two different lower bounds, either of which could be tighter depending on the singular value profile of  $\mathbf{A}$ . The final lower bound will be the maximum of both.
For the first lower bound, let  $\mathbf{A} = \mathbf{U}\pmb{\Sigma}\mathbf{V}^{\top}$  be the full singular value decomposition of  $\mathbf{A}$ , where  $\mathbf{U} \in \mathbb{R}^{n \times n}$ ,  $\pmb{\Sigma} \in \mathbb{R}^{n \times d}$ , and  $\mathbf{V} \in \mathbb{R}^{d \times d}$ . Then, denoting  $\widetilde{\mathbf{S}} = \mathbf{V}^{\top}\mathbf{S}$ , we have,
$$
\begin{array}{l} \sigma_ {\min } (\mathbf {A} \mathbf {S}) = \sigma_ {\min } \left(\mathbf {A} _ {k} \mathbf {S} + (\mathbf {A} - \mathbf {A} _ {k}) \mathbf {S}\right) \stackrel {(i)} {\geq} \sigma_ {\min } (\mathbf {A} _ {k} \mathbf {S}) - \| (\mathbf {A} - \mathbf {A} _ {k}) \mathbf {S} \| _ {2} \\ \geq \sigma_ {\min } \left(\mathbf {A} _ {k} \mathbf {S}\right) - \sigma_ {k + 1} \| \mathbf {S} \| _ {2}, \tag {38} \\ \end{array}
$$
where  $\mathbf{A}_k$  is the best rank- $k$  approximation of  $\mathbf{A}$ . Here, once again, (i) follows from Lemma B.5 and (ii) follows from Lemma B.4. In order to further lower bound  $\sigma_{\mathrm{min}}(\mathbf{A}_k\mathbf{S})$ , let us denote the SVD of  $\mathbf{A}_k$  as  $\mathbf{A}_k = \mathbf{U}\boldsymbol{\Sigma}_k\mathbf{V}_k^\top$ . Since the best rank- $k$  approximation is obtained by retaining the top- $k$  singular values of  $\mathbf{A}$  and zeroing out the rest, we have  $\mathbf{U} \in \mathbb{R}^{n \times n}$ ,  $\boldsymbol{\Sigma}_k \in \mathbb{R}^{n \times k}$  and  $\mathbf{V}_k \in \mathbb{R}^{d \times k}$ . Let us further denote  $\widetilde{\mathbf{S}} = \mathbf{V}_k^\top\mathbf{S}$ . Then we have
$$
\sigma_ {\min } \left(\mathbf {A} _ {k} \mathbf {S}\right) = \sigma_ {\min } \left(\mathbf {U} \boldsymbol {\Sigma} _ {k} \mathbf {V} _ {k} ^ {\top} \mathbf {S}\right) = \sigma_ {\min } \left(\mathbf {U} \boldsymbol {\Sigma} _ {k} \widetilde {\mathbf {S}}\right) \stackrel {\text {(i)}} {=} \sigma_ {\min } \left(\boldsymbol {\Sigma} _ {k} \widetilde {\mathbf {S}}\right) \stackrel {\text {(i i)}} {\geq} \sigma_ {k} \sigma_ {\min } (\widetilde {\mathbf {S}}), \tag {39}
$$
where (i) follows from Lemma B.6, and (ii) follows from Lemma B.4. Note that the  $(i,j)^{\mathrm{th}}$  entry of  $\widetilde{\mathbf{S}}$  is  $\widetilde{S}_{ij} = \mathbf{v}_i^\top \mathbf{s}_j$ , where  $\mathbf{v}_i$  is the  $i^{\mathrm{th}}$  column of  $\mathbf{V}_k$ . Clearly,  $\widetilde{S}_{ij}$  is a Gaussian random variable with mean,  $\mathbb{E}\left[\widetilde{S}_{ij}\right] = \sum_{\ell} V_{\ell i} \mathbb{E}[S_{\ell j}] = 0$ , and variance,  $\operatorname{Var}\left(\widetilde{S}_{ij}\right) = \sum_{\ell} V_{\ell i}^{2} \operatorname{Var}\left(S_{\ell j}\right) = \frac{1}{m} \sum_{\ell} V_{\ell i}^{2} = \frac{1}{m}$ , where the last equality follows from the fact that the columns of  $\mathbf{V}_k$  are orthonormal. In other words, the entries of  $\widetilde{\mathbf{S}} \in \mathbb{R}^{d \times m}$  are distributed according to  $\widetilde{S}_{ij} \sim \mathcal{N}\left(0, \frac{1}{m}\right)$ . Using (39), (38) can be further lower bounded as
$$
\sigma_ {\min } (\mathbf {A} \mathbf {S}) \geq \sigma_ {k} \sigma_ {\min } (\widetilde {\mathbf {S}}) - \sigma_ {k + 1} \| \mathbf {S} \| _ {2} \stackrel {\text {(i)}} {=} \sigma_ {k} \sigma_ {\min } (\mathbf {S}) - \sigma_ {k + 1} \| \mathbf {S} \| _ {2}, \tag {40}
$$
where (i) once again follows from the rotation invariance of spectrum, i.e., Lemma B.6.
On the other hand, let  $r = \mathrm{rank}(\mathbf{A})$  and  $\sigma_r$  by the smallest non-zero singular value of  $\mathbf{A}$ . Then,  $\mathbf{A} = \mathbf{U}\pmb{\Sigma}\mathbf{V}^{\top} = \mathbf{U}\pmb{\Sigma}_r\mathbf{V}_r^{\top}$  and using the same arguments as in (39), we also have
$$
\sigma_ {\min } (\mathbf {A} \mathbf {S}) = \sigma_ {\min } \left(\mathbf {U} \boldsymbol {\Sigma} _ {r} \mathbf {V} _ {r} ^ {\top} \mathbf {S}\right) = \sigma_ {\min } \left(\boldsymbol {\Sigma} _ {r} \mathbf {V} _ {r} ^ {\top} \mathbf {S}\right) \geq \sigma_ {r} \sigma_ {\min } (\mathbf {S}). \tag {41}
$$
Combining (40) and (41), we get
$$
\sigma_ {\min } (\mathbf {A} \mathbf {S}) \geq \max  \left\{\sigma_ {r} \sigma_ {\min } (\mathbf {S}), \sigma_ {k} \sigma_ {\min } (\mathbf {S}) - \sigma_ {k + 1} \| \mathbf {S} \| _ {2} \right\} \tag {42}
$$
All we are left with now is to utilize the concentration bounds on the singular values of  $\mathbf{S}$ . As a consequence of Lemma B.10 and recalling  $\gamma = d / m$ , with probability exceeding  $1 - 2e^{-\frac{mt^2}{2}}$ ,
$$
\sqrt {\gamma} - 1 - t \leq \sigma_ {\min } (\mathbf {S}) \leq \| \mathbf {S} \| _ {2} \leq \sqrt {\gamma} + 1 + t. \tag {43}
$$
Substituting this in (42), with probability exceeding  $1 - 2e^{-\frac{mt^2}{2}}$
$$
\sigma_ {\min } (\mathbf {A S}) \geq \max  \left\{\sigma_ {r} (\sqrt {\gamma} - 1 - t), \sigma_ {k} (\sqrt {\gamma} - 1 - t) - \sigma_ {k + 1} (\sqrt {\gamma} + 1 + t) \right\}. \tag {44}
$$
Upper bounding  $\| \mathbf{E}\| _2$ : Finally, conditioned on the event  $\mathcal{Q}$ , the entries of  $\mathbf{E}$  are bounded. Choosing the dynamic range of  $\tilde{\mathbf{Q}}$  as dictated by the upper bound on  $\| \mathbf{AS}\|_{\max}$  in Lemma E.1, we have, with probability exceeding  $1 - \frac{\epsilon}{8nR^2}$ ,
$$
\left| E _ {i j} \right| \leq \Delta = \frac {2 \mathrm {R}}{\left(2 ^ {\mathrm {B}} - 1\right)} \sqrt {\frac {2 \log \left(\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon}\right)}{m}} \quad \text {f o r a l l} i \in [ n ] \text {a n d} j \in [ m ]. \tag {45}
$$
Since  $E_{ij}$  is a bounded w.h.p., it is also subgaussian w.h.p. (ref. eq. (16)) with subgaussian norm given by,
$$
\left\| E _ {i j} \right\| _ {\psi_ {2}} \leq \frac {\Delta}{\log 2} = \frac {2 \mathrm {R}}{(\log 2) (2 ^ {\mathrm {B}} - 1)} \sqrt {\frac {2 \log \left(\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon}\right)}{m}}. \tag {46}
$$
From Lemma B.12 we get for some absolute constant  $C$
$$
\| \mathbf {E} \| _ {2} \leq \frac {2 C R (\sqrt {d} + \sqrt {m} + \tilde {t})}{(2 ^ {\mathrm {B}} - 1) \log 2} \sqrt {\frac {2 \log (\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon})}{m}} \text {w . p . e x c e e d i n g} 1 - \frac {\epsilon}{8 n \mathrm {R} ^ {2}} - 2 e ^ {- \tilde {t} ^ {2}}
$$
$$
\left. \right.\left\| \mathbf {E} \right\| _ {2} \leq \frac {2 C R (\sqrt {\gamma} + 1 + \tilde {t})}{(2 ^ {\mathrm {B}} - 1) \log 2} \sqrt {2 \log \left(\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon}\right)} \text {w . p . e x c e e d i n g} 1 - \frac {\epsilon}{8 n \mathrm {R} ^ {2}} - 2 e ^ {- m \tilde {t} ^ {2}}. \tag {47}
$$
Completing the proof: Finally, combining (36), (37), (44) and (47), we get with probability exceeding  $1 - \frac{\epsilon}{8n\mathrm{R}^2} - 2e^{-\frac{mt^2}{2}} - 2e^{-mt^2}$ ,
$$
\begin{array}{l} \left\| \mathrm {Q} (\mathbf {A S}) ^ {\dagger} \mathbf {A} \right\| _ {\max } \leq \sigma_ {1} \left[ \max  \left\{\sigma_ {r} (\sqrt {\gamma} - 1 - t), \sigma_ {k} (\sqrt {\gamma} - 1 - t) - \sigma_ {k + 1} (\sqrt {\gamma} + 1 + t) \right\} \right. (48) \\ \left. - \frac {2 C R (\sqrt {\gamma} + 1 + \tilde {t})}{(2 ^ {\mathrm {B}} - 1) \log 2} \sqrt {2 \log \left(\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon}\right)} \right] ^ {- 1} \\ \leq \frac {\sigma_ {1}}{\sqrt {\gamma} - 1 - t} \left[ \max  \left\{\sigma_ {r}, \sigma_ {k} - \sigma_ {k + 1} \left(\frac {\sqrt {\gamma} + 1 + t}{\sqrt {\gamma} - 1 - t}\right) \right\} \right. \\ \left. - \frac {2 C R}{(2 ^ {\mathrm {B}} - 1) \log 2} \left(\frac {\sqrt {\gamma} + 1 + \tilde {t}}{\sqrt {\gamma} - 1 - t}\right) \sqrt {2 \log \left(\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon}\right)} \right] ^ {- 1} \\ = \frac {\sigma_ {1}}{\sqrt {\gamma} - 1 - t} \left[ \mu - \frac {2 C R}{\left(2 ^ {\mathrm {B}} - 1\right) \log 2} \left(\frac {\sqrt {\gamma} + 1 + \tilde {t}}{\sqrt {\gamma} - 1 - t}\right) \sqrt {2 \log \left(\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon}\right)} \right] ^ {- 1}, (49) \\ \end{array}
$$
where we denote  $\mu = \max \left\{\sigma_r, \sigma_k - \sigma_{k+1}\left(\frac{\sqrt{\gamma} + 1 + t}{\sqrt{\gamma} - 1 - t}\right)\right\}$ . Let us choose our bit-budget B of quantizer Q to be such that it satisfies  $\|\mathbf{E}\|_2 \leq \frac{\mu}{2}$ , i.e.,
$$
\mathrm {B} \geq \log_ {2} \left(\frac {4 C \mathrm {R}}{\mu \log 2} \left(\frac {\sqrt {\gamma} + 1 + \tilde {t}}{\sqrt {\gamma} - 1 - t}\right) \sqrt {2 \log \left(\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon}\right)} + 1\right). \tag {50}
$$
$$
\left\| \mathrm {Q} (\mathbf {A S}) ^ {\dagger} \mathbf {A} \right\| _ {\max } \leq \frac {2 \sigma_ {1}}{(\sqrt {\gamma} - 1 - t) \mu} \tag {51}
$$
$$
t = \sqrt {\frac {2 \log \left(\frac {3 2 n R ^ {2}}{\epsilon}\right)}{m}} \quad \text {a n d ,} \quad \tilde {t} = \sqrt {\frac {\log \left(\frac {3 2 n R ^ {2}}{\epsilon}\right)}{m}} = \frac {t}{\sqrt {2}}, \tag {52}
$$
it follows that with probability exceeding  $1 - \frac{\epsilon}{4n\mathrm{R}^2}$
$$
\left\| \mathrm {Q} (\mathbf {A S}) ^ {\dagger} \mathbf {A} \right\| _ {\max } \leq \frac {2}{\sqrt {\gamma} - 1 - t} \min  \left\{\kappa (\mathbf {A}), \frac {\kappa \left(\mathbf {A} _ {k}\right)}{1 - \frac {\sigma_ {k + 1}}{\sigma_ {k}} \left(\frac {\sqrt {\gamma} + 1 + t}{\sqrt {\gamma} - 1 - t}\right)} \right\} \tag {53}
$$
F Sketched least squares with quantized response
Since the approximation error guarantees of sketched least squares with quantized response might be a problem of independent interest, this is a standalone section, and the notations used in this section are independent of the rest of the paper.
Consider the generalized least squares problem
$$
\mathbf {X} ^ {*} = \underset {\mathbf {x} \in \mathbb {R} ^ {p \times q}} {\arg \min } \| \boldsymbol {\Phi} \mathbf {X} - \mathbf {Y} \| _ {\mathrm {F}} ^ {2}, \tag {54}
$$
where  $\Phi \in \mathbb{R}^{\ell \times p}$  and  $\mathbf{Y} \in \mathbb{R}^{\ell \times q}$ . The sketched variant of (54) with quantized response is given by,
$$
\widetilde {\mathbf {X}} = \underset {\mathbf {X} \in \mathbb {R} ^ {p \times q}} {\arg \min } \| \mathbf {G} \boldsymbol {\Phi} \mathbf {X} - \mathrm {Q} (\mathbf {G} \mathbf {Y}) \| _ {\mathrm {F}} ^ {2}, \tag {55}
$$
where  $\mathbf{G} \in \mathbb{R}^{m \times \ell}$  is a Gaussian sketch matrix with entries are distributed as  $G_{ij} \sim \mathcal{N}\left(0, \frac{1}{m}\right)$ , and  $\mathrm{Q} \equiv \mathrm{Q}_{\mathrm{R,B}}$  is the uniformly dithered quantizer as described in (1). We assume that the dynamic range  $\mathrm{R} \geq \| \mathbf{G} \mathbf{Y} \|_{\max}$  so that  $\mathrm{Q}$  is unsaturated. The solution of (55) can be obtained in closed form as,
$$
\tilde {\mathbf {X}} = (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \mathrm {Q} (\mathbf {G} \mathbf {Y}) \tag {56}
$$
The following lemma provides a characterization of the accuracy of  $\tilde{\mathbf{X}}$  with respect to the original problem (54).
Lemma F.1. Let  $\mathbf{G} \in \mathbb{R}^{m \times \ell}$  be a random Gaussian matrix with entries distributed as  $G_{ij} \sim \mathcal{N}(0, \frac{1}{m})$ , and  $\mathrm{Q} \equiv \mathrm{Q}_{\mathrm{R,B}}$  be a uniformly dithered quantizer with dynamic range R and bit-budget B. Furthermore, suppose  $\Phi \in \mathbb{R}^{\ell \times p}$  and  $\mathbf{Y} \in \mathbb{R}^{\ell \times q}$  be given, and let us denote
$$
\mathbf{X}^{*} = \operatorname *{arg  min}_{\mathbf{X}\in \mathbb{R}^{p\times q}}\| \boldsymbol {\Phi}\mathbf{X} - \mathbf{Y}\|_{\mathrm{F}}^{2}\quad and\quad \widetilde{\mathbf{X}} = \operatorname *{arg  min}_{\mathbf{X}\in \mathbb{R}^{p\times q}}\| \mathbf{G}\boldsymbol {\Phi}\mathbf{X} - \mathrm{Q}(\mathbf{G}\mathbf{Y})\|_{\mathrm{F}}^{2}.
$$
Let  $\mathbf{E} = \mathrm{Q}(\mathbf{G}\mathbf{Y}) - \mathbf{G}\mathbf{Y}$  be the quantization error matrix. Then, if  $\mathrm{R} \geq \| \mathbf{G}\mathbf{Y}\|_{\max}$ , we have
$$
\| \boldsymbol {\Phi} \mathbf {X} ^ {*} - \mathbf {Y} \| _ {\mathrm {F}} ^ {2} \leq \mathbb {E} \left\| \boldsymbol {\Phi} \widetilde {\mathbf {X}} - \mathbf {Y} \right\| _ {\mathrm {F}} ^ {2} \leq \frac {m - 1}{m - r - 1} \| \boldsymbol {\Phi} \mathbf {X} ^ {*} - \mathbf {Y} \| _ {\mathrm {F}} ^ {2} + \frac {q \Delta^ {2}}{4} \frac {\sigma_ {\max} ^ {2}}{\sigma_ {\min} ^ {2}} \frac {m ^ {2}}{(\ell - m - 1)},
$$
where  $r = \mathrm{rank}(\Phi)$ , and  $\sigma_{\mathrm{max}}$  and  $\sigma_{\mathrm{min}}$  are the maximum and minimum singular values of  $\Phi$  respectively.
Proof. The solution of the generalized least squares problem (54) can be written as:
$$
\mathbf {X} ^ {*} = \left[ \mathbf {x} _ {1} ^ {*} \dots \mathbf {x} _ {q} ^ {*} \right], \quad \text {w h e r e ,} \quad \mathbf {x} _ {i} ^ {*} = \underset {\mathbf {x} \in \mathbb {R} ^ {p}} {\arg \min } \| \boldsymbol {\Phi} \mathbf {x} - \mathbf {y} _ {i} \| ^ {2}, \tag {57}
$$
where  $\mathbf{y}_i\in \mathbb{R}^p$  denote the  $i^{\mathrm{th}}$  column of  $\mathbf{Y}$ . Consequently, we will first analyze the standard least squares problem