SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {x} ^ {*} = \underset {\mathbf {x} \in \mathbb {R} ^ {p}} {\arg \min } \| \boldsymbol {\Phi} \mathbf {x} - \mathbf {y} \| ^ {2}, \tag {58}
$$
and generalize the results by concatenating  $\mathbf{x}_i^*$  to obtain  $\mathbf{X}^*$ . The sketched variant of (58) with quantized response is given by,
$$
\widetilde {\mathbf {x}} = \underset {\mathbf {x} \in \mathbb {R} ^ {p}} {\arg \min } \| \mathbf {G} \boldsymbol {\Phi} \mathbf {x} - \mathrm {Q} (\mathbf {G} \mathbf {y}) \| ^ {2}. \tag {59}
$$
The solution of (59) is available in closed form as  $\widetilde{\mathbf{x}} = (\mathbf{G}\boldsymbol {\Phi})^{\dagger}\mathbf{Q}(\mathbf{G}\mathbf{y})$ . Let us denote the quantization error as  $\epsilon \triangleq \mathrm{Q}(\mathbf{G}\mathbf{y}) - \mathbf{G}\mathbf{y}\in \mathbb{R}^{m}$ . We then have,
$$
\mathbb {E} \| \boldsymbol {\Phi} \widetilde {\mathbf {x}} - \mathbf {y} \| _ {2} ^ {2} = \mathbb {E} \| \boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \mathbf {Q} (\mathbf {G} \mathbf {y}) - \mathbf {y} \| _ {2} ^ {2} = \mathbb {E} \| \boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} (\mathbf {G} \mathbf {y} + \boldsymbol {\epsilon}) - \mathbf {y} \| _ {2} ^ {2}
$$
$$
\stackrel {\text {(i)}} {=} \mathbb {E} \| \boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \mathbf {G} \mathbf {y} - \mathbf {y} \| _ {2} ^ {2} + \mathbb {E} \| \boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \boldsymbol {\epsilon} \| _ {2} ^ {2}. \tag {60}
$$
Here, (i) follows as the cross term disappears because,
$$
\mathbb {E} \left[ (\boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \mathbf {G} \mathbf {y} - \mathbf {y}) ^ {\top} \boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \boldsymbol {\epsilon} \right] = \mathbb {E} _ {\mathbf {G}} \left[ (\boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \mathbf {G} \mathbf {y} - \mathbf {y}) ^ {\top} \boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \mathbb {E} _ {\mathbf {Q}} [ \boldsymbol {\epsilon} ] \right] = \mathbf {0}, \tag {61}
$$
where the last equality follows from (2), since  $\mathbb{E}_{\mathrm{Q}}[\epsilon] = 0$  when  $\mathrm{Q}$  is a uniformly dithered quantizer.
Let us denote the quantization error matrix by  $\mathbf{E} = \mathrm{Q}(\mathbf{G}\mathbf{Y}) - \mathbf{G}\mathbf{Y} \in \mathbb{R}^{m \times q}$ . Generalizing (60) to the generalized least squares problem by treating each column  $\mathbf{y}_i$  separately and adding, we have,
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {\Phi} \widetilde {\mathbf {X}} - \mathbf {Y} \right\| _ {\mathrm {F}} ^ {2} = \mathbb {E} \left\| \boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \mathbf {G} \mathbf {Y} - \mathbf {Y} \right\| _ {\mathrm {F}} ^ {2} + \mathbb {E} \left\| \boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \mathbf {E} \right\| _ {\mathrm {F}} ^ {2} \\ \stackrel {\mathrm {(i)}} {=} \frac {m - 1}{m - r - 1} \| \boldsymbol {\Phi} \mathbf {X} ^ {*} - \mathbf {Y} \| _ {\mathrm {F}} ^ {2} + \mathbb {E} \| \boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \mathbf {E} \| _ {\mathrm {F}} ^ {2}. \tag {62} \\ \end{array}
$$
Here, (i) follows from known results on the approximation error of sketched generalized least squares Halko et al. [23], Pilanci [48], and  $r$  is the rank of  $\mathbf{A}$ .
Upper bounding  $\mathbb{E}\| \Phi (\mathbf{G}\Phi)^{\dagger}\mathbf{E}\|_{\mathrm{F}}^{2}$ : The remainder of the proof concerns upper bounding the final term in (62). Since  $\| \mathbf{X}\|_{\mathrm{F}}^{2} = \operatorname {Tr}\left[\mathbf{X}^{\top}\mathbf{X}\right]$  for any matrix  $\mathbf{X}$ , using the cyclic property of trace,
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \mathbf {E} \right\| _ {\mathrm {F}} ^ {2} = \mathbb {E} \left[ \operatorname {T r} \left(\mathbf {E} ^ {\top} \left((\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger}\right) ^ {\top} \boldsymbol {\Phi} ^ {\top} \boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \mathbf {E}\right) \right] \\ = \mathbb {E} \left[ \operatorname {T r} \left(\left(\left(\mathbf {G} \boldsymbol {\Phi}\right) ^ {\dagger}\right) ^ {\top} \boldsymbol {\Phi} ^ {\top} \boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \mathbf {E} \mathbf {E} ^ {\top}\right) \right] \\ = \mathbb {E} _ {\mathbf {G}} \left[ \operatorname {T r} \left(\left(\left(\mathbf {G} \boldsymbol {\Phi}\right) ^ {\dagger}\right) ^ {\top} \boldsymbol {\Phi} ^ {\top} \boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \mathbb {E} _ {\mathrm {Q}} \left[ \mathbf {E} \mathbf {E} ^ {\top} \right]\right) \right] \tag {63} \\ \end{array}
$$
Since  $\mathrm{R} \geq \| \mathbf{G}\mathbf{Y}\|_{\max}$ , from (2), the  $(i,j)^{\mathrm{th}}$ -entry of the quantization error matrix  $\mathbf{E}$  satisfies,
$$
\mathbb {E} \left[ E _ {i j} \right] = 0, \quad \text {a n d} \quad \operatorname {V a r} \left(E _ {i j}\right) \leq \frac {\Delta^ {2}}{4} = \frac {\mathrm {R} ^ {2}}{\left(2 ^ {\mathrm {B}} - 1\right) ^ {2}}. \tag {64}
$$
$$
\mathbb {E} \left[ \left(\mathbf {E E} ^ {\top}\right) _ {i j} \right] = \sum_ {k = 1} ^ {q} \mathbb {E} \left[ E _ {i k} E _ {j k} \right] = \left\{ \begin{array}{c c} q \operatorname {V a r} \left(E _ {i k}\right) \leq \frac {q \Delta^ {2}}{4} & \text {f o r} i = j \\ 0 & \text {f o r} i \neq j. \end{array} \right. \tag {65}
$$
In other words,  $\mathbb{E}_{\mathbb{Q}}\left[\mathbf{E}\mathbf{E}^{\top}\right]$  is a diagonal matrix whose diagonal elements are upper bounded by  $\frac{q\Delta^2}{4}$ . Let us denote  $\boldsymbol{\Lambda} = \left((\mathbf{G}\boldsymbol{\Phi})^\dagger\right)^\top \boldsymbol{\Phi}^\top \boldsymbol{\Phi}(\mathbf{G}\boldsymbol{\Phi})^\dagger$ . Then, (63) simplifies to,
$$
\mathbb {E} _ {\mathbf {G}} \left[ \operatorname {T r} \left(\boldsymbol {\Lambda} \cdot \mathbb {E} _ {\mathrm {Q}} \left[ \mathbf {E} ^ {\top} \mathbf {E} \right]\right) \right] = \mathbb {E} _ {\mathbf {G}} \left[ \sum_ {i = 1} ^ {m} \Lambda_ {i i} \left(\mathbb {E} _ {\mathrm {Q}} \left[ \mathbf {E} ^ {\top} \mathbf {E} \right]\right) _ {i i} \right] \leq \frac {q \Delta^ {2}}{4} \mathbb {E} _ {\mathbf {G}} [ \operatorname {T r} (\boldsymbol {\Lambda}) ]. \tag {66}
$$
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {G}} \left[ \operatorname {T r} (\boldsymbol {\Lambda}) \right] = \mathbb {E} _ {\mathbf {G}} \left[ \left\| \boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \right\| _ {\mathrm {F}} ^ {2} \right] \stackrel {\text {(i)}} {\leq} \mathbb {E} _ {\mathbf {G}} \left[ \left\| (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \right\| _ {\mathrm {F}} ^ {2} \right] \sigma_ {\max } ^ {2} (\boldsymbol {\Phi}) \\ = \mathbb {E} _ {\mathbf {G}} \left[ \mathrm {T r} \left[ \left(\mathbf {G} \boldsymbol {\Phi} \boldsymbol {\Phi} ^ {\top} \mathbf {G} ^ {\top}\right) ^ {- 1} \right] \right] \sigma_ {\max} ^ {2} (\boldsymbol {\Phi}) \\ \stackrel {\mathrm {(i i)}} {\leq} \frac {\sigma_ {\max } ^ {2} (\boldsymbol {\Phi})}{\sigma_ {\min } ^ {2} (\boldsymbol {\Phi})} \operatorname {T r} \left(\mathbb {E} \left[ \left(\mathbf {G} \mathbf {G} ^ {\top}\right) ^ {- 1} \right]\right) \\ \stackrel {(\mathrm {i i i})} {=} \frac {\sigma_ {\operatorname* {m a x}} ^ {2}}{\sigma_ {\operatorname* {m i n}} ^ {2}} \frac {m ^ {2}}{(\ell - m - 1)}. \tag {67} \\ \end{array}
$$
Here, (i) follows from Lemma B.1, (ii) is a consequence of Lemma B.2, and (iii) follows from Lemma B.9. Combining (63), (66) and (67) yields,
$$
\mathbb {E} \left\| \boldsymbol {\Phi} (\mathbf {G} \boldsymbol {\Phi}) ^ {\dagger} \mathbf {E} \right\| _ {\mathrm {F}} ^ {2} \leq \frac {q \Delta^ {2}}{4} \frac {\sigma_ {\max } ^ {2}}{\sigma_ {\min } ^ {2}} \frac {m ^ {2}}{(\ell - m - 1)}. \tag {68}
$$
G LPLR algorithm: Approximation error analysis
To prove Thm. 3.2, we will first prove the result when both quantizers  $\mathrm{Q}$  and  $\mathrm{Q}'$  are unsaturated. We first prove Lemma G.1, which gives us an upper bound on the approximation error when  $\mathrm{Q}$  and  $\mathrm{Q}'$  are unsaturated. Since we can ensure that they remain unsaturated with high probability (with appropriate choices of dynamic ranges), the final approximation error upper bound in Thm. 3.2 is slightly worse than Lemma G.1.
Lemma G.1. Let our matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  have non-zero singular values  $\sigma_1, \ldots, \sigma_k, \sigma_{k+1}, \ldots, \sigma_r$ , where  $r = \mathrm{rank}(\mathbf{A})$ , and bounded row norms  $\| \mathbf{a}^{(i)} \| \leq R$ . Let  $\kappa(\mathbf{A}) = \sigma_1 / \sigma_r$  and  $\kappa(\mathbf{A}_k) = \sigma_1 / \sigma_k$  respectively be the condition numbers of  $\mathbf{A}$  and the best rank- $k$  approximation of  $\mathbf{A}$ , and let us denote
$$
t = \sqrt {\frac {2 \log \left(\frac {3 2 n \mathrm {R} ^ {2}}{\epsilon}\right)}{m}}, \kappa = \min \left\{\kappa (\mathbf {A}), \frac {\kappa (\mathbf {A} _ {k})}{1 - \frac {\sigma_ {k + 1}}{\sigma_ {k}} \left(\frac {\sqrt {\gamma} + 1 + t}{\sqrt {\gamma} - 1 - t}\right)} \right\},
$$
for some sufficiently small  $\epsilon$  that satisfies  $0 < \epsilon \leq \frac{4nR^2\kappa(\mathbf{A})^2}{\gamma}$ . Here  $\gamma = d / m$  is the aspect ratio of the sketching matrix  $\mathbf{S} \in \mathbb{R}^{d \times m}$  with  $S_{ij} \stackrel{i.i.d.}{\sim} \mathcal{N}\left(0, \frac{1}{m}\right)$ . Suppose the dynamic ranges of the quantizers  $\mathrm{Q}$  and  $\mathrm{Q}'$  are set to  $\mathrm{R}\sqrt{\frac{2\log\left(\frac{16R^2n^2m}{\epsilon}\right)}{m}}$  and  $\frac{2\kappa}{\sqrt{\gamma - 1 - t}}$  as dictated by Lemmas E.1 and E.2 respectively, and suppose their bit-budgets B and B' satisfy
$$
\mathrm {B} \geq \max  \left\{\mathrm {B} _ {1}, \mathrm {B} _ {2} \right\} \quad a n d, \quad \mathrm {B} ^ {\prime} \geq \log_ {2} \left(\frac {4 \mathrm {R} \kappa}{\left(\sqrt {\gamma} - 1 - t\right)} \sqrt {\frac {n d}{\epsilon}} + 1\right),
$$
$$
\mathrm {B} _ {1} = \log_ {2} \left(\frac {2 \mathrm {R} \kappa (\mathbf {A} _ {k}) \sqrt {2 n}}{\sqrt {\epsilon (\gamma - 1 - \frac {1}{m})}} \sqrt {\log \left(\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon}\right)} + 1\right),
$$
and  $\mathrm{B}_2$  is equal to
$$
\log_ {2} \left(\frac {4 C R}{\max \left\{\sigma_ {r} , \sigma_ {k} - \sigma_ {k + 1} \left(\frac {\sqrt {\gamma} + 1 + t}{\sqrt {\gamma} - 1 - t}\right) \right\} \log 2} \left(\frac {\sqrt {\gamma} + 1 + t / \sqrt {2}}{\sqrt {\gamma} - 1 - t}\right) \sqrt {2 \log \left(\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon}\right)} + 1\right).
$$
Furthermore, let  $\mathcal{E}$  be the event that both quantizers  $\mathrm{Q}$  and  $\mathrm{Q}'$  are unsaturated. Then,
$$
\mathbb {E} \left[ \left\| \mathrm {Q} (\mathbf {A S}) \mathrm {Q} ^ {\prime} \left(\mathrm {Q} (\mathbf {A S}) ^ {\dagger} \mathbf {A}\right) - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \mathbb {1} _ {\mathcal {E}} \right] \leq \left(1 + \frac {k}{m - k - 1}\right) \| \mathbf {A} _ {k} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} + \frac {3 \epsilon}{4},
$$
where  $\mathbb{1}_{(\cdot)}$  is the indicator function.
Proof. Let us denote the quantization error matrices from  $\mathbf{Q}$  and  $\mathbf{Q}'$  as  $\mathbf{E} \in \mathbb{R}^{n \times m}$  and  $\mathbf{E}' \in \mathbb{R}^{m \times d}$  respectively, i.e.,  $\mathbf{E} = \mathrm{Q}(\mathbf{A}\mathbf{S}) - \mathbf{A}\mathbf{S}$  and  $\mathbf{E}' = \mathrm{Q}'\left(\mathrm{Q}(\mathbf{A}\mathbf{S})^\dagger\mathbf{A}\right) - \mathrm{Q}(\mathbf{A}\mathbf{S})^\dagger\mathbf{A}$ . Since  $\mathbb{1}_{\mathcal{E}} = 1$  implies that  $\mathbf{Q}$  and  $\mathbf{Q}'$  are unsaturated,  $\mathbb{E}[\mathbf{E}] = \mathbf{0}$  and  $\mathbb{E}[\mathbf{E}'] = \mathbf{0}$ . Then,
$$
\mathbb {E} \left[ \| Q (\mathbf {A S}) Q ^ {\prime} (Q (\mathbf {A S}) ^ {\dagger} \mathbf {A}) - \mathbf {A} \| _ {F} ^ {2} \mathbb {1} _ {\mathcal {E}} \right] = \mathbb {E} _ {\mathbf {S}} \left[ \mathbb {E} _ {Q, Q ^ {\prime}} \left[ \| Q (\mathbf {A S}) Q ^ {\prime} (Q (\mathbf {A S}) ^ {\dagger} \mathbf {A}) - \mathbf {A} \| _ {F} ^ {2} \right] \mathbb {1} _ {\mathcal {E}} \right].
$$
$$
\begin{array}{l} \mathbb {E} _ {\mathrm {Q}, \mathrm {Q} ^ {\prime}} \left[ \left\| \mathrm {Q} (\mathbf {A S}) \mathrm {Q} ^ {\prime} \left(\mathrm {Q} (\mathbf {A S}) ^ {\dagger} \mathbf {A}\right) - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \right] \mathbb {1} _ {\mathcal {E}} \\ = \mathbb {E} _ {\mathrm {Q}, \mathrm {Q} ^ {\prime}} \left[ \| \mathrm {Q} (\mathbf {A S}) \left(\mathrm {Q} (\mathbf {A S}) ^ {\dagger} \mathbf {A} + \mathbf {E} ^ {\prime}\right) - \mathbf {A} \| _ {\mathrm {F}} ^ {2} \right] \mathbb {1} _ {\mathcal {E}} \\ \stackrel {\text {(i)}} {=} \underbrace {\mathbb {E} _ {\mathrm {Q}} \left[ \| \mathrm {Q} (\mathbf {A S}) \mathrm {Q} (\mathbf {A S}) ^ {\dagger} \mathbf {A} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} \right] \mathbb {1} _ {\mathcal {E}}} _ {\mathrm {T} _ {1}} + \underbrace {\mathbb {E} _ {\mathrm {Q} , \mathrm {Q} ^ {\prime}} \left[ \| \mathrm {Q} (\mathbf {A S}) \mathbf {E} ^ {\prime} \| _ {\mathrm {F}} ^ {2} \right] \mathbb {1} _ {\mathcal {E}}} _ {\mathrm {T} _ {2}} \tag {69} \\ \end{array}
$$
The cross terms disappear in (i) because,
$$
\begin{array}{l} \mathbb {E} _ {\mathrm {Q}, \mathrm {Q} ^ {\prime}} \left[ \operatorname {T r} \left[ \left(\mathrm {Q} (\mathbf {A S}) \mathrm {Q} (\mathbf {A S}) ^ {\dagger} \mathbf {A} - \mathbf {A}\right) ^ {\top} \mathrm {Q} (\mathbf {A S}) \mathbf {E} ^ {\prime} \right] \right] \mathbb {1} _ {\mathcal {E}} \\ = \operatorname {T r} \left(\mathbb {E} _ {\mathrm {Q}} \left[ \left(\mathrm {Q} (\mathbf {A S}) \mathrm {Q} (\mathbf {A S}) ^ {\dagger} \mathbf {A} - \mathbf {A}\right) ^ {\top} \mathrm {Q} (\mathbf {A S}) \mathbb {E} _ {\mathrm {Q} ^ {\prime}} [ \mathbf {E} ^ {\prime} ] \right]\right) \mathbb {1} _ {\varepsilon} \stackrel {\text {(i i)}} {=} 0, \tag {70} \\ \end{array}
$$
where, (ii) follows from  $\mathbb{E}_{\mathbf{Q}'}[\mathbf{E}] = \mathbf{0}$  as the quantizer  $\mathbf{Q}'$  is unbiased when unsaturated.
Analyzing the term  $\mathrm{T}_{1}$ : Recall that  $\mathbf{A}_k$  is the best rank- $k$  approximation of  $\mathbf{A}$  obtained using computing the full-SVD of  $\mathbf{A}$  and subsequently making all singular values of  $\mathbf{A}$  less than  $\sigma_k$  as  $0$ . We next analyze the first term in (69) as
$$
\begin{array}{l} \left\| Q (\mathbf {A} \mathbf {S}) Q (\mathbf {A} \mathbf {S}) ^ {\dagger} \mathbf {A} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \leq \left\| Q (\mathbf {A} \mathbf {S}) \left(\mathbf {A} _ {k} \mathbf {S}\right) ^ {\dagger} \mathbf {A} _ {k} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \\ = \left\| \mathbf {A} _ {k} ^ {\top} \left(\mathbf {S} ^ {\top} \mathbf {A} _ {k} ^ {\top}\right) ^ {\dagger} \mathbf {Q} \left(\mathbf {S} ^ {\top} \mathbf {A} ^ {\top}\right) - \mathbf {A} ^ {\top} \right\| _ {\mathrm {F}} ^ {2} \\ = \left\| \mathbf {A} _ {k} ^ {\top} \widetilde {\mathbf {X}} - \mathbf {A} ^ {\top} \right\| _ {\mathrm {F}} ^ {2}, \tag {71} \\ \end{array}
$$
where  $\widetilde{\mathbf{X}}\triangleq \arg \min_{\mathbf{X}}\| \mathbf{S}^{\top}\mathbf{A}_{k}^{\top}\mathbf{X} - \mathrm{Q}(\mathbf{S}^{\top}\mathbf{A}^{\top})\|_{\mathrm{F}}^{2}$ . This minimization problem is a Gaussian sketched variant of a generalized least squares problem with the response matrix quantized as seen in App. F. Corresponding to the notations in App. F, here we have  $\mathbf{S}^{\top}$  instead of  $\mathbf{G}$ ,  $\mathbf{A}_{k}^{\top}$  instead of  $\boldsymbol{\Phi}$ , and  $\mathbf{A}^{\top}$  instead of  $\mathbf{Y}$ . As a consequence of Lemma F.1, we have the upper bound,
$$
\mathbb {E} _ {\mathrm {Q}} \left[ \| \mathrm {Q} (\mathbf {A S}) \mathrm {Q} (\mathbf {A S}) ^ {\dagger} \mathbf {A} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} \right] \mathbb {1} _ {\mathcal {E}} \leq \frac {m - 1}{m - k - 1} \| \mathbf {A} _ {k} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} + \frac {n \Delta^ {2}}{4} \frac {\sigma_ {1} ^ {2}}{\sigma_ {k} ^ {2}} \frac {m ^ {2}}{(d - m - 1)} \tag {72}
$$
Analyzing the term  $\mathrm{T}_2$ : The term  $\mathrm{T}_2$  can be written as,
$$
\begin{array}{l} \mathbb {E} _ {\mathrm {Q}, \mathrm {Q} ^ {\prime}} \left[ \| \mathrm {Q} (\mathbf {A S}) \mathbf {E} ^ {\prime} \| _ {\mathrm {F}} ^ {2} \right] \mathbb {1} _ {\mathcal {E}} \\ = \mathbb {E} _ {\mathrm {Q}, \mathrm {Q} ^ {\prime}} \left[ \| (\mathbf {A} \mathbf {S} + \mathbf {E}) \mathbf {E} ^ {\prime} \| _ {\mathrm {F}} ^ {2} \right] \mathbb {1} _ {\mathcal {E}} \\ = \mathbb {E} _ {\mathrm {Q} ^ {\prime}} \left[ \| \mathbf {A S E} ^ {\prime} \| _ {\mathrm {F}} ^ {2} \right] \mathbb {1} _ {\mathcal {E}} + \mathbb {E} _ {\mathrm {Q}, \mathrm {Q} ^ {\prime}} \left[ \| \mathbf {E E} ^ {\prime} \| _ {\mathrm {F}} ^ {2} \right] \mathbb {1} _ {\mathcal {E}} + \operatorname {T r} \left(\mathbb {E} _ {\mathrm {Q} ^ {\prime}} \left[ (\mathbf {A S E} ^ {\prime}) ^ {\top} \mathbb {E} _ {\mathrm {Q}} [ \mathbf {E} ] \mathbf {E} ^ {\prime} \right]\right) \mathbb {1} _ {\mathcal {E}} \\ \stackrel {\text {(i)}} {=} \mathbb {E} \left[ \| \mathbf {A S E} ^ {\prime} \| _ {\mathrm {F}} ^ {2} \right] \mathbb {1} _ {\varepsilon} + \mathbb {E} \left[ \| \mathbf {E E} ^ {\prime} \| _ {\mathrm {F}} ^ {2} \right] \mathbb {1} _ {\varepsilon}, \tag {73} \\ \end{array}
$$
where, (i) follows once again since  $\mathbb{E}_{\mathrm{Q}'}[\mathbf{E}] \mathbb{1}_{\mathcal{E}} = \mathbf{0}$ . The first term in (73) can be rewritten as
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {A S E} ^ {\prime} \right\| _ {\mathrm {F}} ^ {2} \mathbb {1} _ {\mathcal {E}} \right] = \mathbb {E} \left[ \operatorname {T r} \left(\mathbf {E} ^ {\prime \top} \mathbf {S} ^ {\top} \mathbf {A} ^ {\top} \mathbf {A S E} ^ {\prime}\right) \mathbb {1} _ {\mathcal {E}} \right] \\ = \mathbb {E} \left[ \operatorname {T r} \left(\mathbf {S} ^ {\top} \mathbf {A} ^ {\top} \mathbf {A S E} ^ {\prime} \mathbf {E} ^ {\prime \top}\right) \mathbf {1} _ {\varepsilon} \right] \\ = \operatorname {T r} \left(\mathbb {E} _ {\mathbf {S}} \left[ \mathbf {S} ^ {\top} \mathbf {A} ^ {\top} \mathbf {A} \mathbf {S} \mathbb {E} _ {\mathrm {Q} ^ {\prime}} \left[ \mathbf {E} ^ {\prime} \mathbf {E} ^ {\prime \top} \right] \mathbb {1} _ {\varepsilon} \right]\right) \\ = \mathbb {E} _ {\mathbf {S}} \left[ \operatorname {T r} \left(\mathbf {S} ^ {\top} \mathbf {A} ^ {\top} \mathbf {A} \mathbf {S} \mathbb {E} _ {\mathrm {Q} ^ {\prime}} \left[ \mathbf {E} ^ {\prime} \mathbf {E} ^ {\prime \top} \right] \mathbb {1} _ {\varepsilon}\right) \right] \tag {74} \\ \end{array}
$$
Similar to (65) in the proof of Lemma F.1,  $\mathbb{E}_{\mathbf{Q}'}\left[\mathbf{E}'\mathbf{E}'^\top\right]\mathbb{1}_{\mathcal{E}}$  is a diagonal matrix with
$$
\mathbb {E} _ {\mathrm {Q} ^ {\prime}} \left[ \left(\mathbf {E} ^ {\prime} \mathbf {E} ^ {\prime} ^ {\top}\right) _ {i j} \right] \mathbb {1} _ {\mathcal {E}} = \sum_ {k = 1} ^ {d} \mathbb {E} _ {\mathrm {Q} ^ {\prime}} \left[ E _ {i k} ^ {\prime} E _ {j k} ^ {\prime} \right] \mathbb {1} _ {\mathcal {E}} = \left\{ \begin{array}{l l} d \cdot \operatorname {V a r} \left(E _ {i k}\right) \leq \frac {d \Delta^ {\prime 2}}{4} & \text {f o r} i = j \\ 0 & \text {f o r} i \neq j. \end{array} \right. \tag {75}
$$
Let us denote  $\Gamma \triangleq \mathbf{S}^{\top}\mathbf{A}^{\top}\mathbf{AS}$ . Then using the fact that  $\mathbb{1}_{\varepsilon} \leq 1$ , we get,
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {S}} \left[ \operatorname {T r} \left(\boldsymbol {\Gamma} \mathbb {E} _ {\mathrm {Q} ^ {\prime}} \left[ \mathbf {E} ^ {\prime} \mathbf {E} ^ {\prime^ {\top}} \right] \mathbb {1} _ {\mathcal {E}}\right) \right] = \mathbb {E} _ {\mathbf {S}} \left[ \sum_ {i = 1} ^ {m} \Gamma_ {i i} \left(\mathbb {E} _ {\mathrm {Q} ^ {\prime}} \left[ \mathbf {E} ^ {\prime} \mathbf {E} ^ {\prime^ {\top}} \right]\right) _ {i i} \right] \mathbb {1} _ {\mathcal {E}} \\ \leq \frac {d \Delta^ {\prime 2}}{4} \operatorname {T r} \left(\mathbb {E} _ {\mathbf {S}} \boldsymbol {\Gamma}\right) \\ = \frac {d \Delta^ {\prime 2}}{4} \operatorname {T r} \left(\mathbf {A} ^ {\top} \mathbf {A} \mathbb {E} _ {\mathbf {S}} \left[ \mathbf {S} \mathbf {S} ^ {\top} \right]\right) \stackrel {\text {(i)}} {=} \frac {d \Delta^ {\prime 2}}{4} \| \mathbf {A} \| _ {\mathrm {F}} ^ {2}. \tag {76} \\ \end{array}
$$
Here, the last equality follows as  $\mathbb{E}_{\mathbf{S}}\left[\mathbf{S}\mathbf{S}^{\top}\right] = \mathbf{I}_d$ . This is because the  $(i,j)^{\mathrm{th}}$  entry of  $\mathbb{E}_{\mathbf{S}}\left[\mathbf{S}\mathbf{S}^{\top}\right]$  is
$$
\mathbb {E} \left[ \left(\mathbf {S S} ^ {\top}\right) _ {i j} \right] = \sum_ {k = 1} ^ {m} \mathbb {E} \left[ S _ {i k} S _ {j k} \right] = \left\{ \begin{array}{l l} m   \mathbb {E} \left[ S _ {i k} ^ {2} \right] = m   \operatorname {V a r} \left(S _ {i k}\right) = 1 & \text {f o r} \quad i = j \\ \sum_ {i = 1} ^ {m} \mathbb {E} [ S _ {i k} ] \mathbb {E} [ S _ {j k} ] = 0 & \text {f o r} \quad i \neq j. \end{array} \right.
$$
Furthermore, the second term  $\mathbb{E}_{\mathrm{Q},\mathrm{Q}'}\left\| \mathbf{EE}'^{\top}\right\|_{\mathrm{F}}^{2}\mathbb{1}_{\mathcal{E}}$  can be simplified as follows,
$$
\begin{array}{l} \mathbb {E} _ {\mathrm {Q}, \mathrm {Q} ^ {\prime}} \left\| \mathbf {E} \mathbf {E} ^ {\prime} \right\| _ {\mathrm {F}} ^ {2} \mathbb {1} _ {\mathcal {E}} = \mathbb {E} _ {\mathrm {Q}, \mathrm {Q} ^ {\prime}} \left[ \operatorname {T r} \left(\mathbf {E} ^ {\prime \top} \mathbf {E} ^ {\top} \mathbf {E} \mathbf {E} ^ {\prime}\right)\right] \mathbb {1} _ {\mathcal {E}} = \mathbb {E} _ {\mathrm {Q}} \left[ \right. \operatorname {T r} \left( \right.\mathbf {E} ^ {\top} \mathbf {E} \left. \mathbb {E} _ {\mathrm {Q} ^ {\prime}} \left(\mathbf {E} ^ {\prime} \mathbf {E} ^ {\prime \top}\right)\right)\left. \right] \mathbb {1} _ {\mathcal {E}} \\ \stackrel {\mathrm {(i)}} {=} \mathbb {E} _ {\mathrm {Q}} \left[ \sum_ {i = 1} ^ {m} \left(\mathbf {E} ^ {\top} \mathbf {E}\right) _ {i i} \left(\mathbb {E} _ {\mathrm {Q} ^ {\prime}} \left[ \mathbf {E} ^ {\prime} \mathbf {E} ^ {\prime \top} \right]\right) _ {i i} \right] \mathbb {1} _ {\mathcal {E}} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {\text {(i i)}} {\leq} \frac {d \Delta^ {\prime 2}}{4} \sum_ {i = 1} ^ {m} \mathbb {E} _ {\mathrm {Q}} \left[ \left(\mathbf {E} ^ {\top} \mathbf {E}\right) _ {i i} \right] \mathbb {1} _ {\varepsilon} \\ \leq \frac {n m d \Delta^ {2} \Delta^ {\prime 2}}{1 6}. \tag {77} \\ \end{array}
$$
Here, (i) follows since  $\mathbb{E}_{\mathbf{Q}'}\left[\mathbf{E}'\mathbf{E}^{\top}\right]\mathbb{1}_{\mathcal{E}}$  is a diagonal matrix as seen before, and (ii) follows from the upper bound derived on  $\mathbb{E}\left[\left(\mathbf{E}^\top \mathbf{E}\right)_{ii}\right]\mathbb{1}_{\mathcal{E}}$  in (65).
Combining all of the above, the approximation error of our low-precision low-rank approximation scheme can be upper bounded as,
$$
\begin{array}{l} \mathbb {E} \| \mathrm {Q} (\mathbf {A S}) \mathrm {Q} ^ {\prime} \left(\mathrm {Q} (\mathbf {A S}) ^ {\dagger} \mathbf {A}\right) - \mathbf {A} \| _ {\mathrm {F}} ^ {2} \\ \leq \underbrace {\frac {(m - 1)}{(m - k - 1)} \| \mathbf {A} _ {k} - \mathbf {A} \| _ {\mathrm {F}} ^ {2}} _ {\mathrm {T} _ {3}} + \underbrace {\frac {n \Delta^ {2}}{4} \frac {\sigma_ {1} ^ {2}}{\sigma_ {k} ^ {2}} \frac {m ^ {2}}{(d - m - 1)}} _ {\mathrm {T} _ {4}} + \underbrace {\frac {d \Delta^ {\prime 2}}{4} \| \mathbf {A} \| _ {\mathrm {F}} ^ {2}} _ {\mathrm {T} _ {5}} + \underbrace {\frac {n m d \Delta^ {2} \Delta^ {\prime 2}}{1 6}} _ {\mathrm {T} _ {6}}. \tag {78} \\ \end{array}
$$
Here, the first term  $\mathrm{T}_3$  is the low-rank approximation error, whereas the remaining terms, i.e.,  $\mathrm{T}_4$ ,  $\mathrm{T}_5$  and  $\mathrm{T}_6$  appear due to quantization. We now analyze each of them separately.
Analyzing the term  $\mathrm{T}_4$ : Since we choose the dynamic range of quantizer Q to be R√ $\frac{2\log\left(\frac{16R^2n^2m}{\epsilon}\right)}{m}$  (ref. Lemma E.1), we have
$$
\frac {n \Delta^ {2}}{4} \frac {\sigma_ {1} ^ {2}}{\sigma_ {k} ^ {2}} \frac {m ^ {2}}{(d - m - 1)} \leq \frac {2 n m R ^ {2} \kappa (\mathbf {A} _ {k}) ^ {2}}{(2 ^ {\mathrm {B}} - 1) ^ {2} (d - m - 1)} \log \left(\frac {1 6 R ^ {2} n ^ {2} m}{\epsilon}\right) \stackrel {(i)} {\leq} \frac {\epsilon}{4}. \tag {79}
$$