SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here, we can ensure (i) holds true, i.e., that this term does not exceed  $\epsilon /4$  if we set the bit-budget B to satisfy,
$$
\mathrm {B} \geq \log_ {2} \left(\frac {2 \mathrm {R} \kappa (\mathbf {A} _ {k}) \sqrt {2 n}}{\sqrt {\epsilon (\gamma - 1 - \frac {1}{m})}} \sqrt {\log \left(\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon}\right)} + 1\right). \tag {80}
$$
Analyzing the term  $\mathrm{T}_5$ : Since we choose the dynamic range of the quantizer  $\mathrm{Q}'$  to be equal to  $\frac{2\kappa}{\sqrt{\gamma - 1 - t}}$  (ref. to Lemma E.2), where  $t$  and  $\kappa$  are defined as in (33), we have
$$
\frac {d \Delta^ {\prime 2}}{4} \| \mathbf {A} \| _ {\mathrm {F}} ^ {2} \leq \frac {n d \mathrm {R} ^ {2}}{\left(2 ^ {\mathrm {B} ^ {\prime}} - 1\right) ^ {2}} \frac {4 \kappa^ {2}}{\left(\sqrt {\gamma} - 1 - t\right) ^ {2}} \stackrel {\text {(i)}} {\leq} \frac {\epsilon}{4}, \tag {81}
$$
where we have made use of Asm. 3.1. Once again, in order to ensure (i), it suffices to choose the bit-budget  $\mathrm{B}^{\prime}$  to be
$$
\mathrm {B} ^ {\prime} \geq \log_ {2} \left(\frac {4 \mathrm {R} \kappa}{(\sqrt {\gamma} - 1 - t)} \sqrt {\frac {n d}{\epsilon}} + 1\right) \tag {82}
$$
Note that in order for the dynamic range of quantizer  $Q'$  to hold true in Lemma E.2, we also require B to satisfy (34), i.e.,
$$
\mathrm {B} \geq \log_ {2} \left(\frac {4 C \mathrm {R}}{\max  \left\{\sigma_ {r} , \sigma_ {k} - \sigma_ {k + 1} \left(\frac {\sqrt {\gamma} + 1 + t}{\sqrt {\gamma} - 1 - t}\right) \right\} \log 2} \left(\frac {\sqrt {\gamma} + 1 + t / \sqrt {2}}{\sqrt {\gamma} - 1 - t}\right) \sqrt {2 \log \left(\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon}\right)} + 1\right). \tag {83}
$$
Analyzing the term  $\mathrm{T}_{6}$ : Using bit-budgets B and  $\mathrm{B}^{\prime}$  as in (80) and (82), the final term is
$$
n m d \frac {\Delta^ {2}}{4} \frac {\Delta^ {\prime 2}}{4} \leq n m d \frac {\epsilon}{4} \left(\frac {\gamma - 1 - \frac {1}{m}}{n m \kappa (\mathbf {A} _ {k})}\right) \frac {\epsilon}{4 n d R ^ {2}} \leq \frac {\epsilon^ {2} \gamma}{1 6 n R ^ {2} \kappa (\mathbf {A} _ {k}) ^ {2}} \overset {(i)} {\leq} \frac {\epsilon}{4} \tag {84}
$$
Here, (i) is ensured by choosing  $\epsilon$  to be sufficiently small - specifically,  $\epsilon \leq \frac{4nR^2\kappa(\mathbf{A})^2}{\gamma}$ .
Tying it all together: Combining (79), (81), and (84), with the bit-budgets set appropriately as dictated by (80), (83), and (82), eq. (78) simplifies to
$$
\mathbb {E} \| \mathrm {Q} (\mathbf {A S}) \mathrm {Q} ^ {\prime} \left(\mathrm {Q} (\mathbf {A S}) ^ {\dagger} \mathbf {A}\right) - \mathbf {A} \| _ {\mathrm {F}} ^ {2} \leq \left(1 + \frac {k}{m - k - 1}\right) \| \mathbf {A} _ {k} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} + \frac {3 \epsilon}{4}. \tag {85}
$$
G.1 LPLR approximation error: Proof of Thm. 3.2
We now formally state our main approximation result.
Theorem G.2. (LPLR approximation error (formal)) Let our matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  have non-zero singular values  $\sigma_1, \ldots, \sigma_k, \sigma_{k+1}, \ldots, \sigma_r$ , where  $r = \mathrm{rank}(\mathbf{A})$ , and bounded row norms  $\|\mathbf{a}^{(i)}\| \leq \mathrm{R}$ . Let  $\kappa(\mathbf{A}) = \sigma_1 / \sigma_r$  and  $\kappa(\mathbf{A}_k) = \sigma_1 / \sigma_k$  respectively be the condition numbers of  $\mathbf{A}$  and the best rank- $k$  approximation of  $\mathbf{A}$ , and let us denote
for some sufficiently small  $\epsilon$  that satisfies  $0 < \epsilon \leq \frac{4n\mathrm{R}^2\kappa(\mathbf{A})^2}{\gamma}$ . Here  $\gamma = d / m$  is the aspect ratio of the sketching matrix  $\mathbf{S} \in \mathbb{R}^{d \times m}$  with  $S_{ij} \stackrel{i.i.d.}{\sim} \mathcal{N}\left(0, \frac{1}{m}\right)$ . Suppose the dynamic ranges of the quantizers Q and Q' are set to  $\mathrm{R}\sqrt{\frac{2\log\left(\frac{16\mathrm{R}^2n^2m}{\epsilon}\right)}{m}}$  and  $\frac{2\kappa}{\sqrt{\gamma - 1 - t}}$  respectively, and suppose their bit-budgets B and B' satisfy
$$
\mathrm {B} \geq \max  \{\mathrm {B} _ {1}, \mathrm {B} _ {2} \} a n d, \mathrm {B} ^ {\prime} \geq \log_ {2} \left(\frac {4 \mathrm {R} \kappa}{\left(\sqrt {\gamma} - 1 - t\right)} \sqrt {\frac {n d}{\epsilon}} + 1\right),
$$
$$
\log_ {2} \left(\frac {4 C R}{\max  \left\{\sigma_ {r} , \sigma_ {k} - \sigma_ {k + 1} \left(\frac {\sqrt {\gamma} + 1 + t}{\sqrt {\gamma} - 1 - t}\right) \right\}} \log 2 \left(\frac {\sqrt {\gamma} + 1 + t / \sqrt {2}}{\sqrt {\gamma} - 1 - t}\right) \sqrt {2 \log \left(\frac {1 6 R ^ {2} n ^ {2} m}{\epsilon}\right)} + 1\right).
$$
Then, the low-precision and low-rank factorization returned by Alg. 1 satisfies
$$
\mathbb {E} \| \mathbf {L R} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} \leq \left(1 + \frac {k}{m - k - 1}\right) \| \mathbf {A} _ {k} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} + \epsilon , \tag {86}
$$
where the expectation is over the random sketching matrix  $\mathbf{S}$ , as well as the inherent stochasticity from quantizers  $\mathrm{Q}$  and  $\mathrm{Q}'$ .
Proof. For the purpose of analysis, we assume that Alg. 1 returns  $\mathbf{L} = \mathbf{0}$  and  $\mathbf{R} = \mathbf{0}$  if either quantizer Q or  $\mathrm{Q^{\prime}}$  gets saturated. In practical implementation, it can easily be checked if either quantizer Q or  $\mathrm{Q^{\prime}}$  gets saturated or not, and the algorithm can be repeated again with a new realization of the sketching matrix S and stochastic quantizer Q. Since the choice of dynamic ranges for Q and  $\mathrm{Q^{\prime}}$  ensures that they remain unsaturated with a high probability, "reasonably few" realizations of S would suffice to get at least one good realization in which Q and  $\mathrm{Q^{\prime}}$  are unsaturated.
However, in what follows, we assume that if either quantizer  $\mathrm{Q}$  or  $\mathrm{Q}'$  gets saturated, then the algorithm returns  $\mathbf{0}$  as an estimate of  $\mathbf{A}$ , resulting in a Frobenius norm error of  $\| \mathbf{A} \|_{\mathrm{F}}$ . Since this happens with a very small probability, we show that its contribution to the expected Frobenius norm error of Alg. 1 is small as well. With this in mind, the expected approximation error can be written as
$$
\mathbb {E} \left\| \mathbf {L R} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} = \mathbb {E} \left[ \left\| \operatorname {Q} (\mathbf {A S}) \operatorname {Q} ^ {\prime} \left(\operatorname {Q} (\mathbf {A S}) ^ {\dagger} \mathbf {A}\right) - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \mathbb {1} _ {\mathcal {E}} \right] + \mathbb {E} \left[ \| \mathbf {A} \| _ {\mathrm {F}} ^ {2} \mathbb {1} _ {\mathcal {E} ^ {C}} \right]
$$
$$
\leq^ {(i)} \mathbb {E} \left[ \left\| Q (\mathbf {A S}) Q ^ {\prime} \left(Q (\mathbf {A S}) ^ {\dagger} \mathbf {A}\right) - \mathbf {A} \right\| _ {F} ^ {2} \mathbb {1} _ {\varepsilon} \right] + n R ^ {2} \Pr \left(\mathbb {1} _ {\varepsilon C}\right). \tag {87}
$$
Inequality (i) follows from Asm. 3.1 and the fact that the expectation of indicator function of an event is the probability of the event. From Lemma G.1, the first term can be upper bounded as:
$$
\mathbb {E} \left[ \left\| Q (\mathbf {A S}) Q ^ {\prime} \left(Q (\mathbf {A S}) ^ {\dagger} \mathbf {A}\right) - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \mathbb {1} _ {\varepsilon} \right] \leq \left(1 + \frac {k}{m - k - 1}\right) \| \mathbf {A} _ {k} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} + \frac {3 \epsilon}{4}. \tag {88}
$$
Since Lemmas E.1 and E.2 state the probabilities of quantizers  $\mathrm{Q}$  and  $\mathrm{Q}'$  being unsaturated,  $\operatorname*{Pr}\left(\mathbb{1}_{\mathcal{E}^c}\right)$  can be obtained by an application of union bound as
$$
\Pr \left(\mathbb {1} _ {\mathcal {E} ^ {C}}\right) \leq \frac {\epsilon}{8 n \mathrm {R} ^ {2}} + \frac {\epsilon}{8 n \mathrm {R} ^ {2}} = \frac {\epsilon}{4 n \mathrm {R} ^ {2}}. \tag {89}
$$
Then, (87) can be written as:
$$
\mathbb {E} \| \mathbf {L R} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} \leq \left(1 + \frac {k}{m - k - 1}\right) \| \mathbf {A} _ {k} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} + \epsilon . \tag {90}
$$
G.2 Informal version of Thm. G.2
From Thm. G.2, we can get a (simplified) asymptotic dependence of the bit-budgets B and  $\mathbf{B}^{\prime}$ . We have  $\mathrm{B}\geq \max \{\mathrm{B}_1,\mathrm{B}_2\}$ ,
$$
\begin{array}{l} \mathrm {B} _ {1} = \log_ {2} \left(\frac {2 \mathrm {R} \kappa (\mathbf {A} _ {k}) \sqrt {2 n}}{\sqrt {\epsilon (\gamma - 1 - \frac {1}{m})}} \sqrt {\log \left(\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon}\right)} + 1\right) \\ \stackrel {\mathrm {(i)}} {=} \frac {1}{2} \log_ {2} \left(\frac {\kappa (\mathbf {A} _ {k}) ^ {2} n m}{\epsilon (d - m - 1)} \log \left(\frac {m n ^ {2}}{\epsilon}\right)\right) \\ \stackrel {\text {(i i)}} {=} \frac {1}{2} \log_ {2} \left(\frac {\kappa (\mathbf {A} _ {k}) ^ {2}}{\epsilon} \frac {n m}{d} \log \left(\frac {m n ^ {2}}{\epsilon}\right)\right), \tag {91} \\ \end{array}
$$
where we have ignored the constant terms inside the  $\log_2(\cdot)$  in (i) and considered the regime  $m\ll d$  in (ii). Furthermore,  $\mathrm{B}_2$  is
$$
\begin{array}{l} \mathrm {B} _ {2} = \log_ {2} \left(\frac {4 C \mathrm {R}}{\max  \left\{\sigma_ {r} , \sigma_ {k} - \sigma_ {k + 1} \left(\frac {\sqrt {\gamma} + 1 + t}{\sqrt {\gamma} - 1 - t}\right) \right\} \log 2} \left(\frac {\sqrt {\gamma} + 1 + t / \sqrt {2}}{\sqrt {\gamma} - 1 - t}\right) \sqrt {2 \log \left(\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon}\right)} + 1\right) \\ \stackrel {\text {(i)}} {=} \frac {1}{2} \log_ {2} \left(\frac {1}{\left(\max  \left\{\sigma_ {r} , \sigma_ {k} - \sigma_ {k + 1} c ^ {\prime} \right\}\right) ^ {2}} \log \left(\frac {m n ^ {2}}{\epsilon}\right)\right). \tag {92} \\ \end{array}
$$
where  $c'$  is approximately a constant. Here, (i) follows because when  $m \ll d$ , we have
$$
\frac {\sqrt {\gamma} + 1 + t}{\sqrt {\gamma} - 1 - t} = \frac {\sqrt {d} + \sqrt {m} + \mathrm {O} \left(\sqrt {\log \left(\frac {n}{\epsilon}\right)}\right)}{\sqrt {d} - \sqrt {m} - \mathrm {O} \left(\sqrt {\log \left(\frac {n}{\epsilon}\right)}\right)} = \mathrm {O} (1). \tag {93}
$$
Similarly,  $\frac{\sqrt{\gamma} + 1 + t / \sqrt{2}}{\sqrt{\gamma} - 1 - t} = \mathrm{O}(1)$ . Comparing (91) and (92), we have
$$
\mathrm {B} \geq \frac {1}{2} \log_ {2} \left(\frac {\kappa \left(\mathbf {A} _ {k}\right) ^ {2}}{\epsilon} \frac {n m}{d} \log \left(\frac {m n ^ {2}}{\epsilon}\right)\right). \tag {94}
$$
Moreover, the bit-budget  $\mathrm{B^{\prime}}$  is
$$
\begin{array}{l} \mathrm {B} ^ {\prime} \geq \log_ {2} \left(\frac {4 \mathrm {R} \kappa}{(\sqrt {\gamma} - 1 - t)} \sqrt {\frac {n d}{\epsilon}} + 1\right) \\ \stackrel {\text {(i)}} {=} \log_ {2} \left(\frac {\kappa \sqrt {n d m}}{\sqrt {\epsilon} \left(\sqrt {d} - \sqrt {m} - \sqrt {\frac {n}{\epsilon}}\right)}\right) \stackrel {\text {(i i)}} {=} \frac {1}{2} \log_ {2} \left(\kappa^ {2} \frac {n m}{\epsilon}\right), \tag {95} \\ \end{array}
$$
where once again, to get (i) we have ignored the constants, and (ii) holds true when  $m \ll d$ . Moreover, suppose  $n \approx d$  (if they are not equal, then the total bit-budget should be computed as a
weighted average, since the  $\mathbf{L}$  has  $nm$  elements and the second low-rank factor has  $md$  elements). Then, for some constant  $c_{3}$  that depends on R, the total bit-budget is
$$
\mathrm {B} \geq \frac {1}{4} \log_ {2} \left(\frac {c _ {3} ^ {2} \kappa (\mathbf {A} _ {k}) ^ {2} \kappa^ {2}}{\epsilon^ {2}} \frac {n ^ {2} m ^ {2}}{d} \log \left(\frac {m n ^ {2}}{\epsilon}\right)\right) = \frac {1}{2} \log_ {2} \left(\frac {c _ {3} \kappa (\mathbf {A} _ {k}) \kappa}{\epsilon} \frac {n m}{\sqrt {d}} \sqrt {\log \left(\frac {m n ^ {2}}{\epsilon}\right)}\right).
$$
Furthermore, the dynamic range of quantizer  $\mathrm{Q}$  is  $c_{1}\sqrt{\frac{\log\left(\frac{n}{\epsilon}\right)}{m}}$  and that of quantizer  $\mathrm{Q}'$  when  $m \ll d$  is
$$
\frac {2 \kappa}{\sqrt {\gamma} - 1 - t} = 2 \sqrt {\frac {m}{d}} \min \left\{\kappa (\mathbf {A}), \frac {\kappa (\mathbf {A} _ {k})}{1 - c _ {4} \frac {\sigma_ {k + 1}}{\sigma_ {k}}} \right\},
$$
for constants  $c_{1}, c_{2}$  and  $c_{4}$  that depend on  $\mathbf{R}$ , where additive logarithmic terms are ignored with respect to  $m$ . This gives us the result in Thm. 3.2.
H Direct-SVD Quantization: Approximation error analysis
In this section, we consider the baseline scheme for obtaining low-precision low-rank factorization by first computing the optimal low-rank factorization using SVD and individually quantizing the low-rank factors with uniform scalar quantizers. For any given matrix,  $\mathbf{A} \in \mathbb{R}^{n \times d}$ , we compute the full SVD as  $\mathbf{A} = \mathbf{U} \boldsymbol{\Sigma} \mathbf{V}^{\top}$ . The best (unquantized) rank- $k$  approximation can be obtained from this by considering the singular vectors corresponding to the top- $k$  singular vectors, and constructing the matrix  $\mathbf{A}_k = (\mathbf{U} \boldsymbol{\Sigma})_k \mathbf{V}_k^{\top}$ . Here,  $(\mathbf{U} \boldsymbol{\Sigma})_k \in \mathbb{R}^{n \times k}$  is the sub-matrix obtained by selecting the first  $k$  columns of  $\mathbf{U} \boldsymbol{\Sigma}$ , and  $\mathbf{V}_k^{\top} \in \mathbb{R}^{k \times d}$  is the sub-matrix obtained by selecting the first  $k$  rows of  $\mathbf{V}^{\top}$ . Subsequently, these low-rank factors are quantized with uniform scalar quantizers  $\mathbf{Q}$  and  $\mathbf{Q}'$  (having bit-budgets B and B' respectively). The algorithm pseudocode is given in Alg. 2 and we provide an upper bound to the approximation error in Proposition H.1. We use the notation  $\widetilde{\mathbf{U}} = \mathbf{U} \boldsymbol{\Sigma} \in \mathbb{R}^{n \times d}$ .
Algorithm 2: Direct-SVD quant.: Directly quantizing the optimal low-rank factorization
Input :Matrix  $\mathbf{A}\in \mathbb{R}^{n\times d}$  , target rank  $k$  , Quantizers Q and  $\mathrm{Q^{\prime}}$
Output: Factorization: LR where  $\mathbf{L} \in \mathbb{R}^{n \times k}$ ,  $\mathbf{R} \in \mathbb{R}^{k \times d}$
1 Compute SVD and get  $\mathbf{A} = \mathbf{U}\pmb{\Sigma}\mathbf{V}^{\top}$
2 Extract the top-  $k$  left (scaled) and right singular vectors and get  $\widetilde{\mathbf{U}}_k = (\mathbf{U}\pmb {\Sigma})_k$  and  $\mathbf{V}_k^\top$
3 Quantize the factors individually and get  $\mathbf{L}\gets \mathrm{Q}(\widetilde{\mathbf{U}}_k)$  and  $\mathbf{R}\gets \mathrm{Q}'(\mathbf{V}_k)$
4 return  $\mathbf{L} \in \mathbb{R}^{n \times k}$ ,  $\mathbf{R} \in \mathbb{R}^{k \times d}$
Proposition H.1. (Direct-SVD quant. approximation error (formal)) Let our matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  have maximum singular value  $\sigma_1$ . Suppose our target rank is  $k$ , and for some small  $\epsilon$  such that  $0 < \epsilon \leq 3k\sigma_1^2$ , the dynamic ranges of quantizers  $\mathrm{Q}$  and  $\mathrm{Q}'$  are set to be  $\sigma_1$  and 1 respectively with bit-budgets satisfying
$$
\mathrm {B} \geq \log_ {2} \left(\sigma_ {1} \sqrt {\frac {3 n k}{\epsilon}} + 1\right) \quad a n d, \quad \mathrm {B} ^ {\prime} \geq \log_ {2} \left(\sigma_ {1} \sqrt {\frac {3 d k}{\epsilon}} + 1\right).
$$
Then, the factorization returned by direct-SVD quantization in Alg. 2 satisfies
$$
\mathbb {E} \left\| \mathbf {L R} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \leq \left\| \mathbf {A} _ {k} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} + \epsilon ,
$$
where the expectation is over the inherent stochasticity of the quantizers  $\mathrm{Q}$  and  $\mathrm{Q}'$ .
Proof. Let us denote the quantization error matrices as  $\mathbf{E} = \mathrm{Q}(\widetilde{\mathbf{U}}_k) - \widetilde{\mathbf{U}}_k$  and  $\mathbf{E}' = \mathrm{Q}'(\mathbf{V}_k^\top) - \mathbf{V}_k^\top$ . The approximation error is given by
$$
\begin{array}{l} \mathbb {E} \left. \left\| \mathrm {Q} (\widetilde {\mathbf {U}} _ {k}) \mathrm {Q} ^ {\prime} (\mathbf {V} _ {k} ^ {\top}) - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} = \mathbb {E} \left\| \mathrm {Q} (\widetilde {\mathbf {U}} _ {k}) (\mathbf {V} _ {k} ^ {\top} + \mathbf {E} ^ {\prime}) - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \right. \\ \stackrel {(\mathrm {i})} {\underbrace {\mathbb {E} \left\| \mathrm {Q} (\widetilde {\mathbf {U}} _ {k}) \mathbf {V} _ {k} ^ {\top} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2}}} + \underbrace {\mathbb {E} \left\| \mathrm {Q} (\widetilde {\mathbf {U}} _ {k}) \mathbf {E} ^ {\prime} \right\| _ {\mathrm {F}} ^ {2}}, \tag {96} \\ \end{array}
$$
where the cross term disappears in (i) because when  $\mathbf{Q}'$  is unsaturated,  $\mathbb{E}_{\mathbf{Q}'}[\mathbf{E}] = \mathbf{0}$ , and
$$
\mathbb {E} _ {\mathrm {Q} ^ {\prime}} \left[ \operatorname {T r} \left(\left(\mathrm {Q} (\widetilde {\mathbf {U}} _ {k}) \mathbf {V} _ {k} ^ {\top} - \mathbf {A}\right) ^ {\top} \mathrm {Q} (\widetilde {\mathbf {U}} _ {k}) \mathbb {E} _ {\mathrm {Q} ^ {\prime}} [ \mathbf {E} ^ {\prime} ]\right) \right] = 0.
$$
Analyzing term  $\mathrm{T}_{1}$ : The first term in (96) can be upper bounded as
$$
\mathbb {E} \left\| \mathrm {Q} \left(\widetilde {\mathbf {U}} _ {k}\right) \mathbf {V} _ {k} ^ {\top} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} = \mathbb {E} \left\| \left(\widetilde {\mathbf {U}} _ {k} + \mathbf {E}\right) \mathbf {V} _ {k} ^ {\top} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \stackrel {\text {(i)}} {=} \| \mathbf {A} _ {k} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} + \mathbb {E} \left\| \mathbf {E} \mathbf {V} _ {k} ^ {\top} \right\| _ {\mathrm {F}} ^ {2}. \tag {97}
$$
Here, in (i), we utilize the fact that  $\mathbf{A}_k = \widetilde{\mathbf{U}}_k\mathbf{V}_k^\top$  and the cross term vanishes once again as quantizer.  $\mathrm{Q}$  is unsaturated, i.e.,  $\mathbb{E}[\mathbf{E}] = \mathbf{0}$  and,
$$
\operatorname {T r} \left(\left(\widetilde {\mathbf {U}} _ {k} \mathbf {V} _ {k} ^ {\top} - \mathbf {A}\right) ^ {\top} \mathbb {E} _ {\mathrm {Q}} [ \mathbf {E} ] \mathbf {V} _ {k} ^ {\top}\right) = 0.
$$
The second term in (97) is
$$
\mathbb {E} \left\| \mathbf {E} \mathbf {V} _ {k} ^ {\top} \right\| _ {\mathrm {F}} ^ {2} = \mathbb {E} \left[ \operatorname {T r} \left(\mathbf {E} \mathbf {V} _ {k} ^ {\top} \mathbf {V} _ {k} \mathbf {E} ^ {\top}\right) \right] \stackrel {\text {(i)}} {=} \operatorname {T r} \left(\mathbb {E} [ \mathbf {E} ^ {\top} \mathbf {E} ]\right) = \sum_ {i = 1} ^ {k} \left(\mathbb {E} [ \mathbf {E} ^ {\top} \mathbf {E} ]\right) _ {i i} \stackrel {\text {(i i)}} {\leq} n k \frac {\Delta^ {2}}{4}. \tag {98}
$$
Here, (i) follows because  $\mathbf{V}_k^\top \mathbf{V}_k = \mathbf{I}_k$ , and (ii) follows because  $\mathbb{E}[\mathbf{E}^\top \mathbf{E}]$  is a diagonal matrix as
$$
\left(\mathbb {E} \left[ \mathbf {E} ^ {\top} \mathbf {E} \right]\right) _ {i j} = \sum_ {\ell = 1} ^ {n} \mathbb {E} \left[ E _ {\ell i} E _ {\ell j} \right] = \left\{ \begin{array}{l l} n \operatorname {V a r} \left(E _ {\ell i} ^ {2}\right) \leq \frac {n \Delta^ {2}}{4} & \text {f o r} i = j, \\ \sum_ {\ell = 1} ^ {n} \mathbb {E} \left[ E _ {\ell i} \right] \mathbb {E} \left[ E _ {\ell j} \right] = 0 & \text {f o r} i \neq j. \end{array} \right. \tag {99}
$$
So, (97) can be upper bounded as
$$
\mathbb {E} \left\| \mathrm {Q} \left(\widetilde {\mathbf {U}} _ {k}\right) \mathbf {V} _ {k} ^ {\top} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \leq \| \mathbf {A} _ {k} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} + n k \frac {\Delta^ {2}}{4}. \tag {100}
$$
Analyzing term  $\mathrm{T}_2$ : We upper bound the second term in (96) as
$$
\mathbb {E} \left\| \mathrm {Q} \left(\widetilde {\mathbf {U}} _ {k}\right) \mathbf {E} ^ {\prime} \right\| _ {\mathrm {F}} ^ {2} = \mathbb {E} \left\| \left(\widetilde {\mathbf {U}} _ {k} + \mathbf {E}\right) \mathbf {E} ^ {\prime} \right\| _ {\mathrm {F}} ^ {2} \stackrel {\text {(i)}} {=} \mathbb {E} \left\| \widetilde {\mathbf {U}} _ {k} \mathbf {E} ^ {\prime} \right\| _ {\mathrm {F}} ^ {2} + \mathbb {E} \| \mathbf {E E} ^ {\prime} \| _ {\mathrm {F}} ^ {2}, \tag {101}
$$
where the cross term vanishes in (i) because  $\mathbb{E}_{\mathrm{Q}'}\left[\mathrm{Tr}\left((\widetilde{\mathbf{U}}_k\mathbf{E}')^\top \mathbb{E}_{\mathrm{Q}}[\mathbf{E}]\mathbf{E}'\right)\right] = 0$ . The first term in (101) is
$$
\begin{array}{l} \mathbb {E} \left\| \widetilde {\mathbf {U}} _ {k} \mathbf {E} ^ {\prime} \right\| _ {\mathrm {F}} ^ {2} = \mathbb {E} \left[ \operatorname {T r} \left(\mathbf {E} ^ {\prime \top} \widetilde {\mathbf {U}} _ {k} ^ {\top} \widetilde {\mathbf {U}} _ {k} \mathbf {E} ^ {\prime}\right) \right] \\ = \mathbb {E} \left[ \operatorname {T r} \left(\widetilde {\mathbf {U}} _ {k} ^ {\top} \widetilde {\mathbf {U}} _ {k} \mathbf {E} ^ {\prime} \mathbf {E} ^ {\prime \top}\right) \right] \\ \stackrel {\mathrm {(i)}} {=} \sum_ {i = 1} ^ {k} \left(\widetilde {\mathbf {U}} _ {k} ^ {\top} \widetilde {\mathbf {U}} _ {k}\right) _ {i i} \left(\mathbb {E} \left[ \mathbf {E} ^ {\prime} \mathbf {E} ^ {\prime \top} \right]\right) _ {i i} \\ \stackrel {\text {(i i)}} {\leq} d \frac {\Delta^ {\prime 2}}{4} \sum_ {i = 1} ^ {k} \left(\widetilde {\mathbf {U}} _ {k} ^ {\top} \widetilde {\mathbf {U}} _ {k}\right) _ {i i} \stackrel {\text {(i i i)}} {\leq} d \frac {\Delta^ {\prime 2}}{4} \sum_ {i = 1} ^ {k} \sigma_ {i} ^ {2} \leq d k \sigma_ {1} ^ {2} \frac {\Delta^ {\prime 2}}{4}. \tag {102} \\ \end{array}
$$
Here, (i) and (ii) follow because  $\mathbb{E}[\mathbf{E}'\mathbf{E}'^\top]$  is a diagonal matrix following the same argument as (99), i.e.,
$$
\left(\mathbb {E} \left[ \mathbf {E} ^ {\prime} \mathbf {E} ^ {\prime \top} \right]\right) _ {i j} = \sum_ {\ell = 1} ^ {n} \mathbb {E} \left[ E _ {i \ell} ^ {\prime} E _ {j \ell} ^ {\prime} \right] = \left\{ \begin{array}{l l} d \operatorname {V a r} \left(E _ {i \ell} ^ {\prime 2}\right) \leq \frac {d \Delta^ {\prime 2}}{4} & \text {f o r} i = j, \\ \sum_ {\ell = 1} ^ {d} \mathbb {E} \left[ E _ {i \ell} ^ {\prime} \right] \mathbb {E} \left[ E _ {j \ell} ^ {\prime} \right] = 0 & \text {f o r} i \neq j. \end{array} \right. \tag {103}
$$
Finally, (iii) follows because the  $i^{\mathrm{th}}$  column of  $\widetilde{\mathbf{U}}_k$  is  $\sigma_i\mathbf{u}_i$ , so  $\left(\widetilde{\mathbf{U}}_k^\top \widetilde{\mathbf{U}}_k\right)_{ii} = (\sigma_i\mathbf{u}_i)^\top (\sigma_i\mathbf{u}_i) = \sigma_i^2\|\mathbf{u}_i\|_2^2 = \sigma_i^2$ .
The second term in (101) is
$$
\begin{array}{l} \mathbb {E} \left\| \mathbf {E} \mathbf {E} ^ {\prime} \right\| _ {\mathrm {F}} ^ {2} = \mathbb {E} \left[ \operatorname {T r} \left(\mathbf {E} ^ {\prime} ^ {\top} \mathbf {E} ^ {\top} \mathbf {E} \mathbf {E} ^ {\prime}\right) \right] \\ = \mathbb {E} _ {\mathrm {Q}} \left[ \operatorname {T r} \left(\mathbf {E} ^ {\top} \mathbf {E} \mathbb {E} _ {\mathrm {Q} ^ {\prime}} \left[ \mathbf {E} ^ {\prime} \mathbf {E} ^ {\prime \top} \right]\right) \right] \\ \stackrel {\mathrm {(i)}} {=} \mathbb {E} _ {\mathbf {Q}} \left[ \sum_ {i = 1} ^ {k} \left(\mathbf {E} ^ {\top} \mathbf {E}\right) _ {i i} \left(\mathbb {E} _ {\mathbf {Q} ^ {\prime}} [ \mathbf {E} ^ {\prime} \mathbf {E} ^ {\prime \top} ]\right) _ {i i} \right] \\ \end{array}
$$
$$
\leq \frac {d \Delta^ {\prime 2}}{4} \sum_ {i = 1} ^ {k} \mathbb {E} _ {\mathrm {Q}} \left[ \left(\mathbf {E} ^ {\top} \mathbf {E}\right) _ {i i} \right] \stackrel {\text {(i i)}} {\leq} k \frac {n \Delta^ {2}}{4} \frac {d \Delta^ {\prime 2}}{4}. \tag {104}
$$
Here, (i) and (ii) follow because  $\mathbb{E}[\mathbf{E}'\mathbf{E}'^\top]$  and  $\mathbb{E}[\mathbf{E}^\top \mathbf{E}]$  are diagonal, which can be seen using similar arguments as in (99). So, (101) is
$$
\mathbb {E} \left\| \mathrm {Q} \left(\widetilde {\mathbf {U}} _ {k}\right) \mathrm {Q} ^ {\prime} \left(\mathbf {V} _ {k} ^ {\top}\right) - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \leq \| \mathbf {A} _ {k} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} + n k \frac {\Delta^ {2}}{4} + d k \sigma_ {1} ^ {2} \frac {\Delta^ {\prime 2}}{4} + n d k \frac {\Delta^ {2}}{4} \frac {\Delta^ {\prime 2}}{4}. \tag {105}
$$