SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} d _ {\mathrm {K L}} \left(\left(P _ {u v} ^ {(j - 1)} \mid \operatorname {G o o d H} _ {u v} ^ {(j - 1)}\right) \| G ^ {(j - 1)}\right) \leq \frac {d _ {\mathrm {K L}} \left(P _ {u v} ^ {(j - 1)} \| G ^ {(j - 1)}\right) + 2}{\Pr_ {\boldsymbol {h} ^ {(j - 1)} \sim P _ {u v} ^ {(j - 1)}} [ \boldsymbol {h} ^ {(j - 1)} \in \operatorname {G o o d H} ^ {(j - 1)} ]} \\ \leq (5 / 4) d _ {\mathrm {K L}} \left(P _ {u v} ^ {(j - 1)} \| G ^ {(j - 1)}\right) + 5 / 2. \\ \end{array}
$$
Here we used the inductive assumption that  $\operatorname*{Pr}_{\pmb{h}^{(j - 1)}\sim P_{uv}^{(j - 1)}}[\pmb{h}^{(j - 1)}\in \mathrm{GoodH}^{(j - 1)}]\geq 1 - 1 / (10\gamma)\geq 9 / 10$  where the last inequality follows as the parameter  $\gamma \geq 1$ . Next, we upper bound the second term in (6). We have that  $\pmb{r}^{(j)}|\pmb{h}^{(j - 1)} = Q^{(j)}[\pmb{h}^{(j - 1)}]\pmb {g} + s(Q^{(j)}[\pmb{h}^{(j - 1)}])(u\otimes v)$  is distributed as  $N(s(Q^{(j)}[h^{(j - 1)}])(u\otimes v),I_k)$  by rotational invariance of the Gaussian distribution. Therefore,
$$
d _ {\mathrm {K L}} \left(\left(\boldsymbol {r} ^ {(j)} \mid \boldsymbol {h} ^ {(j - 1)}\right) \| \boldsymbol {g} _ {k} ^ {(j)}\right) = (1 / 2) s ^ {2} \| Q ^ {(j)} [ \boldsymbol {h} ^ {(j - 1)} ] (u \otimes v) \| _ {2} ^ {2} \leq (\alpha^ {2} / n) f _ {j - 1} (\alpha , \gamma) k.
$$
In the last inequality, we used the fact that  $\pmb{h}^{(j - 1)} \in \mathrm{GoodH}_{uv}^{(j - 1)}$ . Finally,
$$
\begin{array}{l} \operatorname {E} _ {(\boldsymbol {u}, \boldsymbol {v}) \sim w ^ {(j)}} [ d (P _ {\boldsymbol {u v}} ^ {(j)} \| G ^ {(j)}) ] \leq (5 / 4) \operatorname {E} _ {(\boldsymbol {u}, \boldsymbol {v}) \sim w ^ {(j)}} d _ {\mathrm {K L}} (P _ {\boldsymbol {u v}} ^ {(j - 1)} \| G ^ {(j - 1)}) + 5 / 2 + \alpha^ {2} f _ {j - 1} (\alpha , \gamma) (k / n) \\ \leq (5 / 2) \mathrm {E} _ {(\boldsymbol {u}, \boldsymbol {v}) \sim w ^ {(j - 1)}} d _ {\mathrm {K L}} \left(P _ {\boldsymbol {u v}} ^ {(j - 1)} \| G ^ {(j - 1)}\right) + 5 / 2 + \alpha^ {2} f _ {j - 1} (\alpha , \gamma) (k / n) \\ \end{array}
$$
as the distribution  $w^{(j)}$  is obtained by conditioning  $w^{(j - 1)}$  on an event with probability  $\geq 1 - 1 / (10\gamma) \geq 1 / 2$  and  $d_{\mathrm{KL}}(\cdot \| \cdot) \geq 0$ . Now, using the inductive assumption, we have
$$
\begin{array}{l} \operatorname {E} _ {(\boldsymbol {u}, \boldsymbol {v}) \sim w ^ {(j)}} [ d (P _ {\boldsymbol {u v}} ^ {(j)} \| G ^ {(j)}) ] \leq (5 / 2) f _ {j - 1} (\alpha , \gamma) (k / n) + 5 / 2 + \alpha^ {2} f _ {j - 1} (\alpha , \gamma) (k / n) \\ \leq 2 \alpha^ {2} f _ {j - 1} (\alpha , \gamma) (k / n) \leq f _ {j} (\alpha , \gamma) (k / n) \\ \end{array}
$$
where we use  $\alpha \geq 15$ . This proves the third statement in the lemma for round  $j$ . We also have,  $I(w^{(j)},\mathcal{P}^{(j)})\leq \operatorname{E}_{(\boldsymbol{u},\boldsymbol{v})\sim w^{(j)}}[d(P_{\boldsymbol{u}\boldsymbol{v}}^{(j)}\| G^{(j)})]\leq 2\alpha^2 f_{j - 1}(\alpha ,\gamma)(k / n)$ . We now define a loss function  $L^{(j)}$  and use Bayes risk lower bounds to prove the remaining statements. Let
$$
\operatorname {L o s s} ^ {(j)} ((u, v), Q) = \left\{ \begin{array}{l l} 1 & \text {i f} \| Q (u \otimes v) \| _ {2} ^ {2} \leq f _ {j} (\alpha , \gamma) k \\ 0 & \text {i f} \| Q (u \otimes v) \| _ {2} ^ {2} > f _ {j} (\alpha , \gamma) k \end{array} \right.
$$
where  $Q \in \mathbb{R}^{k \times n^2}$  is an orthonormal matrix with  $k \geq n$  rows. We have for  $j$  such that  $f_j(\alpha, \gamma) k \leq 16n^2$ ,
$$
R _ {0} (\operatorname {L o s s} ^ {(j)}, w ^ {(j)}) = \inf  _ {Q} \operatorname {E} _ {(\boldsymbol {u}, \boldsymbol {v}) \sim w ^ {(j)}} [ \operatorname {L o s s} ^ {(j)} ((\boldsymbol {u}, \boldsymbol {v}), Q) ] \geq 1 - (1 - 1 / (1 0 \gamma)) ^ {- j} \exp (- \beta f _ {j} (\alpha , \gamma) k / n)
$$
where we use Lemma B.1 and the fact that the distribution  $w^{(j)}$  is obtained by conditioning  $w^{(1)}$  on an event with probability  $\geq (1 - 1 / (10\gamma))^j$  which we prove in section B.4. By the generalized Fano inequality, we obtain
$$
\begin{array}{l} R _ {\text {B a y e s}} \left(\operatorname {L o s s} ^ {(j)}, w ^ {(j)}\right) \geq 1 + \frac {I \left(w ^ {(j)} , \mathcal {P} ^ {(j)}\right) + \log (2)}{\log \left(1 - R _ {0} \left(\operatorname {L o s s} ^ {(j)} , w ^ {(j)}\right)\right)} \\ \geq 1 - \frac {2 \alpha^ {2} f _ {j - 1} (\alpha , \gamma) k / n + \log (2)}{\beta f _ {j} (\alpha , \gamma) k / n + j \log (1 - 1 / (1 0 \gamma))}. \\ \end{array}
$$
As  $f_{j}(\alpha, \gamma) = K\alpha^{2}\gamma^{2}f_{j-1}(\alpha, \gamma)$  and  $f_{0}(\alpha, \gamma) \geq K$  for a large enough constant  $K$ , we have
$$
\beta f _ {j} (\alpha , \gamma) \geq - 1 0 j \log (1 - 1 / (1 0 \gamma))
$$
and therefore for  $K$  large enough,
$$
R _ {\mathrm {B a y e s}} (\mathrm {L o s s} ^ {(j)}, w ^ {(j)}) \geq 1 - \frac {1}{1 0 0 \gamma^ {2}}.
$$
By definition of  $R_{\mathrm{Bayes}}$ , we conclude that
$$
\operatorname {E} _ {(\boldsymbol {u}, \boldsymbol {v}) \sim w ^ {(j)}} \left[ \operatorname {E} _ {\boldsymbol {h} ^ {(j)} \sim P _ {\boldsymbol {u v}} ^ {(j)}} [ \operatorname {L o s s} ((\boldsymbol {u}, \boldsymbol {v}), Q ^ {(j + 1)} [ \boldsymbol {h} ^ {(j)} ]) ] \right] \geq 1 - 1 / (1 0 0 \gamma^ {2}).
$$
By Markov's inequality, we have that with probability  $\geq 1 - 1 / (10\gamma)$  over  $(\pmb{u},\pmb{v})\sim w^{(j)}$ , it holds that
$$
\operatorname {E} _ {\boldsymbol {h} ^ {(j)} \sim P _ {\boldsymbol {u v}} ^ {(j)}} [ \operatorname {L o s s} ((\boldsymbol {u}, \boldsymbol {v}), Q ^ {(j + 1)} [ \boldsymbol {h} ^ {(j)} ]) ] \geq 1 - 1 / (1 0 \gamma)
$$
$$
\operatorname * {P r} _ {\boldsymbol {h} ^ {(j)} \sim P _ {\boldsymbol {u v}} ^ {(j)}} [ \| Q ^ {(j + 1)} [ \boldsymbol {h} ^ {(j)} ] (\boldsymbol {u} \otimes \boldsymbol {v}) \| _ {2} ^ {2} \leq f _ {j} (\alpha , \gamma) k ] \geq 1 - 1 / (1 0 \gamma).
$$
Thus, we conclude that  $\operatorname{Pr}_{(\boldsymbol{u},\boldsymbol{v})\sim w^{(j)}}[(\boldsymbol{u},\boldsymbol{v})\in \mathrm{Good}^{(j)}]\geq 1 - 1 / (10\gamma)$  and that for  $(u,v)\in \mathrm{Good}^{(j)}$  we have  $\operatorname{Pr}_{\boldsymbol{h}^{(j)}\sim P_{uv}^{(j)}}[\boldsymbol{h}^{(j)}\in \mathrm{GoodH}_{uv}^{(j)}]\geq 1 - 1 / (10\gamma)$ .
B.4 Wrap-up
Let  $j \geq 1$  satisfy  $f_{j}(\alpha, \gamma)k \leq 16n^{2}$ . By definition,  $\mathrm{Good}^{(j)} \subseteq \mathrm{Good}^{(j-1)} \subseteq \ldots \mathrm{Good}^{(1)}$ . We also have
$$
w ^ {(j)} = w ^ {(j - 1)} | \operatorname {G o o d} ^ {(j - 1)} = w ^ {(1)} | \operatorname {G o o d} ^ {(j - 1)}.
$$
Now, using the fact that  $\mathrm{Good}^{(j)}\subseteq \mathrm{Good}^{(j - 1)}$
$$
\frac {\Pr_ {(\boldsymbol {u} , \boldsymbol {v}) \sim w ^ {(1)}} [ (\boldsymbol {u} , \boldsymbol {v}) \in \operatorname {G o o d} ^ {(j)} ]}{\Pr_ {(\boldsymbol {u} , \boldsymbol {v}) \sim w ^ {(1)}} [ (\boldsymbol {u} , \boldsymbol {v}) \in \operatorname {G o o d} ^ {(j - 1)} ]} = \Pr_ {(\boldsymbol {u}, \boldsymbol {v}) \sim w ^ {(1)}} [ (\boldsymbol {u}, \boldsymbol {v}) \in \operatorname {G o o d} ^ {(j)} | (\boldsymbol {u}, \boldsymbol {v}) \in \operatorname {G o o d} ^ {(j - 1)} ]
$$
As  $w^{(j)} = w^{(1)}|\mathrm{Good}^{(j - 1)}$ , we have
$$
\begin{array}{l} \Pr_ {(\boldsymbol {u}, \boldsymbol {v}) \sim w ^ {(1)}} [ (\boldsymbol {u}, \boldsymbol {v}) \in \operatorname {G o o d} ^ {(j)} | (\boldsymbol {u}, \boldsymbol {v}) \in \operatorname {G o o d} ^ {(j - 1)} ] = \Pr_ {(\boldsymbol {u}, \boldsymbol {v}) \sim w ^ {(j)}} [ (\boldsymbol {u}, \boldsymbol {v}) \in \operatorname {G o o d} ^ {(j)} ] \\ \geq (1 - 1 / (1 0 \gamma)) \\ \end{array}
$$
where the last inequality is from Lemma B.2. Thus,  $\operatorname{Pr}_{(\boldsymbol{u},\boldsymbol{v})\sim w^{(1)}}[(\boldsymbol{u},\boldsymbol{v})\in \mathrm{Good}^{(j)}]\geq (1 - 1 / (10\gamma))^j$
Similarly, for  $(u,v)\in \mathrm{Good}^{(j)}$ , we have that
$$
\begin{array}{l} \frac {\Pr_ {\boldsymbol {h} ^ {(j)} \sim H _ {u v} ^ {(j)}} [ \boldsymbol {h} ^ {(j)} \in \operatorname {G o o d H} _ {u v} ^ {(j)} ]}{\Pr_ {\boldsymbol {h} ^ {(j - 1)} \sim H _ {u v} ^ {(j - 1)}} [ \boldsymbol {h} ^ {(j - 1)} \in \operatorname {G o o d H} _ {u v} ^ {(j - 1)} ]} = \Pr_ {\boldsymbol {h} ^ {(j)} \sim P _ {u v} ^ {(j)}} [ \boldsymbol {h} ^ {(j)} \in \operatorname {G o o d H} _ {u v} ^ {(j)} ] \\ \geq (1 - 1 / (1 0 \gamma)) \\ \end{array}
$$
where again, the last inequality follows from Lemma B.2. Thus,  $\operatorname*{Pr}_{\pmb{h}^{(j)}\sim H_{uv}^{(j)}}[\pmb{h}^{(j)}\in \mathrm{GoodH}_{uv}^{(j)}]\geq (1 - 1 / (10\gamma))^j$ . Now, for  $(\pmb {u},\pmb {v})\sim w^{(1)}$  and  $\pmb{h}^{(j)}\sim H_{\pmb{u}\pmb{v}}^{(j)}$ , we have that with probability  $\geq (1 - 1 / (10\gamma))^{2j}$  it holds that  $(\pmb {u},\pmb {v})\in \mathrm{Good}^{(j)}$  and  $\pmb{h}^{(j)}\in \mathrm{GoodH}^{(j)}$ . Thus, with probability  $\geq (1 - 1 / (10\gamma))^{2j}$  over the input matrix  $\pmb {G} + \alpha \sum_{i = 1}^{r}\pmb{u}_{i}\pmb{v}_{i}^{\top}$ , we have that
$$
\sum_ {j ^ {\prime} = 1} ^ {j} \| Q ^ {(j ^ {\prime} + 1)} [ \boldsymbol {h} ^ {(j ^ {\prime})} ] (\boldsymbol {u} \otimes \boldsymbol {v}) \| _ {2} ^ {2} \leq \sum_ {j ^ {\prime} = 1} ^ {j} f _ {j ^ {\prime}} (\alpha , \gamma) k \leq 2 f _ {j} (\alpha , \gamma) k.
$$
With probability  $\geq 1 - 1/\mathrm{poly}(n)$ , using Lemma B.1,  $\|Q^{(0)}(\boldsymbol{u} \otimes \boldsymbol{v})\|_2^2 \leq k \log(n)$ . Using a union bound, we obtain that with probability  $\geq (1 - 1/(10\gamma))^{2j} - 1/\mathrm{poly}(n)$ ,
$$
\begin{array}{l} \sum_ {j ^ {\prime} = 0} ^ {j} \| Q ^ {(j ^ {\prime} + 1)} \left[ \boldsymbol {h} ^ {(j ^ {\prime})} \right] (\boldsymbol {u} \otimes \boldsymbol {v}) \| _ {2} ^ {2} \leq 2 f _ {j} (\alpha , \gamma) k r + K k \log (n) \\ = 2 k f _ {0} (\alpha , \gamma) (K \alpha^ {2} \gamma^ {2}) ^ {j} + K k \log (n) \\ = (3 K) k \left(K \alpha^ {2} \gamma^ {2}\right) ^ {j} \\ \end{array}
$$
where  $K$  is a large enough absolute constant which proves the theorem for  $j$  such that  $(K\alpha^{2}\gamma^{2})^{j} = \Omega (\log (n))$ .
Suppose  $\mathsf{Alg}$  is a  $t$  round algorithm that uses  $k$  liner measurements in each round such that when run on the matrix  $\pmb {G} + (\alpha /\sqrt{n})\pmb {u}\pmb{v}^{\top}$ , it produces a matrix  $\hat{A}$  such that with probability 9/10 over the input matrix,
$$
\| \hat {A} - \boldsymbol {u v} ^ {\mathsf {T}} \| _ {\mathsf {F}} ^ {2} \leq c n ^ {2}
$$
for a small enough constant  $c$ . By making the algorithm to query the vector  $\operatorname{vec}(\hat{A})$  in round  $t + 1$ , we can therefore ensure that if  $Q$  is the overall query space of the algorithm, then
$$
\Pr [ \| Q (\boldsymbol {u} \otimes \boldsymbol {v}) \| _ {2} ^ {2} \geq n ^ {2} / 1 0 0 ] \geq 4 / 5.
$$
By Lemma 3.1, we obtain that  $t + 1 = \Omega(\log(n^2/k) / (\log(\alpha) + \log \log n))$  and hence  $t = \Omega(\log(n^2/k) / (\log(\alpha) + \log \log n))$ .
Now suppose  $\mathsf{Alg}_r$  is a  $t$  round algorithm that uses  $k$  linear measurements of the input random matrix  $\pmb {G} + (\alpha /\sqrt{n})\sum_{i = 1}^{r}\pmb {u}_i\pmb {v}_i^\top$  and outputs a matrix  $\hat{A}$  such that with probability  $\geq 9 / 10$  over the input,
$$
\| \hat {A} - \sum_ {i = 1} ^ {r} \boldsymbol {u} _ {i} \boldsymbol {v} _ {i} ^ {\top} \| _ {\mathsf {F}} ^ {2} \leq c n ^ {2} r
$$
for a small enough constant  $c$ . We obtain a lower bound on  $t$  by reducing the rank-1 plant estimation problem to the rank-  $r$  plant estimation problem. Suppose the input is the random matrix  $G + (\alpha / \sqrt{n}) \mathbf{u}_1 \mathbf{v}_1^\top$ . We sample  $\mathbf{u}_2, \ldots, \mathbf{u}_r$  and  $\mathbf{v}_2, \ldots, \mathbf{v}_r$  so that the coordinates of these vectors are independent standard Gaussian random variables. Now we note that we can perform arbitrary linear measurements of the matrix  $G + (\alpha / \sqrt{n}) \sum_{i=1}^{r} \mathbf{u}_i \mathbf{v}_i^\top$  since we have access to linear measurements of  $G + (\alpha / \sqrt{n}) \mathbf{u}_1 \mathbf{v}_1^\top$  and we know the vectors  $\mathbf{u}_2, \ldots, \mathbf{u}_r$  and  $\mathbf{v}_2, \ldots, \mathbf{v}_r$ . We can then use Algorithm  $\mathrm{Alg}_r$  to obtain a matrix  $\hat{A}$  such that with probability  $\geq 9/10$  over the input and our sampled vectors,
$$
\| \sum_ {i = 1} ^ {r} \boldsymbol {u} _ {i} \boldsymbol {v} _ {i} ^ {\mathsf {T}} - \hat {A} \| _ {\mathbb {F}} ^ {2} \leq c n ^ {2} r.
$$
Condition on this event. We have  $\| \hat{A} - [\hat{A}]_r \|_{\mathbb{F}}^2 \leq \| \sum_{i=1}^{r} \boldsymbol{u}_i \boldsymbol{v}_i^\top - \hat{A} \|_{\mathbb{F}}^2 \leq c n^2 r$  which then implies  $\| \sum_{i=1}^{r} \boldsymbol{u}_i \boldsymbol{v}_i^\top - [\hat{A}]_r \|_{\mathbb{F}}^2 \leq 2 c n^2 r$ . Now, let  $P$  be the at most rank  $r$  projection matrix onto the rowspace of the matrix  $[\hat{A}]_r$  and assume that  $U$  and  $V$  are matrices with columns given by  $\boldsymbol{u}_1, \ldots, \boldsymbol{u}_r$  and  $\boldsymbol{v}_1, \ldots, \boldsymbol{v}_r$  respectively. From above, we get
$$
\| \boldsymbol {U} \boldsymbol {V} ^ {\mathsf {T}} (I - P) \| _ {\mathbb {F}} ^ {2} \leq \| \boldsymbol {U} \boldsymbol {V} ^ {\mathsf {T}} - [ \hat {A} ] _ {r} \| _ {\mathbb {F}} ^ {2} \leq 2 c n ^ {2} r
$$
which then implies that
$$
\sigma_ {\min } (\boldsymbol {U}) ^ {2} \left\| \boldsymbol {V} ^ {\top} (I - P) \right\| _ {\mathsf {F}} ^ {2} \leq \left\| \boldsymbol {U} \boldsymbol {V} ^ {\top} (I - P) \right\| _ {\mathsf {F}} ^ {2} \leq 2 c n ^ {2} r.
$$
Now,  $\pmb{U}$  is an  $n\times r$  matrix with coordinates being independent standard Gaussian random variables. We have from [26] that if  $r\leq n / 2$ , then with probability  $1 - \exp (-n)$ ,  $\sigma_{\mathrm{min}}(\pmb {U})\geq c_1\sqrt{n}$ . From [17] we also have that with probability  $\geq 1 - \exp (-n)$ ,  $c_{2}n\leq \| \pmb{u}_{1}\|_{2}^{2},\ldots ,\| \pmb{u}_{2}\|_{2}^{2}\leq Cn$  for a small enough  $c_{2}$  and large enough  $C$ . Conditioned on all these events, we have
$$
\begin{array}{l} \sum_ {i = 1} ^ {r} \left\| \boldsymbol {u} _ {i} \boldsymbol {v} _ {i} ^ {\mathsf {T}} (I - P) \right\| _ {\mathsf {F}} ^ {2} \leq C n \sum_ {i = 1} ^ {r} \left\| \boldsymbol {v} _ {i} ^ {\mathsf {T}} (I - P) \right\| _ {2} ^ {2} \\ \leq C n \| \boldsymbol {V} ^ {\mathsf {T}} (I - P) \| _ {\mathsf {F}} ^ {2} \leq \frac {(2 c n ^ {2} r) (C n)}{c _ {1} ^ {2} n} \leq (2 c C / c _ {1}) n ^ {2}. \\ \end{array}
$$
Hence, at least  $9r / 10$  indices  $i$  have the property that  $\| \pmb{u}_i\pmb{v}_i^\top (I - P)\| _F^2\leq (20cC / c_1)n^2$ . Since the marginal distributions of  $\pmb{u}_1\pmb{v}_1^\top ,\dots ,\pmb{u}_r\pmb{v}_r^\top$  are identical, we obtain that with probability  $\geq 8 / 10$
$$
\left\| \boldsymbol {u} _ {1} \boldsymbol {v} _ {1} ^ {\top} (I - P) \right\| _ {\mathbb {F}} ^ {2} \leq (2 0 c C / c _ {1}) n ^ {2}.
$$
Note that rank of  $P$  is at most  $r$  and therefore  $P = QQ^{\top}$  for an orthonormal matrix  $Q$  with at most  $r$  columns. In the  $(t + 1)$ -th round, we make  $nr$  linear measurements of the input matrix and obtain the matrix
$$
\boldsymbol {G} \boldsymbol {Q} + (\alpha / \sqrt {n}) \boldsymbol {u} _ {1} \boldsymbol {v} _ {1} ^ {\mathsf {T}} \boldsymbol {Q}
$$
and therefore, we can obtain the matrix  $M = \mathbf{G}P + (\alpha / \sqrt{n})\mathbf{u}_1\mathbf{v}_1^\top P$ . Let  $[M]_1$  be the best rank-1 approximation of  $M$  in operator norm. We have
$$
\begin{array}{l} \| [ M ] _ {1} - (\alpha / \sqrt {n}) \boldsymbol {u} _ {1} \boldsymbol {v} _ {1} ^ {\mathsf {T}} \| _ {2} \leq \| [ M ] _ {1} - M \| _ {2} + \| M - (\alpha / \sqrt {n}) \boldsymbol {u} _ {1} \boldsymbol {v} _ {1} ^ {\mathsf {T}} P \| _ {2} + \| (\alpha / \sqrt {n}) \boldsymbol {u} _ {1} \boldsymbol {v} _ {1} ^ {\mathsf {T}} P - (\alpha / \sqrt {n}) \boldsymbol {u} _ {1} \boldsymbol {v} _ {1} ^ {\mathsf {T}} \| _ {2} \\ \leq \| \boldsymbol {G} P \| _ {2} + \| \boldsymbol {G} P \| _ {2} + (\alpha / \sqrt {n}) \sqrt {(2 0 c C / c _ {1}) n ^ {2}}. \\ \end{array}
$$
Since,  $\| G\| _2\leq 2\sqrt{n}$  with probability  $1 - \exp (-\Theta (n))$  , we get
$$
\left\| \left(\sqrt {n} / \alpha\right) [ M ] _ {1} - \boldsymbol {u} _ {1} \boldsymbol {v} _ {1} ^ {\mathbf {T}} \right\| _ {2} \leq \left(4 / \alpha + \sqrt {2 0 c C / c _ {1}}\right) n
$$
$$
\left\| \left(\sqrt {n} / \alpha\right) [ M ] _ {1} - \boldsymbol {u} _ {1} \boldsymbol {v} _ {1} ^ {\mathsf {T}} \right\| _ {\mathsf {F}} ^ {2} \leq 2 \left(4 / \alpha + \sqrt {2 0 c C / c _ {1}}\right) ^ {2} n ^ {2}.
$$
For  $\alpha$  large enough and  $c$  small enough, by querying the vector  $\operatorname{vec}([M]_1)$ , we can again ensure that if  $Q$  is the query space of the algorithm after  $t + 2$  rounds, we have with probability  $\geq 1/2$  over the input random matrix that
$$
\left\| Q \left(\boldsymbol {u} _ {1} \otimes \boldsymbol {v} _ {1}\right) \right\| _ {2} ^ {2} \geq n ^ {2} / 1 0 0.
$$
We again have from Lemma 3.1 that  $t + 2 = \Omega (\log (n^2 /k) / (\log \log n + \log \alpha))$  and therefore that  $t = \bar{\Omega} (\log (n^{2} / k) / (\log \log n + \log \alpha))$ .
D Proof of Theorem 1.1 using Theorem 3.2
Let  $\mathcal{A}$  be any any randomized algorithm that queries orthonormal measurements of the underlying matrix  $A$  and outputs a reconstruction  $\hat{A}$  of  $A$  satisfying  $\| A - \hat{A} \|_{\mathbb{F}}^2 \leq c \| A \|_{\mathbb{F}}^2$  with probability  $\geq p$ , over both the randomness of the algorithm and randomness of the measurements. Let  $\hat{A} = \mathcal{A}(A, \sigma, \gamma)$  where  $\sigma$  captures the randomness of the algorithm and  $\gamma$  captures the randomness in measurements. First we have the following lemma.
Lemma D.1. If there is a randomized algorithm  $\mathcal{A}(A,\sigma ,\gamma)$  such that for all rank  $\leq r$  matrices  $A$  with  $\| A\|_{\mathsf{F}}^{2} = \Theta (nr)$ ,
$$
\operatorname * {P r} _ {\boldsymbol {\sigma}, \boldsymbol {\gamma}} [ \| \mathcal {A} (A, \boldsymbol {\sigma}, \boldsymbol {\gamma}) - A \| _ {\mathsf {F}} ^ {2} \leq c \| A \| _ {\mathsf {F}} ^ {2} ] \geq p,
$$
then there is a randomized algorithm  $\mathcal{A}'$  such that for all  $A$  with  $\| A \|_{\mathsf{F}}^2 = \Theta (nr)$ ,
$$
\operatorname * {P r} _ {\boldsymbol {G}, \boldsymbol {\sigma}} [ \| \mathcal {A} ^ {\prime} (\boldsymbol {A} + \boldsymbol {G}, \boldsymbol {\sigma}) - \boldsymbol {A} \| _ {\mathsf {F}} ^ {2} \leq c \| \boldsymbol {A} \| _ {\mathsf {F}} ^ {2} ] \geq p.
$$
Proof. Fix a particular  $\sigma$ . The algorithm first queries an orthonormal matrix  $Q^{(1)} \in \mathbb{R}^{k \times n^2}$  and receives a random response  $\pmb{r}^{(1)} = Q^{(1)} \cdot \operatorname{vec}(A) + \pmb{g}^{(1)}$  where  $\pmb{g}^{(1)}$  is a vector with independent Gaussian components of mean 0 and variance 1. As a function of  $\pmb{r}^{(1)}$ , the algorithm queries an orthonormal matrix  $Q^{(1)}[\pmb{r}^{(1)}]$  and receives the random response  $\pmb{r}^{(2)} = Q^{(2)}[\pmb{r}^{(1)}] \cdot \operatorname{vec}(A) + \pmb{g}^{(2)}$  where  $\pmb{g}^{(2)}$  is again a random Gaussian vector independent of  $\pmb{g}^{(1)}$ . Importantly, we also have that  $Q^{(2)}[\pmb{r}^{(1)}]$  is orthogonal to the query matrix  $Q^{(1)}$  in the first round. The algorithm proceeds in further rounds accordingly where it picks query matrices as a function of the responses in all the previous rounds.
Let  $\pmb{G}$  be an  $n \times n$  Gaussian matrix with independent entries of mean 0 and variance 1. We now observe that  $\pmb{r}^{(1)} = Q^{(1)} \cdot \mathrm{vec}(A) + \pmb{g}^{(1)}$  has the same distribution as  $Q^{(1)} \cdot \mathrm{vec}(A + \pmb{G})$  by rotational invariance of Gaussian distribution. Now conditioning on  $\pmb{r}^{(1)}$ , we obtain that  $\pmb{r}^{(2)} = Q^{(2)}[\pmb{r}^{(1)}] \cdot \mathrm{vec}(A) + \pmb{g}^{(2)}$  has the same distribution as  $Q^{(2)}[\pmb{r}^{(1)}] \cdot \mathrm{vec}(A + \pmb{G})$  as  $Q^{(2)}[\pmb{r}^{(1)}]$  is orthogonal to  $Q^{(1)}$  and hence  $Q^{(2)}[\pmb{r}^{(1)}] \cdot \pmb{g}$  is independent of  $Q^{(1)}\pmb{g}$ , again by rotational invariance.
Thus, we can consider a deterministic algorithm  $\mathcal{A}'$  parameterized by the same  $\sigma$  which exactly simulates the behavior of  $\mathcal{A}$  performing matrix recovery with noisy measurements. Thus,
$$
\operatorname * {P r} _ {\boldsymbol {G}} [ \| \mathcal {A} ^ {\prime} (\boldsymbol {A} + \boldsymbol {G}, \sigma) - \boldsymbol {A} \| _ {\mathsf {F}} ^ {2} \leq c \| \boldsymbol {A} \| _ {\mathsf {F}} ^ {2} ] = \operatorname * {P r} _ {\boldsymbol {\gamma}} [ \| \mathcal {A} (\boldsymbol {A}, \sigma , \boldsymbol {\gamma}) - \boldsymbol {A} \| _ {\mathsf {F}} ^ {2} \leq c \| \boldsymbol {A} \| _ {\mathsf {F}} ^ {2} ].
$$
Taking expectation over  $\sigma$ , we obtain the result.
Let  $\pmb{u}_1, \dots, \pmb{u}_r$  and  $\pmb{v}_1, \dots, \pmb{v}_r$  be  $2r$  random vectors with coordinates being independent standard normal random variables. Let  $U$  be an  $n \times r$  matrix with columns given by  $\pmb{u}_1, \dots, \pmb{u}_r$  and  $V$  be an  $n \times r$  matrix with columns given by  $\pmb{v}_1, \dots, \pmb{v}_r$ .
Claim D.2. If  $r \leq n/2$ , With high probability,
$$
\| \sum_ {i = 1} ^ {r} \boldsymbol {u} _ {i} \boldsymbol {v} _ {i} ^ {\mathsf {T}} \| _ {\mathsf {F}} ^ {2} = \Theta (n ^ {2} r).
$$
Proof. Clearly,  $\mathbf{U}\mathbf{V}^{\top} = \sum_{i=1}^{r} \mathbf{u}_{i}\mathbf{v}_{i}^{\top}$ . We now have  $2nr \geq \| \mathbf{V} \|_{\mathsf{F}}^{2} \geq nr / 2$  with probability  $1 - \exp(-\Theta(nr))$  from [17]. From [26], with probability  $\geq 1 - \exp(-\Theta(n))$ ,  $\sigma_{\min}(\mathbf{U}) \geq c\sqrt{n}$  for a constant  $c$  and as seen in preliminaries,  $\| \mathbf{U} \|_{2} \leq 2\sqrt{n}$ . Thus, conditioned on both these events,
$$
\left\| \boldsymbol {U} \boldsymbol {V} ^ {\mathrm {T}} \right\| _ {\mathbb {F}} ^ {2} \geq \sigma_ {\min } (\boldsymbol {U}) ^ {2} \left\| \boldsymbol {V} \right\| _ {\mathbb {F}} ^ {2} \geq c ^ {2} n (n r / 2) \geq (c / 2) n ^ {2} r
$$
$$
\left\| \boldsymbol {U} \boldsymbol {V} ^ {\mathsf {T}} \right\| _ {\mathsf {F}} ^ {2} \leq \left\| \boldsymbol {U} \right\| _ {2} ^ {2} \left\| \boldsymbol {V} \right\| _ {\mathsf {F}} ^ {2} \leq 8 n ^ {2} r.
$$
Hence  $\| (\alpha /\sqrt{n})\sum_{i = 1}^{r}\pmb {u}_i\pmb {v}_i^\top \| _F^2 = \Theta (nr)$  with a large probability. Now, we obtain that
$$
\begin{array}{l} \Pr_ {\boldsymbol {u}, \boldsymbol {v}, \boldsymbol {G}, \boldsymbol {\sigma}} [ \| \mathcal {A} ^ {\prime} ((\alpha / \sqrt {n}) \sum_ {i = 1} ^ {r} \boldsymbol {u} _ {i} \boldsymbol {v} _ {i} ^ {\top} + \boldsymbol {G}, \boldsymbol {\sigma}) - (\alpha / \sqrt {n}) \sum_ {i = 1} ^ {r} \boldsymbol {u} _ {i} \boldsymbol {v} _ {i} ^ {\top} \| _ {\mathsf {F}} ^ {2} \leq c \| (\alpha / \sqrt {n}) \sum_ {i = 1} ^ {r} \boldsymbol {u} _ {i} \boldsymbol {v} _ {i} ^ {\top} \| _ {\mathsf {F}} ^ {2} ] \\ \geq (1 - \exp (- \Theta (n))) p. \\ \end{array}
$$
Thus, there is some  $\sigma$  such that
$$
\Pr_ {\boldsymbol {u}, \boldsymbol {v}, \boldsymbol {G}} [ \| \mathcal {A} ^ {\prime} ((\alpha / \sqrt {n}) \sum_ {i = 1} ^ {r} \boldsymbol {u} _ {i} \boldsymbol {v} _ {i} ^ {\top} + \boldsymbol {G}, \sigma) - (\alpha / \sqrt {n}) \sum_ {i = 1} ^ {r} \boldsymbol {u} _ {i} \boldsymbol {v} _ {i} ^ {\top} \| _ {\mathsf {F}} ^ {2} \leq c \| (\alpha / \sqrt {n}) \sum_ {i = 1} ^ {r} \boldsymbol {u} _ {i} \boldsymbol {v} _ {i} ^ {\top} \| _ {\mathsf {F}} ^ {2} ] \geq p (1 - \exp (- \Theta (n))).
$$
Thus with probability  $\geq p(1 - \exp (-\Theta (n)))$
$$
\| (\sqrt {n} / \alpha) \mathcal {A} ^ {\prime} ((\alpha / \sqrt {n}) \sum_ {i = 1} ^ {r} \boldsymbol {u} _ {i} \boldsymbol {v} _ {i} ^ {\top} + G, \sigma) - \sum_ {i = 1} ^ {r} \boldsymbol {u} _ {i} \boldsymbol {v} _ {i} ^ {\top} \| _ {\mathsf {F}} ^ {2} \leq 2 c n ^ {2} r.
$$
By picking  $c$  small enough, we obtain that  $t = \Omega (\log (n^2 /k) / \log \log n)$  for algorithms with success probability  $p\geq 9 / 10$ .
E Applications to Other Problems
E.1 Spectral Low Rank Approximation