SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Consider the  $n \times n$  random matrix  $M = G + (\alpha / \sqrt{n})\boldsymbol{u}\boldsymbol{v}^{\mathsf{T}}$  for a large enough constant  $\alpha$ . The existence of a  $t$ -round algorithm as in the statement implies that there is a deterministic  $t$ -round algorithm that outputs approximations to singular vectors of the matrix  $M$  with probability  $\geq 9/10$ . We then have that  $10\|G\|_2 \leq \left\|(\alpha / \sqrt{n})\boldsymbol{u}\boldsymbol{v}^{\mathsf{T}}\right\|_2$  and  $\|\boldsymbol{u}\|_2, \|\boldsymbol{v}\|_2 \geq \sqrt{n}/2$  with large probability. Condition on this event. Let  $v_1 \in \mathbb{R}^n$  be the top right singular vector of the matrix  $M$ . We have
$$
(9 / 1 0) \| (\alpha / \sqrt {n}) \boldsymbol {u} \boldsymbol {v} ^ {\mathsf {T}} \| _ {2} \leq \| (\alpha / \sqrt {n}) \boldsymbol {u} \boldsymbol {v} ^ {\mathsf {T}} \| _ {2} - \| \boldsymbol {G} \| _ {2} \leq \| \boldsymbol {M} \| _ {2} = \| (\boldsymbol {G} + (\alpha / \sqrt {n}) \boldsymbol {u} \boldsymbol {v} ^ {\mathsf {T}}) v _ {1} \| _ {2}.
$$
$$
\begin{array}{l} \left\| \left(\alpha / \sqrt {n}\right) \boldsymbol {u} \left(\boldsymbol {v} ^ {\mathsf {T}} v _ {1}\right) \right\| _ {2} \geq \left\| \left(\boldsymbol {G} + \left(\alpha / \sqrt {n}\right) \boldsymbol {u} \boldsymbol {v} ^ {\mathsf {T}}\right) \right\| _ {2} - \left\| \boldsymbol {G} v _ {1} \right\| _ {2} \\ \geq (9 / 1 0) \| (\alpha / \sqrt {n}) \boldsymbol {u} \boldsymbol {v} ^ {\mathsf {T}} \| _ {2} - (1 / 1 0) \| (\alpha / \sqrt {n}) \boldsymbol {u} \boldsymbol {v} ^ {\mathsf {T}} \| _ {2} \\ = (8 / 1 0) \cdot (\alpha / \sqrt {n}) \| \boldsymbol {u} \| _ {2} \| \boldsymbol {v} ^ {\mathsf {T}} \| _ {2}. \\ \end{array}
$$
Thus, we obtain that  $|\pmb{v}^{\mathsf{T}}v_{1}| \geq 4/5\|\pmb{v}\|_{2}$ . Similarly, if  $u_{1}$  is the top left singular vector, we have that  $|\pmb{u}^{\mathsf{T}}u_{1}| \geq 4/5\|\pmb{u}\|_{2}$ . Suppose  $v_{1}'$  is a unit vector such that  $\|v_{1}-v_{1}'\|_{2} \leq 1/10$ . Then,  $|\pmb{v}^{\mathsf{T}}v_{1}'| \geq |\pmb{v}^{\mathsf{T}}v_{1}| - (1/10)\|\pmb{v}\|_{2} \geq (7/10)\|\pmb{v}\|_{2}$ . Similarly,  $|\pmb{u}^{\mathsf{T}}u_{1}'| \geq (7/10)$  which implies
$$
\langle u _ {1} ^ {\prime} \otimes v _ {1} ^ {\prime}, \boldsymbol {u} \otimes \boldsymbol {v} \rangle^ {2} = \langle u _ {1} ^ {\prime}, \boldsymbol {u} \rangle^ {2} \langle v _ {1} ^ {\prime}, \boldsymbol {v} \rangle^ {2} \geq (4 9 / 1 0 0) \| \boldsymbol {u} \| _ {2} ^ {2} \| \boldsymbol {v} \| _ {2} ^ {2} \geq n ^ {2} / 1 0.
$$
Thus, with a large probability over the random matrix  $M$ , approximations to the top left and right singular vectors of the matrix  $M$  can be used to construct an  $n^2$ -dimensional unit vector  $q$  such that  $\langle q, \boldsymbol{u} \otimes \boldsymbol{v} \rangle^2 \geq n^2 / 10$ . Thus, we have a  $t + 1$  round deterministic algorithm such that the squared projection of  $\boldsymbol{u} \otimes \boldsymbol{v}$  onto the query space of the algorithm is  $\geq n^2 / 10$  with probability  $\geq 8 / 10$  over the random matrix  $M$ . Now, (4) implies that  $t + 1 \geq c \log(n^2 / k) / \log \log(n)$  for a small enough constant  $c$ .
To extend the above lower bound to approximating  $i$ -th singular vector for all  $i \leq n/2$ , we can use the following instance:
$$
M ^ {\prime} = \left[ \begin{array}{c c} M & 0 \\ 0 & S \end{array} \right]
$$
where  $M$  is the  $(n/2) \times (n/2)$  random matrix  $\mathbf{G} + (\alpha/\sqrt{n/2})\mathbf{u}\mathbf{v}^{\top}$  and  $S$  is a deterministic diagonal matrix chosen such that the top singular vectors of  $M$  correspond to  $i$ -th singular vectors of the matrix  $M'$ . If  $S$  is chosen to be the diagonal matrix with  $i - 1$  values equal to  $10\sqrt{n}\alpha$  and the rest of the entries are set to 0, we have that with high probability that the  $i$ -th left and right singular vectors of  $M'$  correspond to the top left and right singular vectors of  $M$  as  $\| M\|_2 \leq 10\sqrt{n}\alpha$  with high probability. Now the existence of a  $t$ -round randomized algorithm to approximate  $i$ -th left and right singular vectors of an  $n \times n$  matrix for  $i \leq n/2$  with probability  $\geq 9/10$  implies that there is a deterministic  $t$ -round algorithm that approximates the top left and right singular vectors of the  $(n/2) \times (n/2)$  matrix  $M$  with probability  $\geq 8/10$ . From the above argument we have  $t + 1 = \Omega(\log((n/2)^2/k)/\log\log(n/2)) = \Omega(\log(n^2/k)/\log\log(n))$ .
For  $i \geq n/2$ , we can do a similar reduction. We have that the top left singular vector  $u_1$  of the  $(n/2) \times (n/2)$  matrix  $M$  is same as the top singular (eigen-) vector of the matrix  $MM^{\top}$  and with high probability is equal to the last singular (eigen-) vector of the matrix  $(100n\alpha^2)I - MM^{\top}$ . Using the same trick as above, we can create an  $n \times n$  matrix such that the  $i$ -th singular vector for  $i \geq n/2$  corresponds to  $u_1$ . Instead of  $MM^{\top}$ , if we use the matrix  $M^{\top}M$ , then we can make the top right singular vector  $v_1$  of the matrix  $M$  correspond to the  $i$ -th singular vector of an  $n \times n$  matrix. Thus, for all  $i \in [n]$ , we have an  $\Omega(\log(n^2/k)/\log\log(n))$  lower bound for any randomized algorithm that computes the  $i$ -th left and right singular vectors of an arbitrary  $n \times n$  matrix  $A$  using  $k$  linear measurements of  $A$  in each round of the algorithm.
The lower bounds for algorithms that output approximations to singular vectors with probability  $\geq 1 - 1 / \mathrm{poly}(n)$  follow similarly using the high probability lower bounds from (5).
E.8 Lower Bounds for Sparse Matrices
The following theorem gives a lower bound of  $\Omega (\log (m / k))$  rounds on adaptive algorithms that compute 2-approximate spectral norm LRA for matrices with at most  $m$  nonzero entries.
Theorem E.13. Any  $t$  round randomized algorithm that computes, given an arbitrary  $n \times n$  matrix  $A$  with at most  $m$  nonzero entries, a 2-approximate rank  $r$  spectral norm LRA of  $A$  with probability  $\geq 1 - 1/\mathrm{poly}(m)$ , must have  $t = \Omega(\log(m/k))$ , where  $k$  is the number of general linear measurements queried by the algorithm in each of the  $t$  rounds.
Proof. We consider the distribution of the  $\sqrt{m} \times \sqrt{m}$  random matrix  $\mathbf{G} + (\alpha / m^{1/4})\mathbf{u}\mathbf{v}^{\top}$  for a large enough constant  $\alpha$  and embed this instance into an  $n \times n$  matrix. Clearly, all the matrices drawn from this distribution have at most  $m$  nonzero entries. And from Theorem E.3, any deterministic algorithm using  $k$  linear measurements in each round and computing a 2-approximate spectral norm approximation for a matrix drawn from this distribution, with probability  $\geq 1 - 1/poly(m)$ , must use  $\Omega(\log((\sqrt{m})^2/k)) = \Omega(\log(m/k))$  rounds.
Thus by Yao's minimax lemma, any randomized algorithm that outputs a 2-approximate spectral norm approximation with probability  $\geq 1 - 1 / \mathrm{poly}(m)$  for an arbitrary matrix with  $m$  nonzero entries must use  $t = \Omega (\log (m / k))$  rounds.
The above theorem implies that any algorithm with only  $O(1)$  adaptive rounds must query  $\Omega(m)$  linear measurements. This lower bound is tight up to constant factors as with  $2m$  linear measurements, since all of the  $m$  non-zero entries of any arbitrary matrix can be computed in just 1 round using a Vandermonde matrix [8, Theorem 2.14].