SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Ainesh Bakshi and Shyam Narayanan. Krylov methods are (nearly) optimal for low-rank approximation. arXiv preprint arXiv:2304.03191, 2023. 9
[2] Ainesh Bakshi, Kenneth L. Clarkson, and David P. Woodruff. Low-rank approximation with  $1 / \varepsilon^{1 / 3}$  matrix-vector products. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2022, 2022. 25
[3] Mark Braverman, Elad Hazan, Max Simchowitz, and Blake Woodworth. The gradient complexity of linear regression. In Conference on Learning Theory, pages 627-647. PMLR, 2020. 24
[4] Emmanuel J. Candès and Mark A. Davenport. How well can we estimate a sparse vector? Appl. Comput. Harmon. Anal., 34(2):317-323, 2013. ISSN 1063-5203. doi: 10.1016/j.acha.2012.08.010.2
[5] Emmanuel J Candès and Yaniv Plan. Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements. IEEE Transactions on Information Theory, 57(4):2342-2359, 2011. 2, 3, 4, 8
[6] Emmanuel J. Candès, Justin K. Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements. Comm. Pure Appl. Math., 59(8):1207-1223, 2006. ISSN 0010-3640. doi: 10.1002/cpa.20124. 2
[7] Xi Chen, Adityanand Guntuboyina, and Yuchen Zhang. On bayes risk lower bounds. Journal of Machine Learning Research, 17(218):1-58, 2016. 6, 9, 13
[8] Simon Foucart and Holger Rauhut. A mathematical introduction to compressive sensing. Applied and Numerical Harmonic Analysis. Birkhäuser/Springer, New York, 2013. ISBN 978-0-8176-4947-0; 978-0-8176-4948-7. doi: 10.1007/978-0-8176-4948-7. 29
[9] Anna C. Gilbert, Yi Li, Ely Porat, and Martin J. Strauss. Approximate sparse recovery: optimizing time and measurements. SIAM J. Comput., 41(2):436-453, 2012. ISSN 0097-5397. doi: 10.1137/100816705. 2
[10] Gene H Golub, Franklin T Luk, and Michael L Overton. A block lanczos method for computing the singular values and corresponding singular vectors of a matrix. ACM Transactions on Mathematical Software (TOMS), 7(2):149-169, 1981. 26
[27] Arvind K Saibaba. Randomized subspace iteration: Analysis of canonical angles and unitarily invariant norms. SIAM Journal on Matrix Analysis and Applications, 40(1):23-48, 2019. 25
[28] Max Simchowitz, Ahmed El Alaoui, and Benjamin Recht. Tight query complexity lower bounds for pca via finite sample deformed wigner law. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, pages 1249-1259, 2018. 8, 24
[29] Xiaoming Sun, David P. Woodruff, Guang Yang, and Jialin Zhang. Querying a matrix through matrix-vector products. ACM Trans. Algorithms, 17(4):31:1-31:19, 2021. 4
[30] Jared Tanner and Simon Vary. Compressed sensing of low-rank plus sparse matrices. Applied and Computational Harmonic Analysis, 2023. 8
[31] Roman Vershynin. Concentration inequalities for random tensors. Bernoulli, 26(4):3139 - 3162, 2020. 5, 6, 15
[32] Karl Wimmer, Yi Wu, and Peng Zhang. Optimal query complexity for estimating the trace of a matrix. In International Colloquium on Automata, Languages, and Programming, pages 1051-1062. Springer, 2014. 4
[33] David P Woodruff. Sketching as a tool for numerical linear algebra. arXiv preprint arXiv:1411.4357, 2014. 27
[34] Wei Zhang, Taejoon Kim, Guojun Xiong, and Shu-Hung Leung. Leveraging subspace information for low-rank matrix reconstruction. Signal processing, 163:123-131, 2019. 3
[35] Kai Zhong, Prateek Jain, and Inderjit S Dhillon. Efficient matrix sensing using rank-1 gaussian measurements. In International conference on algorithmic learning theory, pages 3–18. Springer, 2015. 2
A.1 Bayes Risk Lower Bounds
Let  $\Theta$  be a set of parameters and  $\mathcal{P} = \{P_{\theta} : \theta \in \Theta\}$  be a set of distributions over  $\mathcal{X}$ . Let  $w$  be the prior distribution on  $\Theta$  known to a learner. Suppose  $\theta \sim w$ , and  $x \sim P_{\theta}$  and that the learner is given  $x$ . Now the learner wants to learn some information about  $\theta$ . Let  $\mathfrak{d} : \mathcal{X} \to \mathcal{A}$  be a mapping from sample  $x$  to action space  $\mathcal{A}$  and  $L : \Theta \times \mathcal{A} \to \{0,1\}$  be a zero-one loss function. We define
$$
R _ {\mathrm {B a y e s}} (L, \Theta , w) = \inf _ {\mathfrak {d}} \int_ {\Theta} \mathrm {E} _ {x \sim P _ {\theta}} [ L (\theta , \mathfrak {d} (x)) ] w (d \theta) = \inf _ {\mathfrak {d}} E _ {\theta \sim w} [ \mathrm {E} _ {x \sim P _ {\theta}} L (\theta , \mathfrak {d} (x)) ],
$$
$$
R _ {0} (L, \Theta , w) = \inf  _ {a \in \mathcal {A}} \int_ {\Theta} L (\theta , a) w (d \theta) = \inf  _ {a \in \mathcal {A}} \mathrm {E} _ {\theta \sim w} [ L (\theta , a) ].
$$
We drop  $\Theta$  from the notation when it is clear.
$f$ -divergence Let  $\mathcal{C}$  denote the set of all convex functions  $f$  with  $f(1) = 0$ . Let  $P$  and  $Q$  be two distributions with densities  $p$  and  $q$  over a common measure  $\mu$ . Then we have the  $f$ -divergence  $D_{f}(P\| Q)$  defined as
$$
D _ {f} (P \| Q) = \int f \left(\frac {p}{q}\right) q d \mu + f ^ {\prime} (\infty) P \{q = 0 \}
$$
using the convention  $0 \cdot \infty = \infty$ . For  $f = x \log(x)$ , the  $f$ -divergence  $D_f(P \| Q) = d_{\mathrm{KL}}(P \| Q)$ , the Kullback-Leibler (KL) divergence. We frequently use  $d_{\mathrm{KL}}(\mathbf{X} \| \mathbf{Y})$  for some random variables  $\mathbf{X}$  and  $\mathbf{Y}$ , which means the KL divergence between the distributions of  $\mathbf{X}$  and  $\mathbf{Y}$ .
Now we can define the  $f$ -informativity of a set of distributions  $\mathcal{P}$  with respect to a distribution  $w$  as follows:
$$
I _ {f} (\mathcal {P}, w) = \inf  _ {Q} \int D _ {f} (P _ {\theta} \| Q) w (d \theta) = \inf  _ {Q} \mathrm {E} _ {\theta \sim w} [ D _ {f} (P _ {\theta} \| Q) ].
$$
We use  $I(\mathcal{P}, w)$  to denote the  $f$ -informativity for  $f = x \log x$ . We finally have the following theorem from [7].
Theorem A.1. For any 0-1 loss function  $L$ ,
$$
I _ {f} (\mathcal {P}, w) \geq \phi_ {f} \left(R _ {\text {B a y e s}}, R _ {0}\right).
$$
where for  $0 \leq a, b \leq 1$ ,  $\phi_f(a, b)$  denotes the  $f$ -divergence between distributions  $P, Q$  over  $\{0, 1\}$  with  $P\{1\} = a$  and  $Q\{1\} = b$ .
As a corollary of the above theorem, [7] prove the following result which is the generalized Fano inequality.
Theorem A.2. For any prior  $w$  and 0-1 loss function  $L$ ,
$$
R _ {\text {B a y e s}} \geq 1 + \frac {I (\mathcal {P} , w) + \log (1 + R _ {0})}{\log (1 - R _ {0})}.
$$
A.2 KL Divergence
We state some properties of the KL-divergence that we use throughout the paper.
Lemma A.3. Let  $P$  and  $Q$  be two random variables with probability measures  $p$  and  $q$  such that  $p$  is absolutely continuous with respect to  $q$ . Let  $\tilde{P}$ , with probability measure  $\tilde{p}$ , be the restriction of  $P$  to an event  $\mathcal{E}$ , i.e., for all  $\mathcal{E}'$ ,  $\operatorname*{Pr}[\tilde{P} \in \mathcal{E}] = \operatorname*{Pr}[P \in \mathcal{E} \cap \mathcal{E}'] / \operatorname*{Pr}(P \in \mathcal{E})$ . Then
$$
d _ {\mathrm {K L}} (\tilde {p} \| q) \leq \frac {1}{\operatorname * {P r} (P \in \mathcal {E})} (d _ {\mathrm {K L}} (p \| q) + 2).
$$
Proof. For  $x \in \mathcal{E}$ , we have  $\tilde{p}(x) = p(x) / \operatorname*{Pr}[P \in \mathcal{E}]$  and for  $x \notin \mathcal{E}$ ,  $\tilde{p}(x) = 0$ . By definition
$$
d _ {\mathrm {K L}} (\tilde {p} \| q) = \int \tilde {p} (x) \log \left(\frac {\tilde {p} (x)}{q (x)}\right) \mathrm {d} x
$$
using the convention  $0 \cdot \log(0/1) = 0$ . Now,
$$
\begin{array}{l} d _ {\mathrm {K L}} (\tilde {p} \| q) = \int_ {\mathcal {E}} \frac {p (x)}{\operatorname * {P r} (P \in \mathcal {E})} \log \left(\frac {p (x)}{\operatorname * {P r} (P \in \mathcal {E}) q (x)}\right) \mathrm {d} x \\ = \frac {1}{\operatorname * {P r} (P \in \mathcal {E})} \int_ {\mathcal {E}} p (x) \log \left(\frac {p (x)}{q (x)}\right) \mathrm {d} x + \log (1 / \operatorname * {P r} (P \in \mathcal {E})). \\ \end{array}
$$
We also have,
$$
d _ {\mathrm {K L}} (p \| q) = \int_ {\mathcal {E}} p (x) \log \left(\frac {p (x)}{q (x)}\right) \mathrm {d} x + \int_ {\bar {\mathcal {E}}} p (x) \log \left(\frac {p (x)}{q (x)}\right) \mathrm {d} x
$$
$$
\int_ {\mathcal {E}} p (x) \log \left(\frac {p (x)}{q (x)}\right) \mathrm {d} x = d _ {\mathrm {K L}} (p \| q) - \int_ {\bar {\mathcal {E}}} p (x) \log \left(\frac {p (x)}{q (x)}\right) \mathrm {d} x.
$$
Thus, it is enough to lower bound  $\int_{\bar{\varepsilon}}p(x)\log (p(x) / q(x))\mathrm{d}x$  to obtain an upper bound on the left hand side. So,
$$
\begin{array}{l} \int_ {\bar {\mathcal {E}}} p (x) \log \left(\frac {p (x)}{q (x)}\right) \mathrm {d} x = - \operatorname * {P r} (P \in \bar {\mathcal {E}}) \int_ {\bar {\mathcal {E}}} \frac {p (x)}{\operatorname * {P r} (P \in \bar {\mathcal {E}})} \log \left(\frac {q (x)}{p (x)}\right) \mathrm {d} x \\ \geq - \Pr (P \in \bar {\mathcal {E}}) \log \left(\frac {\Pr (Q \in \bar {\mathcal {E}})}{\Pr (P \in \bar {\mathcal {E}})}\right) \\ \end{array}
$$
where the last inequality follows from  $-\log (x)$  being convex. Finally,
$$
\begin{array}{l} d _ {\mathrm {K L}} (\tilde {p}, q) = \frac {1}{\operatorname* {P r} (P \in \mathcal {E})} \int_ {x \in \mathcal {E}} p (x) \log \left(\frac {p (x)}{q (x)}\right) \mathrm {d} x + \log (1 / \operatorname * {P r} (P \in \mathcal {E})) \\ \leq \frac {1}{\operatorname* {P r} (P \in \mathcal {E})} d _ {\mathrm {K L}} (p, q) + \frac {\operatorname* {P r} (P \in \bar {\mathcal {E}})}{\operatorname* {P r} (P \in \mathcal {E})} \log (\operatorname * {P r} (Q \in \bar {\mathcal {E}}) / \operatorname * {P r} (P \in \bar {\mathcal {E}})) + \log (1 / \operatorname * {P r} (P \in \mathcal {E})) \\ \leq \frac {1}{\Pr (P \in \mathcal {E})} d _ {\mathrm {K L}} (p, q) + \frac {1}{\Pr (P \in \mathcal {E})} + \log (1 / \Pr (P \in \mathcal {E})) \\ \end{array}
$$
where we used  $p\log (q / p) = p\log (q) + p\log (1 / p)\leq 0 + 1 = 1$  for  $0\leq p,q\leq 1$ .
The following lemma states the chain rule for KL-divergence.
Lemma A.4 (Chain Rule). Let  $(X, Y)$  be jointly distributed random variables and  $Z, W$  be independent random variables. Then
$$
d _ {\mathrm {K L}} \left(\left(\boldsymbol {X}, \boldsymbol {Y}\right) \| (\boldsymbol {Z}, \boldsymbol {W})\right) = d _ {\mathrm {K L}} (\boldsymbol {X} \| \boldsymbol {Z}) + \operatorname {E} _ {\boldsymbol {X}} \left[ d _ {\mathrm {K L}} \left(\left(\boldsymbol {Y} | \boldsymbol {X}\right) \| \boldsymbol {W}\right) \right].
$$
The following lemma states the KL-divergence between two multivariate Gaussians.
Lemma A.5 (KL-divergence between Gaussians, folklore).
$$
d _ {\mathrm {K L}} (N (\mu_ {1}, \Sigma_ {1}) \| N (\mu_ {2}, \Sigma_ {2})) = \frac {1}{2} [ \log \frac {\det (\Sigma_ {2})}{\det (\Sigma_ {1})} + \operatorname {t r} (\Sigma_ {2} ^ {- 1} \Sigma_ {1}) + (\mu_ {2} - \mu_ {1}) ^ {\mathsf {T}} \Sigma_ {2} ^ {- 1} (\mu_ {2} - \mu_ {1}) - n ]
$$
where  $n$  is the dimension of the Gaussian.
In this article, we use the above lemma only for  $\Sigma_1 = \Sigma_2 = I$  in which case the above lemma implies that  $d_{\mathrm{KL}}(N(\mu_1,I)\| N(\mu_2,I)) = (1 / 2)\| \mu_2 - \mu_1\| _2^2$
A.3 Properties of Gaussian random matrices and vectors
We use the following bounds on the norms of Gaussian matrices and vectors throughout the paper: (i) If  $\pmb{g}$  is an  $n$ -dimensional Gaussian with all its entries independently drawn from  $N(0,1)$ , then with probability  $\geq 1 - \exp(-\Theta(n))$ ,  $(n/2) \leq \| \pmb{g} \|_2^2 \leq (3n/2)$ , and (ii) If  $\pmb{G}$  is an  $m \times n$  ( $m \geq n$ ) Gaussian matrix with i.i.d. entries from  $N(0,1)$ , then  $\operatorname*{Pr}[\| \pmb{G} \|_2 \in [\sqrt{m} - \sqrt{n} - t, \sqrt{m} + \sqrt{n} + t] \geq 1 - \exp(-\Theta(t^2))$ . See [26, 17] and references therein for proofs.
Rotational Invariance We frequently use the rotational invariance property of multivariate Gaussian vectors  $\pmb{g} \sim N(0, I_n)$ , which implies that for any  $n \times n$  orthogonal matrix  $Q$  independent of  $\pmb{g}$ , the vector  $Q\pmb{g}$  is also distributed as  $N(0, I_n)$ . Additionally, if  $q_1, \ldots, q_n$  denote rows of the matrix  $Q$  with  $q_1$  chosen arbitrarily independent of  $\pmb{g}$  and the subsequent vectors  $q_i := f_i(q_1, \ldots, q_{i-1}, \langle q_1, \pmb{g} \rangle, \ldots, \langle q_{i-1}, \pmb{g} \rangle)$  for arbitrary functions  $f_i$  satisfying  $q_i \perp q_1, \ldots, q_{i-1}$ , then  $\langle q_i, \pmb{g} \rangle$  is distributed as  $N(0, 1)$  and is independent of  $\langle q_1, \pmb{g} \rangle, \ldots, \langle q_{i-1}, \pmb{g} \rangle$ . We crucially use this property in the proof of Theorem 3.2.
First we discuss some notation, distributions and random variables that we use throughout the proof. Fix an arbitrary deterministic algorithm  $\mathsf{Alg}$ . Let  $A = G + suv^{\mathsf{T}}$  be a fixed realization of the random matrix  $A$ . Recall  $s = \alpha / \sqrt{n}$ . The algorithm  $\mathsf{Alg}$  queries a fixed orthonormal matrix  $Q^{(1)}$  and receives the response  $r^{(1)} := Q^{(1)} \operatorname{vec}(A) = Q^{(1)} g + sQ^{(1)} (u \otimes v)$  where we use  $g := \operatorname{vec}(G)$ . As  $\mathsf{Alg}$  is deterministic, in the second round it queries the matrix  $Q^{(2)}[r^{(1)}]$ . We use  $Q^{(2)}[r^{(1)}]$  to emphasize that  $\mathsf{Alg}$  picks  $Q^{(2)}$  purely as a function of  $r^{(1)}$ . It receives the response  $r^{(2)} = Q^{(2)}[r^{(1)}] g + sQ^{(2)}[r^{(1)}](u \otimes v)$  and picks queries  $Q^{(3)}[r^{(1)}, r^{(2)}]$  for the third round and so on. Using  $h^{(j)} := (r^{(1)}, \ldots, r^{(j)})$  to denote the history of the algorithm until the end of round  $j$ , we have for  $j = 0, \ldots, t - 1$
$$
r ^ {(j + 1)} := Q ^ {(j + 1)} [ h ^ {(j)} ] (g + s \cdot u \otimes v).
$$
For each  $j = 1, \dots, t$ , the randomness of  $\mathbf{A}$  induces a distribution  $H^{(j)}$  over histories until round  $j$ . Then for  $u$  and  $v$ , the distribution  $H_{uv}^{(j)} \equiv H^{(j)} | \{ \mathbf{u} = u, \mathbf{v} = v \}$  is the distribution of the history of  $\mathsf{Alg}$  after  $j$  rounds conditioned on  $\mathbf{u} = u$  and  $\mathbf{v} = v$  where the randomness in the histories is purely from the randomness of  $\mathbf{G}$ . Recall that without loss of generality, we assume all the general linear queries made by the algorithm are orthogonal to each other. We first prove a tail bound on the amount of information that a non-adaptive query matrix can obtain.
B.1 A Tail Bound
Let  $\mathbf{u}$  and  $\mathbf{v}$  be independent Gaussian random vectors. Let  $Q\in \mathbb{R}^{k\times n^2}$  be an arbitrary matrix with  $k$  orthonormal rows. We have the following lemma.
Lemma B.1. There is a small enough universal constant  $\beta$  such that for all  $k\geq n$  and  $C$  satisfying  $4k\leq Ck\leq 16n^{2}$ , we have for any orthonormal matrix  $Q^{k\times n^2}$  that
$$
\operatorname * {P r} _ {\boldsymbol {u}, \boldsymbol {v}} [ \| Q \sum_ {i} \boldsymbol {u} _ {i} \otimes \boldsymbol {v} _ {i} \| _ {2} ^ {2} \geq C k ] \leq \exp (- \beta C k / n).
$$
Proof. By Theorem 1.4 and Remark 1.5 of [31], we have that
$$
\Pr_ {\boldsymbol {u}, \boldsymbol {v}} [ \| Q (\boldsymbol {u} \otimes \boldsymbol {v}) \| _ {2} \geq \| Q \| _ {\mathsf {F}} + t ] \leq \exp \left(- \frac {c t ^ {2}}{2 n}\right)
$$
for all  $0 \leq t \leq 2n$ . For  $t = \sqrt{Ck / 4}$ , for  $4k \leq Ck \leq 16n^2$ , we have that
$$
\begin{array}{l} \Pr_ {\boldsymbol {u}, \boldsymbol {v}} [ \| Q (\boldsymbol {u} \otimes \boldsymbol {v}) \| _ {2} \geq \sqrt {C k} ] \leq \Pr_ {\boldsymbol {u}, \boldsymbol {v}} [ \| Q (\boldsymbol {u} \otimes \boldsymbol {v}) \| _ {2} \geq \sqrt {k} + \sqrt {C k / 4} ] \\ \leq \exp \left(- \frac {c C k}{8 n}\right) \\ \end{array}
$$
which implies, taking  $\beta = c / 8$ , that  $\operatorname{Pr}_{\tilde{\pmb{u}},\tilde{\pmb{v}}}[\| Q(\tilde{\pmb{u}}\otimes \pmb {v})\| _2^2\geq Ck]\leq \exp (-\beta Ck / n)$ .
B.2 First Round
Let  $w^{(1)}$  be the distribution of  $(u, v)$  where  $u$  and  $v$  are random variables that are independently sampled from  $N(0, I_n)$ . As already defined, the distribution of the history of the algorithm after the first round for a fixed  $u, v$  is given by  $H_{uv}^{(1)}$ . We have that  $H_{uv}^{(1)}$  is the distribution of the random variable
$$
Q ^ {(1)} \boldsymbol {g} + s Q ^ {(1)} u \otimes v.
$$
Let  $P_{uv}^{(1)}$  be the distribution of the above random variable (although  $P_{uv}^{(1)} \equiv H_{uv}^{(1)}$ , we distinguish  $P_{uv}^{(j)}$  and  $H_{uv}^{(j)}$  in later rounds) and let  $\mathcal{P}^{(1)} = \{P_{uv}^{(1)}\}$  be the set of distributions for all  $u, v$ . Then we have
$$
\begin{array}{l} I \left(\mathcal {P} ^ {(1)}, w ^ {(1)}\right) \leq \operatorname {E} _ {\left(\boldsymbol {u}, \boldsymbol {v}\right) \sim w ^ {(1)}} \left[ d _ {\mathrm {K L}} \left(Q ^ {(1)} \boldsymbol {g} + s Q ^ {(1)} (\boldsymbol {u} \otimes \boldsymbol {v}) \| \boldsymbol {g} _ {k}\right) \right] \\ \leq s ^ {2} \operatorname {E} _ {(\boldsymbol {u}, \boldsymbol {v}) \sim w ^ {(1)}} \| Q ^ {(1)} (\boldsymbol {u} \otimes \boldsymbol {v}) \| _ {2} ^ {2} = \left(\alpha^ {2} / n\right) k r. \\ \end{array}
$$
We define the loss function
$$
\operatorname {L o s s} ^ {(1)} ((u, v), Q) = \left\{ \begin{array}{l l} 0 & \text {i f} \| Q (u \otimes v) \| _ {2} ^ {2} \geq f _ {1} (\alpha , \gamma) k \\ 1 & \text {i f} \| Q (u \otimes v) \| _ {2} ^ {2} <   f _ {1} (\alpha , \gamma) k \end{array} \right.
$$
for  $Q \in \mathbb{R}^{k \times n^2}$  with orthonormal rows and some function  $f_1(\alpha, \gamma)$  satisfying  $4 \leq f_1(\alpha, \gamma) \leq 16n^2 / k$  for a parameter  $\gamma \geq 1$ . From Lemma B.1, we have
$$
R _ {0} \left(\operatorname {L o s s} ^ {(1)}, w ^ {(1)}\right) \geq 1 - \exp \left(- \beta f _ {1} (\alpha , \gamma) k / n\right)
$$
$$
R _ {\text {B a y e s}} \geq 1 + \frac {(\alpha^ {2} / n) k + \log (2)}{- \beta f _ {1} (\alpha , \gamma) k / n + \log (2)}.
$$
Picking  $f_{1}(\alpha, \gamma) = K\alpha^{2}\gamma^{2}$ , we have that  $R_{\mathrm{Bayes}} \geq 1 - 1 / (100\gamma^{2})$ . Thus with probability  $1 - 1 / (10\gamma)$  over  $(\pmb{u}, \pmb{v}) \sim w^{(1)}$ , we have that
$$
\operatorname * {P r} _ {\boldsymbol {h} ^ {(1)} \sim P _ {\boldsymbol {u v}} ^ {(1)}} [ \| Q ^ {(2)} [ \boldsymbol {h} ^ {(1)} ] (\boldsymbol {u} \otimes \boldsymbol {v}) \| _ {2} ^ {2} \leq f _ {1} (\alpha , \gamma) k ] \geq 1 - 1 / (1 0 \gamma).
$$
Let  $\mathrm{Good}^{(1)}$  be the set of all  $(u,v)$  that satisfy the above property. Let  $w^{(2)}$  be the distribution of  $(\pmb {u},\pmb {v})\sim w^{(1)}$  conditioned on  $(\pmb {u},\pmb {v})\in \mathrm{Good}^{(1)}$ . For each  $(u,v)\in \mathrm{Good}^{(1)}$ , let
$$
\operatorname {G o o d H} _ {u v} ^ {(1)} = \left\{h ^ {(1)}: \| Q ^ {(2)} [ h ^ {(1)} ] (u \otimes v) \| _ {2} ^ {2} \leq f _ {1} (\alpha , \gamma) k \right\}.
$$
We have for all  $(u,v)\in \mathrm{Good}^{(1)}$  that  $\operatorname*{Pr}_{\pmb{h}^{(1)}\sim P_{uv}^{(1)}}[\pmb{h}^{(1)}\in \mathrm{GoodH}_{uv}^{(1)}]\geq 1 - 1 / (10\gamma)$ . The overall conclusion of this is that with a large probability over  $(\pmb {u},\pmb {v})\sim w^{(1)}$ , the squared projection of  $\pmb {u}\otimes \pmb{v}$  on the query asked by Alg in round 2 is small with large probability over  $G$ .
B.3 Further Rounds
We first define the following objects inductively.
1. For  $j \geq 2$ , let  $w^{(j)}$  be the distribution of  $(\pmb{u},\pmb{v}) \sim w^{(j-1)}$  conditioned on  $(\pmb{u},\pmb{v}) \in \mathrm{Good}^{(j-1)}$ . So  $w^{(j)}$  is the distribution over only those inputs for which the squared projection of  $u \otimes v$  on the query space of Alg until round  $j$  is small with high probability over  $G$ .
2. For  $(u,v)\in \mathrm{Good}^{(j - 1)}$
$$
P _ {u v} ^ {(j)} := \text {d i s t r i b u t i o n} \boldsymbol {h} ^ {(j)} = \left(\boldsymbol {h} ^ {(j - 1)}, \boldsymbol {r} ^ {(j)}\right) \sim \boldsymbol {H} _ {u v} ^ {(j)} \text {c o n d i t i o n e d} \boldsymbol {h} ^ {(j - 1)} \in \operatorname {G o o d H} _ {u v} ^ {(j - 1)}.
$$
$P_{uv}^{(j)}$  denotes the distribution over histories after round  $j$ , conditioned on the queries used until round  $j$  not having a lot of "information" about  $u \otimes v$ .
3. Let  $\mathcal{P}^{(j)}\coloneqq \{P_{uv}^{(j)}:(u,v)\in \mathrm{Good}^{(j - 1)}\}$
4. For  $j \geq 1$ ,
Good  $(^{(j)}\coloneqq$
$$
\{(u,v)\in \operatorname{Good}^{(j - 1)}:\Pr_{\boldsymbol {h}^{(j)}\sim P_{uv}^{(j)}}[||Q^{(j + 1)}[\boldsymbol{h}^{(j)}](u\otimes v)\|_{2}^{2}\leq f_{j}(\alpha ,\gamma)k]\geq 1 - 1 / (10\gamma)\} .
$$
The set  $\mathrm{Good}^{(j)}$  denotes those values of  $(u, v)$  for which with large probability over  $G$ , the queries used by the algorithm until round  $j + 1$  do not have a lot of "information" about  $u \otimes v$ .
5. For  $(u,v)\in \mathrm{Good}^{(j)}$
GoodH  $\begin{array}{r}\mathbf{\Pi}_{uv}^{(j)}\coloneqq \\ \mathbf{\Pi}_{uv}^{(j)}\coloneqq \end{array}$
$$
\left\{h ^ {(j)} = \left(h ^ {(j - 1)}, r ^ {(j)}\right): h ^ {(j - 1)} \in \operatorname {G o o d H} _ {u v} ^ {(j - 1)} \text {a n d} \| Q ^ {(j + 1)} [ h ^ {(j)} ] (u \otimes v) \| _ {2} ^ {2} \leq f _ {j} (\alpha , \gamma) k \right\}.
$$
GoodH(j) denotes those histories, for which the queries used by the algorithm until round  $j + 1$  do not have a lot of "information" about  $u \otimes v$ .
6. Let  $f_0(\alpha, \gamma) = K$  and for  $j \geq 1$ , let  $f_j(\alpha, \gamma) = K\alpha^2\gamma^2 f_{j-1}(\alpha, \gamma)$  for a large enough universal constant  $K$ .
Lemma B.2. For all  $1 \leq j$  satisfying  $f_{j}(\alpha, \gamma)k \leq 16n^{2}$ ,
$$
\Pr_ {(\boldsymbol {u}, \boldsymbol {v}) \sim w ^ {(j)}} [ (\boldsymbol {u}, \boldsymbol {v}) \in \operatorname {G o o d} ^ {(j)} ] \geq 1 - \frac {1}{1 0 \gamma}
$$
2. For all  $(u,v)\in \mathrm{Good}^{(j)}$
$$
\Pr_ {\boldsymbol {h} ^ {(j)} \sim P _ {u v} ^ {(j)}} [ \boldsymbol {h} ^ {(j)} \in \operatorname {G o o d H} _ {u v} ^ {(j)} ] \geq 1 - \frac {1}{1 0 \gamma}
$$
$$
\mathrm {E} _ {(\boldsymbol {u}, \boldsymbol {v}) \sim w ^ {(j)}} \left[ d _ {\mathrm {K L}} \left(P _ {\boldsymbol {u v}} ^ {(j)} \| G ^ {(j)}\right) \right] \leq f _ {j} (\alpha , \gamma) (k / n),
$$
where  $G^{(j)}$  is the joint distribution of  $j$  independent  $k$ -dimensional Gaussian random variables.
Proof. From the previous section, all the above statements hold for  $j = 1$ . Now assume that all the above statements hold for rounds  $1, \ldots, j - 1$ . We prove the statements inductively for round  $j$ . Recall  $w^{(j)}$  is defined to be the distribution of  $(\pmb{u}, \pmb{v}) \sim w^{(j - 1)}$  conditioned on  $(\pmb{u}, \pmb{v}) \in \mathrm{Good}^{(j - 1)}$ . For each  $(u, v) \in \mathrm{Good}^{(j - 1)}$ , we now bound  $d(P_{uv}^{(j)} \| G^{(j)})$ . By definition,
$$
d \left(P _ {u v} ^ {(j)} \| G ^ {(j)}\right) = d _ {\mathrm {K L}} \left(\left(\boldsymbol {h} ^ {(j - 1)}, \boldsymbol {r} ^ {(j)}\right) \| \left(\boldsymbol {g} _ {k} ^ {(1)}, \dots , \boldsymbol {g} _ {k} ^ {(j)}\right)\right)
$$
where  $\pmb{h}^{(j)} = (\pmb{h}^{(j - 1)},\pmb{r}^{(j)})\sim P_{uv}^{(j)}$ . Note that the marginal distribution of  $\pmb{h}^{(j - 1)}$  is given by conditioning the distribution  $P_{uv}^{(j - 1)}$  on the event  $\mathrm{GoodH}_{uv}^{(j - 1)}$ . By Lemma A.4, we have
$$
\begin{array}{l} d _ {\mathrm {K L}} \left(\left(\boldsymbol {h} ^ {(j - 1)}, \boldsymbol {r} ^ {(j)}\right) \| \left(\boldsymbol {g} _ {k} ^ {(1)}, \dots , \boldsymbol {g} _ {k} ^ {(j)}\right)\right) \\ = d _ {\mathrm {K L}} \left(\boldsymbol {h} ^ {(j - 1)} \| \left(\boldsymbol {g} _ {k} ^ {(1)}, \dots , \boldsymbol {g} _ {k} ^ {(j - 1)}\right)\right) + \operatorname {E} _ {\boldsymbol {h} ^ {(j - 1)}} \left[ d _ {\mathrm {K L}} \left(\left(\boldsymbol {r} ^ {(j)}\right) \mid \boldsymbol {h} ^ {(j - 1)}\right) \| \boldsymbol {g} _ {k} ^ {(j)} \right] \\ = d _ {\mathrm {K L}} \left(\left(P _ {u v} ^ {(j - 1)} \mid \operatorname {G o o d H} _ {u v} ^ {(j - 1)}\right) \| G ^ {(j - 1)}\right) + \mathrm {E} _ {\boldsymbol {h} ^ {(j - 1)} \sim P _ {u v} ^ {(j - 1)} \mid \operatorname {G o o d H} _ {u v} ^ {(j - 1)}} \left[ d _ {\mathrm {K L}} \left(\left(\boldsymbol {r} ^ {(j)}\right) \mid \boldsymbol {h} ^ {(j - 1)}\right) \| \boldsymbol {g} _ {n r} ^ {(j)} \right]. \tag {6} \\ \end{array}
$$
Now, using Lemma A.3, we have