SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \max  \left(\left\| \mathcal {P} _ {U _ {\mathcal {I}} ^ {(\Pi_ {n})}} - \mathcal {P} _ {\hat {U} _ {\mathcal {I}} ^ {(\Pi_ {n})}} \right\| _ {\mathrm {o p}}, \left\| \mathcal {P} _ {V _ {\mathcal {I}} ^ {(\Pi_ {n})}} - \mathcal {P} _ {\hat {V} _ {\mathcal {I}} ^ {(\Pi_ {n})}} \right\| _ {\mathrm {o p}}\right) \leq \frac {2 \| \mathbb {E} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] - \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] \| _ {\mathrm {o p}}}{s _ {r _ {n}} - s _ {r _ {n} + 1}} \\ \leq \frac {2 \sqrt {| \mathcal {I} | | \Pi_ {n} |} \| \mathbb {E} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] - \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] \| _ {\max }}{s _ {r _ {n}} - s _ {r _ {n} + 1}} \\ = \frac {2 \sqrt {| \mathcal {I} | | \Pi_ {n} |} \Delta_ {E}}{s _ {r _ {n}} - s _ {r _ {n} + 1}} \\ = \frac {2 \sqrt {\left| \mathcal {I} \right| \left| \Pi_ {n} \right|} \Delta_ {E}}{s _ {r _ {n}}} \tag {25} \\ \end{array}
$$
where the last equality follows from the fact that  $\mathrm{rank}(\mathbb{E}[\mathbf{Y}_{\mathcal{T}^n}^{\Pi_n}]) = r_n$ , hence  $s_{r_n + 1} = 0$ . Now, plugging Assumption 6.4, (25) into (24) gives us
$$
\max  \left(\left\| \mathcal {P} _ {V _ {\mathcal {I}} ^ {(\Pi_ {n})}} - \mathcal {P} _ {\hat {V} _ {\mathcal {I}} ^ {(\Pi_ {n})}} \right\| _ {\mathrm {o p}}, \left\| \mathcal {P} _ {V _ {\mathcal {I}} ^ {(\Pi_ {n})}} - \mathcal {P} _ {\hat {V} _ {\mathcal {I}} ^ {(\Pi_ {n})}} \right\| _ {\mathrm {o p}}\right) \leq C \sqrt {r _ {n}} \Delta_ {E} \tag {26}
$$
Substituting the result of Proposition C.5 and (26) into (24) gives us
$$
\left\| \left(\mathcal {P} _ {V _ {\mathcal {I}} ^ {(\Pi_ {n})}} - \mathcal {P} _ {\hat {V} _ {\mathcal {I}} ^ {(\Pi_ {n})}}\right) \Delta_ {w} ^ {n} \right\| _ {2} = O _ {p} \left(\log^ {3} \left(| \Pi_ {n} | | \mathcal {I} |\right) r _ {n} ^ {3 / 2} \left[ \frac {\| \tilde {\mathbf {w}} ^ {n} \| _ {2} \Delta_ {E}}{\min  \left\{\sqrt {| \Pi_ {n} |} , \sqrt {| \mathcal {I} |} \right\}} + \Delta_ {E} ^ {2} \right]\right) \tag {27}
$$
To further simplify (27), we substitute the result of Lemma C.3
Substituting the Lemma C.3 into (27) gives us
$$
\left\| \left(\mathcal {P} _ {V _ {\mathcal {I}} ^ {(\Pi_ {n})}} - \mathcal {P} _ {\hat {V} _ {\mathcal {I}} ^ {(\Pi_ {n})}}\right) \Delta_ {w} ^ {n} \right\| _ {2} = \tilde {O} _ {p} \left(r _ {n} ^ {2} \left[ \frac {\Delta_ {E}}{\sqrt {| \mathcal {I} |} \min  \left\{\sqrt {| \Pi_ {n} |} , \sqrt {| \mathcal {I} |} \right\}} + \Delta_ {E} ^ {2} \right]\right) \tag {28}
$$
Bounding Term 2. We introduce some necessary notation required to bound the second term. Let  $\hat{\mathbb{E}}[\mathbf{Y}_{\mathcal{I}}^{(\Pi_n),r_n}] = \sum_{l=1}^{r_n}\hat{s}_l\hat{\mu}_l\hat{v}_l'$  denote the  $r_n$  decomposition of  $\hat{\mathbb{E}}[\mathbf{Y}_{\mathcal{I}}^{(\Pi_n)}]$ . Let,  $\hat{\mathbb{E}}[\mathbf{Y}_{\mathcal{I}}^{(\Pi_n),r_n}] = \hat{\mathbf{U}}_{\mathcal{I}}^{(\Pi_n)}\hat{\mathbf{S}}_{\mathcal{I}}^{(\Pi_n)}(\hat{\mathbf{V}}_{\mathcal{I}}^{(\Pi_n)})^T$ . Further, define  $\epsilon^{\Pi_n} = [\epsilon_n^\pi : \pi \in \Pi_n] \in \mathbb{R}^{|\Pi_n|}$ .
Then to begin, note that since  $\hat{\mathbf{V}}_{\mathcal{I}}^{(\Pi_n)}$  is a isometry, we have that
$$
\| \mathcal {P} _ {\hat {V} _ {\mathcal {I}} ^ {(\Pi_ {n})}} \Delta_ {w} ^ {n} \| _ {2} ^ {2} = \| (\hat {\mathbf {V}} _ {\mathcal {I}} ^ {(\Pi_ {n})}) ^ {T} \Delta_ {w} ^ {n} \| _ {2} ^ {2}
$$
To upper bound  $\| (\hat{\mathbf{V}}_{\mathcal{I}}^{\Pi_n})^T\Delta_w^n\| _2^2$  as follows, consider
$$
\| \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] \Delta_ {w} ^ {n} \| _ {2} ^ {2} = \left((\hat {\mathbf {V}} _ {\mathcal {I}} ^ {(\Pi_ {n})}) ^ {T} \Delta_ {w} ^ {n}\right) ^ {T} \left(\hat {\mathbf {S}} _ {\mathcal {I}} ^ {(\Pi_ {n})}\right) ^ {2} \left((\hat {\mathbf {V}} _ {\mathcal {I}} ^ {(\Pi_ {n})}) ^ {T} \Delta_ {w} ^ {n}\right) \geq \hat {s} _ {r _ {n}} ^ {2} \| (\hat {\mathbf {V}} _ {\mathcal {I}} ^ {(\Pi_ {n})}) ^ {T} \Delta_ {w} ^ {n} \| _ {2} ^ {2}
$$
Using the two equations above gives us
$$
\left\| \mathcal {P} _ {\hat {V} _ {\mathcal {I}} ^ {(\Pi_ {n})}} \Delta_ {w} ^ {n} \right\| _ {2} ^ {2} \leq \frac {\left\| \hat {\mathbb {E}} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}) , r _ {n}} \right] \Delta_ {w} ^ {n} \right\| _ {2} ^ {2}}{\hat {s} _ {r _ {n}} ^ {2}} \tag {29}
$$
To bound the numerator in (29), note that by definition  $\mathbb{E}[\mathbf{Y}^{(\Pi_n)}] = \mathbb{E}[\mathbf{Y}_{\mathcal{I}}^{(\Pi_n)}]\tilde{\mathbf{w}}^n$ . Using this observation, we have
$$
\begin{array}{l} \| \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] \Delta_ {w} ^ {n} \| _ {2} ^ {2} \leq 2 \| \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] \tilde {\mathbf {w}} ^ {n} - \mathbb {E} [ \mathbf {Y} ^ {(\Pi_ {n})} ] \| _ {2} ^ {2} + 2 \| \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] \tilde {\mathbf {w}} ^ {n} - \mathbb {E} [ \mathbf {Y} ^ {(\Pi_ {n})} ] \| _ {2} ^ {2} \\ = 2 \| \hat {\mathbb {E}} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {\left(\Pi_ {n}\right), r _ {n}} \right] \hat {\mathbf {w}} ^ {n} - \mathbb {E} \left[ \mathbf {Y} ^ {\left(\Pi_ {n}\right)} \right] \| _ {2} ^ {2} + 2 \| (\hat {\mathbb {E}} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {\left(\Pi_ {n}\right), r _ {n}} \right] - \mathbb {E} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {\left(\Pi_ {n}\right)} \right]) \tilde {\mathbf {w}} ^ {n} \| _ {2} ^ {2} \tag {30} \\ \end{array}
$$
To proceed, we then use the following inequality: for any  $\mathbf{A} \in \mathbb{R}^{a \times b}, v \in \mathbb{R}^b$ , we have
$$
\left\| \mathbf {A} v \right\| _ {2} = \left\| \sum_ {j = 1} ^ {b} \mathbf {A}. _ {j} v _ {j} \right\| _ {2} \leq \left(\max  _ {j \leq b} \| \mathbf {A}. _ {j} \| _ {2}\right) \left(\sum_ {j = 1} ^ {b} v _ {j}\right) = \| \mathbf {A} \| _ {2, \infty} \| v \| _ {1} \tag {31}
$$
Substituting (30) into (29) and then applying inequality (31) gives us
$$
\left. \right.\left\| \mathcal {P} _ {\hat {V} _ {\mathcal {I}} ^ {(\Pi_ {n})}} \Delta_ {w} ^ {n} \right\| _ {2} ^ {2} \leq \frac {2}{\hat {s} _ {r _ {n}} ^ {2}} \left(\left\| \hat {\mathbb {E}} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} \right] \hat {\mathbf {w}} ^ {n} - \mathbb {E} \left[ \mathbf {Y} ^ {(\Pi_ {n})} \right]\right\| _ {2} ^ {2} + \left\| \hat {\mathbb {E}} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} \right] - \mathbb {E} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} \right]\right\| _ {2, \infty} ^ {2} \| \tilde {\mathbf {w}} ^ {n} \| _ {1} ^ {2}\right) \tag {32}
$$
Next, we bound  $\|\hat{\mathbb{E}}[\mathbf{Y}_{\mathcal{I}}^{(\Pi_n), r_n}]\hat{\mathbf{w}}^n - \mathbb{E}[\mathbf{Y}^{(\Pi_n)}]\|_2^2$ . To this end, note that Assumption 3.1 implies that
$$
\begin{array}{l} \| \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] \hat {\mathbf {w}} ^ {n} - \mathbf {Y} ^ {(\Pi_ {n})} \| _ {2} ^ {2} \\ = \left\| \hat {\mathbb {E}} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {\left(\Pi_ {n}\right), r _ {n}} \right] \hat {\mathbf {w}} ^ {n} - \mathbb {E} \left[ \mathbf {Y} ^ {\left(\Pi_ {n}\right)} \right] - \epsilon^ {\Pi_ {n}} \right\| _ {2} ^ {2} \\ = \| \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] \hat {\mathbf {w}} ^ {n} - \mathbb {E} [ \mathbf {Y} ^ {(\Pi_ {n})} ] \| _ {2} ^ {2} + \| \boldsymbol {\epsilon} ^ {\Pi_ {n}} \| _ {2} ^ {2} - 2 \left\langle \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] \hat {\mathbf {w}} ^ {n} - \mathbb {E} [ \mathbf {Y} ^ {(\Pi_ {n})} ], \boldsymbol {\epsilon} ^ {\Pi_ {n}} \right\rangle \tag {33} \\ \end{array}
$$
Next, we proceed by calling upon Property 4.1 of [4] which states that  $\hat{\mathbf{w}}^n$  as given by (4) is the unique solution to the following convex program:
$$
\min_{\substack{\mathbf{w}\in \mathbb{R}^{|\mathcal{I}|}\\ }}\quad \| \mathbf{w}\|_{2}
$$
$$
\operatorname {s. t.} \mathbf {w} \in \underset {\mathbf {w} \in \mathbb {R} ^ {| \mathcal {I} |}} {\operatorname {a r g m i n}} \| \mathbf {Y} ^ {(\Pi_ {n})} - \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] \mathbf {w} \| _ {2} ^ {2} \tag {34}
$$
Using this property, we have that
$$
\begin{array}{l} \| \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] \hat {\mathbf {w}} ^ {n} - \mathbf {Y} ^ {(\Pi_ {n})} \| _ {2} ^ {2} \\ \leq \| \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] \tilde {\mathbf {w}} ^ {n} - \mathbf {Y} ^ {(\Pi_ {n})} \| _ {2} ^ {2} \\ = \| \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] \tilde {\mathbf {w}} ^ {n} - \mathbb {E} [ \mathbf {Y} ^ {(\Pi_ {n})} ] - \epsilon^ {\Pi_ {n}} \| _ {2} ^ {2} \\ = \| \hat {\mathbb {E}} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {\left(\Pi_ {n}\right), r _ {n}} \right] \tilde {\mathbf {w}} ^ {n} - \mathbb {E} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {\left(\Pi_ {n}\right)} \right] \tilde {\mathbf {w}} ^ {n} - \epsilon^ {\Pi_ {n}} \| _ {2} ^ {2} \\ = \left\| \left(\hat {\mathbb {E}} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {\left(\Pi_ {n}\right), r _ {n}} \right] - \mathbb {E} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {\left(\Pi_ {n}\right)} \right]\right) \tilde {\mathbf {w}} ^ {n} \right\| _ {2} ^ {2} + \left\| \epsilon^ {\Pi_ {n}} \right\| _ {2} ^ {2} - 2 \langle \left(\hat {\mathbb {E}} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {\left(\Pi_ {n}\right), r _ {n}} \right] - \mathbb {E} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {\left(\Pi_ {n}\right)} \right]\right) \tilde {\mathbf {w}} ^ {n}, \epsilon^ {\Pi_ {n}} \rangle \tag {35} \\ \end{array}
$$
Substituting (33) and (35) into (32) and using (31), we get
$$
\begin{array}{l} \| \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})}, r _ {n} ] \hat {\mathbf {w}} ^ {n} - \mathbb {E} [ \mathbf {Y} ^ {(\Pi_ {n})} ] \| _ {2} ^ {2} \\ \leq \| (\hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] - \mathbb {E} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ]) \tilde {\mathbf {w}} ^ {n} \| _ {2} ^ {2} + 2 \langle (\hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ]) \Delta_ {w} ^ {n}, \boldsymbol {\epsilon} ^ {\Pi_ {n}} \rangle \\ \leq \| (\hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] - \mathbb {E} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ]) \| _ {2, \infty} ^ {2} \| \tilde {\mathbf {w}} ^ {n} \| _ {1} ^ {2} + 2 \langle (\hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ]) \Delta_ {w} ^ {n}, \boldsymbol {\epsilon} ^ {\Pi_ {n}} \rangle \\ \end{array}
$$
Then substituting this equation into (32) gives us
$$
\left. \right.\left\| \mathcal {P} _ {\hat {V} _ {\mathcal {I}} ^ {(\Pi_ {n})}} \Delta_ {w} ^ {n} \right\| _ {2} ^ {2} \leq \frac {4}{\hat {g} _ {r _ {n}} ^ {2}} \left(\left\| \hat {\mathbb {E}} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} \right] - \mathbb {E} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} \right]\right\| _ {2, \infty} ^ {2} \| \tilde {\mathbf {w}} ^ {n} \| _ {1} ^ {2} + \langle \hat {\mathbb {E}} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} \right] \Delta_ {w} ^ {n}, \boldsymbol {\epsilon} ^ {\Pi_ {n}} \rangle\right) \tag {36}
$$
We state three lemmas that help us bound the equation above with their proofs given in Appendices D.3.1, D.3.2 and D.3.3 respectively.
Lemma D.2. Let the set-up of Theorem 6.6 hold. Then,
$$
\hat {s} _ {r _ {n}} - s _ {r _ {n}} = O _ {p} (1)
$$
Lemma D.3. Let the set-up of Theorem 6.6 hold. Then for a universal constant  $C > 0$ , we have
$$
\left\| \hat {\mathbb {E}} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {\left(\Pi_ {n}\right), r _ {n}} \right] - \mathbb {E} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {\left(\Pi_ {n}\right)} \right] \right\| _ {2, \infty} \leq C \sqrt {r _ {n} \left| \Pi_ {n} \right|} \Delta_ {E} \tag {37}
$$
Lemma D.4. Let the set-up of Theorem 6.6 hold. Then,
$$
\langle \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] \Delta_ {w} ^ {n}, \boldsymbol {\epsilon} ^ {\Pi_ {n}} \rangle = O _ {p} \left(\sqrt {| \Pi_ {n} |} + r _ {n} + \| \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] - \mathbb {E} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] \| _ {2, \infty} \| \tilde {\mathbf {w}} ^ {n} \| _ {1}\right) \tag {38}
$$
Using the results of three lemmas above, applying Assumption 6.4 in (36), and then simplifying gives us
$$
\left\| \mathcal {P} _ {\hat {V} _ {\mathcal {I}} ^ {(\Pi_ {n})}} \Delta_ {w} ^ {n} \right\| _ {2} = O _ {p} \left(\frac {r _ {n} \Delta_ {E} \| \tilde {\mathbf {w}} ^ {n} \| _ {1}}{\sqrt {| \mathcal {I} |}} + \frac {r _ {n} \left(1 + \sqrt {\Delta_ {E} \| \tilde {\mathbf {w}} ^ {n} \| _ {1}}\right)}{\sqrt {| \mathcal {I} |} \| \Pi_ {n} | ^ {1 / 4}}\right) \tag {39}
$$
This concludes the proof for term 2. Using the result of Lemma C.3, we have that  $\| \tilde{\mathbf{w}}^n\| _1\leq \sqrt{r_n}$ . Substituting this into (39) gives us
$$
\left\| \mathcal {P} _ {\hat {V} _ {\mathcal {I}} ^ {(\Pi_ {n})}} \Delta_ {w} ^ {n} \right\| _ {2} = O _ {p} \left(\frac {r _ {n} ^ {3 / 2} \Delta_ {E}}{\sqrt {| \mathcal {I} |}} + \frac {r _ {n} \left(1 + \sqrt {\Delta_ {E}} r _ {n} ^ {1 / 4}\right)}{\sqrt {| \mathcal {I} |} | \Pi_ {n} | ^ {1 / 4}}\right) \tag {40}
$$
Collecting Terms. Combining the results of (28) and (40), gives us
$$
\begin{array}{l} \left\| \mathcal {P} _ {V _ {\mathcal {I}} ^ {(\Pi_ {n})}} \Delta_ {w} ^ {n} \right\| _ {2} \\ = \tilde {O} _ {p} \left(r _ {n} ^ {2} \left[ \frac {\Delta_ {E}}{\sqrt {| \mathcal {I} |} \min \{\sqrt {| \Pi_ {n} |} , \sqrt {| \mathcal {I} |} \}} + \Delta_ {E} ^ {2} \right] + \frac {r _ {n} ^ {3 / 2} \Delta_ {E}}{\sqrt {| \mathcal {I} |}} + \frac {r _ {n} \left(1 + \sqrt {\Delta_ {E}} r _ {n} ^ {1 / 4}\right)}{\sqrt {| \mathcal {I} |} | \Pi_ {n} | ^ {1 / 4}}\right) \\ \end{array}
$$
We first state Weyl's inequality which will be useful for us to establish the results.
Theorem D.5 (Weyl's Inequality). Given two matrices  $\mathbf{A},\mathbf{B}\in \mathbb{R}^{m\times n}$ , let  $s_i$  and  $\hat{s}_i$  denote the  $i$ -th singular values of  $\mathbf{A}$  and  $\mathbf{B}$  respectively. Then, for all,  $i\leq \min \{n,m\}$ , we have  $|s_i - \hat{s}_i|\leq \| \mathbf{A} - \mathbf{B}\|_{op}$
Using Weyl's inequality gives us
$$
\begin{array}{l} \left| \hat {s} _ {r _ {n}} - s _ {r _ {n}} \right| \leq \left\| \mathbb {E} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {\left(\Pi_ {n}\right)} \right] - \hat {E} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {\left(\Pi_ {n}\right)} \right] \right\| _ {\mathrm {o p}} \\ \leq \sqrt {| \mathcal {I} | | \Pi_ {n} |} \Delta_ {E} \\ \end{array}
$$
Using the inequality above and Assumption 6.4, we have
$$
\begin{array}{l} \hat {s} _ {r _ {n}} \geq s _ {r _ {n}} - \sqrt {\left| \mathcal {I} \right| \Pi_ {n}} \Delta_ {E} \\ = s _ {r _ {n}} \left(1 - \frac {\sqrt {| \mathcal {I} | \Pi_ {n}} \Delta_ {E}}{s _ {r _ {n}}}\right) \\ \geq s _ {r _ {n}} \left(1 - \sqrt {r _ {n}} \Delta_ {E}\right) \\ \end{array}
$$
Then, substituting (21) into the equation above gives us that
$$
\frac {\hat {s} _ {r _ {n}}}{s _ {r _ {n}}} \geq \left(1 - C \sqrt {\frac {r _ {n} s ^ {2} p}{M}}\right)
$$
holds with high probability for some universal constant  $C \geq 0$ . Finally, using the assumption that  $M = \omega (r_{n}s^{2}p)$  yields the claimed result.
D.3.2 Proof of Lemma D.3
For notational simplicity, let  $\hat{\mathbf{U}}_{r_n},\hat{\mathbf{S}}_{r_n},\hat{\mathbf{V}}_{r_n}$  denote  $\hat{\mathbf{U}}_{\mathcal{I}}^{(\Pi_n)},\hat{\mathbf{S}}_{\mathcal{I}}^{(\Pi_n)},\hat{\mathbf{V}}_{\mathcal{I}}^{(\Pi_n)}$  respectively. For a matrix  $\mathbf{A}$ , let  $A_{.,j}$  denote its  $j$ -th column. Additionally, denote  $\Delta_j = \mathbb{E}[\mathbf{Y}_{\mathcal{I}}^{(\Pi_n),r_n}]_{.,j} - \mathbb{E}[\mathbf{Y}_{\mathcal{I}}^{(\Pi_n)}]_{.,j}$ . Then,
$$
\begin{array}{l} \hat {\mathbb {E}} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {\left(\Pi_ {n}\right), r _ {n}} \right] _ {., j} - \mathbb {E} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {\left(\Pi_ {n}\right)} \right] _ {., j} \\ = \left(\hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] _ {., j} - \hat {\mathbf {U}} _ {r _ {n}} \hat {\mathbf {U}} _ {r _ {n}} ^ {T} \mathbb {E} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] _ {., j}\right) + \left(\hat {\mathbf {U}} _ {r _ {n}} \hat {\mathbf {U}} _ {r _ {n}} ^ {T} \mathbb {E} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] _ {., j} - \mathbb {E} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] _ {., j}\right) \\ \end{array}
$$
We have that  $\left(\hat{\mathbb{E}}[\mathbf{Y}_{\mathcal{I}}^{(\Pi_n),r_n}]_{.,j} - \hat{\mathbf{U}}_{r_n}\hat{\mathbf{U}}_{r_n}^T\mathbb{E}[\mathbf{Y}_{\mathcal{I}}^{(\Pi_n)}]_{.,j}\right)$  belongs to the subspace spanned by the column vectors of  $\hat{U}_{r_n}$ , while  $\left(\hat{\mathbf{U}}_{r_n}\hat{\mathbf{U}}_{r_n}^T\mathbb{E}[\mathbf{Y}_{\mathcal{I}}^{(\Pi_n)}]_{.,j} - \mathbb{E}[\mathbf{Y}_{\mathcal{I}}^{(\Pi_n)}]_{.,j}\right)$  belongs to its orthogonal complement. Therefore,
$$
\begin{array}{l} \| \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] _ {., j} - \mathbb {E} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] _ {., j} \| _ {2} ^ {2} \\ = \left\| \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] _ {., j} - \hat {\mathbf {U}} _ {r _ {n}} \hat {\mathbf {U}} _ {r _ {n}} ^ {T} \mathbb {E} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] _ {., j} \right\| _ {2} ^ {2} + \left\| \hat {\mathbf {U}} _ {r _ {n}} \hat {\mathbf {U}} _ {r _ {n}} ^ {T} \mathbb {E} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] _ {., j} - \mathbb {E} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] _ {., j} \right\| _ {2} ^ {2} \\ \end{array}
$$
Bounding  $\left\| \hat{\mathbb{E}} [\mathbf{Y}_{\mathcal{I}}^{(\Pi_n),r_n}]_{.,j} - \hat{\mathbf{U}}_{r_n}\hat{\mathbf{U}}_{r_n}^T\mathbb{E}[\mathbf{Y}_{\mathcal{I}}^{(\Pi_n)}]_{.,j}\right\| _2^2$  . Observe that, we have
$$
\begin{array}{l} \hat {\mathbf {U}} _ {r _ {n}} \hat {\mathbf {U}} _ {r _ {n}} ^ {T} \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] _ {., j} = \hat {\mathbf {U}} _ {r _ {n}} \hat {\mathbf {U}} _ {r _ {n}} ^ {T} \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] \mathbf {e _ {j}} = \hat {\mathbf {U}} _ {r _ {n}} \hat {\mathbf {U}} _ {r _ {n}} ^ {T} \hat {\mathbf {U}} _ {\mathcal {I}} ^ {(\Pi_ {n})} \hat {\mathbf {S}} _ {\mathcal {I}} ^ {(\Pi_ {n})} \hat {\mathbf {V}} _ {\mathcal {I}} ^ {(\Pi_ {n})} \mathbf {e _ {j}} \\ = \hat {\mathbf {U}} _ {r _ {n}} \hat {\mathbf {S}} _ {r _ {n}} \hat {\mathbf {V}} _ {r _ {n}} ^ {T} \mathbf {e} _ {j} = \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ].. j \\ \end{array}
$$
$$
\begin{array}{l} \left\| \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] _ {., j} - \hat {\mathbf {U}} _ {r _ {n}} \hat {\mathbf {U}} _ {r _ {n}} ^ {T} \mathbb {E} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] _ {., j} \right\| _ {2} ^ {2} = \left\| \hat {\mathbf {U}} _ {r _ {n}} \hat {\mathbf {U}} _ {r _ {n}} ^ {T} \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] _ {., j} - \hat {\mathbf {U}} _ {r _ {n}} \hat {\mathbf {U}} _ {r _ {n}} ^ {T} \mathbb {E} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] _ {., j} \right\| _ {2} ^ {2} \\ \leq \left\| \hat {\mathbf {U}} _ {r _ {n}} \hat {\mathbf {U}} _ {r _ {n}} ^ {T} \right\| _ {2} ^ {2} \left\| \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] _ {., j} - \mathbb {E} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] _ {., j} \right\| _ {2} ^ {2} \\ \leq \left| \Pi_ {n} \right| \Delta_ {E} ^ {2} \\ \end{array}
$$
Bounding  $\left\| \hat{\mathbf{U}}_{r_n}\hat{\mathbf{U}}_{r_n}^T\mathbb{E}[\mathbf{Y}_{\mathcal{I}}^{(\Pi_n)}]_{.,j} - \mathbb{E}[\mathbf{Y}_{\mathcal{I}}^{(\Pi_n)}]_{.,j}\right\| _2^2.$  Note that  $\mathbb{E}[\mathbf{Y}_{\mathcal{I}}^{(\Pi_n)}]_{.,j}$  =  $\mathbf{U}_{\mathcal{I}}^{(\Pi_n)}\Big(\mathbf{U}_{\mathcal{I}}^{(\Pi_n)}\Big)^T\mathbb{E}[\mathbf{Y}_{\mathcal{I}}^{(\Pi_n)}]_{.,j}.$  Using Assumption 6.1 and (26), we have
$$
\begin{array}{l} \left\| \hat {\mathbf {U}} _ {r _ {n}} \hat {\mathbf {U}} _ {r _ {n}} ^ {T} \mathbb {E} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} \right] _ {., j} - \mathbb {E} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} \right] _ {., j} \right\| _ {2} ^ {2} \leq \left\| \hat {\mathbf {U}} _ {r _ {n}} \hat {\mathbf {U}} _ {r _ {n}} ^ {T} - \mathbf {U} _ {\mathcal {I}} ^ {(\Pi_ {n})} \left(\mathbf {U} _ {\mathcal {I}} ^ {(\Pi_ {n})}\right) ^ {T} \right\| _ {2} ^ {2} \| \mathbb {E} \left[ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} \right] _ {., j} \| _ {2} ^ {2} \\ \leq \left\| \hat {\mathbf {U}} _ {r _ {n}} \hat {\mathbf {U}} _ {r _ {n}} ^ {T} - \mathbf {U} _ {\mathcal {I}} ^ {(\Pi_ {n})} \left(\mathbf {U} _ {\mathcal {I}} ^ {(\Pi_ {n})}\right) ^ {T} \right\| _ {2} ^ {2} | \Pi_ {n} | \\ \leq C r _ {n} | \Pi_ {n} | \Delta_ {E} ^ {2} \\ \end{array}
$$
for a universal constant  $C > 0$ . Combining the two bounds gives us,
$$
\max  _ {j} \Delta_ {j} \leq C \sqrt {r _ {n} | \Pi_ {n} |} \Delta_ {E}
$$
for a universal constant  $C > 0$
D.3.3 Proof of Lemma D.4
The proof of Lemma D.4 follows from adapting the notation of Lemma H.5 in [5] to that used in this paper. Specifically, we have the notational changes established in Table 2, as well as the following changes:  $r_{pre} = r_n,\mathbf{Y}_{pre,\mathcal{I}^{(d)}}^{r_{pre}} = \hat{\mathbb{E}} [\mathbf{Y}_{\mathcal{I}}^{(\Pi_n),r_n}],\mathbb{E}[\mathbf{Y}_{pre,\mathcal{I}^{(d)}}],\epsilon_{pre,n} = \epsilon^{\Pi_n},T_0 = \Pi_n$  where the l.h.s of each equality is the notation used in [5], and the r.h.s is the notation used in this work. Using the notation established, we can use the result of Lemma H.5 in [5] to give us
$$
\langle \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] \Delta_ {w} ^ {n}, \boldsymbol {\epsilon} ^ {\Pi_ {n}} \rangle = O _ {p} \left(\sqrt {| \Pi_ {n} |} + r _ {n} + \| \hat {\mathbb {E}} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n}), r _ {n}} ] - \mathbb {E} [ \mathbf {Y} _ {\mathcal {I}} ^ {(\Pi_ {n})} ] \| _ {2, \infty} \| \tilde {\mathbf {w}} ^ {n} \| _ {1}\right)
$$
D.4 Proof of Proposition C.5
We prove Proposition C.5, which also serves as the finite-sample analysis of the vertical regression procedure (step 2 of Synthetic Combinations) done via principal component regression. The proof follows from adapting the notation of Proposition E.3 in [2] to that used in this paper. Specifically, we present table 2 that matches their notation to ours.
Using the table above, we have that the following holds with probability  $1 - O\left(\left(|\mathcal{I}||\Pi_n|\right)^{-10}\right)$
$$
\| \tilde {\mathbf {w}} ^ {n} - \tilde {\mathbf {w}} ^ {n} \| _ {2} \leq C \sigma^ {3} \log^ {3} \left(\left(| \mathcal {I} | | \Pi_ {n} |\right)\right) \left[ \frac {\| \tilde {\mathbf {w}} ^ {n} \| _ {2}}{\min  \left\{\sqrt {| \Pi_ {n} |} , \sqrt {| \mathcal {I} |} \right\}} + \Delta_ {E} \right]
$$
Notation of [2] Our Notation
Y	YΠn
X	E[Y(Πn)I]
Z	E[Y(Πn)I]
n	Πn
p	I
β*	w
β	w
ΔE	ΔE
r	r
φlr	0
A	1
K	0
Ka,κ,σ	Cσ
ρmin	1
Table 2: A summary of the main notational differences between our setting and that of [2].
E Sparsity of Matrix of Fourier Coefficients
In this section, we prove that the matrix of Fourier coefficients  $\mathcal{A}$  has at most  $rs$  non-zero columns. This result helps establish that the lower bound on the sample complexity for estimating all  $N\times 2^p$  parameters is  $O(Nr + r^2 s)$  as discussed in 6.3.
Lemma E.1. The number of nonzero columns of  $\mathcal{A}$ , which we have denoted  $p'$ , satisfies  $p' \leq rs$ .
Proof. Let  $\mathcal{A}_t$  denote the sub-matrix of  $\mathcal{A}$  formed by taking the first  $t$  rows. Further, let  $p_t^\prime$  denote the number of nonzero columns of  $\mathcal{A}_t$ . We proceed by induction, that is we show for all  $t$ ,
$$
p _ {t} ^ {\prime} - s \cdot \operatorname {r a n k} \left(\mathcal {A} _ {t}\right) \leq 0
$$
In the case of  $t = 1$ , either the first row is the zero vector, in which case  $p_1' = \mathrm{rank}(\mathcal{A}_1) = 0$ , or else  $\mathrm{rank}(\mathcal{A}_1) = 1$  and  $p_1' \leq s$  since the first row is at most  $s$ -sparse. Then, for general  $t$ , note that