SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma A.1 Given  $n \geq d \geq 1$ . Let  $\{(\mathbf{a}_i, b_i)\}_{i=1}^n$  be a sequence of pairs such that  $\mathbf{a}_i \in \mathbb{R}^d$  are  $\mathcal{F}_{i-1}$ -measurable and  $b_i \in \mathbb{R}$  are  $\mathcal{F}_i$ -measurable w.r.t. some filtration  $\{\mathcal{F}_i\}_{i=1}^n$ . Assume in addition that  $b_i$  are zero-mean sub-Gaussian random variables with parameter  $\sigma$  conditioned on  $\mathcal{F}_{i-1}$ , i.e.,
$$
\mathbb {E} [ b _ {i} | \mathcal {F} _ {i - 1} ] = 0, a n d \mathbb {E} [ e ^ {\lambda b _ {i}} | \mathcal {F} _ {i - 1} ] \leq e ^ {\sigma^ {2} \lambda^ {2} / 2}, f o r a l l \lambda \in \mathbb {R}.
$$
Let  $\mathbf{A} = [\mathbf{a}_1, \ldots, \mathbf{a}_n]^\top$  and  $\mathbf{b} = [b_1, \ldots, b_n]^\top$ . Suppose that  $1 \leq \sigma_{\min}(\mathbf{A}^\top \mathbf{A}) \leq \sigma_{\max}(\mathbf{A}^\top \mathbf{A}) \leq nB$  for some constant  $B > 0$ .
(a). (A simplified version of Theorem 1 in Abbasi et al. [1].) With probability over  $1 - \delta$
$$
\left\| \mathbf {P} _ {\mathbf {A}} \mathbf {b} \right\| _ {2} ^ {2} = \mathbf {b} ^ {\top} \mathbf {P} _ {\mathbf {A}} \mathbf {b} \leq c \sigma^ {2} \log (\det  (\mathbf {A} ^ {\top} \mathbf {A}) / \delta) \leq c \sigma^ {2} d \log (n B / \delta)
$$
(b). Let  $\tilde{\mathbf{A}}\coloneqq \mathbf{A} - \mathbf{P}_{1_n}\mathbf{A}$  be the centered matrix, then with probability over  $1 - \delta$
$$
\left\| \mathbf {P} _ {\widetilde {\mathbf {A}}} \mathbf {b} \right\| _ {2} ^ {2} = \mathbf {b} ^ {\top} \mathbf {P} _ {\widetilde {\mathbf {A}}} \mathbf {b} \leq c \sigma^ {2} \log (n \det  (\widetilde {\mathbf {A}} ^ {\top} \widetilde {\mathbf {A}}) / \delta) \leq c \sigma^ {2} d \log (n B / \delta)
$$
In the proofs we choose  $(\mathbf{A},\mathbf{b}) = (\mathbf{X}_{\mathrm{nad}},\varepsilon),(\mathbf{X}_{\mathrm{ad}},\varepsilon),(\mathbf{X}_{\mathrm{ad}},\mathbf{x}_j)$  for  $j\in [k + 1,d]$ . It is readily verified that conditions in Lemma A.1 are satisfied under assumptions in Theorem 3.1 and 3.2. See the proof of this lemma in Section A.7.
Lemma A.2 Suppose the data set  $\{(\pmb{x}_i, y_i)\}_{i=1}^n$  is  $(k, d)$ -adaptive and satisfies Assumption (A1)-(A4). Suppose the sample size condition (6) is in force. Adopt the notations in Section 3.1.2. Define  $\widetilde{\mathbf{X}}_{\mathrm{nad}} = \mathbf{X}_{\mathrm{nad}} - \mathbb{E}[\mathbf{X}_{\mathrm{nad}}]$  and recall that  $\widetilde{\mathbf{X}}_{\mathrm{nad}} = (\mathbf{I}_n - \mathbf{P}_{\mathbf{1}_n})\mathbf{X}_{\mathrm{nad}}$ . For the matrices  $\widetilde{\mathbf{X}}_{\mathrm{nad}}, \widetilde{\mathbf{X}}_{\mathrm{nad}}$ , we have the following with probability over  $1 - \delta$
$$
\frac {1}{\sigma_ {\operatorname* {m i n}} \left(\overline {{\mathbf {X}}} _ {\mathrm {n a d}}\right) ^ {2}} = \| \left(\overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}}\right) ^ {- 1} \| _ {o p} \leq \frac {2}{n \sigma_ {\min }} \tag {14a}
$$
$$
\| \overline {{\mathbf {X}}} _ {\text {n a d}} \| _ {o p} \leq \sqrt {2 n \sigma_ {\max }} \tag {14b}
$$
$$
\left\| \overline {{\mathbf {X}}} _ {\text {n a d}} - \widetilde {\mathbf {X}} _ {\text {n a d}} \right\| _ {o p} \leq C \sqrt {\log (n / \delta)} \sqrt {d - k} \leq \frac {1}{2} \sigma_ {\min } \left(\overline {{\mathbf {X}}} _ {\text {n a d}}\right), \tag {14c}
$$
$$
\| \left(\overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}}\right) ^ {- 1} - \left(\widetilde {\mathbf {X}} _ {\mathrm {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {n a d}}\right) ^ {- 1} \| o p \leq \frac {C \sqrt {\log (n / \delta)} \sqrt {d - k}}{n ^ {3 / 2}}. \tag {14d}
$$
$$
\| \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}} - \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}} \| _ {o p} \leq C \sqrt {\log (n / \delta)} \sqrt {\frac {d - k}{n}} \leq \frac {1}{4} \tag {14e}
$$
$$
\left\| \left(\mathbf {P} _ {\overline {{\mathbf {X}}} _ {\mathrm {n a d}}} - \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}}\right) \boldsymbol {\varepsilon} \right\| _ {2} \leq C \tag {14f}
$$
for some parameter-dependent constants  $C > 0$ . Moreover, equation (14a), (14b) also hold when replacing  $\overline{\mathbf{X}}_{\mathrm{rad}}$  with the zero-mean matrix  $\mathbf{X}_{\mathrm{rad}}$  defined in Section 3.1.1.
See the proof in Section A.8.
Lemma A.3 Under assumptions in Theorem 3.1, with probability over  $1 - \delta$
$$
\mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathrm {n a d}}} \mathbf {X} _ {\mathrm {a d}} \preceq \frac {C (d - k) k \log (n / \delta)}{n} \mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}}
$$
for some parameter-dependent constant  $C > 0$ .
See the proof in Section A.9.
Lemma A.4 Under assumptions in Theorem 3.2, with probability over  $1 - \delta$
$$
\widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}} \widetilde {\mathbf {X}} _ {\mathrm {a d}} \preceq \frac {C (d - k) k \log (n / \delta)}{n} \widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}}
$$
for some parameter-dependent constant  $C > 0$
See the proof in Section A.10.
A.2 Proof of Proposition 2.2
Part (a). Proposition 2.2(a) is a standard result on the minimax optimality of the OLS estimator in linear models. A proof of this result using a Bayes argument can be found in the proof of Theorem 2(a) in Khamaru et al. [20].
By properties of the OLS estimator, we have  $(\widehat{\pmb{\theta}}_{\mathrm{ols}} - \pmb{\theta}^{*})\sim \mathcal{N}(0,\sigma^{2}(\mathbf{S}_{n})^{-1})$  conditioned on  $\mathbf{X}$ . Therefore  $(\widehat{\pmb{\theta}}_{\mathrm{ols},\mathcal{I}} - \pmb{\theta}_{\mathcal{I}}^{*})\sim \mathcal{N}(0,\sigma^{2}(\mathbf{S}_{n}^{-1})_{\mathcal{II}})$  conditioned on  $\mathbf{X}$ , where  $(\mathbf{S}_n^{-1})_{\mathcal{II}}$  denotes the submatrix of  $\mathbf{S}_n^{-1}$  that consists of the coordinates in  $\mathcal{I}$ . It follows immediately that
$$
\mathbb {E} \left\| \widehat {\pmb {\theta}} _ {\mathcal {I}} - \pmb {\theta} _ {\mathcal {I}} ^ {*} \right\| _ {[ (\mathbf {S} _ {n} ^ {- 1}) _ {\mathcal {I I}} ] ^ {- 1}} ^ {2} = \sigma^ {2} \operatorname {t r} (\mathbf {I} _ {| \mathcal {I} |}) = \sigma^ {2} | \mathcal {I} |
$$
for any  $\pmb{\theta}^{*}\in \mathbb{R}^{d}$ . As a result, the OLS estimator attains the minimax lower bound when the data are i.i.d.
Part (b). When  $|\mathcal{I}| = 1$ , part (b) follows immediately from Theorem 2(b) of Khamaru et al. [20] with  $\mathbf{v}$  chosen to be the one-hot vector supported on  $\mathcal{I}$ . When  $|\mathcal{I}| > 1$ , w.l.o.g. assume  $\mathcal{I}$  consists of the first  $|\mathcal{I}|$  coordinates of  $[d]$ . It suffices to show the scaled-MSE for  $\mathcal{I}$  is always no less than the scaled-MSE for the first coordinate.
This follows from properties of Schur complement and the projection operator,
$$
\begin{array}{l} \left\| \widehat {\boldsymbol {\theta}} _ {\mathcal {I}} - \boldsymbol {\theta} _ {\mathcal {I}} ^ {*} \right\| _ {[ (\mathbf {S} _ {n} ^ {- 1}) _ {\mathcal {I I}} ] ^ {- 1}} ^ {2} \\ = \left(\widehat {\boldsymbol {\theta}} _ {\mathcal {I}} - \boldsymbol {\theta} _ {\mathcal {I}} ^ {*}\right) ^ {\top} \mathbf {X} _ {\mathcal {I}} ^ {\top} \left(\mathbf {I} _ {n} - \mathbf {P} _ {\mathbf {X} _ {\mathcal {I} ^ {c}}}\right) \mathbf {X} _ {\mathcal {I}} \left(\widehat {\boldsymbol {\theta}} _ {\mathcal {I}} - \boldsymbol {\theta} _ {\mathcal {I}} ^ {*}\right) \\ = (\widehat {\boldsymbol {\theta}} _ {\mathcal {I}} - \boldsymbol {\theta} _ {\mathcal {I}} ^ {*}) ^ {\top} \left( \begin{array}{c c} 1 & \star \\ 0 & \mathbf {I} _ {k - 1} \end{array} \right) \left( \begin{array}{c c} d _ {1} & \mathbf {0} ^ {\top} \\ \mathbf {0} & \mathbf {D} _ {2} \end{array} \right) \left( \begin{array}{c c} 1 & 0 \\ \star & \mathbf {I} _ {k - 1} \end{array} \right) (\widehat {\boldsymbol {\theta}} _ {\mathcal {I}} - \boldsymbol {\theta} _ {\mathcal {I}} ^ {*}) \\ = \left( \begin{array}{c c} \widehat {\theta} _ {1} - \theta_ {1} ^ {*} & \star \end{array} \right) \left( \begin{array}{c c} d _ {1} & \mathbf {0} ^ {\top} \\ \mathbf {0} & \mathbf {D} _ {2} \end{array} \right) \left( \begin{array}{c} \widehat {\theta} _ {1} - \theta_ {1} ^ {*} \\ \star \end{array} \right) \\ \geq d _ {1} \left(\widehat {\theta_ {1}} - \theta_ {1} ^ {*}\right) ^ {2}, \\ \end{array}
$$
where  $\mathbf{P}_{\mathbf{M}}^{\perp}$  denote the projection onto the orthogonal space of the column space of  $\mathbf{M}$  and
$$
\begin{array}{l} d _ {1} = \mathbf {x} _ {1} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathcal {I} ^ {c}}} ^ {\perp} \mathbf {x} _ {1} - \mathbf {x} _ {1} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathcal {I} ^ {c}}} ^ {\perp} \mathbf {X} _ {\mathcal {I}, - 1} \left(\mathbf {X} _ {\mathcal {I}, - 1} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathcal {I} ^ {c}}} ^ {\perp} \mathbf {X} _ {\mathcal {I}, - 1}\right) ^ {- 1} \mathbf {X} _ {\mathcal {I}, - 1} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathcal {I} ^ {c}}} ^ {\perp} \mathbf {x} _ {1}, \\ \mathbf {D} _ {2} = \mathbf {X} _ {\mathcal {I}, - 1} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathcal {I} ^ {c}}} ^ {\perp} \mathbf {X} _ {\mathcal {I}, - 1}. \\ \end{array}
$$
Using the properties of projection operator and linear space decomposition, it can be verified that
$$
d _ {1} = \mathbf {x} _ {1} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {- 1}} ^ {\perp} \mathbf {x} _ {1} = \mathbf {x} _ {1} ^ {\top} \left(\mathbf {I} _ {n} - \mathbf {P} _ {\mathbf {X} _ {- 1}}\right) \mathbf {x} _ {1}.
$$
Therefore  $\left\| \widehat{\pmb{\theta}}_{\mathcal{I}} - \pmb{\theta}_{\mathcal{I}}^{*}\right\|_{[(\mathbf{S}_n^{-1})_{\mathcal{I}\mathcal{I}}]^{-1}}^2 \geq \left\| \widehat{\theta}_1 - \theta_1^*\right\|_{[(\mathbf{S}_n^{-1})_{11}]^{-1}}^2$ . This completes the proof.
By the definition of the OLS estimator, we have
$$
\widehat {\boldsymbol {\theta}} - \boldsymbol {\theta} ^ {*} = \left( \begin{array}{c c} \mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}} & \mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}} \\ \mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}} & \mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}} \end{array} \right) ^ {- 1} \cdot \left( \begin{array}{c} \mathbf {X} _ {\mathrm {a d}} ^ {\top} \boldsymbol {\varepsilon} \\ \mathbf {X} _ {\mathrm {n a d}} ^ {\top} \boldsymbol {\varepsilon} \end{array} \right).
$$
Applying the block matrix inverse formula, we obtain
$$
\widehat {\pmb {\theta}} _ {\mathrm {a d}} - \pmb {\theta} _ {\mathrm {a d}} ^ {*} = (\mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}} - \mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathrm {n a d}}} \mathbf {X} _ {\mathrm {a d}}) ^ {- 1} (\mathbf {X} _ {\mathrm {a d}} ^ {\top} \pmb {\varepsilon} - \mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathrm {n a d}}} \pmb {\varepsilon}).
$$
To simplify notation, we define
$$
\overline {{R}} _ {1} := \mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}} - \mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathrm {n a d}}} \mathbf {X} _ {\mathrm {a d}}, \qquad \overline {{R}} _ {2} := \mathbf {X} _ {\mathrm {a d}} ^ {\top} \boldsymbol {\varepsilon} - \mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathrm {n a d}}} \boldsymbol {\varepsilon},
$$
$$
\boldsymbol {R} _ {1} := \mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}}, \quad \quad \boldsymbol {R} _ {2} := \mathbf {X} _ {\mathrm {a d}} ^ {\top} \varepsilon .
$$
$$
\left(\widehat {\pmb {\theta}} _ {\mathrm {a d}} - \pmb {\theta} _ {\mathrm {a d}} ^ {*}\right) ^ {\top} \overline {{\mathbf {R}}} _ {1} \left(\widehat {\pmb {\theta}} _ {\mathrm {a d}} - \pmb {\theta} _ {\mathrm {a d}} ^ {*}\right) = \overline {{\mathbf {R}}} _ {2} ^ {\top} \overline {{\mathbf {R}}} _ {1} ^ {- 1} \overline {{\mathbf {R}}} _ {2}.
$$
We claim the following results which we prove later. With probability over  $1 - \delta$
$$
\mathbf {0} \preceq \frac {1}{2} \mathbf {R} _ {1} \preceq \overline {{\mathbf {R}}} _ {1} \preceq \mathbf {R} _ {1}, \tag {15a}
$$
$$
\left| \overline {{\boldsymbol {R}}} _ {2} ^ {\top} \boldsymbol {R} _ {1} ^ {- 1} \overline {{\boldsymbol {R}}} _ {2} - \boldsymbol {R} _ {2} ^ {\top} \boldsymbol {R} _ {1} ^ {- 1} \boldsymbol {R} _ {2} \right| \leq C \log (n \det (\mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}}) / \delta), \tag {15b}
$$
$$
\boldsymbol {R} _ {2} ^ {\top} \boldsymbol {R} _ {1} ^ {- 1} \boldsymbol {R} _ {2} \leq C \log (\det (\mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}}) / \delta), \tag {15c}
$$
Taking these claims as given, we establish
$$
\left(\widehat {\boldsymbol {\theta}} _ {\mathrm {a d}} - \boldsymbol {\theta} _ {\mathrm {a d}} ^ {*}\right) ^ {\top} \overline {{\boldsymbol {R}}} _ {1} \left(\widehat {\boldsymbol {\theta}} _ {\mathrm {a d}} - \boldsymbol {\theta} _ {\mathrm {a d}} ^ {*}\right) = \overline {{\boldsymbol {R}}} _ {2} ^ {\top} \overline {{\boldsymbol {R}}} _ {1} ^ {- 1} \overline {{\boldsymbol {R}}} _ {2} \leq 2 \overline {{\boldsymbol {R}}} _ {2} ^ {\top} \boldsymbol {R} _ {1} ^ {- 1} \overline {{\boldsymbol {R}}} _ {2}, \tag {16}
$$
$$
\overline {{\boldsymbol {R}}} _ {2} ^ {\top} \boldsymbol {R} _ {1} ^ {- 1} \overline {{\boldsymbol {R}}} _ {2} \leq | \overline {{\boldsymbol {R}}} _ {2} ^ {\top} \boldsymbol {R} _ {1} ^ {- 1} \overline {{\boldsymbol {R}}} _ {2} - \boldsymbol {R} _ {2} ^ {\top} \boldsymbol {R} _ {1} ^ {- 1} \boldsymbol {R} _ {2} | + \boldsymbol {R} _ {2} ^ {\top} \boldsymbol {R} _ {1} ^ {- 1} \boldsymbol {R} _ {2} \leq C \log (n \det (\mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}}) / \delta) \tag {17}
$$
where equation (16) uses claim (15a) and equation (17) uses claim (15b), (15c). Putting the last two displays together completes the proof.
Proof of claim (15a) The first and third inequality follows from the definition of  $\overline{R}_1$  and  $R_1$ . The second inequality follows from Lemma A.3 and noting that  $\frac{C(d - k)k\log(n / \delta)}{n} < 1 / 2$  under the sample size assumption (6) with  $C$  in (6) chosen sufficiently large.
Proof of claim (15b) Define  $\widetilde{\mathbf{P}}_{\mathbf{A}} := (\mathbf{A}^{\top}\mathbf{A})^{-1/2}\mathbf{A}^{\top}$  for any  $\mathbf{A} \in \mathbb{R}^{n \times d}$ . Then we have  $\|\widetilde{\mathbf{P}}_{\mathbf{A}}\mathbf{b}\|_2 = \|\mathbf{P}_{\mathbf{A}}\mathbf{b}\|_2$  for any  $\mathbf{b} \in \mathbb{R}^n$ . Note that
$$
\begin{array}{l} \left| \bar {R} _ {2} ^ {\top} R _ {1} ^ {- 1} \bar {R} _ {2} - R _ {2} ^ {\top} R _ {1} ^ {- 1} R _ {2} \right| \\ \leq | (\bar {\boldsymbol {R}} _ {2} - \boldsymbol {R} _ {2}) ^ {\top} \boldsymbol {R} _ {1} ^ {- 1} (\bar {\boldsymbol {R}} _ {2} - \boldsymbol {R} _ {2}) | + 2 | (\bar {\boldsymbol {R}} _ {2} - \boldsymbol {R} _ {2}) ^ {\top} \boldsymbol {R} _ {1} ^ {- 1} \boldsymbol {R} _ {2} | \\ = \varepsilon^ {\top} \mathbf {P} _ {\mathbf {X} _ {\text {n a d}}} \mathbf {P} _ {\mathbf {X} _ {\text {a d}}} \mathbf {P} _ {\mathbf {X} _ {\text {n a d}}} \varepsilon + 2 | \varepsilon^ {\top} \mathbf {P} _ {\mathbf {X} _ {\text {n a d}}} \mathbf {P} _ {\mathbf {X} _ {\text {a d}}} \varepsilon | \\ = \varepsilon^ {\top} \widetilde {\mathbf {P}} _ {\mathbf {X} _ {\mathrm {n a d}}} (\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}}) ^ {- 1 / 2} (\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathrm {a d}}} \mathbf {X} _ {\mathrm {n a d}}) (\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}}) ^ {- 1 / 2} \widetilde {\mathbf {P}} _ {\mathbf {X} _ {\mathrm {n a d}}} \varepsilon \\ + 2 \left| \varepsilon^ {\top} \widetilde {\mathbf {P}} _ {\mathbf {X} _ {\text {n a d}}} \left(\mathbf {X} _ {\text {n a d}} ^ {\top} \mathbf {X} _ {\text {n a d}}\right) ^ {- 1 / 2} \mathbf {X} _ {\text {n a d}} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\text {a d}}} \varepsilon \right| \\ \leq \frac {\| \mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathrm {a d}}} \mathbf {X} _ {\mathrm {n a d}} \| \mathrm {o p}}{\sigma_ {\min} (\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}})} \| \widetilde {\mathbf {P}} _ {\mathbf {X} _ {\mathrm {n a d}}} \boldsymbol {\varepsilon} \| _ {2} ^ {2} + 2 \frac {\| \mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathrm {a d}}} \boldsymbol {\varepsilon} \| _ {2}}{\sigma_ {\min} (\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}}) ^ {1 / 2}} \| \widetilde {\mathbf {P}} _ {\mathbf {X} _ {\mathrm {n a d}}} \boldsymbol {\varepsilon} \| _ {2}. \\ \end{array}
$$
Applying equation (14a) and Lemma A.1 in the last display, we continue
$$
\begin{array}{l} \left| \bar {R} _ {2} ^ {\top} R _ {1} ^ {- 1} \bar {R} _ {2} - R _ {2} ^ {\top} R _ {1} ^ {- 1} R _ {2} \right| \\ \leq \frac {C}{n} \operatorname {t r} \left(\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathrm {a d}}} \mathbf {X} _ {\mathrm {n a d}}\right) \| \widetilde {\mathbf {P}} _ {\mathbf {X} _ {\mathrm {n a d}}} \boldsymbol {\varepsilon} \| _ {2} ^ {2} + \frac {C \| \left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left.\left.\left.\left.\left.\left.\left. | | \widetilde {\mathbf {P}} _ {\mathbf {X} _ {\mathrm {a d}}} | | | \widetilde {\mathbf {P}} _ {\mathbf {X} _ {\mathrm {a d}}} | | | \boldsymbol {\varepsilon} | | _ {2} \right| | | \widetilde {\mathbf {P}} _ {\mathbf {X} _ {\mathrm {a d}}} | | | \boldsymbol {\varepsilon} | | _ {2} \right| | | \widetilde {\mathbf {P}} _ {\mathbf {X} _ {\mathrm {a d}}} | | | \boldsymbol {\varepsilon} | | _ {2} \right| | | \widetilde {\boldsymbol {\varepsilon}} | | _ {2} \right| | | \widetilde {\mathbf {P}} _ {\mathbf {X} _ {\mathrm {a d}}} | | | \boldsymbol {\varepsilon} | | _ {2} \right| | | \widetilde {\mathbf {P}} _ {\mathbf {X} _ {\mathrm {a d}}} | | | \boldsymbol {\varepsilon} | | _ {2}\right)} \| 0 \| _ {2} ^ {2}\right)} \\ \leq \frac {C (d - k)}{n} \max  _ {k + 1 \leq j \leq d} (\mathbf {x} _ {j} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathrm {a d}}} \mathbf {x} _ {j}) \| \mathbf {P} _ {\mathbf {X} _ {\mathrm {n a d}}} \varepsilon \| _ {2} ^ {2} \\ + \frac {C \sqrt {k} \max  _ {k + 1 \leq j \leq d} \left(\sqrt {\mathbf {x} _ {j} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathrm {a d}}} \mathbf {x} _ {j}}\right) \| \mathbf {P} _ {\mathbf {X} _ {\mathrm {a d}}} \boldsymbol {\varepsilon} \| _ {2}}{\sqrt {n}} \| \mathbf {P} _ {\mathbf {X} _ {\mathrm {n a d}}} \boldsymbol {\varepsilon} \| _ {2} \\ \leq \frac {C (d - k) ^ {2}}{n} \log (n \det (\mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}}) / \delta) \log (n / \delta) + \frac {C \sqrt {k (d - k)}}{\sqrt {n}} \log (n \det (\mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}}) / \delta) \log^ {1 / 2} (n / \delta) \\ \leq \log (n \det (\mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}}) / \delta), \\ \end{array}
$$
where the fourth line uses Lemma A.1 and equation (14a), the last line follows from the sample size assumption (6).
Proof of claim (15c) This is a direct consequence of Lemma A.1 since  $\varepsilon_{i}$  are conditionally zero-mean sub-Gaussian by Assumption (A4).
Let  $\pmb{\mu}^{*} = (\pmb{\mu}_{\mathrm{ad}}^{*\top},\pmb{\mu}_{\mathrm{rad}}^{*\top})^{\top}$  denote the mean vector of  $\mathbb{E}[\pmb{x}_i]$  and define
$$
\bar {\mathbf {X}} _ {\text {n a d}} := \mathbf {X} _ {\text {n a d}} - \mathbf {1} _ {n} \boldsymbol {\mu} _ {\text {n a d}} ^ {* \top}.
$$
We write  $\overline{\mathbf{X}}_{\mathrm{rad}} = [\overline{\mathbf{x}}_{k + 1}\quad \dots \quad \overline{\mathbf{x}}_d]$ . The proof of this theorem follows the same basic steps as the proof of Theorem 3.1.
Recall that  $\widehat{\theta}_{\mathrm{ad}} \in \mathbb{R}^k$  denotes the non-adaptive component of the centered OLS estimator. By definition and the matrix inverse formula, we have
$$
\widehat {\pmb {\theta}} _ {\mathrm {a d}} - \pmb {\theta} _ {\mathrm {a d}} ^ {*} = (\widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} (\mathbf {I} _ {n} - \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}}) \widetilde {\mathbf {X}} _ {\mathrm {a d}}) ^ {- 1} \cdot \widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} (\mathbf {I} _ {n} - \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}}) \pmb {\varepsilon}.
$$
To simplify notation, we introduce
$$
\widetilde {\boldsymbol {R}} _ {3} := \widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} (\mathbf {I} _ {n} - \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}}) \widetilde {\mathbf {X}} _ {\mathrm {a d}}, \qquad \widetilde {\boldsymbol {R}} _ {4} := \widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} (\mathbf {I} _ {n} - \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}}) \boldsymbol {\varepsilon},
$$
$$
\overline {{\boldsymbol {R}}} _ {3} := \widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} (\mathbf {I} _ {n} - \mathbf {P} _ {\overline {{\mathbf {X}}} _ {\mathrm {n a d}}}) \widetilde {\mathbf {X}} _ {\mathrm {a d}}, \qquad \overline {{\boldsymbol {R}}} _ {4} := \widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} (\mathbf {I} _ {n} - \mathbf {P} _ {\overline {{\mathbf {X}}} _ {\mathrm {n a d}}}) \boldsymbol {\varepsilon},
$$
$$
\boldsymbol {R} _ {3} := \widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}}, \qquad \qquad \boldsymbol {R} _ {4} := \widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \boldsymbol {\varepsilon},
$$
$$
\left(\widehat {\pmb {\theta}} _ {\mathrm {a d}} - \pmb {\theta} _ {\mathrm {a d}} ^ {*}\right) ^ {\top} \widetilde {\pmb {R}} _ {3} \left(\widehat {\pmb {\theta}} _ {\mathrm {a d}} - \pmb {\theta} _ {\mathrm {a d}} ^ {*}\right) = \widetilde {\pmb {R}} _ {4} ^ {\top} \widetilde {\pmb {R}} _ {3} ^ {- 1} \widetilde {\pmb {R}} _ {4}.
$$
Again, we claim the following results which we prove later. With probability over  $1 - \delta$
$$
\mathbf {0} \preceq \frac {1}{2} \boldsymbol {R} _ {3} \preceq \widetilde {\boldsymbol {R}} _ {3} \preceq \boldsymbol {R} _ {3}, \tag {18a}
$$
$$
\left| \widetilde {\boldsymbol {R}} _ {4} ^ {\top} \boldsymbol {R} _ {3} ^ {- 1} \widetilde {\boldsymbol {R}} _ {4} - \boldsymbol {R} _ {4} ^ {\top} \boldsymbol {R} _ {3} ^ {- 1} \boldsymbol {R} _ {4} \right| \leq C \log \left(n \det \left(\widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}}\right) / \delta\right), \tag {18b}
$$
$$
\boldsymbol {R} _ {4} ^ {\top} \boldsymbol {R} _ {3} ^ {- 1} \boldsymbol {R} _ {4} \leq C \log (n \det (\widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}}) / \delta). \tag {18c}
$$
With the claims at hand, we obtain
$$
\left(\widehat {\boldsymbol {\theta}} _ {\mathrm {a d}} - \boldsymbol {\theta} _ {\mathrm {a d}} ^ {*}\right) ^ {\top} \widetilde {\boldsymbol {R}} _ {3} \left(\widehat {\boldsymbol {\theta}} _ {\mathrm {a d}} - \boldsymbol {\theta} _ {\mathrm {a d}} ^ {*}\right) = \widetilde {\boldsymbol {R}} _ {4} ^ {\top} \widetilde {\boldsymbol {R}} _ {3} ^ {- 1} \widetilde {\boldsymbol {R}} _ {4} \leq 2 \widetilde {\boldsymbol {R}} _ {4} ^ {\top} \boldsymbol {R} _ {3} ^ {- 1} \widetilde {\boldsymbol {R}} _ {4}, \tag {19}
$$
$$
\tilde {\boldsymbol {R}} _ {4} ^ {\top} \boldsymbol {R} _ {3} ^ {- 1} \tilde {\boldsymbol {R}} _ {4} \leq | \tilde {\boldsymbol {R}} _ {4} ^ {\top} \boldsymbol {R} _ {3} ^ {- 1} \tilde {\boldsymbol {R}} _ {4} - \boldsymbol {R} _ {4} ^ {\top} \boldsymbol {R} _ {3} ^ {- 1} \boldsymbol {R} _ {4} | + \boldsymbol {R} _ {4} ^ {\top} \boldsymbol {R} _ {3} ^ {- 1} \boldsymbol {R} _ {4} \leq C \log (n \det (\tilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \tilde {\mathbf {X}} _ {\mathrm {a d}}) / \delta) \quad (2 0)
$$