SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where equation (19) uses claim (18a) and equation (20) uses claim (18b), (18c). Combining the last two displays concludes the proof.
Proof of claim (18a) The first and third inequality follows from the definition of  $\widetilde{R}_3$  and  $R_3$ . For the second inequality, we have
$$
\begin{array}{l} \boldsymbol {R} _ {3} - \tilde {\boldsymbol {R}} _ {3} = (\boldsymbol {R} _ {3} - \bar {\boldsymbol {R}} _ {3}) + (\bar {\boldsymbol {R}} _ {3} - \tilde {\boldsymbol {R}} _ {3}) \\ = \tilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \mathbf {P} _ {\bar {\mathbf {X}} _ {\mathrm {n a d}}} \tilde {\mathbf {X}} _ {\mathrm {a d}} + \tilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} (\mathbf {P} _ {\tilde {\mathbf {X}} _ {\mathrm {n a d}}} - \mathbf {P} _ {\bar {\mathbf {X}} _ {\mathrm {n a d}}}) \tilde {\mathbf {X}} _ {\mathrm {a d}} \\ \preceq \frac {1}{4} \widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}} + \frac {1}{4} \widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}} = \frac {1}{2} \mathbf {R} _ {3}, \\ \end{array}
$$
where the last line uses Lemma A.2, A.4 and noting that  $\frac{C(d - k)k\log(n / \delta)}{n} < 1 / 4$  under our sample size assumption (6) with  $C$  in (6) chosen sufficiently large.
Proof of claim (18b) Recall that we define  $\widetilde{\mathbf{P}}_{\mathbf{A}} := (\mathbf{A}^{\top}\mathbf{A})^{-1/2}\mathbf{A}^{\top}$  for any  $\mathbf{A} \in \mathbb{R}^{n \times d}$ . Note that
$$
\begin{array}{l} \left| \tilde {\boldsymbol {R}} _ {4} ^ {\top} \boldsymbol {R} _ {3} ^ {- 1} \tilde {\boldsymbol {R}} _ {4} - \boldsymbol {R} _ {4} ^ {\top} \boldsymbol {R} _ {3} ^ {- 1} \boldsymbol {R} _ {4} \right| \\ \leq | (\widetilde {\boldsymbol {R}} _ {4} - \boldsymbol {R} _ {4}) ^ {\top} \boldsymbol {R} _ {3} ^ {- 1} (\widetilde {\boldsymbol {R}} _ {4} - \boldsymbol {R} _ {4}) | + 2 | (\widetilde {\boldsymbol {R}} _ {4} - \boldsymbol {R} _ {4}) ^ {\top} \boldsymbol {R} _ {3} ^ {- 1} \boldsymbol {R} _ {4} | \\ \leq 2 \left[ \left(\widetilde {\boldsymbol {R}} _ {4} - \overline {{\boldsymbol {R}}} _ {4}\right) ^ {\top} \boldsymbol {R} _ {3} ^ {- 1} \left(\widetilde {\boldsymbol {R}} _ {4} - \overline {{\boldsymbol {R}}} _ {4}\right) + \left(\overline {{\boldsymbol {R}}} _ {4} - \boldsymbol {R} _ {4}\right) ^ {\top} \boldsymbol {R} _ {3} ^ {- 1} \left(\overline {{\boldsymbol {R}}} _ {4} - \boldsymbol {R} _ {4}\right) \right. \\ + \left| \left(\bar {\boldsymbol {R}} _ {4} - \bar {\boldsymbol {R}} _ {4}\right) ^ {\top} \boldsymbol {R} _ {3} ^ {- 1} \boldsymbol {R} _ {4} \right| + \left| \left(\bar {\boldsymbol {R}} _ {4} - \boldsymbol {R} _ {4}\right) ^ {\top} \boldsymbol {R} _ {3} ^ {- 1} \boldsymbol {R} _ {4} \right| ] \\ =: 2 \left[ W _ {1} + W _ {2} + W _ {3} + W _ {4} \right], \\ \end{array}
$$
$$
W _ {1} := \varepsilon^ {\top} \left(\mathbf {P} _ {\tilde {\mathbf {X}} _ {\mathrm {n a d}}} - \mathbf {P} _ {\tilde {\mathbf {X}} _ {\mathrm {n a d}}}\right) \mathbf {P} _ {\tilde {\mathbf {X}} _ {\mathrm {a d}}} \left(\mathbf {P} _ {\bar {\mathbf {X}} _ {\mathrm {n a d}}} - \mathbf {P} _ {\tilde {\mathbf {X}} _ {\mathrm {n a d}}}\right) \varepsilon
$$
$$
W _ {2} := | \boldsymbol {\varepsilon} ^ {\top} (\mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}} - \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}}) \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {a d}}} \boldsymbol {\varepsilon} |
$$
$$
W _ {3} := \varepsilon^ {\top} \mathbf {P} _ {\overline {{\mathbf {X}}} _ {\mathrm {n a d}}} \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {a d}}} \mathbf {P} _ {\overline {{\mathbf {X}}} _ {\mathrm {n a d}}} \varepsilon
$$
$$
W _ {4} := | \varepsilon^ {\top} \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}} \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {a d}}} \varepsilon |.
$$
We next bound  $W_{i}(i = 1,2,3,4)$  respectively.
For  $W_{1}$  and  $W_{2}$ , we have from Lemma A.1 and equation (14f) that
$$
W _ {1} \leq \| (\mathbf {P} _ {\overline {{\mathbf {X}}} _ {\mathrm {n a d}}} - \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}}) \boldsymbol {\varepsilon} \| _ {2} ^ {2} \leq C
$$
$$
W _ {2} \leq \| \pmb {\varepsilon} ^ {\top} (\mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}} - \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}}) \| _ {2} \| \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {a d}}} \pmb {\varepsilon} \| _ {2} \leq C \sqrt {\log (n \det (\widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}}) / \delta)}.
$$
For  $W_{3}$ , similar to the proof of claim (15b), we have
$$
\begin{array}{l} W _ {3} = \varepsilon^ {\top} \widetilde {\mathbf {P}} _ {\overline {{\mathbf {X}}} _ {\mathrm {n a d}}} (\overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}}) ^ {- 1 / 2} \overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {a d}}} \overline {{\mathbf {X}}} _ {\mathrm {n a d}} (\overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}}) ^ {- 1 / 2} \widetilde {\mathbf {P}} _ {\overline {{\mathbf {X}}} _ {\mathrm {n a d}}} \varepsilon \\ \leq \frac {(d - k) \max  _ {k + 1 \leq j \leq d} \bar {\mathbf {x}} _ {j} ^ {\top} \mathbf {P} _ {\tilde {\mathbf {X}} _ {\mathrm {a d}}} \bar {\mathbf {x}} _ {j}}{\sigma_ {\min } (\bar {\mathbf {X}} _ {\mathrm {n a d}} ^ {\top} \bar {\mathbf {X}} _ {\mathrm {n a d}})} \| \widetilde {\mathbf {P}} _ {\bar {\mathbf {X}} _ {\mathrm {n a d}}} \boldsymbol {\varepsilon} \| _ {2} ^ {2} \\ \leq \frac {C (d - k) ^ {2}}{n} \log (n \det (\widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}}) / \delta) \log (n / \delta) \\ \leq C \log (n \det (\widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}}) / \delta), \\ \end{array}
$$
where third line follows from Lemma A.1 and the last line follows from the sample size assumption (6). Likewise,
$$
\begin{array}{l} W _ {4} = | \boldsymbol {\varepsilon} ^ {\top} \mathbf {P} _ {\overline {{\mathbf {X}}} _ {\mathrm {n a d}}} \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {a d}}} \boldsymbol {\varepsilon} | \\ \leq \| \tilde {\mathbf {P}} _ {\overline {{\mathbf {X}}} _ {\mathrm {n a d}}} \varepsilon \| _ {2} \| (\overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}}) ^ {- 1 / 2} \| _ {\mathrm {o p}} \| \overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \mathbf {P} _ {\tilde {\mathbf {X}} _ {\mathrm {a d}}} \varepsilon \| _ {2} \\ \leq \frac {c \sqrt {d - k}}{\sigma_ {\min } (\overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}}) ^ {1 / 2}} \max  _ {k + 1 \leq j \leq d} \| \overline {{\mathbf {x}}} _ {j} ^ {\top} \widetilde {\mathbf {P}} _ {\widetilde {\mathbf {X}} _ {\mathrm {a d}}} \| _ {2} \| \widetilde {\mathbf {P}} _ {\widetilde {\mathbf {X}} _ {\mathrm {a d}}} \boldsymbol {\varepsilon} \| _ {2} \| \widetilde {\mathbf {P}} _ {\overline {{\mathbf {X}}} _ {\mathrm {n a d}}} \boldsymbol {\varepsilon} \| _ {2} \\ \leq \frac {C (d - k)}{\sqrt {n}} \log (n \det (\widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}}) / \delta) \sqrt {\log (n / \delta)} \\ \leq C \log (n \det (\widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}}) / \delta). \\ \end{array}
$$
where the third inequality follows from Lemma A.1 again. Putting pieces together yields the desired result.
Proof of claim (18c) This is a direct consequence of Lemma A.1.
A.5 Proof of Corollary 3.3
W.l.o.g. assume  $\ell = 1$ . By Schur decomposition, we have
$$
\begin{array}{l} \left(\widetilde {\boldsymbol {\theta}} _ {\mathrm {a d}} - \boldsymbol {\theta} _ {\mathrm {a d}} ^ {*}\right) ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}} \left(\widetilde {\boldsymbol {\theta}} _ {\mathrm {a d}} - \boldsymbol {\theta} _ {\mathrm {a d}} ^ {*}\right) \\ = (\widetilde {\boldsymbol {\theta}} _ {\mathrm {a d}} - \boldsymbol {\theta} _ {\mathrm {a d}} ^ {*}) ^ {\top} \left( \begin{array}{c c} 1 & \widetilde {\mathbf {x}} _ {1} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}, - 1} \left(\widetilde {\mathbf {X}} _ {\mathrm {a d}, - 1} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}, - 1}\right) ^ {- 1} \\ 0 & \mathbf {I} _ {k - 1} \end{array} \right) \left( \begin{array}{c c} \widetilde {\mathbf {x}} _ {1} ^ {\top} (\mathbf {I} _ {n} - \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {a d}, - 1}}) \widetilde {\mathbf {x}} _ {1} & \mathbf {0} ^ {\top} \\ \mathbf {0} & \widetilde {\mathbf {X}} _ {\mathrm {a d}, - 1} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}, - 1} \end{array} \right) \\ \cdot \left( \begin{array}{c c} 1 & 0 \\ \left(\widetilde {\mathbf {X}} _ {\mathrm {a d}, - 1} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}, - 1}\right) ^ {- 1} \widetilde {\mathbf {X}} _ {\mathrm {a d}, - 1} ^ {\top} \widetilde {\mathbf {x}} _ {1} & \mathbf {I} _ {k - 1} \end{array} \right) (\widetilde {\boldsymbol {\theta}} _ {\mathrm {a d}} - \boldsymbol {\theta} _ {\mathrm {a d}} ^ {*}) \\ = \left( \begin{array}{c c} \widetilde {\theta} _ {\mathrm {a d}, 1} - \theta_ {\mathrm {a d}, 1} ^ {*} & \star \end{array} \right) \left( \begin{array}{c c} \widetilde {\mathbf {x}} _ {1} ^ {\top} (\mathbf {I} _ {n} - \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {a d}, - 1}}) \widetilde {\mathbf {x}} _ {1} & \mathbf {0} ^ {\top} \\ \mathbf {0} & \widetilde {\mathbf {X}} _ {\mathrm {a d}, - 1} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}, - 1} \end{array} \right) \left( \begin{array}{c} \widetilde {\theta} _ {\mathrm {a d}, 1} - \theta_ {\mathrm {a d}, 1} ^ {*} \\ \star \end{array} \right) \\ \geq (\widetilde {\boldsymbol {\theta}} _ {\mathrm {a d}, 1} - \boldsymbol {\theta} _ {\mathrm {a d}, 1} ^ {*}) ^ {2} (\widetilde {\mathbf {x}} _ {1} ^ {\top} (\mathbf {I} _ {n} - \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {a d}, - 1}}) \widetilde {\mathbf {x}} _ {1}). \\ \end{array}
$$
Therefore by Lemma A.4, the sample size assumption (6), and Theorem 3.2, we establish
$$
\begin{array}{l} \left(\widetilde {\boldsymbol {\theta}} _ {\mathrm {a d}, 1} - \boldsymbol {\theta} _ {\mathrm {a d}, 1} ^ {*}\right) ^ {2} (\widetilde {\mathbf {x}} _ {1} ^ {\top} (\mathbf {I} _ {n} - \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {a d}, - 1}}) \widetilde {\mathbf {x}} _ {1}) \\ \leq (\widetilde {\pmb {\theta}} _ {\mathrm {a d}} - \pmb {\theta} _ {\mathrm {a d}} ^ {*}) ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}} (\widetilde {\pmb {\theta}} _ {\mathrm {a d}} - \pmb {\theta} _ {\mathrm {a d}} ^ {*}) \leq C (\widetilde {\pmb {\theta}} _ {\mathrm {a d}} - \pmb {\theta} _ {\mathrm {a d}} ^ {*}) ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} (\mathbf {I} _ {n} - \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}}) \widetilde {\mathbf {X}} _ {\mathrm {a d}} (\widetilde {\pmb {\theta}} _ {\mathrm {a d}} - \pmb {\theta} _ {\mathrm {a d}} ^ {*}) \\ \leq C \log (n \det (\widetilde {\mathbf {X}} _ {\mathrm {a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {a d}}) / \delta) \leq C k \log (n / \delta). \\ \end{array}
$$
Corollary 3.3 follows immediately from the fact that  $\mathbf{I}_n - \mathbf{P}_{\widetilde{\mathbf{X}}_{-1}} \preceq \mathbf{I}_n - \mathbf{P}_{\widetilde{\mathbf{X}}_{\mathrm{ad}, - 1}}$  since  $\widetilde{\mathbf{X}}_{\mathrm{ad}, - 1}$  consists of some columns of  $\widetilde{\mathbf{X}}_{-1}$ .
A.6 Proof of Theorem 3.4
We start with the following decomposition
$$
\sum_ {i = 1} ^ {n} w _ {i} x _ {i} ^ {\mathrm {a d}} \Big (\widehat {\theta} _ {\mathrm {T A L E}} - \theta_ {\mathrm {a d}} ^ {*} \Big) = \underbrace {\sum_ {i = 1} ^ {n} w _ {i} \epsilon_ {i}} _ {v _ {n}} + \underbrace {\sum_ {i = 1} ^ {n} w _ {i} \pmb {x} _ {i} ^ {\mathrm {n a d} \top} (\pmb {\theta} _ {\mathrm {n a d}} ^ {*} - \widehat {\pmb {\theta}} _ {\mathrm {n a d}} ^ {\mathrm {P r}})} _ {b _ {n}}.
$$
To prove the theorem, it suffices to show
$$
v _ {n} \xrightarrow {d} \mathcal {N} (0, \alpha \cdot \sigma^ {2}) \qquad \text {a n d} \qquad b _ {n} \xrightarrow {p} 0,
$$
where  $\xrightarrow{d}$  stands for convergence in distribution and  $\alpha$  is a constant that is specified in equation (21).
Proof of  $v_{n} \xrightarrow{d} \mathcal{N}(0, \alpha \cdot \sigma^{2})$ : The proof of this part directly follows from [39, Theorem 3.1]. For completeness, we provide a proof here. Note that function  $f$  is a positive decreasing function and satisfies properties
$$
\int_ {1} ^ {\infty} f (x) d x = \infty \quad \text {a n d} \quad \int_ {1} ^ {\infty} f ^ {2} (x) d x = \alpha . \tag {21}
$$
Furthermore, it can be shown that
$$
\max  _ {1 \leq i \leq n} f ^ {2} \left(\frac {s _ {i}}{s _ {0}}\right) \frac {\left(x _ {i} ^ {\mathrm {a d}}\right) ^ {2}}{s _ {0}} = o _ {p} (1) \quad \text {a n d} \quad \max  _ {1 \leq i \leq n} \left(1 - \frac {f \left(s _ {i} / s _ {0}\right)}{f \left(s _ {i - 1} / s _ {0}\right)}\right) = o _ {p} (1). \tag {22}
$$
Next we compute
$$
\begin{array}{l} \sum_ {i = 1} ^ {n} w _ {i} ^ {2} = \sum_ {i = 1} ^ {n} f ^ {2} (s _ {i} / s _ {0}) \frac {(x _ {i} ^ {\mathrm {a d}}) ^ {2}}{s _ {0}} = \int_ {1} ^ {s _ {n} / s _ {0}} f ^ {2} (x) d x \cdot \frac {\sum_ {t \leq n} f ^ {2} (s _ {i} / s _ {0}) (x _ {i} ^ {\mathrm {a d}}) ^ {2} / s _ {0}}{\int_ {1} ^ {s _ {n} / s _ {0}} f ^ {2} (x) d x} \\ \stackrel {(i)} {=} \int_ {1} ^ {s _ {n} / s _ {0}} f ^ {2} (x) d x \cdot (1 + \frac {\sum_ {i \leq n} (\frac {f ^ {2} (s _ {i} / s _ {0})}{f ^ {2} (\xi_ {i} / s _ {0})} - 1) f ^ {2} (\xi_ {i} / s _ {0}) (x _ {i} ^ {\mathrm {a d}}) ^ {2} / s _ {0}}{\sum_ {i \leq n} f ^ {2} (\xi_ {i} / s _ {0}) (x _ {i} ^ {\mathrm {a d}}) ^ {2} / s _ {0}}). \\ \end{array}
$$
In equation (i), we consider the mean value theorem where  $\int_{s_{i - 1} / s_0}^{s_i / s_0}f^2 (x)dx = f^2 (\xi_i / s_0)(x_i^{\mathrm{ad}})^2 /s_0$ . Consequently, we have
$$
\begin{array}{l} \frac {\sum_ {i \leq n} | \frac {f ^ {2} (s _ {i} / s _ {0})}{f ^ {2} (\xi_ {i} / s _ {0})} - 1 | f ^ {2} (\xi_ {i} / s _ {0}) (x _ {i} ^ {\mathrm {a d}}) ^ {2} / s _ {0}}{\sum_ {i \leq n} f (\xi_ {i} / s _ {0}) (x _ {i} ^ {\mathrm {a d}}) ^ {2} / s _ {0}} \leq \frac {\sum_ {i \leq n} | \frac {f ^ {2} (s _ {i} / s _ {0})}{f ^ {2} (s _ {i - 1} / s _ {0})} - 1 | f ^ {2} (\xi_ {i} / s _ {0}) (x _ {i} ^ {\mathrm {a d}}) ^ {2} / s _ {0}}{\sum_ {i \leq n} f ^ {2} (\xi_ {i} / s _ {0}) (x _ {i} ^ {\mathrm {a d}}) ^ {2} / s _ {0}} \\ \leq \max _ {i \leq n} \left(1 - \frac {f ^ {2} (s _ {i} / s _ {0})}{f ^ {2} (s _ {i - 1} / s _ {0})}\right) = o _ {p} (1). \\ \end{array}
$$
$$
\sum_ {i = 1} ^ {n} w _ {i} ^ {2} = \left(1 + o _ {p} (1)\right) \int_ {1} ^ {s _ {n} / s _ {0}} f ^ {2} (x) d x = \alpha + o _ {p} (1) \tag {23}
$$
By noticing  $\max_{1\leq i\leq n}w_i^2 = \max_{1\leq i\leq n}f^2 (s_i / s_0)(x_i^{\mathrm{ad}})^2 /s_0^2 = o_p(1)$ , we conclude from martingale central limit theorem that
$$
\sum_ {i = 1} ^ {n} w _ {i} \varepsilon_ {i} \xrightarrow {d} \mathcal {N} (0, \alpha \cdot \sigma^ {2}).
$$
Moreover, applying Slutsky's theorem yields
$$
\frac {1}{\widehat {\sigma} \left(\sum_ {1 \leq i \leq n} w _ {i} ^ {2}\right) ^ {1 / 2}} \left(\sum_ {i = 1} ^ {n} w _ {i} \varepsilon_ {i}\right) \xrightarrow {d} \mathcal {N} (0, 1). \tag {24}
$$
Proof of  $b_n \xrightarrow{p} 0$ : To simplify notations, let  $\boldsymbol{w} = (w_1, \ldots, w_n)^\top$ . Without loss of generality, we consider the first column of the design matrix is collected adaptively. By the definition of  $b_n$ , we observe that
$$
\begin{array}{l} \left| \sum_ {i = 1} ^ {n} w _ {i} \boldsymbol {x} _ {i} ^ {\mathrm {n a d} \top} \left(\widehat {\boldsymbol {\theta}} _ {\mathrm {n a d}} ^ {\mathrm {P r}} - \boldsymbol {\theta} _ {\mathrm {n a d}} ^ {*}\right) \right| \leq \| \sum_ {i = 1} ^ {n} w _ {i} \boldsymbol {x} _ {i} ^ {\mathrm {n a d}} \| _ {2} \cdot \| \widehat {\boldsymbol {\theta}} _ {\mathrm {n a d}} ^ {\mathrm {P r}} - \boldsymbol {\theta} _ {\mathrm {n a d}} ^ {*} \| _ {2} \\ = \underbrace {\sqrt {\sum_ {i = 2} ^ {d} \left(\boldsymbol {w} ^ {\top} \mathbf {x} _ {i} ^ {\text {n a d}}\right) ^ {2}}} _ {\triangleq b _ {n, 1}} \cdot \underbrace {\| \widehat {\boldsymbol {\theta}} _ {\text {n a d}} ^ {\Pr} - \boldsymbol {\theta} _ {\text {n a d}} ^ {*} \| _ {2}} _ {\triangleq b _ {n, 2}} \tag {25} \\ \end{array}
$$
Analysis of  $b_{n,1}$ : By the construction of the weights  $\{w_i\}_{1 \leq i \leq n}$ , we have
$$
\left\| \boldsymbol {w} \right\| _ {2} ^ {2} \leq \int_ {1} ^ {\infty} f ^ {2} (x) d x = \alpha . \tag {26}
$$
Applying Lemma A.1 with  $\mathbf{A} = \mathbf{w}$  and  $b = \mathbf{x}_i^{\mathrm{rad}}$ , we conclude that with probability at least  $1 - \delta$
$$
\left(\boldsymbol {w} ^ {\top} \mathbf {x} _ {i} ^ {\text {n a d}}\right) ^ {2} \leq c \nu^ {2} \| \boldsymbol {w} \| _ {2} ^ {2} \log \left(\| \boldsymbol {w} \| _ {2} ^ {2} / \delta\right) \leq c \nu^ {2} \alpha \log (\alpha / \delta), \tag {27}
$$
where  $c$  is a universal constant. Therefore, with probability at least  $1 - \delta$ ,
$$
b _ {n, 1} \leq \sqrt {d} \cdot \sqrt {c \nu^ {2} \alpha \log (d \alpha / \delta)}. \tag {28}
$$
Analysis of  $b_{n,2}$ : note  $\widehat{\pmb{\theta}}_{\mathrm{nad}}^{\mathrm{Pr}} = \widehat{\pmb{\theta}}_{\mathrm{nad}}^{\mathrm{ols}}$  is the OLS estimate. Therefore, we can use block-wise matrix inverse formula to get its expression. Precisely, we have
$$
\begin{array}{l} \widehat {\boldsymbol {\theta}} _ {\mathrm {n a d}} ^ {\mathrm {P r}} - \boldsymbol {\theta} _ {\mathrm {n a d}} ^ {*} = - \frac {\left(\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}}\right) ^ {- 1} \mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {x} _ {1} \mathbf {x} _ {1} ^ {\top} \boldsymbol {\varepsilon}}{\| \mathbf {x} _ {1} - \mathbf {P} _ {\mathbf {X} _ {\mathrm {n a d}}} \mathbf {x} _ {1} \| _ {2} ^ {2}} + \left(\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}}\right) ^ {- 1} \mathbf {X} _ {\mathrm {n a d}} ^ {\top} \boldsymbol {\varepsilon} \\ + \frac {\left(\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}}\right) ^ {- 1} \mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {x} _ {1} \mathbf {x} _ {1} ^ {\top} \mathbf {X} _ {\mathrm {n a d}} \left(\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}}\right) ^ {- 1} \mathbf {X} _ {\mathrm {n a d}} ^ {\top} \varepsilon}{\| \mathbf {x} _ {1} - \mathbf {P} _ {\mathbf {X} _ {\mathrm {n a d}}} \mathbf {x} _ {1} \| _ {2} ^ {2}}. \tag {29} \\ \end{array}
$$
Therefore, we can upper bound  $b_{n,2}$  by
$$
\begin{array}{l} b _ {n, 2} \leq \underbrace {\frac {1}{\| \mathbf {x} _ {1} - \mathbf {P} _ {\mathbf {X} _ {\mathrm {n a d}}} \mathbf {x} _ {1} \| _ {2} ^ {2}} \| (\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}}) ^ {- 1} \| _ {\mathrm {o p}} \cdot \| \mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {x} _ {1} \| _ {2} \cdot \| \mathbf {x} _ {1} ^ {\top} \boldsymbol {\varepsilon} \| _ {2}} _ {:= b _ {n, 2} ^ {(1)}} \\ + \underbrace {\left\| \left(\mathbf {X} _ {\text {n a d}} ^ {\top} \mathbf {X} _ {\text {n a d}}\right) ^ {- 1} \mathbf {X} _ {\text {n a d}} ^ {\top} \boldsymbol {\varepsilon} \right\| _ {2}} _ {:= b _ {n, 2} ^ {(2)}} \tag {30} \\ + \underbrace {\frac {1}{\| \mathbf {x} _ {1} - \mathbf {P} _ {\mathbf {X} _ {\mathrm {n a d}}} \mathbf {x} _ {1} \| _ {2} ^ {2}} \| (\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}}) ^ {- 1} \| _ {\mathrm {o p}} \cdot \| \mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {x} _ {1} \| _ {2} ^ {2} \cdot \| (\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}}) ^ {- 1} \mathbf {X} _ {\mathrm {n a d}} ^ {\top} \boldsymbol {\varepsilon} \| _ {2}} _ {:= b _ {n, 2} ^ {(3)}}. \\ \end{array}
$$
To analyze terms  $b_{n,2}^{(1)}, b_{n,2}^{(2)}$ , and  $b_{n,2}^{(3)}$ , we make use of the results from Lemma A.1, Lemma A.2, and Lemma A.3. Specifically, we have with probability  $1 - \delta$ , the following statements hold
$$
\begin{array}{l} b _ {n, 2} ^ {(1)} = \frac {1}{\| \mathbf {x} _ {1} - \mathbf {P} _ {\mathbf {X} _ {\mathrm {n a d}}} \mathbf {x} _ {1} \| _ {2} ^ {2}} \cdot \| \big (\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}} \big) ^ {- 1} \| _ {\mathrm {o p}} \cdot \| \mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {x} _ {1} \| _ {2} \cdot \| \mathbf {x} _ {1} ^ {\top} \boldsymbol {\varepsilon} \| _ {2} \\ \stackrel {(i)} {\leq} c ^ {\prime} \left(\frac {1}{(1 - C d \log (n / \delta) / n) \cdot \| \mathbf {x} _ {1} \| _ {2} ^ {2}}\right) \cdot \frac {1}{n} \cdot \| \mathbf {x} _ {1} \| _ {2} \sqrt {d} \sqrt {\log (d \| \mathbf {x} _ {1} \| _ {2} ^ {2} / \delta)} \\ \cdot \| \mathbf {x} _ {1} \| _ {2} \sqrt {\log (\| \mathbf {x} _ {1} \| _ {2} ^ {2} / \delta)} \\ \leq \frac {c ^ {\prime} \sqrt {d} \log (n ^ {2} d / \delta)}{(1 - C d \log (n / \delta) / n) n} \\ \end{array}
$$
$$
\begin{array}{l} b _ {n, 2} ^ {(2)} = \left\| \left(\mathbf {X} _ {\text {n a d}} ^ {\top} \mathbf {X} _ {\text {n a d}}\right) ^ {- 1} \mathbf {X} _ {\text {n a d}} ^ {\top} \boldsymbol {\varepsilon} \right\| _ {2} \leq \left\| \left(\mathbf {X} _ {\text {n a d}} ^ {\top} \mathbf {X} _ {\text {n a d}}\right) ^ {- 1} \right\| _ {\text {o p}} \cdot \left\| \mathbf {X} _ {\text {n a d}} ^ {\top} \boldsymbol {\varepsilon} \right\| _ {2} \tag {31} \\ \stackrel {(i i)} {\leq} c ^ {\prime \prime} \sqrt {\frac {d \log (d / \delta)}{n}} \\ \end{array}
$$
$$
\begin{array}{l} b _ {n, 2} ^ {(3)} = \frac {1}{\| \mathbf {x} _ {1} - \mathbf {P} _ {\mathbf {X} _ {\mathrm {n a d}}} \mathbf {x} _ {1} \| _ {2} ^ {2}} \| (\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}}) ^ {- 1} \| _ {\mathrm {o p}} \cdot \| \mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {x} _ {1} \| _ {2} ^ {2} \cdot \| (\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}}) ^ {- 1} \mathbf {X} _ {\mathrm {n a d}} ^ {\top} \boldsymbol {\varepsilon} \| _ {2} \\ \stackrel {(i i i)} {\leq} \frac {1}{(1 - C d \log (n / \delta) / n) \cdot \| \mathbf {x} _ {1} \| _ {2} ^ {2}} \cdot \frac {c ^ {\prime \prime \prime}}{n} \cdot d \| \mathbf {x} _ {1} \| _ {2} ^ {2} \log (d \| \mathbf {x} _ {1} \| _ {2} ^ {2} / \delta) \cdot \sqrt {\frac {d \log (d / \delta)}{n}} \\ \leq c ^ {\prime \prime \prime} \frac {1}{(1 - C d \log (n / \delta) / n)} \frac {d ^ {3 / 2} \{\log (d n ^ {2} / \delta) \} ^ {3 / 2}}{n ^ {3 / 2}}, \\ \end{array}
$$
where  $c'$ ,  $c''$  and  $c'''$  are universal constants that are independent of  $n$  and  $d$ . In inequality  $(i)$ , we use Lemma A.3 to obtain a lower bound for  $\| \mathbf{x}_1 - \mathbf{P}_{\mathbf{X}_{\mathrm{rad}}}\mathbf{x}_1\|_2^2$  and apply Lemma A.1 and Lemma A.2 to control the other three terms separately. Inequality  $(ii)$  makes use of the fact that  $\pmb{x}_{ij}\varepsilon_i$  is sub-exponential with parameter  $(cvv, cvv)$  conditioned on  $\mathcal{F}_{i-1}$  for  $j = 2, \ldots, d$ . Therefore, by Azuma-Bernstein inequality and the sample size assumption, we obtain for  $2 \leq j \leq d$ ,
$$
| \mathbf {x} _ {j} ^ {\top} \boldsymbol {\varepsilon} | \leq \nu v \sqrt {\log (d / \delta)} \left(\sqrt {n} \vee \sqrt {\log (d / \delta)}\right) = \nu v \sqrt {n \log (d / \delta)}
$$
with probability over  $1 - \delta /d$  for any  $2\leq j\leq d$  . Applying a union bound to  $j = 2,\ldots ,d$  , we have
$$
\left\| \mathbf {X} _ {\text {n a d}} ^ {\top} \boldsymbol {\varepsilon} \right\| _ {2} \leq C ^ {\prime} n d \log (d / \delta), \tag {32}
$$
for some constant  $C'$ . Inequality (iii) makes use of the bound for  $b_{n,2}^{(2)}$  and Lemmas A.1 and A.2. Therefore, when  $d^2 \log^2(n) / n \to 0$ , we conclude
$$
b _ {n, 1} \cdot b _ {n, 2} = o _ {p} (1). \tag {33}
$$
With  $b_n \xrightarrow{p} 0$  at hand, a direct application of Slutsky's theorem yields