SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {1}{\widehat {\sigma} \sqrt {\sum_ {1 \leq i \leq n} w _ {i} ^ {2}}} \left(\sum_ {i = 1} ^ {n} w _ {i} \boldsymbol {x} _ {i} ^ {\text {n a d} \top}\right) \cdot \left(\widehat {\boldsymbol {\theta}} _ {\text {n a d}} ^ {\text {P r}} - \boldsymbol {\theta} _ {\text {n a d}} ^ {*}\right) \xrightarrow {p} 0. \tag {34}
$$
Putting things together, we conclude that
$$
\frac {1}{\widehat {\sigma} \sqrt {\sum_ {1 \leq i \leq n} w _ {i} ^ {2}}} \left(\sum_ {i = 1} ^ {n} w _ {i} x _ {i} ^ {\mathrm {a d}}\right) \cdot \left(\widehat {\theta} _ {\mathrm {T A L E}} - \theta_ {1} ^ {*}\right) \xrightarrow {d} \mathcal {N} (0, 1). \tag {35}
$$
A.7 Proof of Lemma A.1
Proof of Part (a) The proof follows immediately from choosing  $V = \mathbf{I}_d$  in Theorem 1 of Abbasi et al. [1].
Proof of Part (b) Define  $\bar{\mathbf{S}}_{\mathbf{1}} := \widetilde{\mathbf{A}}^{\top} \widetilde{\mathbf{A}}$  and  $\mathbf{A}_{\mathrm{aug}} := [\mathbf{A} \quad \mathbf{1}_n]$ . Then
$$
\begin{array}{l} \mathbf {b} ^ {\top} \mathbf {P} _ {\widetilde {\mathbf {A}}} \mathbf {b} = \mathbf {b} ^ {\top} \widetilde {\mathbf {A}} \bar {\mathbf {S}} _ {1} ^ {- 1} \bar {\mathbf {S}} _ {1} \bar {\mathbf {S}} _ {1} ^ {- 1} \widetilde {\mathbf {A}} ^ {\top} \mathbf {b} \\ = \mathbf {b} ^ {\top} \mathbf {A} _ {\mathrm {a u g}} (\mathbf {A} _ {\mathrm {a u g}} ^ {\top} \mathbf {A} _ {\mathrm {a u g}}) ^ {- 1} \left( \begin{array}{c} \mathbf {I} _ {k} \\ 0 \end{array} \right) (\mathbf {I} _ {k} \quad 0)   \overline {{\mathbf {S}}} _ {\mathbf {1}} \left( \begin{array}{c} \mathbf {I} _ {k} \\ 0 \end{array} \right) (\mathbf {I} _ {k} \quad 0)   (\mathbf {A} _ {\mathrm {a u g}} ^ {\top} \mathbf {A} _ {\mathrm {a u g}}) ^ {- 1} \mathbf {A} _ {\mathrm {a u g}} ^ {\top} \mathbf {b} \\ \leq \mathbf {b} ^ {\top} \mathbf {A} _ {\mathrm {a u g}} (\mathbf {A} _ {\mathrm {a u g}} ^ {\top} \mathbf {A} _ {\mathrm {a u g}}) ^ {- 1} (\mathbf {A} _ {\mathrm {a u g}} ^ {\top} \mathbf {A} _ {\mathrm {a u g}}) (\mathbf {A} _ {\mathrm {a u g}} ^ {\top} \mathbf {A} _ {\mathrm {a u g}}) ^ {- 1} \mathbf {A} _ {\mathrm {a u g}} ^ {\top} \mathbf {b} \\ = \mathbf {b} ^ {\top} \mathbf {A} _ {\mathrm {a u g}} (\mathbf {A} _ {\mathrm {a u g}} ^ {\top} \mathbf {A} _ {\mathrm {a u g}}) ^ {- 1} \mathbf {A} _ {\mathrm {a u g}} ^ {\top} \mathbf {b} \\ \leq c \sigma^ {2} \log (\det  (\mathbf {A} _ {\mathrm {a u g}} ^ {\top} \mathbf {A} _ {\mathrm {a u g}}) / \delta) \\ = c \sigma^ {2} \log (n \det (\widetilde {\mathbf {A}} ^ {\top} \widetilde {\mathbf {A}}) / \delta) \\ \end{array}
$$
for all  $k + 1 \leq j \leq d$  with probability over  $1 - \delta$ . Here the first equality comes from the definition of  $\mathbf{P}$ ; the second equality uses the fact that the coefficients of  $\mathbf{A}$  in the ordinary least squares (OLS) estimator for the linear model  $\mathbf{b} \sim \mathbf{A} + \mathbf{1}$  equals the OLS estimator for the centered linear model  $\mathbf{b} \sim \widetilde{\mathbf{A}}$ , i.e.,
$$
\left( \begin{array}{c} \mathbf {I} _ {k} \\ 0 \end{array} \right) (\mathbf {I} _ {k} \quad 0) (\mathbf {A} _ {\mathrm {a u g}} ^ {\top} \mathbf {A} _ {\mathrm {a u g}}) ^ {- 1} \mathbf {A} _ {\mathrm {a u g}} ^ {\top} \mathbf {b} = \bar {\mathbf {S}} _ {1} ^ {- 1} (\mathbf {A} - \mathbf {P} _ {1 _ {n}} \mathbf {A}) ^ {\top} \mathbf {b};
$$
the third line is due to the fact that  $\left[ \begin{array}{cc}\overline{\mathbf{S}}_1 & \mathbf{0}_k\\ \mathbf{0}_k^\top & 0 \end{array} \right]\preceq \mathbf{A}_{\mathrm{aug}}^\top \mathbf{A}_{\mathrm{aug}}$  ; the fifth line follows from Lemma A.1 and the last line exploits the Schur complement of  $\widetilde{\mathbf{A}}^{\top}\widetilde{\mathbf{A}}$
A.8 Proof of Lemma A.2
Let  $\pmb{\mu}^{*} = (\pmb{\mu}_{\mathrm{ad}}^{*\top},\pmb{\mu}_{\mathrm{nad}}^{*\top})^{\top}$  denote the mean vector of  $\mathbb{E}[\pmb{x}_i]$  and define
$$
\overline {{\mathbf {X}}} _ {\mathrm {n a d}} := \mathbf {X} _ {\mathrm {n a d}} - \mathbf {1} _ {n} \boldsymbol {\mu} _ {\mathrm {n a d}} ^ {* ^ {\top}}.
$$
Proof of (14a) and (14b). Applying the concentration result for a sample covariance matrix of sub-Gaussian ensemble (see e.g., Theorem 6.5 in Wainwright [37]), we obtain
$$
\| \overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}} - n \boldsymbol {\Sigma} \| _ {\mathrm {o p}} \leq c \nu^ {2} n \Big (\sqrt {\frac {d - k + \log (1 / \delta)}{n}} + \frac {d - k + \log (1 / \delta)}{n} \Big)
$$
with probability over  $1 - \delta$ . Using Weyl's theorem (see e.g., Theorem 4.3.1 in Horn et al. [18]), the sample size assumption and the last display, we find that
$$
\sigma_ {\min } \left(\overline {{\mathbf {X}}} _ {\text {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\text {n a d}}\right) \geq n \sigma_ {\min } (\boldsymbol {\Sigma}) - \| \overline {{\mathbf {X}}} _ {\text {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\text {n a d}} - n \boldsymbol {\Sigma} \| _ {\text {o p}} \geq \frac {n \sigma_ {\min }}{2}.
$$
Similarly, we have  $\| \overline{\mathbf{X}}_{\mathrm{nad}}\|_{\mathrm{op}} = (\sigma_{\mathrm{max}}(\overline{\mathbf{X}}_{\mathrm{nad}}^{\top}\overline{\mathbf{X}}_{\mathrm{nad}}))^{1 / 2}\leq \sqrt{n(\sigma_{\mathrm{max}} + \sigma_{\mathrm{min}} / 2)}\leq \sqrt{2n\sigma_{\mathrm{max}}}$  This concludes (14a), (14b).
Proof of (14c). Recall that we use  $\hat{\pmb{\mu}} = (\hat{\pmb{\mu}}_{\mathrm{ad}}^{*\top},\hat{\pmb{\mu}}_{\mathrm{nad}}^{*\top})^{\top}$  to denote the empirical average of  $\{\pmb {x}_i\}_{i = 1}^n$  From Assumption (A2) and properties of sub-Gaussian vectors, we have with probability over  $1 - \delta$
$$
\left\| \hat {\boldsymbol {\mu}} _ {\mathrm {n a d}} - \boldsymbol {\mu} _ {\mathrm {n a d}} ^ {*} \right\| _ {2} \leq c \nu \sqrt {\log ((d - k) / \delta)} \sqrt {\frac {d - k}{n}}. \tag {36}
$$
It follows immediately that
$$
\| \overline {{\mathbf {X}}} _ {\mathrm {n a d}} - \widetilde {\mathbf {X}} _ {\mathrm {n a d}} \| _ {\mathrm {o p}} = \| \mathbf {1} _ {n} (\hat {\boldsymbol {\mu}} _ {\mathrm {n a d}} - \boldsymbol {\mu} _ {\mathrm {n a d}} ^ {*}) ^ {\top} \| _ {\mathrm {o p}} = \| \hat {\boldsymbol {\mu}} _ {\mathrm {n a d}} - \boldsymbol {\mu} _ {\mathrm {n a d}} ^ {*} \| _ {2} \| \mathbf {1} _ {n} \| _ {2} \leq C \sqrt {\log (n / \delta)} \sqrt {d - k}.
$$
The second inequality in (14c) follows from equation (14a) and the sample size assumption 6.
Proof of equation (14d) and (14e). By Woodbury's matrix identity, we have
$$
\begin{array}{l} \| \left(\overline {{\mathbf {X}}} _ {\text {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\text {n a d}}\right) ^ {- 1} - \left(\widetilde {\mathbf {X}} _ {\text {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\text {n a d}}\right) ^ {- 1} \| \mathrm {o p} \\ \leq \| (\overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}}) ^ {- 1} \| _ {\mathrm {o p}} \| \overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}} - \widetilde {\mathbf {X}} _ {\mathrm {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {n a d}} \| _ {\mathrm {o p}} \| (\widetilde {\mathbf {X}} _ {\mathrm {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {n a d}}) ^ {- 1} \| _ {\mathrm {o p}}. \\ \end{array}
$$
By equation (14a), (14b), (14c) and a standard triangular inequality, we find
$$
\| \overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}} - \widetilde {\mathbf {X}} _ {\mathrm {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {n a d}} \| _ {\mathrm {o p}} \leq C \sqrt {\log (n / \delta)} \sqrt {(d - k) n} \leq \frac {n \sigma_ {\min}}{4} \leq \frac {1}{2} \sigma_ {\min} (\overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}}),
$$
where the second inequality follows from the sample size assumption (6). Therefore, we have
$$
\| (\widetilde {\mathbf {X}} _ {\mathrm {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {n a d}}) ^ {- 1} \| _ {\mathrm {o p}} \leq \frac {2}{\sigma_ {\min } (\overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}})}
$$
$$
\begin{array}{l} \| (\overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}}) ^ {- 1} - (\widetilde {\mathbf {X}} _ {\mathrm {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {n a d}}) ^ {- 1} \| _ {\mathrm {o p}} \leq \frac {C}{\sigma_ {\min} (\overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}}) ^ {2}} \| \overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}} - \widetilde {\mathbf {X}} _ {\mathrm {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {n a d}} \| _ {\mathrm {o p}} \\ \leq \frac {C \sqrt {\log (n / \delta)} \sqrt {d - k}}{n ^ {3 / 2}}. \tag {37} \\ \end{array}
$$
This gives equation (14d). Moreover, note that
$$
\begin{array}{l} \| \left. \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}} - \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}} \right\| \text {o p} \\ \leq \| \widetilde {\mathbf {X}} _ {\mathrm {n a d}} [ (\overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}}) ^ {- 1} - (\widetilde {\mathbf {X}} _ {\mathrm {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {n a d}}) ^ {- 1} ] \widetilde {\mathbf {X}} _ {\mathrm {n a d}} ^ {\top} \| \mathrm {o p} \\ + \left\| \left(\overline {{\mathbf {X}}} _ {\mathrm {n a d}} - \widetilde {\mathbf {X}} _ {\mathrm {n a d}}\right) \left(\overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}}\right) ^ {- 1} \left(\overline {{\mathbf {X}}} _ {\mathrm {n a d}} - \widetilde {\mathbf {X}} _ {\mathrm {n a d}}\right) ^ {\top} \right\| \mathrm {o p} \\ + 2 \| \left(\overline {{\mathbf {X}}} _ {\text {n a d}} - \widetilde {\mathbf {X}} _ {\text {n a d}}\right) ^ {\top} \left(\overline {{\mathbf {X}}} _ {\text {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\text {n a d}}\right) ^ {- 1} \overline {{\mathbf {X}}} _ {\text {n a d}} ^ {\top} \| \mathrm {o p} \\ \leq \| \widetilde {\mathbf {X}} _ {\text {n a d}} \| _ {\text {o p}} ^ {2} \| (\overline {{\mathbf {X}}} _ {\text {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\text {n a d}}) ^ {- 1} - (\widetilde {\mathbf {X}} _ {\text {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\text {n a d}}) ^ {- 1} \| _ {\text {o p}} \\ + \| \left\| \overline {{\mathbf {X}}} _ {\text {n a d}} - \widetilde {\mathbf {X}} _ {\text {n a d}} \right\|\left\| \operatorname {o p} \left(\left\|\left\| \overline {{\mathbf {X}}} _ {\text {n a d}} - \widetilde {\mathbf {X}} _ {\text {n a d}} \right\|\right\| \operatorname {o p} + 2 \left\|\left\| \overline {{\mathbf {X}}} _ {\text {n a d}} \right\|\right\| \operatorname {o p}\right)\right\|\left(\overline {{\mathbf {X}}} _ {\text {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\text {n a d}}\right) ^ {- 1} \| \left. \right\| _ {\mathrm {o p}}. \\ \end{array}
$$
It follows immediately from equation (14a), (14b), (14c), (14d) and (37) that with probability over  $1 - \delta$
$$
\| \mathbf {P} _ {\tilde {\mathbf {X}} _ {\mathrm {n a d}}} - \mathbf {P} _ {\tilde {\mathbf {X}} _ {\mathrm {n a d}}} \| _ {\mathrm {o p}} \leq \frac {C \sqrt {\log (n / \delta)} \sqrt {d - k}}{\sqrt {n}}.
$$
This yields equation (14e).
Proof of equation (14f). Define  $\pmb{\Delta} \coloneqq \hat{\pmb{\mu}} - \pmb{\mu}^{*}$ . We have
$$
\begin{array}{l} \left\| \left(\mathbf {P} _ {\tilde {\mathbf {X}} _ {\mathrm {n a d}}} - \mathbf {P} _ {\tilde {\mathbf {X}} _ {\mathrm {n a d}}}\right) \boldsymbol {\varepsilon} \right\| _ {2} \\ \leq \| \widetilde {\mathbf {X}} _ {\mathrm {n a d}} (\widetilde {\mathbf {X}} _ {\mathrm {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {n a d}}) ^ {- 1} \boldsymbol {\Delta} \mathbf {1} _ {n} ^ {\top} \boldsymbol {\varepsilon} \| _ {2} + \| (\widetilde {\mathbf {X}} _ {\mathrm {n a d}} (\widetilde {\mathbf {X}} _ {\mathrm {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {n a d}}) ^ {- 1} - \overline {{\mathbf {X}}} _ {\mathrm {n a d}} (\overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}}) ^ {- 1}) \overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \boldsymbol {\varepsilon} \| _ {2} \\ \leq \| \widetilde {\mathbf {X}} _ {\mathrm {n a d}} (\widetilde {\mathbf {X}} _ {\mathrm {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {n a d}}) ^ {- 1} \| _ {\mathrm {o p}} \| \boldsymbol {\Delta} \| _ {2} | \mathbf {1} _ {n} ^ {\top} \boldsymbol {\varepsilon} | + \| \widetilde {\mathbf {X}} _ {\mathrm {n a d}} (\widetilde {\mathbf {X}} _ {\mathrm {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {n a d}}) ^ {- 1} \\ - \overline {{\mathbf {X}}} _ {\mathrm {n a d}} (\overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}}) ^ {- 1} \| | _ {\mathrm {o p}} \| | | \overline {{\mathbf {X}}} _ {\mathrm {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\mathrm {n a d}} \| | _ {\mathrm {o p}} ^ {1 / 2} \| | \widetilde {\mathbf {P}} _ {\overline {{\mathbf {X}}} _ {\mathrm {n a d}}} \varepsilon \| | _ {2}. \\ \end{array}
$$
Since  $|\mathbf{1}_n^\top \pmb {\varepsilon}|\leq cv\sqrt{n\log(1 / \delta)}$  with probability over  $1 - \delta$  by Assumption (A4) and concentration of sub-Gaussian variables, and
$$
\begin{array}{l} \| \widetilde {\mathbf {X}} _ {\text {n a d}} \left(\widetilde {\mathbf {X}} _ {\text {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\text {n a d}}\right) ^ {- 1} - \overline {{\mathbf {X}}} _ {\text {n a d}} \left(\overline {{\mathbf {X}}} _ {\text {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\text {n a d}}\right) ^ {- 1} \| \mathrm {o p} \\ \leq \| \widetilde {\mathbf {X}} _ {\mathrm {n a d}} - \overline {{\mathbf {X}}} _ {\mathrm {n a d}} \| _ {\mathrm {o p}} \| (\widetilde {\mathbf {X}} _ {\mathrm {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\mathrm {n a d}}) ^ {- 1} \| _ {\mathrm {o p}} \\ + \| \left\| \overline {{\mathbf {X}}} _ {\text {n a d}} \right\| \left\| \operatorname {o p} \right\| \left(\widetilde {\mathbf {X}} _ {\text {n a d}} ^ {\top} \widetilde {\mathbf {X}} _ {\text {n a d}}\right) ^ {- 1} - \left(\overline {{\mathbf {X}}} _ {\text {n a d}} ^ {\top} \overline {{\mathbf {X}}} _ {\text {n a d}}\right) ^ {- 1} \| \left\| \operatorname {o p} \right. \\ \end{array}
$$
by the triangular inequality, it follows immediately from combining equation (14a), (14b), (14c), (14d), (36), (37) that
$$
\left\| \left(\mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}} - \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {n a d}}}\right) \boldsymbol {\varepsilon} \right\| _ {2} \leq C \left[ \sqrt {\frac {(d - k) \log (n / \delta) \log (1 / \delta)}{n}} + \sqrt {\frac {(d - k) ^ {2} \log^ {2} (n / \delta)}{n}} \right] \leq C
$$
with probability over  $1 - \delta$
A.9 Proof of Lemma A.3
Denote  $(\mathbf{X}_{\mathrm{ad}}^{\top}\mathbf{X}_{\mathrm{ad}})^{-1 / 2}\mathbf{X}_{\mathrm{ad}}^{\top}\mathbf{P}_{\mathbf{X}_{\mathrm{nad}}}\mathbf{X}_{\mathrm{ad}}(\mathbf{X}_{\mathrm{ad}}^{\top}\mathbf{X}_{\mathrm{ad}})^{-1 / 2}$  by B. By our construction and noting that  $\| \mathbf{B}\|_{\mathrm{op}}\leq \operatorname {tr}(\mathbf{B})$  , it suffices to show
$$
\operatorname {t r} (\mathbf {B}) \leq \frac {C (d - k) k \log (n / \delta)}{n}
$$
with probability over  $1 - \delta$  for some  $C > 0$ . Plugging in the definition of  $\mathbf{P}_{\mathbf{X}_{\mathrm{rad}}}$ , we obtain
$$
\begin{array}{l} \operatorname {t r} (\mathbf {B}) = \operatorname {t r} \left(\left(\mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}}\right) ^ {- 1 / 2} \mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}} \left(\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}}\right) ^ {- 1} \mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}} \left(\mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}}\right) ^ {- 1 / 2}\right) \\ = \operatorname {t r} \left(\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}} \left(\mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}}\right) ^ {- 1} \mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}} \left(\mathbf {X} _ {\mathrm {n a d}} ^ {\top} \mathbf {X} _ {\mathrm {n a d}}\right) ^ {- 1}\right) \\ \leq \operatorname {t r} \left(\mathbf {X} _ {\text {n a d}} ^ {\top} \mathbf {X} _ {\text {a d}} \left(\mathbf {X} _ {\text {a d}} ^ {\top} \mathbf {X} _ {\text {a d}}\right) ^ {- 1} \mathbf {X} _ {\text {a d}} ^ {\top} \mathbf {X} _ {\text {n a d}}\right) \cdot \| \left(\mathbf {X} _ {\text {n a d}} ^ {\top} \mathbf {X} _ {\text {n a d}}\right) ^ {- 1} \| _ {\mathrm {o p}} \\ \leq \frac {c}{n \sigma_ {\operatorname* {m i n}}} \operatorname {t r} \left(\mathbf {X} _ {\text {n a d}} ^ {\top} \mathbf {X} _ {\text {a d}} \left(\mathbf {X} _ {\text {a d}} ^ {\top} \mathbf {X} _ {\text {a d}}\right) ^ {- 1} \mathbf {X} _ {\text {a d}} ^ {\top} \mathbf {X} _ {\text {n a d}}\right) \tag {38} \\ \end{array}
$$
with probability over  $1 - \delta$ , where the third line follows from von Neumann's trace inequality (see e.g., Theorem A.15 in Bai et al. [3]), and the last line uses equation (14a). Write  $\mathbf{X}_{\mathrm{ad}} = [\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_k] \in \mathbb{R}^{n \times k}$  and  $\mathbf{X}_{\mathrm{nad}} = [\mathbf{x}_{k+1}, \mathbf{x}_{k+2}, \ldots, \mathbf{x}_d] \in \mathbb{R}^{n \times (d-k)}$ . Following the calculation, we further have
$$
\operatorname {t r} \left(\mathbf {X} _ {\text {n a d}} ^ {\top} \mathbf {X} _ {\text {a d}} \left(\mathbf {X} _ {\text {a d}} ^ {\top} \mathbf {X} _ {\text {a d}}\right) ^ {- 1} \mathbf {X} _ {\text {a d}} ^ {\top} \mathbf {X} _ {\text {n a d}}\right) = \sum_ {j = k + 1} ^ {d} \mathbf {x} _ {j} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\text {a d}}} \mathbf {x} _ {j}. \tag {39}
$$
It follows from Lemma A.1 that
$$
\mathbf {x} _ {j} ^ {\top} \mathbf {P} _ {\mathbf {X} _ {\mathrm {a d}}} \mathbf {x} _ {j} \leq C k \log (n / \delta) \tag {40}
$$
for all  $k + 1 \leq j \leq d$  with probability over  $1 - \delta$ . The desired result follows immediately from combining equation (40) with (38) and (39).
A.10 Proof of Lemma A.4
Following the same arguments as in the proof of Lemma A.3 with  $\mathbf{X}_{\mathrm{ad}}$  replaced by  $\widetilde{\mathbf{X}}_{\mathrm{ad}} := \mathbf{X}_{\mathrm{ad}} - \mathbf{P}_{\mathbf{1}_n}\mathbf{X}_{\mathrm{ad}}$ , it suffices to show
$$
\mathbf {x} _ {j} ^ {\top} \mathbf {P} _ {\widetilde {\mathbf {X}} _ {\mathrm {a d}}} \mathbf {x} _ {j} \leq C k \log (n / \delta). \tag {41}
$$
for all  $k + 1 \leq j \leq d$  with probability over  $1 - \delta$ . This follows immediately from Lemma A.1 (b).
B Simulation set up
We provide implementation details of our simulations in this section. The code is available at https://github.com/licong-lin/low-dim-debias.
B.1 Single coordinate estimation
In this section, we detail the simulation set up that was used to generate the Figure 1. The goal of this simulation is to display the effect of degree of adaptivity  $k$  on the estimation of a single coordinate, and provide empirical validation to the theory developed in the paper (c.f. Corollary 3.3).
We want to design an adaptive data collection mechanism, which can capture the estimation lower bound. Therefore, we adopt a similar data collection procedure as provided in Khamaru et al. [20]. We also refer readers to Lattimore [25] for related information.
Simulation set-up:
- Sample size  $n = {1000},d = {300}$  .
- The degree of adaptivity  $k$  varies from 2 to 200 with step size equal to 3.
- Replication number 20 for each level of adaptivity  $(k, d)$ .
-  $\theta_1^{\mathrm{ad}} = 1$  and other coefficients are generated independently from  $\mathcal{N}(0,1)$ .
-  $\pmb{x}_i^{\mathrm{rad}}$  is generated independently from uniform distribution on the sphere  $S^{d - k - 1}$ . If  $\pmb{x}_i^{\mathrm{rad}}$  has mean not equal to zero, then we consider  $\pmb{x}_i^{\mathrm{rad}}$  plus  $\mathbb{E}\pmb{x}_i^{\mathrm{rad}}$ , where  $\mathbb{E}\pmb{x}_i^{\mathrm{rad}}$  is generated from  $\mathcal{N}(\mathbf{1},\mathbf{I}_{d - k})$ .
Data collection method: Here we modified the data collection algorithm from Section 5.2.2 in Khamaru et al. [20]. The only difference between our data collection algorithm and the one in Khamaru et al. [20] is that we replace  $m_{u,v} \coloneqq \sum_{w=1}^{v} b_w (y_{u,w} - a_{u,w})$  by
$$
m _ {u, v} := \sum_ {w = 1} ^ {v} b _ {w} \left(y _ {u, w} - a _ {u, w} - \boldsymbol {\theta} ^ {\text {n a d} \top} \boldsymbol {x} _ {u + (w - 1) (d - 1)} ^ {\text {n a d}}\right). \tag {42}
$$
Figure 1 shows that empirical relation between the MSE of the centered OLS estimate of the first coordinate and the degree of dependence  $k$ .
B.2 Single coordinate inference
In this section, we detail the simulation set up that is used to generate the Figure 2 and 3.
We generate a dataset  $\{(x_i, y_i)\}_{i=1}^n$  that satisfies the assumptions in Theorem 3.4. On this simulated dataset, we compare our method with the ordinary least squares (OLS) estimator, W-decorrelation proposed by Deshpande [13], and the non-asymptotic confidence intervals derived from Theorem 8 in Lattimore et al. [26].
We begin by describing our data generating mechanism. We assume the data  $\{(\pmb{x}_i, y_i)\}_{i=1}^n \in \mathbb{R}^d \times \mathbb{R}$  are generated from a linear model  $y_i = \pmb{x}_i^\top \pmb{\theta}^* + \varepsilon_i$ , where  $\varepsilon_i \sim \mathcal{N}(0, \sigma^2)$ . We generate the covariates  $\{\pmb{x}_i\}_{i=1}^n$  in the following way
1. We assume the non-adaptive component  $\pmb{x}_{i,2:d}$  are i.i.d  $\mathcal{N}(0,\mathbf{I}_{d - 1})$  across  $i\in [n]$ .
2. For the adaptive coordinate, we choose  $x_{1,1} = 1$  and assume  $x_{i,1} \in \{0,1\}$  for all  $i \in [n]$ .
3. At each stage  $i \geq 2$ , denote by  $\widehat{\theta}_{1}^{(i)}$  the OLS estimator for the first coordinate  $\theta_{1}^{*}$  obtained using the first  $i - 1$  samples  $(\mathbf{X}_{1:i - 1},\mathbf{y}_{1:i - 1})$ . With probability  $p$  we choose  $x_{i,1} = 1$  if  $\widehat{\theta}_{1}^{(i)} > 0$  and  $x_{i,1} = 0$  if otherwise; with probability  $1 - p$  we simply choose  $x_{i,1} = 1$  to encourage exploration.
Recalling Example 2.1 on treatment assignment, in the simulated data, we use the OLS estimator to obtain an prior estimate of the treatment effect  $\theta_{1}^{*}$  and assign the treatment to the  $i$ -th patient if the prior estimation suggests that the treatment has a positive effect (i.e.,  $\widehat{\theta}_{1}^{(i)} > 0$ ). Moreover, to encourage exploration, we assign the treatment (i.e.,  $x_{i,1} = 1$ ) with some small probability  $1 - p$  regardless of the prior estimation.
Throughout the simulation we choose  $\theta_{2:d}^{*} = \mathbf{1}_{d - 1} / \sqrt{d - 1}$  and  $\theta_1^* = 0$ , which corresponds to the case where no treatment effect is presented. We choose the noise level  $\sigma = 0.3$  and the probability  $p = 0.8$ . In the simulations we assume the noise level is known for simplicity. We run our simulations on both a low-dimensional model ( $n = 1000$ ,  $d = 10$ ) and a high-dimensional model ( $n = 500$ ,  $d = 50$ ).
Comparison with W-decorrelation by Deshpande et al. [13] In our implementation of W-decorrelation, we follow Algorithm 1 in [13], with the parameter  $\lambda \cdot \log (n)$  be the  $1 / n$ -quantile of  $\sigma_{\mathrm{min}}(\mathbf{X}^\top \mathbf{X})$ . To estimate the quantile, we use the sample estimate from 1000 i.i.d. data matrices  $\mathbf{X}$ 's to estimate the quantile.