SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ITE methods. We discuss two distinct approaches for ITE methods. (i) Baseline: The ideal scenario or "oracle" baseline involves utilizing a linear function, trained on vector  $\mathbf{t}$ . Naturally, this is not feasible in practical applications. (ii) leverage score based: This is based on learning two different vectors by ridge leverage score sampling and using the difference of the two linear functions for estimating  $\mathbf{t}$ . This is an adaptation of the algorithm in [1] to the full observation setting — we make this adaptation because the exact algorithm of [1] cannot be applied to the full-observation setting due to sampling with replacement. We do not compare to [28] because their guarantees are only for an infinite population.
Synthetic Datasets. For synthetic datasets, we will construct  $\mathbf{y}^{(0)},\mathbf{y}^{(1)}$  such that  $\pmb{\mu}$  (for the ATE case) or t (for the ITE case) is a linear function of the covariate matrix.
1. ATE Dataset. There are 50 covariates (i.e., features). Each entry of  $\mathbf{X} \in \mathbb{R}^{n \times d}$  is a uniform random number in  $[0, 0.01]$ . Each entry of  $\mathbf{b} \in \mathbb{R}^{50}$  is a uniform random number in  $[0, 1]$ . The individual treatment effect vector  $\pmb{\mu}$  is  $\mathbf{Xb} + \mathbf{r}$ , where each entry of  $\mathbf{r} \in \mathbb{R}^n$  is randomly picked from a mean zero Gaussian distribution with standard deviation of 0.2. Each entry of  $\mathbf{y}^{(0)}$  is picked uniformly at random from  $[0, 5]$ . Then  $\mathbf{y}^{(1)} = -\mathbf{y}^{(0)} + \pmb{\mu}$  and  $\mathbf{t} = \mathbf{y}^{(1)} - \mathbf{y}^{(0)}$ . The results for this dataset with different numbers of samples are shown in the right plot of Figure 1. Our experiments illustrate that on the synthetic dataset, our approach outperforms all other approaches except the GSW design. However, we note that our approach is computationally more efficient than GSW design (see Table 6) and gives a much simpler design that can be used in different settings.
2. ITE Dataset. The covariate matrix  $\mathbf{X}$  and the vectors  $\mathbf{b},\mathbf{r},\mathbf{y}^{(0)}$  are picked similar to the ATE dataset. Then individual treatment effect vector  $\mathbf{t}$  is  $\mathbf{X}\mathbf{b} + \mathbf{r}$  and  $\mathbf{y}^{(1)} = \mathbf{y}^{(0)} + \mathbf{t}$ . The results for this dataset with different number of samples is shown in the left plot of Figure 1. ITE error for our method is consistently smaller than leverage score based method.
Real-World Datasets We analyze the following three distinct real-world datasets. A comprehensive breakdown, including variance and other measurements for each dataset, is available in Table 5 in the appendix.
1. Boston Dataset [17]: This is a dataset of housing prices in the Boston area, consisting of 506 samples and 13 features. Since it has only one label, we set  $\mathbf{y}^{(1)} = \mathbf{y}^{(0)}$ , i.e., ATE is zero.
2. IHDP Dataset [16, 11]: Derived from the characteristics of children and their mothers, this dataset comprises 747 samples and 25 features. ATE for is -4.016.
3. Twins Dataset [3]: This dataset is constructed based on the characteristics and mortality rates of twin births in the US. We specifically selected samples that have complete feature values, resulting in a dataset of 32,120 samples with 50 features. The ATE of this dataset is 0.0064.
In this paper, we considered the problems of mean estimation, ATE and ITE estimations in the presence of covariates. We considered the finite population setting and provided non-asymptotic variance bounds for several novel variants of the classical regression adjustment method-based estimators. Our guarantees are model-free, even if the covariates are arbitrary (are not informative of the estimand and may be chosen adversarily), the variance bounds are still meaningful and almost match the variance bounds of widely used estimators not based on covariates.
Our algorithms are simple and efficient, and we believe they readily extend to many related settings like arbitrary assignment probabilities and online treatment assignments. We also believe the results can be stated for the kernel setting. These extensions will be part of future work.
[1] Raghavendra Addanki, David Arbour, Tung Mai, Cameron N Musco, and Anup Rao. Sample constrained treatment effect estimation. In Advances in Neural Information Processing Systems, 2022.
[2] Ahmed Alaoui and Michael W Mahoney. Fast randomized kernel ridge regression with statistical guarantees. Advances in neural information processing systems, 28, 2015.
[3] Douglas Almond, Kenneth Y Chay, and David S Lee. The costs of low birth weight. The Quarterly Journal of Economics, 120(3):1031-1083, 2005.
[4] David Arbour, Drew Dimmery, Tung Mai, and Anup Rao. Online balanced experimental design. In International Conference on Machine Learning, pages 844-864. PMLR, 2022.
[5] David Arbour, Drew Dimmery, and Anup Rao. Efficient balanced treatment assignments for experimentation. In International Conference on Artificial Intelligence and Statistics, pages 3070-3078. PMLR, 2021.
[6] Haim Avron, Kenneth L Clarkson, and David P Woodruff. Sharper bounds for regularized data fitting. Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, 2017.
[7] Adam Bloniarz, Hanzhong Liu, Cun-Hui Zhang, Jasjeet S Sekhon, and Bin Yu. Lasso adjustments of treatment effect estimates in randomized experiments. Proceedings of the National Academy of Sciences, 113(27):7383-7390, 2016.
[8] Xue Chen and Eric Price. Active regression via linear-sample sparsification. In Conference on Learning Theory, pages 663-695. PMLR, 2019.
[9] William G Cochran. Sampling techniques. John Wiley & Sons, 1977.
[10] Michael B Cohen, Yin Tat Lee, Cameron Musco, Christopher Musco, Richard Peng, and Aaron Sidford. Uniform sampling for matrix approximation. In Proceedings of the 2015 Conference on Innovations in Theoretical Computer Science, pages 181-190, 2015.
[11] Vincent Dorie. NPCI: non-parametrics for causal inference. https://github.com/vdorie/npci, 2016.
[12] Matthew Fahrbach, Gang Fu, and Mehrdad Ghadiri. Subquadratic kronecker regression with applications to tensor decomposition. In Advances in Neural Information Processing Systems, 2022.
[13] David A Freedman. On regression adjustments to experimental data. Advances in Applied Mathematics, 40(2):180-193, 2008.
[14] Alan S Gerber and Donald P Green. Field experiments and natural experiments, 2008.
[15] Robert Greevy, Bo Lu, Jeffrey H Silber, and Paul Rosenbaum. Optimal multivariate matching before randomization. *Biostatistics*, 5(2):263-275, 2004.
[16] Ruth T Gross et al. Infant health and development program (ihdp): Enhancing the outcomes of low birth weight, premature infants in the united states, 1985-1988, 1993.
[17] David Harrison Jr and Daniel L Rubinfeld. Hedonic housing prices and the demand for clean air. Journal of environmental economics and management, 5(1):81-102, 1978.
[18] Christopher Harshaw, Fredrik Savje, Daniel Spielman, and Peng Zhang. Balancing covariates in randomized experiments with the gram-schmidt walk design. arXiv preprint arXiv:1911.03071, 2019.
[19] Daniel G Horvitz and Donovan J Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American statistical Association, 47(260):663-685, 1952.
[20] Andrew Jesson, Panagiotis Tigas, Joost van Amersfoort, Andreas Kirsch, Uri Shalit, and Yarin Gal. Causal-bald: Deep bayesian active learning of outcomes to infer treatment-effects from observational data. Advances in Neural Information Processing Systems, 34:30465-30478, 2021.
[21] Nathan Kallus. Optimal a priori balance in the design of controlled experiments. Journal of the Royal Statistical Society. Series B (Statistical Methodology), 80(1):85-112, 2018.
[22] Edward H Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. arXiv preprint arXiv:2004.14497, 2020.
[23] Xinran Li and Peng Ding. Rerandomization and regression adjustment. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(1):241-268, 2020.
[24] Winston Lin. Agnostic notes on regression adjustments to experimental data: Reexamining freedman's critique. THE ANNALS OF APPLIED STATISTICS, pages 295-318, 2013.
[25] Ping Ma, Michael Mahoney, and Bin Yu. A statistical perspective on algorithmic leveraging. In International Conference on Machine Learning, pages 91-99. PMLR, 2014.
[26] Michael W Mahoney et al. Randomized algorithms for matrices and data. Foundations and Trends® in Machine Learning, 3(2):123-224, 2011.
[27] Kari Lock Morgan and Donald B Rubin. *Rerandomization to improve covariate balance in experiments.* The Annals of Statistics, pages 1263–1282, 2012.
[28] Wenlong Mou, Martin J Wainwright, and Peter L Bartlett. Off-policy estimation of linear functionals: Non-asymptotic theory for semi-parametric efficiency. arXiv preprint arXiv:2209.13075, 2022.
[29] Akanksha Negi and Jeffrey M Wooldridge. Revisiting regression adjustment in experiments with heterogeneous treatment effects. *Econometric Reviews*, 40(5):504–534, 2021.
[30] Tian Qin, Tian-Zuo Wang, and Zhi-Hua Zhou. Budgeted heterogeneous treatment effect estimation. In International Conference on Machine Learning, pages 8693-8702. PMLR, 2021.
[31] Mark Rudelson and Roman Vershynin. Hanson-wright inequality and sub-gaussian concentration. Electronic Communications in Probability, 18:1-9, 2013.
[32] Iiris Sundin, Peter Schulam, Eero Siivola, Aki Vehtari, Suchi Saria, and Samuel Kaski. Active learning for decision-making from imbalanced observational data. In International conference on machine learning, pages 6046-6055. PMLR, 2019.
[33] Stefan Wager, Wenfei Du, Jonathan Taylor, and Robert J Tibshirani. High-dimensional regression adjustments in randomized experiments. Proceedings of the National Academy of Sciences, 113(45):12673-12678, 2016.
[34] Martin B Wilk. The randomization analysis of a generalized randomized block design. Biometrika, 42(1/2):70-79, 1955.
[35] David P Woodruff et al. Sketching as a tool for numerical linear algebra. Foundations and Trends® in Theoretical Computer Science, 10(1-2):1-157, 2014.
A Warm-up: Mean Estimation
In this section, we consider the problem of estimating the mean of a single vector  $\mathbf{y}$  by observing only a few entries (i.e.,  $\overline{y} := \frac{1}{n} \sum_{i=1}^{n} y_i$ ). The most classic approach is to simply uniformly sample  $m$  units from the population and report their average. This results in a variance of at most  $\frac{1}{mn} \left\| \mathbf{y} - \overline{\mathbf{y}} \right\|_2^2$ , where  $\overline{\mathbf{y}}$  is a vector of the same size as  $\mathbf{y}$  with all of its entries equal to  $\overline{y}$ .
The main theme of regression adjustment is to replace the dependence of variance on  $\| \mathbf{y} - \overline{\mathbf{y}}\| _2^2$  with a dependence on the error of the best linear fit, i.e.,  $\min_{\mathbf{b}\in \mathbb{R}^d}\left\| \mathbf{y} - \overline{\mathbf{y}} -(\mathbf{X} - \overline{\mathbf{X}})\mathbf{b}\right\| _2^2$ , where  $\overline{\mathbf{X}}$  is a vector of the same size as  $\mathbf{X}$  with all of its rows equal to  $\overline{\mathbf{x}} = \frac{1}{n}\sum_{i = 1}^{n}\mathbf{x}_i$ . We wish to do so without making any modeling assumptions — a common modeling assumption in statistics is that the response vector is a linear function of the covariates plus a Gaussian noise. However, our goal is to achieve significant variance reduction if  $\mathbf{y} - \overline{\mathbf{y}}$  can be well approximated by a vector in the span of  $\mathbf{X} - \overline{\mathbf{X}}$  without any such modeling assumptions. In the worst case, when  $\mathbf{y} - \overline{\mathbf{y}}$  is orthogonal to the column span of  $(\mathbf{X} - \overline{\mathbf{X}})$ ,  $\min_{\mathbf{b}\in \mathbb{R}^d}\left\| \mathbf{y} - \overline{\mathbf{y}} -(\mathbf{X} - \overline{\mathbf{X}})\mathbf{b}\right\| _2^2 = \| \mathbf{y} - \overline{\mathbf{y}}\| _2^2$  which recovers the result of the classic approach.
For the mean estimation problem, if an oracle provided  $\mathbf{b}^{*} = \arg \min_{\mathbf{b}\in \mathbb{R}^{d}}\left\| \mathbf{y} - \overline{\mathbf{y}} -(\mathbf{X} - \overline{\mathbf{X}})\mathbf{b}\right\|_{2}^{2}$ , the estimator  $\widehat{y} = \frac{1}{|S|}\sum_{i\in S}y_i - (\mathbf{x}_i - \overline{\mathbf{x}})^\top \mathbf{b}^*$  is unbiased with its variance bounded by  $\frac{1}{mn}\min_{\mathbf{b}\in \mathbb{R}^d}\left\| \mathbf{y} - \overline{\mathbf{y}} -(\mathbf{X} - \overline{\mathbf{X}})\mathbf{b}\right\|_2^2$ , where  $S$  is a uniformly sampled subset of the population of size  $m$ . This is a classic result (see Chapter 7 of [9]) that also follows from the following Lemma.
Lemma 9 (9 - Chapter 2). Let  $\mathbf{y} \in \mathbb{R}^n$  and  $\widehat{y}$  be the average of a uniformly sampled set of entries of  $\mathbf{y}$  of size  $m$ . Then
$$
\mathbb {E} \left[ (\widehat {y} - \overline {{y}}) ^ {2} \right] = \frac {\| \mathbf {y} - \overline {{\mathbf {y}}} \| _ {2} ^ {2}}{m (n - 1)} (1 - \frac {m}{n}),
$$
where  $\overline{\mathbf{y}}$  is a vector of size equal to  $\mathbf{y}$  for which all the entries are equal to  $\overline{y}$ .
One can easily see that by applying Lemma 9 to  $\tilde{\mathbf{y}}$ , where  $\tilde{\mathbf{y}}_i = \mathbf{y}_i - (\mathbf{x}_i - \overline{\mathbf{x}})^{\top}\mathbf{b}$ , a variance of  $\frac{\|\mathbf{y} - \overline{\mathbf{y}} - (\mathbf{X} - \overline{\mathbf{X}})\mathbf{b}\|_2^2}{m(n - 1)} (1 - \frac{m}{n})$  can be obtained.
Key challenge. The main challenge for the mean estimation problem is to compute  $\mathbf{b}^*$  (or a vector close to it) without observing all the entries of vector  $\mathbf{y}$ . Note that we do not have access to any entry of  $\mathbf{y} - \overline{\mathbf{y}}$  because we cannot compute  $\overline{\mathbf{y}}$  without observing all the entries of  $\mathbf{y}$ .
We resolve the issue of not being able to compute  $\overline{y}$  by performing the regression on a modified matrix. We add a column of all ones to the matrix  $\mathbf{X} - \overline{\mathbf{X}}$  and perform a regression on  $\mathbf{y}$  using the modified matrix — see the proof of Theorem 10. More specifically, we prove the following.
$$
\min  _ {\mathbf {b} \in \mathbb {R} ^ {d}} \left\| (\mathbf {X} - \overline {{\mathbf {X}}}) \mathbf {b} - (\mathbf {y} - \overline {{\mathbf {y}}}) \right\| _ {2} ^ {2} = \min  _ {\mathbf {b} \in \mathbb {R} ^ {d + 1}} \left\| \left[ (\mathbf {X} - \overline {{\mathbf {X}}}) \quad \mathbf {1} \right] \mathbf {b} - \mathbf {y} \right\| _ {2} ^ {2}
$$
To solve the regression problem on the right-hand side without observing all the entries of  $\mathbf{y}$ , we use leverage score sampling [25, 10]. Leverage score sampling is a numerical linear algebraic tool that allows for solving a linear regression problem approximately by subsampling about  $d$  units from the population and solving a regression problem over these units, where  $d$  is the number of features [8] — see Section 3 for details. Combining these techniques gives the following results.
Theorem 10 (Leverage score sampling based regression adjustment). Let  $\epsilon, \delta \in (0,1]$ , and  $m \in \mathbb{N}$  and  $\mathbf{X} \in \mathbb{R}^{n \times d}$  be a covariate matrix. Then, Algorithm 2 computes an unbiased estimator of the mean of an outcome vector  $\mathbf{y}$  (on a population of size  $n$ ) with  $O(d\log(d / \delta) / \epsilon^2 + m)$  samples. Moreover, there is an event  $\mathcal{E}$  with  $\operatorname{Pr}(\mathcal{E}) \geq 1 - \delta$  (over the randomness of the algorithm) such that the variance of the estimator conditioned to  $\mathcal{E}$  is bounded by
$$
(1 + \epsilon) \frac {1}{m (n - 1)} \left(1 - \frac {m}{n}\right) \min  _ {\mathbf {b}} \| (\mathbf {X} - \overline {{\mathbf {X}}}) \mathbf {b} - (\mathbf {y} - \overline {{\mathbf {y}}}) \| _ {2} ^ {2}. \tag {2}
$$
Note that since the dependence of number of samples on  $\delta$  is logarithmic, we can pick  $\delta$  polynomially small, e.g.,  $\delta = 1 / n^2$ . The variance upper bound achieved by our method is compared to other methods in Table 3.
Regression adjustment. Our estimators are based on a classic technique in statistics called regression adjustment [9]. Given a vector  $\widehat{\mathbf{b}}$  (either given by an oracle or learned from the data), the classical technique uses the sample covariates to "adjust" the estimate of the sample mean  $\frac{1}{|S'|} \sum_{i \in S'} y_i$  and outputs the following estimate
$$
\frac {1}{| S ^ {\prime} |} \sum_ {i \in S ^ {\prime}} y _ {i} - (\mathbf {x} _ {i} - \overline {{\mathbf {x}}}) \widehat {\mathbf {b}}.
$$
If  $S'$  is a random sample independent of  $\widehat{\mathbf{b}}$ , the adjusted estimator is unbiased. The adjusted estimate will have a lower variance than the sample mean estimator if  $\| \mathbf{y} - \overline{\mathbf{y}} - (\mathbf{X} - \overline{\mathbf{X}}) \widehat{\mathbf{b}} \|_2^2 < \| \mathbf{y} - \overline{\mathbf{y}} \|_2^2$ .
Lemma 11. Let  $\mathbf{y} \in \mathbb{R}^n$  and  $\mathbf{X} \in \mathbb{R}^{n \times d}$ . Let  $\widehat{\mathbf{b}}$  be a fixed/preassigned vector. Then the variance of  $\widehat{y} := \frac{1}{|S'|} \sum_{i \in S'} y_i - (\mathbf{x}_i - \overline{\mathbf{x}})^{\top} \widehat{\mathbf{b}}$ , where  $S'$  is a uniformly sampled subset of  $[n]$  of size  $m$ , is
$$
\frac {1}{m (n - 1)} \left(1 - \frac {m}{n}\right) \left\| (\mathbf {y} - \overline {{\mathbf {y}}}) - (\mathbf {X} - \overline {{\mathbf {X}}}) \widehat {\mathbf {b}} \right\| _ {2} ^ {2}.
$$
Proof. Since  $\widehat{\mathbf{b}}$  is fixed, by linearity of expectation
$$
\mathbb {E} [ \widehat {y} ] = \mathbb {E} \left[ \frac {1}{| S ^ {\prime} |} \sum_ {i \in S ^ {\prime}} y _ {i} \right] + \mathbb {E} \left[ \frac {1}{| S ^ {\prime} |} \sum_ {i \in S ^ {\prime}} (\overline {{\mathbf {x}}} - \mathbf {x} _ {i}) ^ {\top} \widehat {\mathbf {b}} \right] = \bar {y} + (\overline {{\mathbf {x}}} - \frac {1}{n} \sum_ {i \in S ^ {\prime}} \mathbf {x} _ {i}) ^ {\top} \widehat {\mathbf {b}} = \bar {y}.
$$
Therefore by Lemma 9 (applied to the quantities  $y_{i} + (\overline{\mathbf{x}} -\mathbf{x}_{i})^{\top}\widehat{\mathbf{b}})$
$$
\mathbb {E} \left[ (\widehat {y} - \overline {{y}}) ^ {2} \right] = \frac {1}{m} \left(1 - \frac {m}{n}\right) \sum_ {i = 1} ^ {n} \frac {\left(y _ {i} - \overline {{y}} - \left(\mathbf {x} _ {i} - \overline {{\mathbf {x}}}\right) ^ {\top} \widehat {\mathbf {b}}\right) ^ {2}}{n - 1},
$$
Therefore our goal is to find  $\widehat{\mathbf{b}}$  that minimizes  $\| \mathbf{y} - \overline{\mathbf{y}} - (\mathbf{X} - \overline{\mathbf{X}}) \widehat{\mathbf{b}} \|_2^2$ . Our approach is to use leverage score sampling (Theorem 7) to find a vector  $\widehat{\mathbf{b}}$  that approximately minimizes this error. A main technique in our approach is that the samples picked for leverage score sampling are independent of the samples drawn for estimation. More specifically, our estimator is based on two sets of samples. We use one set of samples  $S$  to estimate  $\widehat{\mathbf{b}}$  and we use the another set  $S'$  of  $m$  uniform samples (that is independent of  $S$ ) in the following estimator  $\widehat{y} = \frac{1}{|S'|} \sum_{i \in S'} y_i + (\overline{\mathbf{x}} - \mathbf{x}_i)^{\top} \widehat{\mathbf{b}}$ . By Lemma 11, this is unbiased and achieves a variance of
$$
\frac {1}{m (n - 1)} \left(1 - \frac {m}{n}\right) \left\| (\mathbf {X} - \overline {{\mathbf {X}}}) \widehat {\mathbf {b}} - (\mathbf {y} - \overline {{\mathbf {y}}}) \right\| _ {2} ^ {2}.
$$
Although it is straightforward to carry this independence for the mean estimation problem, as we will see, we require a more involved algorithm to provide it for the treatment effect estimation problems.
Algorithm 2: (Ridge) leverage score sampling based regression adjustment
1 Input:  $\mathbf{X} \in \mathbb{R}^{n \times d}$ ,  $\mathbf{y} \in \mathbb{R}^n$ ,  $m \in \mathbb{N}$ ,  $0 < \epsilon < 1$ .
2 For Theorem 10, set  $\lambda = 0$ , and for Theorem 12, set  $\lambda = \zeta^2 \cdot n / m$  //  $\zeta = \max_i \| \mathbf{X}_{i:} \|_2$
3 Compute the leverage scores  $\ell_i^{(\lambda)}$  of  $\widetilde{\mathbf{X}} = \left[(\mathbf{X} - \overline{\mathbf{X}})\quad \mathbf{1}\right]$
4 Set  $p_i = \min \{1, 3 \cdot \ell_i^{(\lambda)} \cdot \log (d / \delta) / \epsilon^2\}$  for  $i \in [n]$ .
5 Let  $\mathbf{S} \in \mathbb{R}^{n \times n}$  be a random diagonal matrix with  $\mathbf{S}_{ii} = 1 / \sqrt{p_i}$  with probability  $p_i$ , and  $\mathbf{S}_{ii} = 0$ , otherwise.
6 Let  $\widetilde{\mathbf{b}} = \arg \min_{\mathbf{b} \in \mathbb{R}^{d+1}} \left\| \mathbf{S}\widetilde{\mathbf{X}}\mathbf{b} - \mathbf{S}\mathbf{y} \right\|_2^2 + \lambda \| \mathbf{b}_{1:d} \|_2^2$ , and  $\widehat{\mathbf{b}} \in \mathbb{R}^d$  be a vector consisting of the first  $d$  entries of  $\widetilde{\mathbf{b}}$ .
7 Let  $S^{\prime}$  be a uniformly sampled subset of size  $m$  of  $[n]$  and  $\widehat{\tau} = \frac{1}{m}\sum_{i\in S^{\prime}}y_{i} + (\overline{\mathbf{x}} -\mathbf{x}_{i})^{\top}\widehat{\mathbf{b}}$  8 Return  $\widehat{\tau}$
Proof. Let  $\widetilde{\mathbf{X}} \coloneqq \left[(\mathbf{X} - \overline{\mathbf{X}}) \quad \mathbf{1}\right]$ . We first show that
$$
\min  _ {\mathbf {b} \in \mathbb {R} ^ {d}} \left\| (\mathbf {X} - \overline {{\mathbf {X}}}) \mathbf {b} - (\mathbf {y} - \overline {{\mathbf {y}}}) \right\| _ {2} ^ {2} = \min  _ {\mathbf {b} \in \mathbb {R} ^ {d + 1}} \left\| \tilde {\mathbf {X}} \mathbf {b} - \mathbf {y} \right\| _ {2} ^ {2}. \tag {3}
$$
Note that  $\min_{\mathbf{b} \in \mathbb{R}^{d+1}} \left\| \widetilde{\mathbf{X}} \mathbf{b} - \mathbf{y} \right\|_2^2 = \left\| \pi_{\widetilde{\mathbf{X}}} \mathbf{y} - \mathbf{y} \right\|_2^2$ . Since  $\mathbf{1}$  is orthogonal to  $\mathbf{X} - \overline{\mathbf{X}}$ ,
$$
\pi_ {\widetilde {\mathbf {X}}} \mathbf {y} = \pi_ {(\mathbf {X} - \overline {{\mathbf {X}}})} \mathbf {y} + \pi_ {\mathbf {1}} \mathbf {y}.
$$
Moreover  $\pi_1\mathbf{y} = \overline{\mathbf{y}}$ . Since 1 is orthogonal to  $\mathbf{X} - \overline{\mathbf{X}}$ ,  $\overline{\mathbf{y}} = \overline{\mathbf{y}} \cdot \mathbf{1}$  is orthogonal to  $\mathbf{X} - \overline{\mathbf{X}}$ , and  $\pi_{(\mathbf{X} - \overline{\mathbf{X}})}\overline{\mathbf{y}} = 0$ . Therefore
$$
\pi_ {\widetilde {\mathbf {X}}} \mathbf {y} = \pi_ {(\mathbf {X} - \overline {{\mathbf {X}}})} (\mathbf {y} - \overline {{\mathbf {y}}}) + \overline {{\mathbf {y}}}.
$$
Subtracting both sides by  $\mathbf{y}$  and taking the norm, we have
$$
\left\| \pi_ {\widetilde {\mathbf {X}}} \mathbf {y} - \mathbf {y} \right\| _ {2} ^ {2} = \left\| \pi_ {(\mathbf {X} - \overline {{\mathbf {X}}})} (\mathbf {y} - \overline {{\mathbf {y}}}) - (\mathbf {y} - \overline {{\mathbf {y}}}) \right\| _ {2} ^ {2} = \min _ {\mathbf {b} \in \mathbb {R} ^ {d}} \left\| (\mathbf {X} - \overline {{\mathbf {X}}}) \mathbf {b} - (\mathbf {y} - \overline {{\mathbf {y}}}) \right\| _ {2} ^ {2},
$$
which implies our claim. Now note that by sampling  $O(d\log (d / \delta) / \epsilon^2)$  entries of  $\mathbf{y}$ , by Theorem 7, with high probability, we can find  $\widetilde{\mathbf{b}}$  such that
$$
\left\| \widetilde {\mathbf {X}} \widetilde {\mathbf {b}} - \mathbf {y} \right\| _ {2} ^ {2} \leq (1 + \epsilon) \min  _ {\mathbf {b} \in \mathbb {R} ^ {d + 1}} \left\| \widetilde {\mathbf {X}} \mathbf {b} - \mathbf {y} \right\| _ {2} ^ {2} = (1 + \epsilon) \min  _ {\mathbf {b} \in \mathbb {R} ^ {d}} \left\| (\mathbf {X} - \overline {{\mathbf {X}}}) \mathbf {b} - (\mathbf {y} - \overline {{\mathbf {y}}}) \right\| _ {2} ^ {2}. \tag {4}
$$
Let  $\widehat{\mathbf{b}}\in \mathbb{R}^d$  be a vector obtained by taking the first  $d$  entries of  $\widetilde{\mathbf{b}}$ . Then, we have
$$
\widetilde {\mathbf {X}} \widetilde {\mathbf {b}} - \mathbf {y} = (\mathbf {X} - \overline {{\mathbf {X}}}) \widehat {\mathbf {b}} - \mathbf {y} + \widetilde {b} _ {d + 1} \cdot \mathbf {1}.
$$
Now consider the following regression problem:
$$
\min  _ {t \in \mathbb {R}} \left\| \mathbf {1} \cdot t - (\mathbf {y} - (\mathbf {X} - \overline {{\mathbf {X}}}) \widehat {\mathbf {b}}) \right\| _ {2} ^ {2}.
$$