SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now let  $\mathcal{E}_1$  be the event that  $\mathbf{X}^{\top}\mathbf{S}^{\top}\mathbf{S}\mathbf{X}$  is not a spectral approximation of  $\mathbf{X}^{\top}\mathbf{X}$ . Let  $\mathcal{E}_2$  be the event that  $\mathbf{X}^{\top}\mathbf{S}^{\top}\mathbf{S}\mathbf{X}$  is a spectral approximation of  $\mathbf{X}^{\top}\mathbf{X}$ , but  $\| \mathbf{S}\mathbf{X}\mathbf{b} - \mathbf{S}\boldsymbol{\mu}\|_2^2 + \lambda \| \mathbf{b}\|_2^2 > (1 + \epsilon) \cdot (\| \mathbf{X}\mathbf{b} - \boldsymbol{\mu}\|_2^2 + \lambda \| \mathbf{b}\|_2^2)$ . Finally let  $\mathcal{E}_3$  be the event that  $\mathbf{X}^{\top}\mathbf{S}^{\top}\mathbf{S}\mathbf{X}$  is a spectral approximation of  $\mathbf{X}^{\top}\mathbf{X}$ , but  $\| \mathbf{S}\mathbf{X}\mathbf{b} - \mathbf{S}\boldsymbol{\mu}\|_2^2 + \lambda \| {\mathbf{b}}\|_2^2 \leq (1 + \epsilon) \cdot (\| \mathbf{X}\mathbf{b} - \boldsymbol{\mu}\|_2^2 + \lambda \| {\mathbf{b}}\|_2^2)$ . Then, by the law of total expectation,
$$
\mathbb {E} [ (\widehat {\tau} - \tau) ^ {2} ] = \mathbb {E} [ (\widehat {\tau} - \tau) ^ {2} | \mathcal {E} _ {1} ] \cdot \mathbb {P} [ \mathcal {E} _ {1} ] + \mathbb {E} [ (\widehat {\tau} - \tau) ^ {2} | \mathcal {E} _ {2} ] \cdot \mathbb {P} [ \mathcal {E} _ {2} ] + \mathbb {E} [ (\widehat {\tau} - \tau) ^ {2} | \mathcal {E} _ {3} ] \cdot \mathbb {P} [ \mathcal {E} _ {3} ]
$$
Note that  $\mathbb{P}[\mathcal{E}_1],\mathbb{P}[\mathcal{E}_2]\leq \delta$  and therefore by the above argument and Theorem 2,
$$
\begin{array}{l} \mathbb {E} \left[ (\widehat {\tau} - \tau) ^ {2} \right] \leq \delta \cdot \| \boldsymbol {\mu} \| _ {2} ^ {2} + \delta \cdot \mathbb {E} \left[ (\widehat {\tau} - \tau) ^ {2} \mid \mathcal {E} _ {2} \right] \tag {15} \\ + (1 - \delta) \left(\frac {8 (1 + \epsilon)}{n ^ {2}} \min _ {\mathbf {b} \in \mathbb {R} ^ {d}} \left(\| \mathbf {X b} - \boldsymbol {\mu} \| _ {2} ^ {2} + 1 0 0 \log (n / \delta) \cdot \zeta^ {2} \cdot \| \mathbf {b} \| _ {2} ^ {2}\right) + \frac {3 2 d}{n ^ {2}} \cdot \| \mathbf {y} ^ {(1)} - \mathbf {y} ^ {(0)} \| _ {\infty} ^ {2}\right) \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ (\widehat {\tau} - \tau) ^ {2} | \mathcal {E} _ {2} ] \\ \leq \mathbb {E} \left[ \frac {8 (1 + \epsilon)}{n ^ {2}} \cdot \left(\left\| \mathbf {X} \widetilde {\mathbf {b}} - \boldsymbol {\mu} \right\| _ {2} ^ {2} + 1 0 0 \log (n / \delta) \cdot \zeta^ {2} \cdot \left\| \widetilde {\mathbf {b}} \right\| _ {2} ^ {2}\right) + \frac {3 2 d}{n ^ {2}} \cdot \| \mathbf {y} ^ {(1)} - \mathbf {y} ^ {(0)} \| _ {\infty} ^ {2} | \mathcal {E} _ {2} \right] \\ \leq \frac {1}{\mathbb {P} [ \mathcal {E} _ {2} ]} \cdot \mathbb {E} \left[ \frac {8 (1 + \epsilon)}{n ^ {2}} \cdot \left(\left\| \mathbf {X} \widetilde {\mathbf {b}} - \boldsymbol {\mu} \right\| _ {2} ^ {2} + 1 0 0 \log (n / \delta) \cdot \zeta^ {2} \cdot \left\| \widetilde {\mathbf {b}} \right\| _ {2} ^ {2}\right) + \frac {3 2 d}{n ^ {2}} \cdot \| \mathbf {y} ^ {(1)} - \mathbf {y} ^ {(0)} \| _ {\infty} ^ {2} \right] \\ \leq \frac {1}{\mathbb {P} [ \mathcal {E} _ {2} ]} \cdot \mathbb {E} \left[ \frac {8 (1 + \epsilon)}{n ^ {2}} \cdot \left(\| \mathbf {X b} ^ {*} - \boldsymbol {\mu} \| _ {2} ^ {2} + 1 0 0 \log (\frac {n}{\delta}) \cdot \zeta^ {2} \cdot \| \mathbf {b} ^ {*} \| _ {2} ^ {2}\right) + \frac {3 2 d}{n ^ {2}} \cdot \| \mathbf {y} ^ {(1)} - \mathbf {y} ^ {(0)} \| _ {\infty} ^ {2} \right] \\ + \frac {1}{\mathbb {P} [ \mathcal {E} _ {2} ]} \cdot \mathbb {E} \left[ \frac {8 (1 + \epsilon)}{n ^ {2}} \cdot \frac {2}{1 - \epsilon} \cdot \left(\| \mathbf {S X b} ^ {*} - \mathbf {S} \boldsymbol {\mu} \| _ {2} ^ {2} + 1 0 0 \log \left(\frac {n}{\delta}\right) \cdot \zeta^ {2} \cdot \| \mathbf {b} ^ {*} \| _ {2} ^ {2}\right) \right], \\ \end{array}
$$
where the last inequality follows from (13) and (14). Now since leverage score sampling preserves the norms in expectation, we have  $\mathbb{E}\left[\| \widetilde{\mathbf{X}}\mathbf{b}^{*} - \widetilde{\boldsymbol{\mu}}\|_{2}\right] = \| \widehat{\mathbf{X}}\mathbf{b}^{*} - \widehat{\boldsymbol{\mu}}\|_{2}$ . Therefore
$$
\begin{array}{l} \mathbb {E} [ (\widehat {\tau} - \tau) ^ {2} | \mathcal {E} _ {2} ] \cdot \mathbb {P} [ \mathcal {E} _ {2} ] \\ \leq \frac {8 (1 + \epsilon)}{n ^ {2}} \cdot \left(1 + \frac {2}{1 - \epsilon}\right) \cdot \left(\| \mathbf {S X b} ^ {*} - \mathbf {S} \boldsymbol {\mu} \| _ {2} ^ {2} + 1 0 0 \log \left(\frac {n}{\delta}\right) \cdot \zeta^ {2} \cdot \| \mathbf {b} ^ {*} \| _ {2} ^ {2}\right) \\ + \frac {3 2 d}{n ^ {2}} \cdot \| \mathbf {y} ^ {(1)} - \mathbf {y} ^ {(0)} \| _ {\infty} ^ {2}, \\ \end{array}
$$
Combining this with (15), we have
$$
\begin{array}{l} \mathbb {E} [ (\widehat {\tau} - \tau) ^ {2} ] \\ \leq \left(\frac {6 4 d}{n ^ {2}} \cdot \| \mathbf {y} ^ {(1)} - \mathbf {y} ^ {(0)} \| _ {\infty} ^ {2} + \frac {8 (2 + \frac {2}{1 - \epsilon})}{n ^ {2}} \min  _ {b} \left(\| \mathbf {X b} - \boldsymbol {\mu} \| _ {2} ^ {2} + 1 0 0 \log (n / \delta) \cdot \zeta^ {2} \cdot \| \mathbf {b} \| _ {2} ^ {2}\right)\right) \\ + \delta \cdot \| \boldsymbol {\mu} \| _ {2} ^ {2}. \\ \end{array}
$$
D ATE with Partial Observation
A simple approach for estimating ATE using partial observations is to first uniformly sample a subset of size  $m$  of the population and then apply any of the methods for ATE with full observation to this subset. If we observe the outcomes of the samples in the subset independently and with equal probability and apply the Horvitz-Thompson estimator, we achieve an squared error of less than  $\frac{\|\mu\|_2^2}{mn} + \frac{\|\mathbf{t} - \overline{\mathbf{t}}\|_2^2}{m} \left(1 - \frac{m}{n}\right)$ . If we use the Gram-Schmidt walk design and the Horvitz-Thompson estimator on the uniformly sampled subset, then we achieve an unbiased estimator with a variance of less than
$$
\mathbb {E} \left[ (\widehat {\tau} - \tau) ^ {2} \right] \leq \frac {1}{m ^ {2}} \min _ {b \in \mathbb {R} ^ {d}} \left[ \frac {m}{n} \cdot \frac {1}{\phi} \| \mathbf {X b} - \boldsymbol {\mu} \| _ {2} ^ {2} + \frac {1}{1 - \phi} \zeta^ {2} \| \mathbf {b} \| _ {2} ^ {2} \right] + \frac {\left\| \mathbf {t} - \bar {\mathbf {t}} \right\| _ {2} ^ {2}}{m} \left(1 - \frac {m}{n}\right).
$$
Our approach is to adjust the Horvitz-Thompson estimator itself. We call this partial observation regression adjusted Horvitz-Thompson (RAHT) estimator. The first result we prove in this section is independent of the algorithm we use for estimation over the uniformly selected subset of the population. We introduce the following novel estimator by applying regression adjustment to HT estimator.
Definition 17 (Partial Observation Regression-Adjusted Horvitz-Thompson (RAHT) Estimator). Given a matrix of covariates  $X \in \mathbb{R}^{n \times d}$ , a fixed vector  $\widehat{\mathbf{b}}$ , and an integer  $1 \leq m \leq n$ , we introduce the partial observation regression adjusted Horvitz-Thompson estimator as the following. Estimation is based on a uniformly sampled set  $S$  of size  $m$  that we partition to two sets  $Z^{+}$  and  $Z^{-}$  by a random process (e.g., Gram-Schmidt walk). Then the ATE estimate is the following.
$$
\widehat {\tau} = \frac {1}{m} \left[ \sum_ {i \in Z ^ {+}} \frac {y _ {i} ^ {(1)}}{\mathbb {P} [ i \in Z ^ {+} ]} - \sum_ {i \in Z ^ {-}} \frac {y _ {i} ^ {(0)}}{\mathbb {P} [ i \in Z ^ {-} ]} \right] - \frac {1}{m} \sum_ {i \in S} \mathbf {x} _ {i} ^ {\top} \widehat {\mathbf {b}}. \tag {16}
$$
The following result indicates that the partial observation regression-adjusted Horvitz-Thompson estimator is a powerful tool for estimating ATE with partial observation.
Lemma 18. Let  $\mathbf{y}^{(1)},\mathbf{y}^{(0)}\in \mathbb{R}^n$ $\widehat{\mathbf{b}}\in \mathbb{R}^d$  be a fixed/preassigned vector and  $\mathbf{X}\in \mathbb{R}^{n\times d}$ . Given an unbiased estimator  $\widehat{\tau}_S$  for  $\tau_{S}\coloneqq \frac{1}{|S|}\sum_{i\in S}y_{i}^{(1)} - y_{i}^{(1)}$ , and a uniformly sampled set  $S$  of size  $m$ , the estimator  $\widehat{\tau} := \widehat{\tau}_{S} - \frac{1}{m}\sum_{i\in S}(\mathbf{x}_{i} - \overline{\mathbf{x}})^{\top}\widehat{\mathbf{b}}$  is unbiased and its variance is
$$
\mathbb {E} \left[ (\widehat {\tau} - \tau) ^ {2} \right] \leq \mathbb {E} _ {S \sim \mathcal {D}} \left[ \mathbb {E} \left[ (\widehat {\tau} _ {S} - \tau_ {S}) ^ {2} \right] \right] + \frac {1}{m n} \left\| (\mathbf {X} - \overline {{\mathbf {X}}}) ^ {\top} \widehat {\mathbf {b}} - (\mathbf {t} - \bar {\mathbf {t}}) \right\| _ {2} ^ {2}, \tag {17}
$$
where the second expectation on the right-hand side is over the randomness of  $\widehat{\tau}_S$ , and  $\mathcal{D}$  denotes the distribution of choosing a uniformly random set of size  $m$ .
Proof. We have  $\mathbb{E}\left[(\widehat{\tau} -\tau)^2\right] = \mathbb{E}_{S\sim \mathcal{D}}\left[\mathbb{E}\left[(\widehat{\tau}_S - (\frac{1}{m}\sum_{i\in S}(\mathbf{x}_i - \overline{\mathbf{x}})^\top \widehat{\mathbf{b}}) - \tau)^2\right]\right]$ . Moreover
$$
\begin{array}{l} \big (\widehat {\tau} _ {S} - \big (\frac {1}{m} \sum_ {i \in S} (\mathbf {x} _ {i} - \overline {{\mathbf {x}}}) ^ {\top} \widehat {\mathbf {b}} \big) - \tau \big) ^ {2} = \big (\widehat {\tau} _ {S} - \tau_ {S} + \tau_ {S} - \big (\frac {1}{m} \sum_ {i \in S} (\mathbf {x} _ {i} - \overline {{\mathbf {x}}}) ^ {\top} \widehat {\mathbf {b}} \big) - \tau \big) ^ {2} \\ = (\widehat {\tau} _ {S} - \tau_ {S}) ^ {2} + (\tau_ {S} - (\frac {1}{m} \sum_ {i \in S} (\mathbf {x} _ {i} - \overline {{\mathbf {x}}}) ^ {\top} \widehat {\mathbf {b}}) - \tau) ^ {2} + 2 (\widehat {\tau} _ {S} - \tau_ {S}) (\tau_ {S} - (\frac {1}{m} \sum_ {i \in S} (\mathbf {x} _ {i} - \overline {{\mathbf {x}}}) ^ {\top} \widehat {\mathbf {b}}) - \tau). \\ \end{array}
$$
Since  $\widehat{\tau}_S$  is an unbiased estimator of  $\tau_S$ ,
$$
\begin{array}{l} \mathbb {E} _ {S \sim \mathcal {D}} \left[ \mathbb {E} \left[ 2 (\widehat {\tau} _ {S} - \tau_ {S}) (\tau_ {S} - (\frac {1}{m} \sum_ {i \in S} (\mathbf {x} _ {i} - \overline {{\mathbf {x}}}) ^ {\top} \widehat {\mathbf {b}}) - \tau) \right] \right] \\ = \mathbb {E} _ {S \sim \mathcal {D}} \left[ \mathbb {E} \left[ 2 (\widehat {\tau} _ {S} - \tau_ {S}) \right] (\tau_ {S} - (\frac {1}{m} \sum_ {i \in S} (\mathbf {x} _ {i} - \overline {{\mathbf {x}}}) ^ {\top} \widehat {\mathbf {b}}) - \tau) \right] = 0. \\ \end{array}
$$
$$
\mathbb {E} _ {S \sim \mathcal {D}} \left[ \mathbb {E} \left[ (\tau_ {S} - (\frac {1}{m} \sum_ {i \in S} (\mathbf {x} _ {i} - \overline {{\mathbf {x}}}) ^ {\top} \widehat {\mathbf {b}}) - \tau) ^ {2} \right] \right] = \mathbb {E} _ {S \sim \mathcal {D}} \left[ (\tau_ {S} - (\frac {1}{m} \sum_ {i \in S} (\mathbf {x} _ {i} - \overline {{\mathbf {x}}}) ^ {\top} \widehat {\mathbf {b}}) - \tau) ^ {2} \right].
$$
Since  $\mathbb{E}_{S\sim \mathcal{D}}\left[\tau_S - \left(\frac{1}{m}\sum_{i\in S}(\mathbf{x}_i - \overline{\mathbf{x}})^{\top}\widehat{\mathbf{b}}\right)\right] = \tau$  by Lemma 9,
$$
\begin{array}{l} \mathbb {E} _ {S \sim \mathcal {D}} \left[ \big (\tau_ {S} - \big (\frac {1}{m} \sum_ {i \in S} (\mathbf {x} _ {i} - \overline {{\mathbf {x}}}) ^ {\top} \widehat {\mathbf {b}} \big) - \tau \big) ^ {2} \right] = \frac {1 - m / n}{m (n - 1)} \left\| (\overline {{\mathbf {X}}} - \mathbf {X}) \widehat {\mathbf {b}} + (\mathbf {y} ^ {(1)} - \mathbf {y} ^ {(0)}) - \overline {{\mathbf {t}}} \right\| _ {2} ^ {2} \\ \leq \frac {1}{m n} \left\| (\overline {{\mathbf {X}}} - \mathbf {X}) \widehat {\mathbf {b}} - (\overline {{\mathbf {t}}} - \mathbf {t}) \right\| _ {2} ^ {2}. \\ \end{array}
$$
We next prove a variance bound if the Gram-Schmidt walk design is applied for estimation over the uniformly selected subset.
Theorem 19. Let  $\widehat{\tau}_{S}$  be an estimate obtained by the Horvitz-Thompson estimator on the Gram-Schmidt walk design. Let  $\widehat{\mathbf{b}}$  be a fixed/preassigned vector and  $0 < \phi < 1$ . Then, the partial observation regression-adjusted Horvitz-Thompson estimator with  $m$  samples is an unbiased estimator with variance of
$$
\mathbb {E} \left[ (\widehat {\tau} - \tau) ^ {2} \right] \leq \frac {1}{m n} \frac {1}{\phi} \| \mathbf {X} \mathbf {b} ^ {*} - \boldsymbol {\mu} \| _ {2} ^ {2} + \frac {1}{m ^ {2}} \frac {\zeta^ {2}}{(1 - \phi)} \| \mathbf {b} ^ {*} \| _ {2} ^ {2} + \frac {1}{m n} \left\| (\mathbf {X} - \overline {{\mathbf {X}}}) ^ {\top} \widehat {\mathbf {b}} - (\mathbf {t} - \overline {{\mathbf {t}}}) \right\| _ {2} ^ {2},
$$
where  $\mathbf{b}^{*} = \arg \min_{\mathbf{b}\in \mathbb{R}^{d}}\left[\frac{1}{\phi}\| \mathbf{X}\mathbf{b} - \boldsymbol {\mu}\|_{2}^{2} + \frac{\zeta^{2}}{(1 - \phi)}\| \mathbf{b}\|_{2}^{2}\right]$
Proof. By Theorem 13 and Lemma 18,
$$
\begin{array}{l} \mathbb {E} \left[ (\widehat {\tau} - \tau) ^ {2} \right] \leq \mathbb {E} _ {S \sim \mathcal {D}} \left[ \frac {1}{m ^ {2}} \min _ {\mathbf {b} \in \mathbb {R} ^ {n}} \left[ \frac {1}{\phi} \| \mathbf {X} _ {S}; \mathbf {b} - \boldsymbol {\mu} _ {S} \| _ {2} ^ {2} + \frac {\zeta^ {2}}{1 - \phi} \| \mathbf {b} \| _ {2} ^ {2} \right] \right] \\ + \frac {1}{m n} \left\| (\mathbf {X} - \overline {{\mathbf {X}}}) ^ {\top} \widehat {\mathbf {b}} - (\mathbf {t} - \overline {{\mathbf {t}}}) \right\| _ {2} ^ {2} \\ \leq \frac {1}{m ^ {2}} \mathbb {E} _ {S \sim \mathcal {D}} \left[ \frac {1}{\phi} \left\| \mathbf {X} _ {S:} \mathbf {b} ^ {*} - \boldsymbol {\mu} _ {S} \right\| _ {2} ^ {2} + \frac {\zeta^ {2}}{1 - \phi} \left\| \mathbf {b} ^ {*} \right\| _ {2} ^ {2} \right] + \frac {1}{m n} \left\| (\mathbf {X} - \overline {{\mathbf {X}}}) ^ {\top} \widehat {\mathbf {b}} - (\mathbf {t} - \overline {{\mathbf {t}}}) \right\| _ {2} ^ {2} \\ = \frac {1}{m ^ {2}} \mathbb {E} _ {S \sim \mathcal {D}} \left[ \frac {1}{\phi} \| \mathbf {X} _ {S :} \mathbf {b} ^ {*} - \boldsymbol {\mu} _ {S} \| _ {2} ^ {2} \right] + \frac {1}{m ^ {2}} \frac {\zeta^ {2}}{1 - \phi} \| \mathbf {b} ^ {*} \| _ {2} ^ {2} \\ + \frac {1}{m n} \left\| (\mathbf {X} - \overline {{\mathbf {X}}}) ^ {\top} \widehat {\mathbf {b}} - (\mathbf {t} - \overline {{\mathbf {t}}}) \right\| _ {2} ^ {2} \\ \end{array}
$$
$$
\left\| \mathbf {X} _ {S:} \mathbf {b} ^ {*} - \boldsymbol {\mu} _ {S} \right\| _ {2} ^ {2} = \sum_ {i \in S} \left(\mathbf {x} _ {i} ^ {\top} \mathbf {b} ^ {*} - \boldsymbol {\mu} _ {i}\right) ^ {2}.
$$
The probability of any  $i \in [n]$  appearing in a uniformly sampled set of size  $m$  is
$$
\frac {\binom {n - 1} {m - 1}}{\binom {n} {m}} = \frac {m}{n}.
$$
$$
\mathbb {E} _ {S \sim \mathcal {D}} \left[ \| \mathbf {X} _ {S:} \mathbf {b} ^ {*} - \boldsymbol {\mu} _ {S} \| _ {2} ^ {2} \right] = \frac {m}{n} \sum_ {i \in [ n ]} (\mathbf {x} _ {i} ^ {\top} \mathbf {b} ^ {*} - \boldsymbol {\mu} _ {i}) ^ {2} = \frac {m}{n} \| \mathbf {X} \mathbf {b} ^ {*} - \boldsymbol {\mu} \| _ {2} ^ {2},
$$
and the result follows by combining the above.
In Lemma 18 and Theorem 19, we have assumed that the vector  $\widehat{\mathbf{b}}$  (which is used for adjusting the Horvitz-Thompson estimator) is given. For our final result, we combine our approach for regression on random vectors (with partial observation according to leverage score sampling) with partial observation RAHT to obtain a complete algorithm for estimation.
$$
\begin{array}{l} \mathbb {E} \left[ (\widehat {\tau} - \tau) ^ {2} | \mathcal {E} \right] \leq \frac {1}{m n} \frac {1}{\phi} \| \mathbf {X b} ^ {*} - \boldsymbol {\mu} \| _ {2} ^ {2} + \frac {1}{m ^ {2}} \frac {\zeta^ {2}}{(1 - \phi)} \| \mathbf {b} ^ {*} \| _ {2} ^ {2} + \frac {1 0 0 d \cdot \log (d / \delta)}{n ^ {2} \epsilon^ {2}} \| \boldsymbol {\mu} \| _ {\infty} ^ {2} \\ + (1 + \epsilon) \cdot \left(\frac {1}{m} \| \pmb {\mu} \| _ {\infty} ^ {2} + \frac {1}{m n} \| (\mathbf {X} - \overline {{\mathbf {X}}}) \widehat {\mathbf {b}} - (\mathbf {t} - \overline {{\mathbf {t}}}) \| _ {2} ^ {2} + \frac {\lambda}{m n} \| \widehat {\mathbf {b}} \| _ {2} ^ {2}\right), \\ \end{array}
$$
Algorithm 4: Regression Adjusted Horvitz-Thompson Estimation
1 Input:  $\mathbf{X} \in \mathbb{R}^{n \times d}$ ,  $\mathbf{y}^{(1)}, \mathbf{y}^{(0)} \in \mathbb{R}^n$ ,  $m \in \mathbb{N}$ ,  $0 < \epsilon < 1$ ,  $0 < \phi < 1$ .
2 Set  $\alpha = d / n$  and  $\lambda = \frac{6\cdot\log(d / \delta)}{\epsilon^2}\cdot \zeta^2$
3 Compute the  $\lambda$ -ridge leverage scores  $\ell_i^{(\lambda)}$  of  $\widetilde{\mathbf{X}} = \left[(\mathbf{X} - \overline{\mathbf{X}})\quad \mathbf{1}\right]$ .
4 Set  $p_i = \min \{1, \max \{\alpha, 3 \cdot \ell_i \cdot \log(d / \delta) / \epsilon^2\}\}$ .
5 For all  $i \in [n]$  define the random variable  $q_{i}$  that is 1 with probability  $p_{i}$ , and is zero, otherwise. Let  $S$  be the set of all indices  $i$  with  $q_{i} = 1$ .
6 Let  $\mathbf{S} \in \mathbb{R}^{n \times n}$  be a random diagonal matrix with  $\mathbf{S}_{ii} = 1 / \sqrt{p_i}$  if  $q_i = 1$ , and  $\mathbf{S}_{ii} = 0$ , otherwise.
7 Let  $\mathbf{y} \in \mathbb{R}^n$  be a random vector where  $y_i$  is independently equal to either  $y_i^{(1)}$  or  $-y_i^{(0)}$  with equal probability.
8 Let  $\widetilde{\mathbf{v}} = \arg \min_{\mathbf{b} \in \mathbb{R}^{d + 1}} \left\| \mathbf{S}\widetilde{\mathbf{X}}\mathbf{b} - \mathbf{S}\mathbf{y} \right\|_2^2 + \lambda \| \mathbf{b}_{1:d} \|_2^2$ , and  $\widehat{\mathbf{v}} \in \mathbb{R}^d$  be a vector consisting of the first  $d$  entries of  $\widetilde{\mathbf{v}}$
9 Let  $T$  be a uniformly sampled set of size  $m$  from  $[n] \setminus S$ .
10 Obtain  $\widehat{\tau}_{\overline{S}}$  by Horvitz-Thompson estimator and Gram-Schmidt walk design on  $T$  with parameter  $\phi$ . // See Theor
11 Let  $\widehat{\tau}_S = 2\sum_{i\in S}y_i$
12 Return  $\frac{|\overline{S}|}{n} \cdot \left( \widehat{\tau}_{\overline{S}} - \frac{1}{m} \sum_{i \in T} (\mathbf{x}_i - \overline{\mathbf{x}})^{\top} \widehat{\mathbf{v}} \right) + \frac{|S|}{n} \cdot \widehat{\tau}_S$ .
Proof. We first prove that the estimator is unbiased. By the law of total expectation, we have
$$
\mathbb {E} \left[ \widehat {\tau} - \tau \right] = \sum_ {S, \mathbf {y}} \mathbb {E} \left[ \widehat {\tau} - \tau | S \right] \cdot \mathbb {P} \left[ S \right],
$$
where the summation is over all sets  $S$  that are selected from leverage score sampling. Note that given  $S$  and  $\mathbf{y}$ , the solution vector  $\widetilde{\mathbf{v}}$  is fixed. We denote this fixed solution vector with  $\widetilde{\mathbf{v}}^{S,\mathbf{y}}$ . Let  $\widehat{\mathbf{v}}^{S,\mathbf{y}} \in \mathbb{R}^d$  be the vector obtained by taking the first  $d$  entries of  $\widetilde{\mathbf{v}}^{S,\mathbf{y}}$ . Let  $\overline{\mathbf{x}}^S$  be the average row of  $\mathbf{X}_{\overline{S}}$ . Then we have
$$
\begin{array}{l} \mathbb {E} \left[ \widehat {\tau} - \tau | S \right] = \sum_ {\mathbf {y}} \mathbb {E} \left[ \frac {| \overline {{S}} |}{n} \cdot \left(\widehat {\tau} _ {\overline {{S}}} - \frac {1}{m} \sum_ {i \in T} \left(\mathbf {x} _ {i} - \overline {{\mathbf {x}}} ^ {\overline {{S}}}\right) ^ {\top} \widetilde {\mathbf {v}} ^ {S, \mathbf {y}}\right) - \frac {| \overline {{S}} |}{n} \cdot \tau_ {\overline {{S}}} | S, \mathbf {y} \right] \cdot \mathbb {P} [ \mathbf {y} ] \\ + \mathbb {E} \left[ \frac {| S |}{n} \cdot \widehat {\tau} _ {S} - \frac {| S |}{n} \cdot \tau_ {S} | S \right], \\ \end{array}
$$
where the summation is over the assignments of the random vector  $\mathbf{y}$ ,  $\overline{\mathbf{x}^S}$  is the average row over the set of rows indexed by  $\overline{S}$ ,  $T$  is a uniformly random subset of  $\overline{S}$  of size  $m$ ,  $\widehat{\tau}_{\overline{S}}$  is estimated by using GSW design and Horvitz-Thompson estimator on set  $T$ , and  $\widehat{\tau}_S$  is estimated by the Bernoulli design and Horvitz-Thompson estimator (the observation is compatible with vector  $\mathbf{y}$ ). Trivially since  $\widehat{\tau}_S$  is an unbiased estimator of  $\tau_S$ , the second term is equal to zero. Moreover since  $\widehat{\tau}_{\overline{S}}$  is an unbiased estimator of  $\tau_{\overline{S}}$ , over the expectation, they cancel out and therefore,
$$
\mathbb {E} \left[ \widehat {\tau} - \tau | S \right] = \sum_ {\mathbf {y}} - \frac {| \overline {{S}} |}{n} \cdot \mathbb {E} \left[ \left(\frac {1}{m} \sum_ {i \in T} \left(\mathbf {x} _ {i} - \overline {{\mathbf {x}}} ^ {\overline {{S}}}\right) ^ {\top} \widetilde {\mathbf {v}} ^ {S, \mathbf {y}}\right) | S, \mathbf {y} \right] \cdot \mathbb {P} [ \mathbf {y} ].
$$
Now since  $\widetilde{\mathbf{v}}^{S,\mathbf{y}}$  is fixed given  $S$  and  $\mathbf{y}$ , we have
$$
\mathbb {E} \left[ \widehat {\tau} - \tau | S \right] = \sum_ {\mathbf {y}} - \frac {| \overline {{S}} |}{m n} \cdot \mathbb {E} \left[ \sum_ {i \in T} \left(\mathbf {x} _ {i} - \overline {{\mathbf {x}}} ^ {\overline {{S}}}\right) ^ {\top} | S, \mathbf {y} \right] \widetilde {\mathbf {v}} ^ {S, \mathbf {y}} \cdot \mathbb {P} \left[ \mathbf {y} \right].
$$
$$
\mathbb {E} \left[ \sum_ {i \in T} (\mathbf {x} _ {i} - \overline {{\mathbf {x}}} ^ {\overline {{S}}}) ^ {\top} | S, \mathbf {y} \right] = \left(\sum_ {i \in \overline {{S}}} \frac {\binom {| \overline {{S}} | - 1} {m - 1}}{\binom {| \overline {{S}} |} {m}} \mathbf {x} _ {i} ^ {\top}\right) - \frac {m}{| \overline {{S}} |} \left(\sum_ {i \in \overline {{S}}} \mathbf {x} _ {i} ^ {\top}\right) ^ {\top} = 0.
$$
Therefore  $\mathbb{E}\left[\widehat{\tau} -\tau |S\right] = 0$  and  $\mathbb{E}\left[\widehat{\tau} -\tau \right] = 0$  . Thus, the estimator is unbiased.
We define the event  $\mathcal{E}$  to be the event where the regression error is within a factor of  $(1 + \epsilon)$  of error and the number of nonzero entries in the matrix  $\mathbf{S}$  is less than  $100d\cdot \log (d / \delta) / \epsilon^2$ . Note that
$$
\mathbb {E} \left[ (\widehat {\tau} - \tau) ^ {2} | \mathcal {E} \right] = \sum_ {S, y} \mathbb {E} \left[ (\widehat {\tau} - \tau) ^ {2} | S, \mathbf {y} \right] \cdot \mathbb {P} \left[ S, \mathbf {y} | \mathcal {E} \right]
$$
Note that condition to  $S$  and  $\mathbf{y}$
$$
\widehat {\tau} = \frac {| \overline {{S}} |}{n} \cdot \left(\widehat {\tau} _ {\overline {{S}}} - \frac {1}{m} \sum_ {i \in T} \left(\mathbf {x} _ {i} - \overline {{\mathbf {x}}} ^ {\overline {{S}}}\right) ^ {\top} \widetilde {\mathbf {v}} ^ {S, \mathbf {y}}\right) + \frac {| S |}{n} \cdot \widehat {\tau} _ {S},
$$
where  $\widehat{\tau}_{\overline{S}}$  and  $\widehat{\tau}_{S}$  are unbiased estimators for  $\tau_{\overline{S}}$  and  $\tau_{S}$ , respectively. Then since  $\tau = \frac{|\overline{S}|}{n} \cdot \tau_{\overline{S}} + \frac{|S|}{n} \cdot \tau_{S}$ , this is an unbiased estimator. Now we have
$$
\begin{array}{l} \mathbb {E} \left[ (\widehat {\tau} - \tau) ^ {2} | S, \mathbf {y} \right] \\ = \mathbb {E} \left[ \left(\frac {| \overline {{S}} |}{n} \cdot \left(\widehat {\tau} _ {\overline {{S}}} - \frac {1}{m} \sum_ {i \in T} (\mathbf {x} _ {i} - \overline {{\mathbf {x}}} ^ {\overline {{S}}}) ^ {\top} \widetilde {\mathbf {v}} ^ {S, \mathbf {y}}\right) + \frac {| S |}{n} \cdot \widehat {\tau} _ {S} - \frac {| \overline {{S}} |}{n} \cdot \tau_ {\overline {{S}}} - \frac {| S |}{n} \cdot \tau_ {S}\right) ^ {2} | S, \mathbf {y} \right] \\ = \mathbb {E} \left[ \left(\frac {| \overline {{S}} |}{n} \cdot \left(\widehat {\tau} _ {\overline {{S}}} - \frac {1}{m} \sum_ {i \in T} \left(\mathbf {x} _ {i} - \overline {{\mathbf {x}}} ^ {\overline {{S}}}\right) ^ {\top} \widetilde {\mathbf {v}} ^ {S, \mathbf {y}}\right) - \frac {| \overline {{S}} |}{n} \cdot \tau_ {\overline {{S}}}\right) ^ {2} | S, \mathbf {y} \right] \\ + \mathbb {E} \left[ (\frac {| S |}{n} \cdot \widehat {\tau} _ {S} - \frac {| S |}{n} \cdot \tau_ {S}) ^ {2} | S, \mathbf {y} \right] \\ + 2 \mathbb {E} \left[ \left(\frac {| \overline {{S}} |}{n} \cdot \left(\widehat {\tau} _ {\overline {{S}}} - \frac {1}{m} \sum_ {i \in T} \left(\mathbf {x} _ {i} - \overline {{\mathbf {x}}} ^ {\overline {{S}}}\right) ^ {\top} \widetilde {\mathbf {v}} ^ {S, \mathbf {y}}\right) - \frac {| \overline {{S}} |}{n} \cdot \tau_ {\overline {{S}}}\right) \left(\frac {| S |}{n} \cdot \widehat {\tau} _ {S} - \frac {| S |}{n} \cdot \tau_ {S}\right) | S, \mathbf {y} \right] \tag {18} \\ \end{array}
$$
Note that condition to  $S$ ,  $\mathbf{y}$ ,  $\left(\frac{|S|}{n} \cdot \widehat{\tau}_S - \frac{|S|}{n} \cdot \tau_S\right)$  is a fixed number. Therefore