Title: Finite Population Regression Adjustment and Non-asymptotic Guarantees for Treatment Effect Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: regression adjustment; treatment effect estimation; average treatment effect
Abstract: The design and analysis of randomized experiments is fundamental to many areas, from the physical and social sciences to industrial settings. 
Regression adjustment is a popular technique to reduce the variance of estimates obtained from experiments, by utilizing information contained in auxiliary covariates. 
While there is a large literature within the statistics community studying various approaches to regression adjustment and their asymptotic properties, little focus has been given to approaches in the finite population setting with non-asymptotic accuracy bounds. 
Further, prior work typically assumes that an entire population is exposed to an experiment, whereas practitioners often seek to minimize the number of subjects exposed to an experiment, for ethical and pragmatic reasons.
In this work, we study the problems of estimating the sample mean, individual treatment effects, and average treatment effect with regression adjustment. 
We propose approaches that use techniques from randomized numerical linear algebra to sample a subset of the population on which to perform an experiment. We give non-asymptotic accuracy bounds for our methods and demonstrate that they compare favorably with prior approaches.

Finite Population Regression Adjustment and Non-asymptotic Guarantees for Treatment Effect Estimation

Abstract

The design and analysis of randomized experiments is fundamental to many areas, from the physical and social sciences to industrial settings. Regression adjustment is a popular technique to reduce the variance of estimates obtained from experiments, by utilizing information contained in auxiliary covariates. While there is a large literature within the statistics community studying various approaches to regression adjustment and their asymptotic properties, little focus has been given to approaches in the finite population setting with non-asymptotic accuracy bounds. Further, prior work typically assumes that an entire population is exposed to an experiment, whereas practitioners often seek to minimize the number of subjects exposed to an experiment, for ethical and pragmatic reasons. In this work, we study the problems of estimating the sample mean, individual treatment effects, and average treatment effect with regression adjustment. We propose approaches that use techniques from randomized numerical linear algebra to sample a subset of the population on which to perform an experiment. We give non-asymptotic accuracy bounds for our methods and demonstrate that they compare favorably with prior approaches.

1 Introduction

Randomized experiments play a crucial role in estimating counterfactual outcomes. The strength of randomization lies in its ability to ensure independence between the treatment group and pretreatment covariates [14, 21]. This independence, in turn, results in unbiased estimation of treatment effects through the comparison of sample averages. However, in finite sample settings, the use of simple randomization techniques result in high variance of the estimated quantities. There have been two broad approaches to address this issue by reducing the variance of the estimated effects when subject level covariates are available: (1) design-based approaches, which focus on the mechanism for assigning individuals to treatment and control groups in order to minimize imbalance (c.f. [21, 18, 27, 34, 5, 15]), and (2) regression adjustment based approaches, which correct for imbalances post hoc by incorporating a regression of the covariates on the outcome (c.f. [9, 24, 7, 13, 29]).

Much of the prior work examining variance reduction for experimentation has focused on asymptotic properties of approaches [23, 24, 13, 21, 23]. Recently, there has been work focusing on design-based variance reduction, which ties the treatment assignment problem to discrepancy minimization and gives finite sample error bounds for estimating causal estimands such as the average treatment effect (ATE) [18, 1, 5] and individual treatment effect (ITE) [1]. In particular, Harshaw et al. propose a

design based on the Gram-Schmidt random walk (GSW) and give finite sample guarantees on the mean squared error for average treatment effect estimation when using this design with the classic Horvitz-Thompson estimator. Addanki, et al. [1] use GSW and leverage score sampling to provide algorithms and finite sample guarantees for both average and individual treatment effect estimation in the partial observation setting where only a subset of subjects are included in the experiment.

However, little work has examined the finite sample properties for experimental designs based on regression adjustment. Mou, et al. [28] provide non-asymptotic guarantees, but the analysis is in infinite population settings. The main contribution of this work is to analyze the finite population behavior of regression adjustment for variance reduction in treatment effect estimation, for ATE and ITE estimation in both the full and partial (sub-population) settings.

1.1 Problem Statement: Finite Population Treatment Effect Estimation

In the finite population treatment effect estimation problem, we have n units (or individuals) associated with covariates x 1 , . . . , x n ∈ R d which are the rows of a covariate matrix X ∈ R n × d . Each unit i is associated with two real numbers y (1) i , y (0) i called the potential outcomes under the treatment and control , respectively. We denote the corresponding potential outcome vectors with y (1) , y (0) ∈ R n . The Individual Treatment Effect (ITE) vector t ∈ R n is defined as t := y (1) -y (0) and the Average Treatment Effect (ATE) τ is defined as the mean of this vector, i.e., τ := 1 n ∑ n i =1 t i .

In treatment effect estimation, we seek to estimate either the ITE or ATE, under the constraint that at most one of the potential outcomes, either y (1) i or y (0) i , can be observed for each unit i . E.g., that unit may be assigned to a treatment group in a controlled trial, and thus y (1) i is observed while y (0) i is not.

In full observation treatment effect estimation, we observe one potential outcome for each unit in the population [9, 13, 24, 18]. In partial observation treatment effect estimation, we further restrict ourselves to observing outcomes for just a small subsample of the population. This setting is important when the goal is to minimize experimental costs by e.g., limiting the size of a controlled trial [30, 20, 32, 1].

To reduce variance, many approaches to ATE and ITE estimation, including our own, attempt to leverage the population covariates X to make inferences about the treatment outcomes. In particular, throughout this work, we will target error bounds that depend on the best linear fit of the treatment outcomes to our covariates. Such linear effects models are common in the literature on treatment effect estimation [33, 18, 1] and provide a useful baseline for establishing theoretical bounds.

1.2 Our Contributions

Wegive new approaches to both ITE and ATE estimation in the full and partial observation settings that combine the classic statistical technique of regression adjustment with techniques from randomized numerical linear algebra. Our algorithms give natural, non-asymptotic error bounds depending on the best linear fit for either the ITE vector t := y (1) -y (0) or the sum of the potential outcome vectors µ := y (1) + y (0) to our covariates. µ is a natural quantity that arises e.g., in variance bounds for classic approaches to ATE estimation, such as the Hovitz-Thompson estimator [19].

1.2.1 Individual Treatment Effect Estimation

Our first result is for individual treatment effect estimation in the full and partial observation settings.

Theorem 1 (ITE Estimation) . For any ϵ, δ, α ∈ (0 , 1] , there exists a randomized algorithm (Algorithm 3) that observes a potential outcome for O ( d log( d/δ ) /ϵ 2 + α · n ) units and outputs a vector ̂ t . Moreover there is an event E such that Pr( E ) ≥ 1 -δ (over the randomness of the algorithm) and

Theorem 1 shows that with high probability (at least 1 -δ ), our algorithm achieves an expected error bound that nearly matches the best linear fit of the ITE vector to our covariates (i.e., min b ∈ R d ∥ Xb -t ∥ 2 2 ) , up to an additive term depending on the maximum magnitude of the sum of potential outcomes for any individual. In the full observation setting, when α = 1 , this additive

term is O ( d ) , assuming that the maximum magnitude does not scale with the population size n . In contrast, we expect the best fit error ( min b ∈ R d ∥ Xb -t ∥ 2 2 ) to grow as Θ( n ) , and thus the additive term is lower order. In the partial observation setting, as long as we set α = ω (1 /n ) , we still expect this additive term to be lower order. Thus, the case when d ≪ n , we are able to nearly match the best fit error while only observing a very small subset of the full population.

We sketch the proof of Theorem 1 in Section 2. Our algorithm is based on pseudo-outcome regression, which is studied in the asymptotic setting by Kennedy et al. for conditional ATE estimation [22]. Roughly, in the full observation setting, we construct a vector v where v i = 2 y (1) i with probability 1 / 2 and v i = -2 y (0) i with probability 1 / 2 . We can see that E [ v ] = t (i.e., v is equal to the ITE vector in expectation), and importantly, that constructing v only requires observing one potential outcome for each individual. By regressing v onto our covariates, we obtain our estimate ̂ t for t .

In the partial observation setting, we further subsample individuals according to the leverage scores of the covariate matrix X . This is a standard technique in randomized linear algebra [26, 10, 35], which allows us to approximately solve the above regression problem while only observing a subset of ˜ O ( d/ϵ 2 ) entries of v (and thus only observing outcomes for a subset of the population). Addanki et al. [1] similarly propose an algorithm for the partial observation setting based on leverage score sampling. However, instead of pseudo-outcome regression, they learn separate approximations to both outcome vectors y (1) and y (0) . Thus, their final error depends on the best linear fit error for both these vectors, rather than the more natural best linear fit error for the ITE vector t .

1.2.2 Average Treatment Effect Estimation

We can extend our general approach to give bounds for average treatment effect estimation. In the full observation setting we show:

Theorem 2 (ATE Estimation - Full Observation) . For any ϵ, δ ∈ (0 , 1] , there exists a randomized algorithm (Algorithm 1) that computes an unbiased estimate ̂ τ of the ATE τ , and for which there is an event E with Pr( E ) ≥ 1 -δ (over the randomness of the algorithm) such that

where ζ := max i ∈ [ n ] ∥ x i ∥ 2 and µ := y (0) + y (1) is the total outcome vector.

As with our result in ITE estimate, the error bound of Theorem 2 matches the error of the best linear fit of the total outcome vector µ to our covariates up to a constant factor plus additive terms depending on 1) the maximum magnitude of the sum of potential outcomes for any individual, and 2) the norm of the coefficient vector used to approximately reconstruct µ from the covariates. Again, we expect these additive terms to scale as O (1 /n 2 ) while we expect the term depending on the best fit error to scale as O (1 /n ) . Thus, we generally expect the additive error to be lower order.

Harshaw et al. [18] proposed an algorithm based on the Gram-Schmidt walk (GSW) design (see Algorithm 5 in the appendix) for balancing assignments to treatment and control groups that achieves an unbiased estimator for ATE with variance of 1 n 2 min b [ 1 ϕ ∥ µ -Xb ∥ 2 2 + ζ 2 1 -ϕ ∥ b ∥ 2 2 ] , where ϕ ∈ (0 , 1) is chosen by the experiment designer. This guarantee is comparable to but stronger than ours, e.g., when one sets ϕ = 1 / 2 . If we ignore the additive error terms and focus just on the best linear fit error ∥ Xb -µ ∥ 2 2 , then GSW is better than our guarantees by about a factor of 8 . But the GSW design is computationally more expensive and, since is based on balancing the covariates, requires availability of all covariates before running the experiment and hence cannot be applied in an online setting [4]. In contrast, our approach can be applied directly in the online setting since we place each unit independently in the treatment or control groups with equal probability. Moreover, as we show in our experiments, the empirical performance of our algorithm is much closer to GSW than the theoretical bounds we are able to show.

We sketch the ideas behind Theorem 2 in Section 2. Our algorithm is based on the classic HorvitzThompson estimator [19]. Roughly, this estimator randomly assigns individuals to control and treatment groups, and estimates ATE as the difference in the average outcome between these groups, appropriately weighted by the assignment probabilities. It is well known that the variance of this estimator is bounded by 1 n 2 ∥ µ ∥ 2 2 . To reduce this variance, we introduce full-observation regression

adjusted Horvitz-Thompson (RAHT) estimator , which estimates ATE as the following. We partition the units into two groups S and S (this is a different partitioning than the partitioning into control and treatment), and we regress µ S and µ S onto their corresponding covariates. As in the ITE case, since we cannot directly form µ , we instead perform the above regressions using a random vector u with E [ u ] = µ , and where u can be formed by only observing one potential outcome per individual. We then use the solution vector of group S to adjust the outcomes of the group S and vice versa, and we apply the HT estimator to the adjusted outcomes. This gives the bound in Theorem 2.

Again, following a similar approach as in the ITE case, we can apply leverage score based subsampling to tackle ATE estimation in the partial observation setting as well, obtaining:

Theorem 3 (ATE Estimation - Partial Observation) . For any ϵ, δ ∈ (0 , 1] , ϕ ∈ (0 , 1) , and m ∈ [ n ] , there exists a randomized algorithm (Algorithm 4) that observes a potential outcome for O ( d log( d/δ ) /ϵ 2 + m ) units and outputs an unbiased estimate ˆ τ of the ATE τ , for which there is an event E with Pr( E ) ≥ 1 -δ (over the randomness of the algorithm), such that

where ζ := max i ∈ [ n ] ∥ x i ∥ 2 , µ := y (0) + y (1) , λ = 6 · log( d/δ ) ϵ 2 · ζ 2 , t ∈ R n is a vector where all the entries are equal to t = 1 n ∑ n i =1 t i , and

The error bound of Theorem 3 depends on 1) the error of best linear fits of the covariates onto µ and t -t , 2) the norm of the coefficient vector used to approximately reconstruct µ and t -t from the covariates, and 3) the largest component of the total outcome vector µ . We expect both the best linear fit and ∥ µ ∥ ∞ terms to scale as O ( 1 m ) . All other terms are lower order. Namely, The terms depending on the norms of the coefficient vectors scale as O ( 1 mn ) and O ( 1 m 2 ) . For the case where md ≪ n , by increasing the number of samples to O ( d log( d/δ ) /ϵ 2 + md ) , we can make the ∥ µ ∥ ∞ term to scale as O (1 /m 2 ) as well.

We deploy a different regression adjustment technique for achieving Theorem 3, which we call partialobservation regression adjusted Horvitz-Thompson (RAHT) estimator . In the partial observation setting, a simple approach is just to apply the full observation algorithm to a uniformly selected subset of m units and then report the ATE estimate for these m units. However, this leads to a variance bound depending on ∥ ∥ t -t ∥ ∥ 2 2 . To reduce this variance, the partial observation RAHT adjusts the estimate obtained from the m units using regression adjustment techniques. We give the details behind the full algorithm in Section 2.3 and provide the full analysis in the appendix.

1.2.3 Experimental Evaluation

We compliment the above theoretical bounds with experiments using both synthetic and real data, focusing on the full observation ATE and ITE estimation problems. For ATE, we compare our method with classical regression adjustment [24], the GSW method [18], and the algorithm of [28]. We observe that our estimator (which is unbiased) performs as well or better than all other estimators except the classic regression adjustment, which is a biased estimator. For ITE, we compare our approach with the method of [1], and observe superior performance. A discussion about the running time of our algorithm and comparison to GSW design is included in the appendix.

1.3 Roadmap

We present the high-level ideas and techniques for ITE and ATE estimation for the full and partial observation settings in Section 2. We then present notation and preliminaries required for a more in-depth discussion of our result in Section 3. In Section 4, we give a detailed non-asymptotic analysis of the random vector (i.e., pseudo-outcome) regression technique, which forms the basis for all of our algorithms. We present our algorithm for the full observation ATE estimation and sketch the proof of its accuracy (Theorem 2). Finally, we present our experimental results in Section 5.

We relegate the details of other algorithms and proofs to the appendix, along with additional details and results for the experimental results. In the appendix, we also include a warm-up result for the mean estimation problem in the partial observation setting (where there is only one outcome y and we seek to estimate 1 n ∑ n i =1 y i from a small number of samples). This result illustrates the key ideas behind leverage score sampling, which is used for our results on both ITE and ATE estimation. In addition we discuss how to remove the 1 -δ probability from our results in Section C.1.

2 Technical Overview

In this section, we sketch the key ideas behind our main results on ITE estimation (Theorem 1) and ATE estimation (Theorems 2 and 3).

2.1 ITE Estimation

We first consider ITE estimation in the full observation setting. The key challenge here is that we must infer the full treatment effect vector t = y (1) -y (0) , while only observing one outcome, y (1) i or y (0) i for any unit i . To do so, we use the idea of pseudo-outcome regression in [22]. We construct a random vector v ∈ R n that is equal to t in expectation, i.e., E [ v ] = t but only requires observing one outcome per unit. Specifically, we independently set v i = 2 y (1) i with probability 0 . 5 and v i = -2 y (0) i with probability 0 . 5 . We then regress v onto our covariates, and use the result to estimate t . In particular, we set ̂ b := arg min b ∈ R d ∥ Xb -v ∥ 2 2 and construct our ITE estimate as ̂ t = X ̂ b .

To give a non-asymptotic error bound for this approach, we observe that we can write v = t + z ⊙ µ , where z ∈ { 1 , -1 } n is a random sign vector, µ = y (1) + y (0) is the sum of potential outcome vectors, and ⊙ denotes the Hadamard (entrywise) product. Thus, letting π X ∈ R n × n be the projection matrix onto the column span of X , we can write ̂ b = π X v = π X t + π X ( z ⊙ µ ) . In turn, we have

The first term on the righthand side is simply the best linear fit error for t , ∥ π X t -t ∥ 2 2 = min b ∈ R d ∥ Xb -t ∥ 2 2 . The third term is 0 in expectation since E [ z ⊙ µ ] = 0 . Finally, since the entries of z ◦ µ are independent and mean 0 , the expectation of the second term can be bounded by tr( π X ) · ∥ µ ∥ 2 ∞ = d ∥ µ ∥ 2 ∞ . Putting these all together, we have

which gives the bound of Theorem 1 in the full observation setting. We note that this expected error bound can also be turned into a high probability bound using concentration inequalities, i.e., Hanson-wright inequality [31].

ITE Estimation with Partial Observation. Our next step is to extend the above to the partial observation setting. To do so, we apply leverage score sampling, which is a standard technique for approximate regression via subsampling [10, 26, 35]. It is well known that if we sample O ( d log( d/δ ) /ϵ 2 ) rows of X and the corresponding entries of v according to the leverage scores of X , then we can solve a reweighted regression problem on just these rows to find ̂ b satisfying ∥ ∥ ∥ X ̂ b -v ∥ ∥ ∥ 2 2 ≤ (1 + ϵ ) · min b ∈ R d ∥ Xb -v ∥ 2 2 . Observe that to compute ̂ b , we only need to observe potential outcomes for the units (i.e., the entries of v ) that are sampled. Thus, this almost allows us to recover a similar bound to (1), up to a multiplicative (1 + ϵ ) factor, in the partial observation setting. However, the leverage score of any one row may be very small, and if sampled, that row may be scaled up by a large factor in our reweighted regression problem (to preserve expectation). It thus becomes difficult to control to error introduced by approximating t by v .

To handle this issue, we mix leverage score sampling, with uniform sampling at rate α . This increases our sample complexity to O ( d log( d/δ ) /ϵ 2 + αn ) , but allows us to bound the error due to approximating t by v by (1+ ϵ ) d α ∥ µ ∥ 2 2 . This yields the final sample complexity and error bound of Theorem 1. For a full proof, see Appendix B.

2.2 ATE with Full Observation

We build on our techniques for ITE estimation to tackle the ATE estimation problem. We start with the classic Horvitz-Thompson (HT) estimator for ATE, defined below.

Definition 4 (Horvitz-Thompson estimator) . Given two outcome vectors y (1) , y (0) ∈ R n , let Z + and Z -= [ n ] \ Z + be a random partitioning of the units to two groups under a distribution P . Then the Horvitz-Thompson (HT) estimator is ̂ τ := 1 n ( ∑ i ∈ Z + y (1) i P P [ i ∈ Z + ] -∑ i ∈ Z -y (0) i P P [ i ∈ Z -] ) , where P P [ i ∈ Z + ] denotes the probability of placing i in Z + when the partitioning is performed according to distribution P .

Note that if units are assigned independently with equal probability to treatment and control, the HT estimator is equivalent to outputting the average of the random vector v used in our regression-based estimate for ITE. In this case, the HT estimator is unbiased and well known to have variance bounded by 1 n 2 ∥ µ ∥ 2 2 , where µ := y (1) + y (0) . Our goal is to improve this variance to instead depend on the error of the best linear fit to µ , min b ∈ R d ∥ Xb -µ ∥ 2 2 , as in Theorem 2. To do so, we will employ the classic technique of regression adjustment.

Horvitz-Thompson on Regression Adjusted Outcomes. Suppose we were given a vector b ∈ R d determined independently of the randomness in the HT estimator. Then applying the HT estimator directly on the regression adjusted vectors ˜ y ( i ) := y ( i ) -Xb (which have the same ATE as the original vectors), would yield an unbiased estimate for the ATE with variance at most 1 n 2 ∥ Xb -µ ∥ 2 2 . Naturally, we would like to find such a b making this variance bound as small as possible.

To do so, we will apply the same approach as in ITE estimation. We cannot directly regress µ onto our covariates to find an optimal b , as we cannot observe any entry of µ since it is the sum of the two potential outcomes. Instead, we can construct a vector u with E [ u ] = µ , and use it as a surrogate in our regression problem. In particular, we let u i = 2 y (1) i with probability 1 / 2 and u i = 2 y (0) i with probability 1 / 2 . As in the ITE setting, we can bound the error incurred by approximating µ by u in our regression problem, and thus bound the variance of our regression adjusted HT estimator. We note that to the best of our knowledge, this idea has not been leveraged in ATE estimation.

Unfortunately, a difficulty arises in this approach. We need to assign each unit to a treatment or control group both when computing u to solve our regression problem, and later when applying the HT estimator. If we use the same assignment for both steps (as is required since we can only observe one of y (1) i or y (0) i for each unit), we introduce bias in our estimator.

Avoiding Estimator Bias. To alleviate this issue, we use ridge leverage score sampling to first partition the units into two groups. Ridge leverage score sampling is a technique similar to leverage score sampling that gives similar approximation guarantees to linear regression but for ridge regression problems (i.e., linear regression problems with an ℓ 2 regularization term). We pick a ridge (regularization) parameter that guarantees that each unit only needs to be in the first or second group with a probability of at most 0 . 5 while guaranteeing that the solution obtained from this sampling is a (1 + ϵ ) approximation of the optimal ridge regression solution. Then we solve ridge regression problems on the groups separately to obtain two vectors ̂ b (1) and ̂ b (2) , for the first and the second group, respectively. We then use ̂ b (1) to adjust the outcomes of the second group and use ̂ b (2) to adjust the outcomes of the first group. Since the vector of each group is not used to adjust the outcomes of itself, this gives an unbiased estimator.

2.3 ATE with Partial Observation

We now consider ATE estimation where we desire to only observe y (1) i or y (0) i for a small subset of the population. If we draw a subset of size m uniformly at random and observe the outcomes of the samples in the subset independently and with equal probability and apply the Horvitz-Thompson estimator, the variance is E [ ( ̂ τ -τ ) 2 ] ≤ ∥ µ ∥ 2 2 mn + ∥ t -t ∥ 2 2 m . We would like to replace terms ∥ µ ∥ 2 2 and ∥ t -t ∥ 2 2 with min b ∥ µ -Xb ∥ 2 and min b ∥ t -t -( X -X ) b ∥ 2 , respectively. To deal with the ∥ µ ∥ 2 2 term, we use the HT estimator applied to the GSW design [18] on the uniformly selected subset of size m of the data. Then for ϕ ∈ (0 , 1) that the experiment designer selects, The variance reduces to 1 m 2 min b ∈ R d [ m n 1 ϕ ∥ Xb -µ ∥ 2 2 + ζ 2 1 -ϕ ∥ b ∥ 2 2 ] + ∥ t -t ∥ 2 2 mn . To replace the ∥ t -t ∥ 2 2 mn term with the error

of the best linear fit, we use the partial observation RAHT estimator that given a solution vector ˜ b adjusts the estimate obtained by the GSW design on the sample of size m . Similar to our approach for partial observation ITE estimation, we compute ˜ b by sampling about d units according to leverage scores of the matrix and performing the random vector regression (pseudo-outcome regression) on a vector that is equal to t / 2 in expectation. Let S be the set of units selected by leverage score sampling and S = [ n ] \ S . We denote the estimate obtained by sampling m units from S (we denote this set with T ) and applying GSW design with ̂ τ S . Then, the RAHT estimator estimates the ATE as ̂ τ S -1 m ∑ i ∈ T ( x i -x S ) ˜ b . This estimate is biased because some units have a higher probability of being included in S and, therefore, would contribute to the estimate with a lower probability. To resolve this issue, we also estimate the ATE on the set S with the same random vector of outcomes that is used in regression to learn ˜ b . We use HT estimator for this estimate, and we denote it by ̂ τ S . Then, our final estimate is a convex combination of the two estimates as the following.

This estimator is unbiased and achieves the variance bounds stated in Theorem 3.

3 Preliminaries

Notation. Vectors and matrices are denoted with bold lowercase and capital letters, respectively. the Hadamard (entrywise) product of x , y is denoted by x ⊙ y . We denote the vectors of all ones and all zeros in R n by 1 n and 0 n , respectively. We drop the subscript if the dimension is clear. We denote the identity matrix of dimension n by I n . For X ∈ R n × d we denote the projection matrix of X by π X := X ( X ⊤ X ) + X , where + denotes the pseudoinverse. For x ∈ R n and i ≤ n , x i denotes its i th entry and x 1: i denotes a vector in R i that is equal to its first i entries. We use X i, : and x i interchangeably to denote row i of X . Similarly for S ⊆ [ n ] , X S, : ∈ R | S |× d denotes the submatrix with row indices in S and for y ∈ R n , y S ∈ R | S | denotes the vector restricted to indices in S .

Definition 5 (Ridge leverage scores [2]) . Let X ∈ R n × d , λ ≥ 0 , and x i ∈ R d be the i th row of X . Then the λ -ridge leverage score of row i of X is defined as ℓ ( λ ) i := x ⊤ i ( X ⊤ X + λ I ) -1 x i . We denote the leverage scores (i.e., λ = 0 ) with ℓ i . If the corresponding matrix X is not clear from the context, we denote the λ -ridge leverage scores with ℓ ( λ ) i ( X ) .

Note that ℓ i is the i 'th diagonal entry of the projection matrix π X = X ( X ⊤ X ) -1 X ⊤ . Thus, since tr ( π X ) = rank ( X ) , ∑ n i =1 ℓ i = rank ( X ) . Since in our approach, we need to sample different sets of population for different purposes, we require a bound on the ridge leverage scores that can be achieved by taking the regularization factor λ to be large enough.

The following theorem provides a sampling procedure based on λ -ridge leverage scores to solve a linear ridge regression problem approximately.

Theorem 7 (6, 12) . Let ϵ, δ ∈ (0 , 1] , and u ∈ R n be a vector that upper bounds the λ -ridge leverage scores of a matrix X ∈ R n × d , i.e., ℓ ( λ ) i ≤ u i . Let S = R n × n be a random diagonal matrix in which S ii = 1 / √ p i with probability p i = min { 1 , 3 · u i · log( d/δ ) /ϵ } , and S ii = 0 , otherwise. Then for any y ∈ R n , letting ̂ b = arg min b ∈ R d [ ∥ SXb -Sy ∥ 2 2 + λ ∥ b ∥ 2 2 ] , with probably at least 1 -δ ,

4 Average Treatment Effect Estimation

In this section, we first analyze the random vector regression (i.e., pseudo-outcome regression) approach which is used both for our ITE and ATE estimation. We then discuss our full observation ATE estimation using this approach and provide a sketch of the proof of Theorem 2.

4.1 Random Vector Regression

For both ATE and ITE estimation, regression adjustment allows us to reduce the variance of estimation to the error of the best linear fit on vectors µ and t , respectively. Since we do not have access to the

entries of these vectors, we cannot compute a solution vector directly. Therefore, for ATE (or ITE), our approach is to instead compute a solution vector b by performing regression on a random vector that, in expectation, is equal to µ (or t ). The following theorem characterizes the error of the solution vector obtained in this way.

The following is a more general result that combines this random assignment technique with leverage score sampling to allow us to observe only a small subset of the population. Algorithm 3 and the proof of the following theorem are presented in the appendix.

Theorem 8 (Random vector regression) . Let y (0) , y (1) ∈ R n and y ∈ R n be a random vector such that for each i ∈ [ n ] , y i is independently and with equal probability is either equal to y (0) i or y (1) i . Moreover, let b ∗ = arg min b ∥ Xb -y ∥ 2 2 . Let µ := y (1) + y (0) . Then

Proof. Let z ∈ {-1 , +1 } n , where z i = +1 if y i = y (1) i , and z i = -1 , otherwise. First, note that 2 y = µ + z ⊙ t , where µ = y (1) + y (0) , and t = y (1) -y (0) . Therefore

Moreover since I -π X and π X are orthogonal to each other,

Now note that ∥ ( µ -π X µ ) ∥ 2 2 = min b ∥ Xb -µ ∥ 2 2 . So we only need to bound E [ ∥ π X ( z ⊙ t ) ∥ 2 2 ] .

̸

We have ∥ π X ( z ⊙ t ) ∥ 2 2 = ( z ⊙ t ) ⊤ π X ( z ⊙ t ) = z ⊤ T π X Tz , where T is the diagonal matrix associated with the vector t . Since z i and z j are independent for i = j , we have

Then the result follows by noting that ( π X ) ii is equal to the leverage score of row i , and the sum of leverage scores is less than or equal to d .

We can use the same theorem to characterize the regression error for t by changing the sign of y (0) .

4.2 Full Observation ATE

We now sketch a proof of our result for ATE estimation with full observation.

Proof sketch of Theorem 2. Here, we only sketch our proof for the variance bound. For S ⊆ [ n ] and z ∈ {-1 , +1 } n (where Z + = { i : z i = +1 } and Z -= { i : z i = -1 } ), we define

Similarly, define τ S and ̂ τ S . Then for ̂ τ as defined in Algorithm 1, we have,

By Cauchy-Schwarz inequality, E z [( ̂ τ S -τ S ) · ( ̂ τ S -τ S )] ≤ √ E z [( ̂ τ S -τ S ) 2 ] · E z [( ̂ τ S -τ S ) 2 ] . Therefore we only need to bound E z [ ( ̂ τ S -τ S ) 2 ] and E z [ ( ̂ τ S -τ S ) 2 ] . We can bound E z [ ( ̂ τ S -τ S ) 2 ] by Theorem 7 (ridge leverage score sampling) and using the fact that ̂ τ S is obtained by ˜ y (2 , 1) and ˜ y (2 , 0) which are adjusted by ̂ b (2) , a vector that is learned from the units in the set S . Since S and S are disjoint, ̂ b (2) is independent of z S and this allows us to bound E z [ ( ̂ τ S -τ S ) 2 ] . The bound on E z [ ( ̂ τ S -τ S ) 2 ] follows from a similar argument.

5 Experiments

In this section, we compare our method for estimating ATE and ITE with full observation with prior art and baselines on synthetic and real-world datasets. For ATE estimation, our experiments demonstrate that our approach either outperforms other unbiased estimation approaches or has comparable performance in terms of the achieved variance.

Algorithm 1: ATE estimation with leverage score sampling and cross adjustment

1 Input: X ∈ R n × d , y (0) , y (1) ∈ R n , ϵ, δ ∈ (0 , 1] .

2 Set y ∈ R n to a random vector such that for each i ∈ [ n ] , y i is independently and with equal probability either equal to y (0) i or y (1) i .

3 Set Z + to the set of entries i such that y i is equal to y (1) i and Z -= [ n ] \ Z + .

4 Set λ = 6 ϵ 2 · ζ 2 log( n/δ ) , where ζ = max i ∈ [ n ] ∥ x i ∥ 2 .

5 Set S ⊆ [ n ] to a random set of indices where each i ∈ [ n ] is included in S independently and with probability 0 . 5 . Set S = [ n ] \ S .

6 Set S to an n × n diagonal matrix corresponding to S where each entry S ii = 2 if i ∈ S , and S ii = 2 , otherwise. Similarly, set S .

7 Set ̂ b (1) = arg min b ∥ SXb -Sy ∥ 2 2 + λ · ∥ b ∥ 2 2 , ̂ b (2) = arg min b ∥ SXb -Sy ∥ 2 2 + λ · ∥ b ∥ 2 2 .

8 Set ˜ y (1 , 0) = y (0) -X ̂ b (1) , ˜ y (1 , 1) = y (1) -X ̂ b (1) , ˜ y (2 , 0) = y (0) -X ̂ b (2) , ˜ y (2 , 1) = y (1) -X ̂ b (2) .

9 Return ̂ τ := 2 n ( ∑ i ∈ Z + ∩ S ˜ y (1 , 1) i + ∑ i ∈ Z + ∩ S ˜ y (2 , 1) i -∑ i ∈ Z -∩ S ˜ y (1 , 0) i -∑ i ∈ Z -∩ S ˜ y (2 , 0) i ) .

ATE methods. Our ATE estimation method is compared against five other approaches. (i) HT Uniform: This employs the Hurvitz Thompson estimator, using a uniform assignment of units to treatment and control groups. (ii) GSW: This design follows the methodology in [18]. (iii) Classic Regression Adjustment: As described in [24], this estimator is inherently biased. (iv) Leverage Score-Based: In this method, we learn two distinct vectors for y (1) and y (0) through leverage score sampling. The difference between these vectors predicts the treatment effect. Though this method has its advantages, it is biased and was previously employed by Addanki et al. [1]. (v) 4 Vectors: Obtained by specializing Mou et al. [28] to linear regressors, this approach adopts the same cross-adjustment mechanism to mitigate bias as in this paper. However, instead of random vector regression, it determines separate vectors for y (1) and y (0) for each group, leading to four vectors obtained by linear regressions.

ITE methods. We discuss two distinct approaches for ITE methods. (i) Baseline: The ideal scenario or "oracle" baseline involves utilizing a linear function, trained on vector t . Naturally, this is not feasible in practical applications. (ii) leverage score based: This is based on learning two different vectors by ridge leverage score sampling and using the difference of the two linear functions for estimating t . This is an adaptation of the algorithm in [1] to the full observation setting - we make this adaptation because the exact algorithm of [1] cannot be applied to the full-observation setting due to sampling with replacement. We do not compare to [28] because their guarantees are only for an infinite population.

Synthetic Datasets. For synthetic datasets, we will construct y (0) , y (1) such that µ (for the ATE case) or t (for the ITE case) is a linear function of the covariate matrix.

ATE Dataset. There are 50 covariates (i.e., features). Each entry of X ∈ R n × d is a uniform random number in [0 , 0 . 01] . Each entry of b ∈ R 50 is a uniform random number in [0 , 1] . The individual treatment effect vector µ is Xb + r , where each entry of r ∈ R n is randomly picked from a mean zero Gaussian distribution with standard deviation of 0.2. Each entry of y (0) is picked uniformly at random from [0 , 5] . Then y (1) = -y (0) + µ and t = y (1) -y (0) . The results for this dataset with different number of samples is shown in the right plot of Figure 1. Our experiments illustrate that on the synthetic dataset, our approach outperforms all other approaches except the GSWdesign. However, we note that our approach is computationally more efficient than GSW design (see Table 6) and gives a much simpler design that can be used in different settings.

ITE Dataset. The covariate matrix X and the vectors b , r , y (0) are picked similar to the ATE dataset. Then individual treatment effect vector t is Xb + r and y (1) = y (0) + t . The results for this dataset with different number of samples is shown in the left plot of Figure 1. ITE error for our method is consistently smaller than leverage score based method.

Real-World Datasets We analyze the following three distinct real-world datasets. A comprehensive breakdown, including variance and other measurements for each dataset, is available in Table 5 in the appendix.

leveragescorebased

sno

0.025

baseline

0.020

ITE error

0.015

0.010

0.005

0.000

2500

5000

75001

1000012500150001750020000

PopulationSize

HT uniform

classicregression adjustment leveragescorebased

4 vectors sno

125015001750

2000

PopulationSize

Boston Dataset [17]: This is a dataset of housing prices in the Boston area, consisting of 506 samples and 13 features. Since it has only one label, we set y (1) = y (0) , i.e., ATE is zero.

IHDP Dataset [16, 11]: Derived from the characteristics of children and their mothers, this dataset comprises 747 samples and 25 features. ATE for is -4.016.

Twins Dataset [3]: This dataset is constructed based on the characteristics and mortality rates of twin births in the US. We specifically selected samples that have complete feature values, resulting in a dataset of 32,120 samples with 50 features. The ATE of this dataset is 0.0064.

6 Conclusion

In this paper, we considered the problems of mean estimation, ATE and ITE estimations in the presence of covariates. We considered the finite population setting and provided non-asymptotic variance bounds for several novel variants of the classical regression adjustment method-based estimators. Our guarantees are model-free, even if the covariates are arbitrary (are not informative of the estimand and may be chosen adversarially), the variance bounds are still meaningful and almost match the variance bounds of widely used estimators not based on covariates.

Our algorithms are simple and efficient, and we believe they readily extend to many related settings like arbitrary assignment probabilities and online treatment assignments. We also believe the results can be stated for the kernel setting. These extensions will be part of future work.

ATE error

0.08

0.06

0.04

0.02

0.00

500

750

1000

2250

2500

References

Raghavendra Addanki, David Arbour, Tung Mai, Cameron N Musco, and Anup Rao. Sample constrained treatment effect estimation. In Advances in Neural Information Processing Systems , 2022.

Ahmed Alaoui and Michael W Mahoney. Fast randomized kernel ridge regression with statistical guarantees. Advances in neural information processing systems , 28, 2015.

Douglas Almond, Kenneth Y Chay, and David S Lee. The costs of low birth weight. The Quarterly Journal of Economics , 120(3):1031-1083, 2005.

David Arbour, Drew Dimmery, Tung Mai, and Anup Rao. Online balanced experimental design. In International Conference on Machine Learning , pages 844-864. PMLR, 2022.

David Arbour, Drew Dimmery, and Anup Rao. Efficient balanced treatment assignments for experimentation. In International Conference on Artificial Intelligence and Statistics , pages 3070-3078. PMLR, 2021.

Haim Avron, Kenneth L Clarkson, and David P Woodruff. Sharper bounds for regularized data fitting. Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques , 2017.

Adam Bloniarz, Hanzhong Liu, Cun-Hui Zhang, Jasjeet S Sekhon, and Bin Yu. Lasso adjustments of treatment effect estimates in randomized experiments. Proceedings of the National Academy of Sciences , 113(27):7383-7390, 2016.

Xue Chen and Eric Price. Active regression via linear-sample sparsification. In Conference on Learning Theory , pages 663-695. PMLR, 2019.

William G Cochran. Sampling techniques . John Wiley & Sons, 1977.

Michael B Cohen, Yin Tat Lee, Cameron Musco, Christopher Musco, Richard Peng, and Aaron Sidford. Uniform sampling for matrix approximation. In Proceedings of the 2015 Conference on Innovations in Theoretical Computer Science , pages 181-190, 2015.

Vincent Dorie. NPCI: non-parametrics for causal inference. https://github.com/vdorie/ npci , 2016.

Matthew Fahrbach, Gang Fu, and Mehrdad Ghadiri. Subquadratic kronecker regression with applications to tensor decomposition. In Advances in Neural Information Processing Systems , 2022.

David A Freedman. On regression adjustments to experimental data. Advances in Applied Mathematics , 40(2):180-193, 2008.

Alan S Gerber and Donald P Green. Field experiments and natural experiments, 2008.

Robert Greevy, Bo Lu, Jeffrey H Silber, and Paul Rosenbaum. Optimal multivariate matching before randomization. Biostatistics , 5(2):263-275, 2004.

Ruth T Gross et al. Infant health and development program (ihdp): Enhancing the outcomes of low birth weight, premature infants in the united states, 1985-1988, 1993.

David Harrison Jr and Daniel L Rubinfeld. Hedonic housing prices and the demand for clean air. Journal of environmental economics and management , 5(1):81-102, 1978.

Christopher Harshaw, Fredrik Sävje, Daniel Spielman, and Peng Zhang. Balancing covariates in randomized experiments with the gram-schmidt walk design. arXiv preprint arXiv:1911.03071 , 2019.

Daniel G Horvitz and Donovan J Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American statistical Association , 47(260):663-685, 1952.

Andrew Jesson, Panagiotis Tigas, Joost van Amersfoort, Andreas Kirsch, Uri Shalit, and Yarin Gal. Causal-bald: Deep bayesian active learning of outcomes to infer treatment-effects from observational data. Advances in Neural Information Processing Systems , 34:30465-30478, 2021.

Nathan Kallus. Optimal a priori balance in the design of controlled experiments. Journal of the Royal Statistical Society. Series B (Statistical Methodology) , 80(1):85-112, 2018.

Edward H Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. arXiv preprint arXiv:2004.14497 , 2020.

Xinran Li and Peng Ding. Rerandomization and regression adjustment. Journal of the Royal Statistical Society Series B: Statistical Methodology , 82(1):241-268, 2020.

Winston Lin. Agnostic notes on regression adjustments to experimental data: Reexamining freedman's critique. THE ANNALS of APPLIED STATISTICS , pages 295-318, 2013.

Ping Ma, Michael Mahoney, and Bin Yu. A statistical perspective on algorithmic leveraging. In International Conference on Machine Learning , pages 91-99. PMLR, 2014.

Michael W Mahoney et al. Randomized algorithms for matrices and data. Foundations and Trends® in Machine Learning , 3(2):123-224, 2011.

Kari Lock Morgan and Donald B Rubin. Rerandomization to improve covariate balance in experiments. The Annals of Statistics , pages 1263-1282, 2012.

Wenlong Mou, Martin J Wainwright, and Peter L Bartlett. Off-policy estimation of linear functionals: Non-asymptotic theory for semi-parametric efficiency. arXiv preprint arXiv:2209.13075 , 2022.

Akanksha Negi and Jeffrey M Wooldridge. Revisiting regression adjustment in experiments with heterogeneous treatment effects. Econometric Reviews , 40(5):504-534, 2021.

Tian Qin, Tian-Zuo Wang, and Zhi-Hua Zhou. Budgeted heterogeneous treatment effect estimation. In International Conference on Machine Learning , pages 8693-8702. PMLR, 2021.

Mark Rudelson and Roman Vershynin. Hanson-wright inequality and sub-gaussian concentration. Electronic Communications in Probability , 18:1-9, 2013.

Iiris Sundin, Peter Schulam, Eero Siivola, Aki Vehtari, Suchi Saria, and Samuel Kaski. Active learning for decision-making from imbalanced observational data. In International conference on machine learning , pages 6046-6055. PMLR, 2019.

Stefan Wager, Wenfei Du, Jonathan Taylor, and Robert J Tibshirani. High-dimensional regression adjustments in randomized experiments. Proceedings of the National Academy of Sciences , 113(45):12673-12678, 2016.

Martin B Wilk. The randomization analysis of a generalized randomized block design. Biometrika , 42(1/2):70-79, 1955.

David P Woodruff et al. Sketching as a tool for numerical linear algebra. Foundations and Trends® in Theoretical Computer Science , 10(1-2):1-157, 2014.

A Warm-up: Mean Estimation

In this section, we consider the problem of estimating the mean of a single vector y by observing only a few entries (i.e., y := 1 n ∑ n i =1 y i ). The most classic approach is to simply uniformly sample m units from the population and report their average. This results in a variance of at most 1 mn ∥ y -y ∥ 2 2 , where y is a vector of the same size as y with all of its entries equal to y .

The main theme of regression adjustment is to replace the dependence of variance on ∥ y -y ∥ 2 2 with a dependence on the error of the best linear fit, i.e., min b ∈ R d ∥ ∥ y -y -( X -X ) b ∥ ∥ 2 2 , where X is a vector of the same size as X with all of its rows equal to x = 1 n ∑ n i =1 x i . We wish to do so without making any modeling assumptions -acommon modeling assumption in statistics is that the response vector is a linear function of the covariates plus a Gaussian noise. However, our goal is to achieve significant variance reduction if y -y can be well approximated by a vector in the span of X -X without any such modeling assumptions. In the worst case, when y -y is orthogonal to the column span of ( X -X ) , min b ∈ R d ∥ ∥ y -y -( X -X ) b ∥ ∥ 2 2 = ∥ y -y ∥ 2 2 which recovers the result of the classic approach.

For the mean estimation problem, if an oracle provided b ∗ = arg min b ∈ R d ∥ ∥ y -y -( X -X ) b ∥ ∥ 2 2 , the estimator ̂ y = 1 | S | ∑ i ∈ S y i -( x i -x ) ⊤ b ∗ is unbiased with its variance bounded by

1 mn min b ∈ R d ∥ ∥ y -y -( X -X ) b ∥ ∥ 2 2 , where S is a uniformly sampled subset of the population of size m . This is a classic result (see Chapter 7 of [9]) that also follows from the following Lemma.

Lemma 9 (9 - Chapter 2) . Let y ∈ R n and ̂ y be the average of a uniformly sampled set of entries of y of size m . Then

where y is a vector of size equal to y for which all the entries are equal to y .

One can easily see that by applying Lemma 9 to ˜ y , where ˜ y i = y i -( x i -x ) ⊤ b , a variance of ∥ y -y -( X -X ) b ∥ 2 2 m ( n -1) (1 -m n ) can be obtained.

Key challenge. The main challenge for the mean estimation problem is to compute b ∗ (or a vector close to it) without observing all the entries of vector y . Note that we do not have access to any entry of y -y because we cannot compute y without observing all the entries of y .

We resolve the issue of not being able to compute y by performing the regression on a modified matrix. We add a column of all ones to the matrix X -X and perform a regression on y using the modified matrix - see the proof of Theorem 10. More specifically, we prove the following.

To solve the regression problem on the right-hand side without observing all the entries of y , we use leverage score sampling [25, 10]. Leverage score sampling is a numerical linear algebraic tool that allows for solving a linear regression problem approximately by subsampling about d units from the population and solving a regression problem over these units, where d is the number of features [8] see Section 3 for details. Combining these techniques gives the following results.

Theorem 10 (Leverage score sampling based regression adjustment) . Let ϵ, δ ∈ (0 , 1] , and m ∈ N and X ∈ R n × d be a covariate matrix. Then, Algorithm 2 computes an unbiased estimator of the mean of an outcome vector y (on a population of size n ) with O ( d log( d/δ ) /ϵ 2 + m ) samples. Moreover, there is an event E with Pr( E ) ≥ 1 -δ (over the randomness of the algorithm) such that the variance of the estimator conditioned to E is bounded by

Note that since the dependence of number of samples on δ is logarithmic, we can pick δ polynomially small, e.g., δ = 1 /n 2 . The variance upper bound achieved by our method is compared to other methods in Table 3.

Regression adjustment. Our estimators are based on a classic technique in statistics called regression adjustment [9]. Given a vector ̂ b (either given by an oracle or learned from the data), the classical technique uses the sample covariates to 'adjust' the estimate of the sample mean 1 | S ′ | ∑ i ∈ S ′ y i and outputs the following estimate

If S ′ is a random sample independent of ̂ b , the adjusted estimator is unbiased. The adjusted estimate will have a lower variance than the sample mean estimator if ∥ y -y -( X -X ) ̂ b ∥ 2 2 < ∥ y -y ∥ 2 2 .

Lemma 11. Let y ∈ R n and X ∈ R n × d . Let ̂ b be a fixed/preassigned vector. Then the variance of ̂ y := 1 | S ′ | ∑ i ∈ S ′ y i -( x i -x ) ⊤ ̂ b , where S ′ is a uniformly sampled subset of [ n ] of size m , is

Proof. Since ̂ b is fixed, by linearity of expectation

Therefore by Lemma 9 (applied to the quantities y i +( x -x i ) ⊤ ̂ b ),

and the result follows.

Therefore our goal is to find ̂ b that minimizes ∥ y -y -( X -X ) ̂ b ∥ 2 2 . Our approach is to use leverage score sampling (Theorem 7) to find a vector ̂ b that approximately minimizes this error. A main technique in our approach is that the samples picked for leverage score sampling are independent of the samples drawn for estimation. More specifically, our estimator is based on two sets of samples. We use one set of samples S to estimate ̂ b and we use the another set S ′ of m uniform samples (that is independent of S ) in the following estimator ̂ y = 1 | S ′ | ∑ i ∈ S ′ y i +( x -x i ) ⊤ ̂ b . By Lemma 11, this is unbiased and achieves a variance of

Although it is straightforward to carry this independence for the mean estimation problem, as we will see, we require a more involved algorithm to provide it for the treatment effect estimation problems.

Algorithm 2: (Ridge) leverage score sampling based regression adjustment

1 Input: X ∈ R n × d , y ∈ R n , m ∈ N , 0 < ϵ < 1 .

2 For Theorem 10, set λ = 0 , and for Theorem 12, set λ = ζ 2 · n/m // ζ = max i ∥ X i : ∥ 2

3 Compute the leverage scores ℓ ( λ ) i of ˜ X = [ ( X -X ) 1 ] .

4 Set p i = min { 1 , 3 · ℓ ( λ ) i · log( d/δ ) /ϵ 2 } for i ∈ [ n ] .

6 Let ˜ b = arg min b ∈ R d +1 ∥ ∥ ∥ S ˜ Xb -Sy ∥ ∥ ∥ 2 2 + λ ∥ b 1: d ∥ 2 2 , and ̂ b ∈ R d be a vector consisting of

5 Let S ∈ R n × n be a random diagonal matrix with S ii = 1 / √ p i with probability p i , and S ii = 0 , otherwise.

the first

d

entries of

b

.

̂ m i ∈ S 8 Return ̂ τ .

˜ 7 Let S ′ be a uniformly sampled subset of size m of [ n ] and τ = 1 ∑ ′ y i +( x -x i ) ⊤ ̂ b

Theorem 10 (Leverage score sampling based regression adjustment) . Let ϵ, δ ∈ (0 , 1] , and m ∈ N and X ∈ R n × d be a covariate matrix. Then, Algorithm 2 computes an unbiased estimator of the mean of an outcome vector y (on a population of size n ) with O ( d log( d/δ ) /ϵ 2 + m ) samples. Moreover, there is an event E with Pr( E ) ≥ 1 -δ (over the randomness of the algorithm) such that the variance of the estimator conditioned to E is bounded by

Proof. Let ˜ X := [ ( X -X ) 1 ] . We first show that

Note that min b ∈ R d +1 ∥ ∥ ∥ ˜ Xb -y ∥ ∥ ∥ 2 2 = ∥ ∥ π ˜ X y -y ∥ ∥ 2 2 . Since 1 is orthogonal to X -X ,

Moreover π 1 y = y . Since 1 is orthogonal to X -X , y = y · 1 is orthogonal to X -X , and π ( X -X ) y = 0 . Therefore

Subtracting both sides by y and taking the norm, we have

which implies our claim. Now note that by sampling O ( d log( d/δ ) /ϵ 2 ) entries of y , by Theorem 7, with high probability, we can find ˜ b such that

Let ̂ b ∈ R d be a vector obtained by taking the first d entries of ˜ b . Then, we have

Now consider the following regression problem:

The optimal solution is 1 n 1 ⊤ ( y -( X -X ) ̂ b ) = 1 n 1 ⊤ y = y . Therefore

Thus by (4) and (5),

and the result follows by Lemma 11.

Note that the number of samples in Theorem 10 depends on the number of covariates (also called features). This dependence can be alleviated by using ridge leverage score sampling at the expense of a higher variance bound. If we want to sample m<d log( d/δ ) /ϵ 2 , we can use ridge leverage score sampling, which gives a bound on the ridge regression error instead of the regression error. Using λ -ridge leverage scores, for λ = ζ 2 · n m , where ζ is the maximum ℓ 2 norm over rows of X , gives the following result. Before providing this result, note that the block leverage score sampling approach of [12] shows that to solve

we can use the same ridge leverage sampling as for min b ∈ R d [ ∥ Xb -y ∥ 2 2 + λ ∥ b ∥ 2 2 ] , and the number of samples only needs to be increased by a constant factor. We require this for our ridge leverage score sampling-based mean estimation result.

Theorem 12 (Ridge leverage score sampling based regression adjustment) . Let 0 < ϵ < 1 , 0 < δ < 0 . 5 , m ∈ N , and X be a covariate matrix. Then Algorithm 2 computes an unbiased estimator of the mean of an outcome vector y (on a population of size n ) by using O ( m log( m/δ ) /ϵ 2 ) samples. Moreover, there is an event E with Pr( E ) ≥ 1 -δ (over the randomness of the algorithm) such that the variance of the estimator conditioned on E is bounded by

Proof. Let ˜ X := [ ( X -X ) 1 ] , and

Matrix ˜ Z corresponds to the regression problem

In other words, defining the vector ˜ y ∈ R n + d as a vector that is equal to y on the first n entries and is zero on other entries, we have

for any vector b ∈ R d +1 . Now note that the solution to arg min b ∈ R d +1 ∥ ∥ ∥ ˜ Zb -˜ y ∥ ∥ ∥ 2 2 is

by definition of ˜ y . Moreover by definition,

Therefore since 1 ⊤ ( X -X ) = 0 ⊤ , we have

Since this is a block diagonal matrix

Therefore

Thus

Note that ( ( X -X ) ⊤ ( X -X ) + λ I d ) -1 ( X -X ) ⊤ ( y -y ) is the solution to

and because 1 is in the null space of X -X ,

(

(

X

-

X

)

) +

Therefore and

Now note that by ridge leverage score sampling, we find a vector ˜ b ∈ R d +1 such that, with high probability,

Therefore

Now let ̂ b ∈ R d be the vector that is equal to the first d entries of ˜ b . Then we have

Now consider a regression problem of the following form.

The optimal solution to this problem is 1 n 1 ⊤ (( y -( X -X ) ̂ b )) = 1 n 1 ⊤ y = y . Therefore

Thus by (8) and (9), and because λ ∥ ∥ ∥ ̂ b ∥ ∥ ∥ 2 2 ≥ 0 ,

Therefore by Lemma 11, the variance of the regression-adjusted estimator with m samples is less than,

Finally by definition λ = ζ 2 n/m = ζ 2 2 , and the result follows.

mn nm m

)

-

1

(

X

-

X

)

⊤

(

y

-

y

) =

(

(

X

-

X

)

⊤

(

X

-

⊤

(

X

-

X

λ

I

d

X

) +

λ

I

d

)

-

1

(

X

-

X

)

⊤

y

.

We now explain previous approaches to mean estimation. The classical approach to regression adjustment [9] is to learn a vector ̂ b from the same set of samples that are used for estimation.

The Gram-Schmidt walk design of Harshaw et al. [18] has originally been developed for the average treatment effect estimation and achieves the following bound using the Horvitz-Thompson estimator -see Algorithm 5.

Theorem 13 (Gram-Schmidt walk design [18]) . Let 0 < ϕ < 1 . Given y (1) , y (0) ∈ R n and X ∈ R n × d , the Horvitz-Thompson estimator under the Gram-Schmidt walk design is unbiased and has a variance of

where µ = y (1) + y (0) .

A special setting of ATE estimation problem reduces to mean estimation. Suppose y (1) = y and y (0) = 0 . Then the average treatment effect for this problem equals the mean of y . This approach partitions the samples into two groups. The first outcome is observed for the first group, and the second outcome is observed for the second group. The estimate is then produced by the HorvitzThompson estimator. Note that in our special case, only the samples in the first group amount to observation since y (0) is the fixed vector of all zeros. This yields the following guarantee for mean estimation

Theorem 14 (Gram-Schmidt walk based design [18]) . The Horvitz-Thompson estimator applied to the output of Algorithm 5 gives an unbiased estimator of the mean of y with a variance of less than or equal to

with O ( m ) samples.

In comparison, the bound of Theorem 12 is better by a factor of m/n on the ∥ Xb -y ∥ 2 2 term.

B Individual Treatment Effect Estimation

In this section, we give an ITE estimation algorithm with partial observation (Algorithm 3). Our approach is to learn a vector b that nearly optimally fits t := y (1) -y (0) linearly onto X , i.e., ∥ Xb -t ∥ 2 is small. Since we do not have access to t / 2 , we perform regression on a random vector for which the expectation is t . To this end, we construct the random vector by setting y i independently to either y (1) i or -y (0) i with equal probability. If we observe all samples, and use Theorem 8 on vectors y (1) and -y (0) , this approach gives the following expected error bound.

We can achieve a similar bound using leverage score sampling by thresholding the probability of sampling rows from below while we guarantee that the number of samples does not increase too much with high probability. The following is the main technical contribution of this part.

Theorem 1 (ITE Estimation) . For any ϵ, δ, α ∈ (0 , 1] , there exists a randomized algorithm (Algorithm 3) that observes a potential outcome for O ( d log( d/δ ) /ϵ 2 + α · n ) units and outputs a vector ̂ t . Moreover there is an event E such that Pr( E ) ≥ 1 -δ (over the randomness of the algorithm) and

Proof. Note that 2 y = t + z ⊙ µ . Then

Algorithm 3: ITE estimation with random vector regression and leverage score sampling

1 Input: X ∈ R n × d , y (1) , y (2) ∈ R n , 0 < ϵ < 1 , 0 < α ≤ 1 .

2 Compute the leverage scores ℓ i of X .

3 Set p i = min { 1 , max { α, 3 · ℓ i · log( d/δ ) /ϵ 2 }} for i ∈ [ n ] .

5 Let ̂ b = arg min b ∈ R d +1 ∥ SXb -Sy ∥ 2 2 .

4 Let S ∈ R n × n be a random diagonal matrix with S ii = 1 / √ p i with probability p i , and S ii = 0 , otherwise.

6 Return ̂ t := 2 X ̂ b .

Moreover

Therefore by Theorem 7, and since ∥ π X t -t ∥ 2 2 = min b ∥ Xb -t ∥ 2 2 ,

Now note that

̸

where the last inequality follows from the fact that the sum of leverage scores is less than d , and the penultimate inequality follows from the definition of p i . Finally, the result follows by noting that,

C ATE with Full Observation

Our approach for ATE with full observation is similar to ITE in the sense that we perform regression on a random vector to learn a solution vector ̂ b . Then, for estimation, we apply regression adjustment based on ̂ b to adjust the outcomes. We first present a naive version of this approach that is biased and achieves a factor proportional to d 2 /n 2 in one of the terms bounding the variance. We improve this to d/n 2 by partitioning the units into two groups and using the learned vector for each group to adjust the estimation in the other group. With this approach, our estimator is unbiased.

Before discussing our results, we discuss the other approaches in the literature. The Gram-Schmidt walk design of [18] that uses the Horvitz-Thompson estimator gives the following variance for the ATE estimation.

where 0 < ϕ < 1 is an input parameter. The classic regression adjustment with interaction term gives an asymptotic variance proportional to 1 m ( σ 2 1 + 2 σ 2 0 + σ 1 , 0 ) , where σ 2 1 is the variance of ˜ y (1) i = y (1) i -y (1) +( X i -x ) b (1) [24], where b (1) is the best linear fit for y (1) -y (1) using X -X when n → ∞ [24]. Similarly, σ 2 1 is the variance of ˜ y (0) i = y (0) i -y (0) +( X i -x ) b (0) , and σ 1 , 0 is the covariance of ˜ y (1) i · ˜ y (0) i . Note that the above mentioned bound is achieved by learning two separate vectors for regression over y (1) i -y (1) and y (0) i -y (0) . Moreover, note that because these bounds are asymptotic, we need to assume a model of data generation.

For our naive approach, we first need the following technical lemma.

̸

Lemma 15. Let X ∈ R n × d , T ∈ R n × n be a diagonal matrix and z ∈ {-1 , +1 } n be a random vector where each entry is picked independently and uniformly at random. Then

where t ∈ R n is the vector corresponding to the diagonal matrix T .

Proof. Let M ∈ R n,n and. Then

Note that for any term such that there exists one of the i, j, k, l that is not equal to any of the other three, the expectation is equal to zero. Therefore we need to look at terms where for each i, j, k, l , at least one of the other three is equal to it. Therefore we have four cases:

i = j = k = l . This gives an expectation of M 2 ii . So total from these terms is ∑ n i =1 M 2 ii .

̸

i = j and k = l but i = k . This gives an expectation of M ii M kk . So total from these terms is 2 ∑ n -1 i =1 ∑ n k = i +1 M ii M kk .

̸

i = k , j = l and i = j . This gives an expectation of M 2 ij . So total from these terms is 2 ∑ n -1 i =1 ∑ n j = i +1 M 2 ij .

̸

i = l , j = k , i = j . This gives an expectation of M ij M ji . So total from these terms is 2 ∑ n -1 i =1 ∑ n j = i +1 M ij M ji .

Recall that we denote the leverage scores of matrix X with ℓ i . Therefore taking the above over our matrix M = T π X , since M ij = t i ( π X ) ij , we have

where the last two inequalities follow from the fact that the sum of leverage scores (which are the diagonal entries of the projection matrix π X ) is at most the rank of the matrix.

We are now equipped to analyze the ATE estimation obtained by purely using random vector regression and regression adjustment.

Theorem 16 (ATE with full observation) . Let y (0) , y (1) ∈ R n and y ∈ R n be a random vector such that for each i ∈ [ n ] , y i is independently and with equal probability either equal to y (0) i or y (1) i . Moreover let Z + be the set of entries i such that y i is equal to y (1) i and Z -= [ n ] \ Z + . Let ̂ b = arg min b ∥ Xb -y ∥ 2 . Let ˜ y (0) = y (0) -X ̂ b and ˜ y (1) = y (1) -X ̂ b . Let ̂ τ be the following Horvitz-Thompson estimator, i.e.,

Then | E [ ̂ τ -τ ] | ≤ d n ∥ y (1) -y (0) ∥ ∞ , and

where µ = y (1) + y (0) .

Proof. Let z ∈ {-1 , +1 } n , where z i = +1 if i ∈ Z + , and z i = -1 , otherwise. Moreover let ˜ µ := ˜ y (0) + ˜ y (1) = y (0) + y (1) -2 X ̂ b . First, note that 2 y = µ + z ⊙ t . Therefore

Moreover

Therefore

Thus by the linearity of expectation and since z i is equal to +1 and -1 with an equal probability, E [ ( µ -π X µ ) ⊤ z ] = 0 . Let T be a diagonal matrix associated with t . Then z ⊙ t = Tz . Therefore

̸

Therefore since z i and z j are independent for i = j , and z 2 i = 1 with probability one.

Therefore by the triangle inequality

Then the first part of the theorem follows by the fact that the sum of leverage scores is less than d , and noting that by definition, t = y (1) -y (0) .

̸

We now prove the second part of the theorem. First, note that since z i and z j are independent for i = j and z i = ± 1 with equal probability,

Therefore by (11) and Lemma 15,

The next step is two use a subset of samples for regression and the rest of the samples for estimating the ATE.

Since we want to have a guarantee on the regression error, we need to make sure that the sampling (i.e., partitioning) to two groups is done according to leverage scores. However, if the leverage score of a row is greater than half, then it cannot have a probability larger than its leverage score for being assigned to each group. To reduce this probability, we use the ridge leverage score. The following lemma states that by picking a large enough ridge parameter, we can achieve a small enough ridge leverage score for all rows (therefore, small enough sampling probabilities). This is at the expense of a regularization term of the form λ · ∥ b ∥ 2 2 in the regression error.

Equipped with this, we present the main result of this section.

Theorem 2 (ATE Estimation - Full Observation) . For any ϵ, δ ∈ (0 , 1] , there exists a randomized algorithm (Algorithm 1) that computes an unbiased estimate ̂ τ of the ATE τ , and for which there is an event E with Pr( E ) ≥ 1 -δ (over the randomness of the algorithm) such that

where ζ := max i ∈ [ n ] ∥ x i ∥ 2 and µ := y (0) + y (1) is the total outcome vector.

Proof. For S ⊆ [ n ] and z ∈ {-1 , +1 } n (where Z + = { i : z i = +1 } and Z -= { i : z i = -1 } ), we define

Similarly, define ̂ τ S . Then we have,

We now bound all of the terms above. Let

Note that

Similarly, we have

Therefore

where the inequality follows from Cauchy-Schwarz inequality. Moreover

Therefore we only need to bound E z [ z ⊤ S ˜ µ (2) S ( ˜ µ (2) S ) ⊤ z S ] and E z [ z ⊤ S ˜ µ (1) S ( ˜ µ (1) S ) ⊤ z S ] . Since S and S are disjoint, ˜ µ (1) S is independent of z S . Moreover, the entries of z are independent from each other. Therefore, since 2 y = µ + z ⊙ t ,

where λ = 100 ζ 2 log( n ) . We have

Note that due to guarantees of leverage score sampling,

Moreover

we have

Therefore

With a similar argument, one can show that

Thus

We can get a high probability bound for the above result using the Hanson-Wright inequality [31]. Adjusting the vectors y (1) and y (0) using Xb ∗ as ˜ y (0) = y (0) -X ̂ b and ˜ y (1) = y (1) -X ̂ b , and using the Horvitz-Thompson estimator on the adjusted outcomes ˜ y (0) and ˜ y (1) , we achieve the following result.

C.1 Unconditional Bound for ATE

The only reason the 1 -δ probability appears in our bound for the variance of ATE estimation (Theorem 2) is that leverage score sampling only gives a high probability guarantee for the regression error. Moreover, our variance only depends on the regression error, and the estimator is always unbiased, even if the leverage score sampling does not give the (1+ ϵ ) approximation guarantee on the regression. This 'bad event' happens with probability at most δ . Therefore, it is enough to bound the regression error when the bad event happens. Here we present such an approach. However, to keep our argument simple, we just work with a single solution vector. To actually apply this to our ATE estimation approach, one needs to consider both solution vectors for groups S and S and consider the bad event for both. Such an argument follows similar to the following and provides the same result (up to constant factors). This approach essentially can make all of our results unconditional (i.e., remove the 1 -δ probability) at the expense of increasing the variance/error bounds by a constant factor and adding an additive term proportional to δ .

If X ⊤ S ⊤ SX is not a spectral approximation of X ⊤ X , i.e., (1 -ϵ ) X ⊤ X ⪯ X ⊤ S ⊤ SX ⪯ (1+ ϵ ) X ⊤ X does not hold (this can be checked since we have access to matrix X and only happens with probability at most δ ), we just set b = 0 . In this case, ∥ Xb -µ ∥ 2 2 + λ ∥ b ∥ 2 2 = ∥ µ ∥ 2 2 . Otherwise (i.e., (1 -ϵ ) X ⊤ X ⪯ X ⊤ S ⊤ SX ⪯ (1 + ϵ ) X ⊤ X ) let ̂ X ∈ R ( n + d ) × d be the matrix X concatenated with factor λ of the identity matrix and ̂ µ ∈ R n + d be the vector µ concatenated with a zero vector and

Let ˜ X ∈ R ( n + d ) × d be the matrix SX concatenated with the factor λ of the identity matrix and ˜ µ ∈ R ( n + d ) be the vector S µ concatenated with a zero vector and

Then since ˜ X is a spectral approximation of ̂ X (i.e., (1 -ϵ ) ̂ X ⊤ ̂ X ⪯ ˜ X ⊤ ˜ X ⪯ (1 + ϵ ) ̂ X ⊤ ̂ X ),

where the first inequality follows by triangle inequality. Moreover, by triangle inequality and optimality of ˜ b

Now let E 1 be the event that X ⊤ S ⊤ SX is not a spectral approximation of X ⊤ X . Let E 2 be the event that X ⊤ S ⊤ SX is a spectral approximation of X ⊤ X , but ∥ SXb -S µ ∥ 2 2 + λ ∥ b ∥ 2 2 > (1+ ϵ ) · ( ∥ Xb -µ ∥ 2 2 + λ ∥ b ∥ 2 2 ) . Finally let E 3 be the event that X ⊤ S ⊤ SX is a spectral approximation of X ⊤ X , but ∥ SXb -S µ ∥ 2 2 + λ ∥ b ∥ 2 2 ≤ (1 + ϵ ) · ( ∥ Xb -µ ∥ 2 2 + λ ∥ b ∥ 2 2 ) . Then, by the law of total expectation,

E [( ̂ τ -τ ) 2 ] = E [( ̂ τ -τ ) 2 |E 1 ] · P [ E 1 ] + E [( ̂ τ -τ ) 2 |E 2 ] · P [ E 2 ] + E [( ̂ τ -τ ) 2 |E 3 ] · P [ E 3 ] Note that P [ E 1 ] , P [ E 2 ] ≤ δ and therefore by the above argument and Theorem 2, E [( ̂ τ -τ ) 2 ] ≤ δ · ∥ µ ∥ 2 2 + δ · E [( ̂ τ -τ ) 2 |E 2 ] (15) +(1 -δ ) ( 8(1 + ϵ ) n 2 min b ∈ R d ( ∥ Xb -µ ∥ 2 2 +100log( n/δ ) · ζ 2 · ∥ b ∥ 2 2 ) + 32 d n 2 · ∥ y (1) -y (0) ∥ 2 ∞ )

Now note that

where the last inequality follows from (13) and (14). Now since leverage score sampling preserves the norms in expectation, we have E [ ∥ ˜ Xb ∗ -˜ µ ∥ 2 ] = ∥ ̂ Xb ∗ -̂ µ ∥ 2 . Therefore

Combining this with (15), we have

D ATE with Partial Observation

A simple approach for estimating ATE using partial observations is to first uniformly sample a subset of size m of the population and then apply any of the methods for ATE with full observation to this subset. If we observe the outcomes of the samples in the subset independently and with equal probability and apply the Horvitz-Thompson estimator, we achieve an squared error of less than ∥ µ ∥ 2 2 mn + ∥ t -t ∥ 2 2 m ( 1 -m n ) . If we use the Gram-Schmidt walk design and the Horvitz-Thompson estimator on the uniformly sampled subset, then we achieve an unbiased estimator with a variance of less than

Our approach is to adjust the Horvitz-Thompson estimator itself. We call this partial observation regression adjusted Horvitz-Thompson (RAHT) estimator. The first result we prove in this section is independent of the algorithm we use for estimation over the uniformly selected subset of the population. We introduce the following novel estimator by applying regression adjustment to HT estimator.

Definition 17 (Partial Observation Regression-Adjusted Horvitz-Thompson (RAHT) Estimator) . Given a matrix of covariates X ∈ R n × d , a fixed vector ̂ b , and an integer 1 ≤ m ≤ n , we introduce the partial observation regression adjusted Horvitz-Thompson estimator as the following. Estimation is based on a uniformly sampled set S of size m that we partition to two sets Z + and Z -by a random process (e.g., Gram-Schmidt walk). Then the ATE estimate is the following.

The following result indicates that the partial observation regression-adjusted Horvitz-Thompson estimator is a powerful tool for estimating ATE with partial observation.

Lemma 18. Let y (1) , y (0) ∈ R n , ̂ b ∈ R d be a fixed/preassigned vector and X ∈ R n × d . Given an unbiased estimator ̂ τ S for τ S := 1 | S | ∑ i ∈ S y (1) i -y (1) i , and a uniformly sampled set S of size m , the estimator ̂ τ := ̂ τ S -1 m ∑ i ∈ S ( x i -x ) ⊤ ̂ b is unbiased and its variance is

where the second expectation on the right-hand side is over the randomness of ̂ τ S , and D denotes the distribution of choosing a uniformly random set of size m .

Since ̂ τ S is an unbiased estimator of τ S ,

Moreover

Since

E

S

τ

S

-

(

1

m

i

∈

S

(

x

i

-

x

)

̂

b

)

=

[

∑

⊤

]

We next prove a varaince bound if the Gram-Schmidt walk design is applied for estimation over the uniformly selected subset.

Theorem 19. Let ̂ τ S be an estimate obtained by the Horvitz-Thompson estimator on the GramSchmidt walk design. Let ̂ b be a fixed/preassigned vector and 0 < ϕ < 1 . Then, the partial observation regression-adjusted Horvitz-Thompson estimator with m samples is an unbiased estimator with variance of

Proof. By Theorem 13 and Lemma 18,

Now note that

The probability of any i ∈ [ n ] appearing in a uniformly sampled set of size m is

Therefore

and the result follows by combining the above.

In Lemma 18 and Theorem 19, we have assumed that the vector ̂ b (which is used for adjusting the Horvitz-Thompson estimator) is given. For our final result, we combine our approach for regression on random vectors (with partial observation according to leverage score sampling) with partial observation RAHT to obtain a complete algorithm for estimation.

Theorem 3 (ATE Estimation - Partial Observation) . For any ϵ, δ ∈ (0 , 1] , ϕ ∈ (0 , 1) , and m ∈ [ n ] , there exists a randomized algorithm (Algorithm 4) that observes a potential outcome for O ( d log( d/δ ) /ϵ 2 + m ) units and outputs an unbiased estimate ˆ τ of the ATE τ , for which there is an event E with Pr( E ) ≥ 1 -δ (over the randomness of the algorithm), such that

where ζ := max i ∈ [ n ] ∥ x i ∥ 2 , µ := y (0) + y (1) , λ = 6 · log( d/δ ) ϵ 2 · ζ 2 , t ∈ R n is a vector where all the entries are equal to t = 1 n ∑ n i =1 t i , and

Algorithm 4: Regression Adjusted Horvitz-Thompson Estimation

1 Input: X ∈ R n × d , y (1) , y (0) ∈ R n , m ∈ N , 0 < ϵ < 1 , 0 < ϕ < 1 .

2 Set α = d/n and λ = 6 · log( d/δ ) ϵ 2 · ζ 2 .

3 Compute the λ -ridge leverage scores ℓ ( λ ) i of ˜ X = [ ( X -X ) 1 ] .

4 Set p i = min { 1 , max { α, 3 · ℓ i · log( d/δ ) /ϵ 2 }} .

5 For all i ∈ [ n ] define the random variable q i that is 1 with probability p i , and is zero, otherwise. Let S be the set of all indices i with q i = 1 .

7 Let y ∈ R n be a random vector where y i is independently equal to either y (1) i or -y (0) i with equal probability.

6 Let S ∈ R n × n be a random diagonal matrix with S ii = 1 / √ p i if q i = 1 , and S ii = 0 , otherwise.

8 Let ˜ v = arg min b ∈ R d +1 ∥ ∥ ∥ S ˜ Xb -Sy ∥ ∥ ∥ 2 2 + λ ∥ b 1: d ∥ 2 2 , and ̂ v ∈ R d be a vector consisting of the first d entries of v

˜ 9 Let T be a uniformly sampled set of size m from [ n ] \ S .

10 Obtain ̂ τ S by Horvitz-Thompson estimator and Gram-Schmidt walk design on T with parameter ϕ . // See Theorem 13

11

Let

̂ i ∈ S 12 Return | S | n · ( ̂ τ S -1 m ∑ i ∈ T ( x i -x ) ⊤ ̂ v ) + | S | n · ̂ τ S .

τ

S

= 2

∑

y

i

.

Proof. We first prove that the estimator is unbiased. By the law of total expectation, we have

where the summation is over all sets S that are selected from leverage score sampling. Note that given S and y , the solution vector ˜ v is fixed. We denote this fixed solution vector with ˜ v S, y . Let ̂ v S, y ∈ R d be the vector obtained by taking the first d entries of ˜ v S, y . Let x S be the average row of X S . Then we have

where the summation is over the assignments of the random vector y , x S is the average row over the set of rows indexed by S , T is a uniformly random subset of S of size m , ̂ τ S is estimated by using GSW design and Horvitz-Thompson estimator on set T , and ̂ τ S is estimated by the Bernoulli design and Horvitz-Thompson estimator (the observation is compatible with vector y ). Trivially since ̂ τ S is an unbiased estimator of τ S , the second term is equal to zero. Moreover since ̂ τ S is an unbiased estimator of τ S , over the expectation, they cancel out and therefore,

Now since ˜ v S, y is fixed given S and y , we have

We have

Therefore E [ ̂ τ -τ | S ] = 0 and E [ ̂ τ -τ ] = 0 . Thus, the estimator is unbiased.

We define the event E to be the event where the regression error is within a factor of (1 + ϵ ) of error and the number of nonzero entries in the matrix S is less than 100 d · log( d/δ ) /ϵ 2 . Note that

Note that condition to S and y ,

where ̂ τ S and ̂ τ S are unbiased estimators for τ S and τ S , respectively. Then since τ = | S | n · τ S + | S | n · τ S , this is an unbiased estimator. Now we have

Note that condition to S, y , ( | S | n · ̂ τ S -| S | n · τ S ) is a fixed number. Therefore

Since E [ ̂ τ S ] = τ S and 1 | S | ∑ i ∈ S x i = x S , the above quantity is equal to zero. Now note that since we use the Horvitz-Thompson estimator to estimate ̂ τ , we have

Now note that ̂ b is fixed given S, y . Let

Then by Theorem 19,

We now use the guarantees of leverage score sampling to bound the following

The matrix X S denotes a matrix with a size equal to X S such that all of its rows are equal to the average row of X S . Similarly, t S is a vector with a size equal to t S such that all of its entries are equal to the average entry of t S . Similar to the proof of Theorem 12, let ˜ X S := [ ( X S -X S ) 1 ] , and

Moreover, let ˜ y (1 ,S ) , ˜ y (0 ,S ) ∈ R | S | + d be vectors that are equal to y (1) S , y (0) S , respectively, on their first | S | entries, and the rest of their entries are zero. Also let ˜ t S = ˜ y (1 ,S ) -˜ y (0 ,S ) . Then by Theorem 1, the vector ˜ v in Algorithm 4 satisfies

Note that

and similar to the proof of Theorem 12, one can see that

Moreover

Again similar to the proof of Theorem 12, one can show that

Combining the above and noting that λ · ∥ ̂ v ∥ 2 2 ≥ 0 , we have

Now since ( X S -X S ) ̂ b -( t S -t S ) is orthogonal to the vector of all ones, we have

Table 4: Variance of different methods for estimating ATE with few samples. µ = y (1) + y (0) , t = y (1) -y (0) . 0 < ϕ < 1 is an input parameter. 0 < ϵ < 1 is an error parameter. ζ is the maximum ℓ 2 norm over rows of X . b ∗ = arg min b ∈ R d [ 1 ϕ ∥ Xb -µ ∥ 2 2 + 1 1 -ϕ ζ 2 ∥ b ∥ 2 2 ] , and ̂ b = arg min b ∈ R d ∥ ∥ ∥ ( X -X ) ̂ b -( t -t ) ∥ ∥ ∥ 2 2 + λ ∥ ∥ ∥ ̂ b ∥ ∥ ∥ 2 2 , where λ = 6 · log( d/δ ) ϵ 2 · ζ 2 .

Therefore

the results follows by picking α = d/n and combining (18), (19), (20), and (21).

We compared the variance bounds obtained by our approach for ATE estimation with partial observation with other methods in Table 4.

E Detailed Description of Datasets and Extra Experiments

The costliest operation in our algorithms is to compute (ridge) leverage scores. This can be performed in input-sparsity time (i.e., ˜ O ( nnz + d 3 ) , where nnz denotes the number of nonzero entries in the covariate matrix X [10]). Therefore, our algorithms are much more efficient than to the GSW design, which has a running time of O ( n 2 d ) [18]. This significant difference in the running time is also illustrated in our experiments, as shown in Table 6.

Some details of the datasets we used in our experiments are presented in Table 5. Moreover, the comparison of our ATE approach with two other methods (Lin's [24] and difference in means) is presented in Table 7 and Figure 2.

HT uniform

0.08

classic regression adjustment

leveragescorebased

4vectors

0.06

error

diff inmeans

ATE

0.04

0.02

0.00

500

750

1000

1250

1500

1750

2000

2250

2500

PopulationSize

F Gram-Schmidt Walk Design

Harshaw et al [18] introduced Algorithm 5 for balanced experimental design.

Algorithm 5: Non-Uniform Gram-Schmidt Walk

1 Input: Matrix X ∈ R n × d , 0 < ϕ < 1 , vector p ∈ R n .

2 Initialize the vector of fractional assignments z (1) ← (2 p 1 -1 , . . . , 2 p n -1) .

3 Initialize an index j ← 1 .

4 Select an initial pivot unit k uniformly at random from [ n ] .

5 Set ζ ← max i ∈ [ n ] ∥ X i : ∥ 2 .

6 Let B ∈ R n × ( n + d ) be a matrix such that B i : := [ √ ϕ · e i ζ -1 √ 1 -ϕ · X i : ] , where e i is the i 'th

basis vector of dimension n .

7 while z ( j ) / ∈ {-1 , +1 } n do

8 Create the set S ←{ i ∈ [ n ] : | z ( j ) i | < 1 } .

9 If k / ∈ S , select a new pivot k from S uniformly at random.

10 Compute a step direction as u ( j ) ← arg min u {∥ Bu ∥ 2 : u i = 0 for all i / ∈ S, u k = 1 } .

11 Let ∆ = { δ ∈ R : z ( j ) + δ · u ( j ) ∈ [ -1 , 1] n } .

12 Set δ + ←| max∆ | and δ -←| min∆ | .

13 Pick a random step size δ j that is equal to δ + with probability δ -/ ( δ + + δ -) , and is equal to -δ -with probability δ + / ( δ + + δ -) .

14 Update fractional assignmenyt: z ( j +1) ← z ( j ) + δ j u ( j ) .

15 Increment the index j ← j +1 .

16 Return the assigment vector z ( j ) ∈ {-1 , +1 } n .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, authors present regression adjusted estimators for estimating the average treatment effect under the Bernoulli design.\nIn particular, they show that by using the leverage scores and a ridge regression adjustment, favorable finite sample bounds on the (conditional) variance may be obtained.\nThis contributes nicely to a long-standing debate about the use of regression adjustment in randomized control trials.\nFinally, simulations are performed which coroborate theoretical results.\n\nSTRENGTHS:\nThe main strength of this paper is a finite sample analysis of regression adjusted estimators under the Bernoulli design.\nThere has been much debate about whether regression adjusted estimators are appropriate for estimates of the average treatment effect, [12, 20].\nHowever, that entire literature has been studied under asymptotic analyses, which provides little insight into finite sample performance.\n\n(Harshaw et al, 2019) have shown that under a sophisticated experimental design, favorable finite sample bounds on the variance may be obtained.\nMoreover, in large samples, (using a well-chosen parameter), the Gram--Schmidt Walk design achieves the same asymptotic variance as Lin's regression adjustment. \nThis puts regression adjustment on the defensive and raises a natural question: can regression adjusted estimators achieve similar finite sample results?\n\nThis paper does a really remarkable job of answering this question in the affirmative, up to some minor details.\nIt will be of interest not only to the NeurIPS community, but also to the statistical causal inference community more broadly.\n\nWEAKNESSES:\nThe paper has a few minor issues, but I believe they can all be addressed by the authors in a satisfactory way without too much work.\n\n ## Weakness 1 -- Conditional Analysis\n  \nThe major weakness is a certain ambiguity that arises in a few of the theorems.\nFor example, Theorem 3 does not analyze the estimator in Algorithm 1 unconditionally.\nRather, it analyses it conditioned on a high probability event (i.e. \"with probability $1 - \\delta$, Algorithm 1 computes an unbiased estimator of the average treatment effect with variance...\")\nAlthough this is natural in computer science, this is very unnatural in this area of causal inference.\nThe reason is that we place larger value on the unconditional statistical inference; the validity / width of confidence intervals depends on the bias and variance, not a conditional bias or variance.\nMoreover, Theorem 3 suggests that in order to get small variance, we should be setting $\\delta$ very large; however, this will incur a large bias, which is not discussed in the Theorem nor in the paper.\n\nIn order for the causal inference community to fully appreciate these results (and they will!) the authors should derive bounds on the unconditional bias / variance.\nThis will have the additional benefit of giving experimenters advice on how to choose $\\delta$ to minimize the MSE.\nThis critique applies to Theorem 4 as well.\n\n## Weakness 2 -- Constant $C$\n\nAs it stands, the constant in Theorem 7 is presently unspecified.\nThis means that the Algorithm 1 cannot be run!\nAuthors should give a precise constant here.\n\n## Weakness 3 -- ITE Results are Weak\n\nTheorem 2 shows that the collection of ITEs can be estimated up to a constant error.\nThis is unsurprising because ITE estimation is considered impossible due to the fundamental problem of causal inference.\nI find this estimator and these results not very exciting because it's not clear what a practitioner would do with such a guarantee -- not a single ITE is really known.\nIf authors are pressed on space, I recommend moving this to the appendix; otherwise, it's fine to leave in the paper.\n\n## Weakness 4 -- Comparisons to GSW\n\nI think that the comparison to (Harshaw et al 2019) is great and highlights the relevance of this work.\nHowever, some of the comparisons are not relevant or could be improved.\nHere is a list of improvements:\n\n- (Line 137): Authors write that the variance of GSW is proportional to \"XYZ\". However, that's missing the important aspect of (Harshaw et al 2019), which is that these terms can be traded-off! So, I think you have to address (if only very briefly) that GSW-Design allows for trade-off of these terms, but that for $\\phi = 1/2$ it is proportional to \"XYZ\".\n- (Lines 33, 39, 141, etc): Authors write that GSW Design does not have an online analogue (which is currently true), but I don't think that this is how I would compare your estimator. The practical benefit of your regression adjustment estimator is that the data collection process is just so much simpler -- the Bernoulli design is decentralized, asynchronous, you name it! So although (Harshaw et al 2019) raised an online design as an open problem, it's really only interesting for covariate balancing. Being able to use such a simple design is beneficial much beyond online assignment. I think focusing on the practical simplicity of the Bernoulli design over the practical complexities of the GSW Design is much stronger argument in favor of your estimator, not the fact that Bernoulli design can be implemented online.\n- (Line 264): Authors write that GSW Design \"is a much slower algorithm\". This doesn't seem true. (Harshaw et al 2019) give an implementation of GSW-Design that runs in $O(n^2 d)$, which would seem to also be required by for the linear system solve required to compute the ridge regression and scores.\n\n## Minor Misc Comments\n\nThe following are minor miscellaneous comments\n\n- (Line 126): \"work\" seems like the wrong word, a typo.\n- (Line 242): The notation $\\mathbf{P}_{\\mathcal{P}}$ is used but not defined. I think it's fine to just remove the subscript.\n- (Line 203): I think it is confusing / inappropriate to call Algorithm 1 \"leverage score sampling\" because that seems to suggest that the assignment to treatment and control (i.e. \"sampling\") is based on leverage scores. Can you call this something different? Perhaps \"cross leverage score adjustment\" or something?\n- (Section 4.2): authors don't specify the value of $\\phi$ used for the Gram--Schmidt Walk Design.\n- It's a shame that most of the algorithms are in the appendix. If possible, I would recommend bringing them to the main body for camera ready version, perhaps at the cost of putting ITE estimation results in the appendix.\n\nQUESTIONS:\n- Can you address my comment on conditional versus unconditional analysis?\n- Can you address my comment on the constant $C$ that is used?\n- How does the Regression Adjusted Horvitz---Thompson estimator relate to double robust estimators?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on estimation of individual and average treatment effects (ITE and ATE) with regression adjustment, which is combined with the method of ridge leverage score sampling in order to obtain the desirable variance bounds. The leading case is algorithm 1, which estimates ATE with leverage score sampling and cross adjustment. In this algorithm, for each observational unit, either treated or control unit is observed with equal probability. Then observed outcomes are adjusted via cross regression adjustments for which the regression coefficients are computed using ridge leverage score samples. The paper provides a number of theoretical results regarding the variance bounds and reports experimental results using both synthetic and real-data datasets.\n\nSTRENGTHS:\n- The main strength of the paper is that it provides finite sample variance bounds for estimating the sample mean, individual treatment effects, and the average treatment effect with regression adjustment.\n- The research question addressed in the paper is of general nature and is important in many applications.\n\nWEAKNESSES:\n- To my reading of the paper, the most important issue is that it is unclear whether the advances in this paper are substantial relative to Harshaw et al [16]. It seems that theoretically, the variance bounds are comparable between the proposed methods in this paper and the Gram-Schmidt walk (GSW) design method of Harshaw et al [16]. Furthermore, the right panel of Figure 1 shows that GSW outperforms the current paper.\n- The paper criticizes the GSW design, saying that it is not suitable for online experimental design (line 33 and lines 139-142). However, the proposed method is not immediately suitable for online settings. For example, it is unclear how to implement ridge leverage score sampling in a fully online fashion because the leverage score changes as more observations are available.  \n- The one advantage of the proposed method over GSW might be computational speed in view of table 6 in the supplement. However, there is no theoretical analysis in terms of computational aspects.\n\nQUESTIONS:\n- Is there any chance to obtain lower bounds for variance? It would complement the existing results on the upper bounds.\n- Would it be possible to estimate the variance bounds?\n- Line 192, is the transpose missing for the last $X$?\n- Steps 5 and 6 in algorithm 1 may contain typos. I think these steps are meant for leverage score sampling but the stated steps are unclear. Please check them.\n- Line 264. It might be helpful to provide a summary of running times in the main text.\n- There is very little discussion on the experimental results in section 4. Probably this is due to the page limit. It would be helpful if there is a short summary of numerical findings in the main text.\n\nLIMITATIONS:\nOne important limitation that is not mentioned in the paper is how to conduct statistical inference on ATE. This is typically done using asymptotic arguments. For example, Harshaw et al [16] provides asymptotic normality and suggests a confidence interval for ATE based on asymptotic normality. Although this paper focuses on finite sample variance bounds, it might be fair to mention the lack of inference methods.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the design and analysis of randomized experiments for treatment effect estimation, which is an important problem in causal inference. The goal of treatment effect estimation is to estimate the effect of a specific treatment on individual subjects or the average effect in the population, using only one of the two potential outcomes (either treatment or control) per subject. The objective is to obtain a precise estimator of the treatment effect, which is unbiased and has a smaller variance. However, it is often also desirable to minimize the number of subjects exposed to the experiment due to practical or ethical considerations. To address this, recent research has focused on leveraging linear algebraic techniques to design optimal experiments and estimate treatment effects.\n\nThe authors identify three main limitations in previous approaches: (1) inefficiency resulting from using the entire population in the experiment, (2) sub-optimal variance bound of the resulting estimator, and (3) inflexibility in experimental design, particularly in online settings. To overcome these challenges, the authors propose combining subsampling techniques from numerical linear algebra with classical statistical methods of regression adjustment. This integrated approach aims to obtain an unbiased estimator with a variance comparable to the optimal value, while also being suitable for online experiment settings.\n\nThe paper is organized as follows. In Section 2, the authors demonstrate the effectiveness of their proposed approach by considering four types of problems: (1) mean estimation of a vector, (2) individual treatment effect (ITE) estimation, (3) average treatment effect estimation (ATE) with full observation of both potential outcomes, and (4) ATE with partial observation, where only one potential outcome per subject is available. The primary focus is on problems (2) and (4), while problems (1) and (3) serve as precursors to facilitate the exposition of ideas. In Section 3, the authors describe the key techniques employed in their approach, providing a brief outline of the proofs for some theorems. Subsequently, they present experimental results from synthetic datasets (Section 4.1) and real-world datasets (Section 4.2) to validate the effectiveness of their proposed method.\n\nSTRENGTHS:\nThe paper demonstrates several strengths that contribute to its overall quality and significance. Firstly, the problem of treatment effect estimation addressed in the paper holds substantial importance within the field of causal inference. The research questions explored are highly relevant and carry practical implications, underscoring the significance of the study.\n\nSecondly, the proposed approach of integrating subsampling techniques rooted in randomized numerical linear algebra with regression adjustment is a well-founded and suitable strategy for tackling the problem at hand. Additionally, the authors' systematic approach in Section 2 effectively illuminates the challenges associated with addressing the problem and provides a rationale for the proposed approach.\n\nFurthermore, the technical tools employed in the paper are well-established and widely utilized in the field. The authors' adept development of methods and analytical arguments reflects a strong grasp of these tools, bolstering the credibility of their findings.\n\nIn summary, the strengths of the paper lie in the significance of the addressed problem, the logical and appropriate nature of the proposed approach, and the authors' proficient utilization of standard technical tools. These strengths collectively enhance the quality and validity of the research presented in the paper.\n\nWEAKNESSES:\nWhile  the paper demonstrates strengths, there are several areas where improvements can be made to enhance clarity and strengthen the overall presentation. The weaknesses can be broadly categorized into two main areas: clarity and organization, as well as the soundness of the arguments. \n\n\nI.  Concerns regarding clarity and organization\n\nFirstly, the paper lacks clarity, potentially resulting from a lack of organization. To improve clarity, the authors should clearly identify the specific problems they are focusing on and indicate which parts of the paper address these issues. This could be achieved by adding a dedicated subsection or table in the Introduction that summarizes the problems addressed, the solutions proposed, and references to specific sections and theorems.\n\nAdditionally, restructuring sections would contribute to better organization and understanding. Merging Sections 2.1 and 2.2 into a single subsection titled \"ITE Estimation\" and merging Sections 2.3 and 2.4 under the title \"ATE Estimation\" would streamline the presentation and emphasize the two treatment effect estimation problems. Furthermore, Section 3 would benefit from reorganization and clearer subsection headings to provide explicit explanations of the specific techniques being described. For example, Section 3.1, currently titled \"Random vector regression,\" contains a combination of an auxiliary theorem (Theorem 8) used to prove Theorem 2 (specifically, the display equation in line 113) related to ITE estimation, as well as a proof sketch of Theorem 3 in Section 2.3 related to ATE estimation. The rationale for grouping these two contents together is unclear, and it would be beneficial to clarify this and consider separating them into distinct subsections.\n\nMoreover, the paper contains numerous theorem statements that, although likely mathematically correct, pose challenges in interpretation for several reasons. Many of these theorems reference algorithms that are either presented in later parts of the paper or not included in the main text at all. This creates logical inconsistencies and forces the reader to navigate back and forth, hindering comprehension. Furthermore, the authors do not provide explicit remarks on the optimal variance or the origin of the terms used in the error bounds, particularly, neither in close proximity to the theorem statements nor in a designated discussion section. Consequently, it is difficult to grasp the significance of the error bounds in terms of their semantic interpretation and their implications, including how tight or loose they are compared to optimal errors.\n\n\nII. Concerns regarding soundness\n\nMoreover, there are concerns regarding the soundness of the work, which may be further exacerbated by the clarity issues. The authors claim that their proposed method successfully addresses the aforementioned three challenges of previous approaches, but this claim lacks adequate support. To strengthen their arguments, the authors should provide explicit discussions or remarks, either following relevant technical propositions or in a dedicated discussion section. Specifically, they should elaborate on the following aspects in clear and comprehensible language:\n\n1.  Clarify why previous approaches, such as [16], require the entire population in the experiment, while the proposed method does not.\n\n2.  Discuss the optimal values or lower bounds of variance and how the authors' error bounds can be considered \"comparable to the optimal.\"\n\n3.  Explain how the proposed method allows for adaptation to the online experiment design setting.\n\nAdditionally, it would be beneficial to appropriately compare the proposed method to baseline approaches, such as classical regression adjustment estimators, Lin's interacted regression adjustment [20], or the crude difference-in-means estimator. Conducting such comparisons, particularly when the sample size is equal to the entire population, would provide a clearer understanding of the strengths and limitations (if any) of the proposed approach.\n\n\nAddressing these weaknesses would significantly enhance the clarity, soundness, and overall quality of the paper.\n\nQUESTIONS:\n1.  I would like to request the authors to address the the concerns raised in the “Weaknesses” section. \n\n2.  It appears that the error bound in Theorem 2 may be trivial and not particularly informative because $ d \\| y^{(1)} + y^{(0)} \\|_{\\infty}^{2} \\geq \\| y^{(1)} + y^{(0)}\\|_{2}^{2} \\geq \\| y^{(1)} - y^{(0)}\\|_{2}^{2} = \\| t \\|_{2}^{2} $ assuming $\\|  \\| y^{(1)} - y^{(0)}\\|_{2} \\| \\leq \\|  \\| y^{(1)} + y^{(0)}\\|_{2} \\| $. I wonder if the authors clarify the value of Theorem 2? t would be beneficial for the authors to clarify the significance and value of Theorem 2, particularly in relation to its practical implications. \n\n3.  In Section 4.1, it would be helpful to provide information about the population size (n) for each dataset used in the experiments. Additionally, it would be valuable for the authors to evaluate the methods by quantifying the required sample size (i.e., the size of the sub-population) to achieve comparable performance to the method based on the entire population. This analysis would provide insights into the practical efficiency of the proposed method and its advantages in terms of minimizing the required sample size. \n\n4.  In Section 4.2, it appears that the classic regression adjustment method outperforms all the other methods, including the authors' proposed method. In light of these results, it would be important for the authors to provide a clear justification for the value and effectiveness of their proposed method, especially in comparison to the superior performance of the classic regression adjustment method.  \n\n5.  In line 267, the authors mention that the classic regression adjustment technique is biased. While this is true, it is worth noting that there are at least two standard unbiased techniques available, namely the difference-in-means method and Lin's interacted regression adjustment [20]. It would be valuable for the authors to include these unbiased techniques in their experiments to provide a comprehensive comparison and analysis of the proposed method against these established approaches.\n\nAddressing these concerns would greatly enhance the understanding, validity, and practical applicability of the research presented in the paper.\n\nLIMITATIONS:\nIt would be valuable for the authors to provide a short discussion on the technical limitations of their methods and offer insights into the potential adverse consequences that may arise when applying these approaches in real-world settings, specifically in experiment design and treatment effect estimation. Nonetheless, it is important to note that given the paper's primary focus on theoretical aspects, a comprehensive and extensive examination of these limitations may not be deemed critical.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of ATE and ITE estimation in the presence of covariates. In particular, the authors provide finite-sample variance bounds for regression adjustment method-based estimators and novel variants thereof. The core of the methodology is using leverage scores, a randomized numerical linear algebra technique. This approach has been previously employed in selecting a sample of the population on which to perform an experiment.\n\nSTRENGTHS:\nTo the best of my knowledge, using leverage scores for regression adjustment and offering finite-sample variance guarantees are novel contributions to the ATE and ITE estimation literature. The paper is strong in its technical aspects, as many of the theoretical contributions are robustly presented and accompanied by thorough proofs.\n\nWEAKNESSES:\nThe idea of the paper is overall good, but the execution is poor. The main issue is that the paper very technically dense and is largely unreadable without the appendix. It reads like a stream of (highly technical) consciousness rather than a scientific report. It also looks like the authors sacrificed intuition in favor of technical details of questionable relevance. \n\nOther notes:\n\n* The algorithm links are broken (as they are in the appendix, but that is not mentioned anywhere)\n* The algorithms referenced from the appendix contain quantities not yet defined which makes the paper difficult to read\n* Many details in experimental section are relegated to the appendix so it's difficult to assess its soundness. \n\nIn my opinion, while the paper has potential to be a valuable addition to the conference, it would require substantial rewriting in order to meet the required standards and expectations.\n\nQUESTIONS:\n* How is the random vector in Key Challenge 2 built given that we don't observe both $\\textbf{y}^{(0)}$ and $\\textbf{y}^{(1)}$?\n* It looks like both potential outcomes are observed in the experiments, is that correct? Again, the experimental section is difficult to read. \n* It appears that, in several experiments, the performance of the proposed method is comparable to that of the leverage scores method. In light of this observation, what is the significance or unique contribution of this approach?\n\nLIMITATIONS:\nI was unable to find a discussion by the authors about the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}