Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper propose novel algorithm to estimate ITE and ATE, in the setup where the budget of targeted persons is constraints. Namely we can only touch s<<n persons, and would like to select these s people to estimate the ITE and ATE the best as possible.
strengths_and_weaknesses: originality:
this is one of the first approach using other tools such as leverage score sampling, in a potential outcome setup, willing to estimate ITE and ATE.

quality:
the mathematical sections are roughly interesting, and the theorem+corrolary are easy to follow. All definitions are clear.

clarity:
the paper is roughly smooth to read, although there could be incremental improvement

significance:
the significance of such sampling constraints could be better motivated, with various concrete example.
questions: - All bounds in probability are or fixed delta, like delta=1/30. why not letting delta unfixed, so that we can check some limit cases, and then expliciting the same result for delta=1/30?
- the choice of removing j from S^1 if j \ in S^0 and S^1 sound odd. why so?
- the case where n=2^k -1 and s=2^(k-1) wil only use s'=2^(k-2) samples, and "waste" s-s' samples if I am right?
- I am not convinced by the explainations on line 234-235. The empirical analysis could be more various to assess this
- the Horvitz-Thomp!son estimator could be more general, with sampling probablility explicited: A/pi-B/(1-pi). The factor 2 looks very unexplained, and the work could be easily generalized to other pi
- Remark line 284 is nonsense, if n=s, then previous bound is -infinity
- code is not working, and would deserve a more precise readme file: "Requires installing python3, julia and interfacing it with other open source code mentioned in the paper."
-  proof Th A9, does not hold of n is not 2^k
-  30-70% percentile looks not standard at all why so?
-  all plots are not vectorised: difficult to read them

typos:
line 54 They->The 
line 55 both select-> select both
line 118 each-> each of them
line 184 simple-> simply
line 242 that: bad formulation
limitations: ITE and ATE are deeply linked with privacy of data and could lead to privacy leak, especially for ITE.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed methods to estimate individual treatment effects (ITEs) and average treatment effects (ATEs) under a budget constraint and a linear model assumption on the potential outcomes. The method for estimating ITEs is based on the idea of leverage score  sampling [51] in numerical algebra. The authors apply the idea to sample a limited set of observed vectors to span the column space of the design matrix in the linear regression models for the treated and control outcomes. The authors then extend their method to estimate ATE, building upon the recent Gram-Schmidt work [20] on covariate balancing in experimental design. The authors provide theoretical guarantees and experimental results for their methods in both settings.


strengths_and_weaknesses: Strengths:
1. I believe the problem studied in this paper is important for treatment effect estimation. 
2. The method based on leverage score is an interesting and good contribution to the field.
3. The theoretical results under a linear model give good guarantees for the method's performance. 

Weaknesses:
1. The definition 2.2 of ITE is correct, but the return of Algorithm 1 is $\mathbf X^* \beta_1 - \mathbf X^* \beta_0$, which is the conditional average treatment effect, i.e. the expectation of the ITE conditional on X. In other words, the noise in the outcome goes to 0 after taking the expectation. In fact, the ITE for a given individual is a random variable rather than a parameter, so does take a fixed value. So we can not estimate it exactly, instead, methods should provide a prediction interval to quantify the randomness in ITE.
2. I don't fully understand the transition from Lemma 3.3 (Follows from [41]) to Theorem 3.6. First Is $S = S^0\cup S^1$ in Theorem 3.6? I thought this is the case after reading the proof in Appendix. I thought the main idea is still the same about controlling the expectation of $|S|$. Could you explain what has been designed differently for causal inference in which we only observe one outcome for every individual? I feel like the difference is just adding the step with pi(1-pi) probability in Algorithm 1
3. It is interesting to see a theoretical guarantee on estimation errors. It is unclear to me whether this method still allows users to do 
finite population inference for the treatment effects, i.e.,  construct valid confidence intervals for the treatment effects.
4. Section 4 is not very easy to understand if readers (like me) don't know the Gram-Schmidt-Walk design very well. Assuming a linear model seems like a limitation when working with the Horvitz-Thompson Estimator. 


questions: In line 184, if one row gets included in both $S^0$ and $S^1$, the authors remove the row from $S^1$. Why not further randomize this with 1/2 and 1/2 probabilities for removing the row in  $S^0$ and $S^1$, respectively.  In your proposal, we are expected to end up with less treated samples.
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes two novel algorithms for designing efficient randomized controlled trials (RCT), so that they can be then used to estimate average and individual treatment effects. The algorithms provide ways to sample s << n points and assign them to treatment and control groups. The paper assumes linearity in the outcome functions, and shows both theoretically and empirically, that the proposed algorithms can design RCTs from which the treatment effects can be estimated accurately.

strengths_and_weaknesses: The paper is addressing a very interesting and important problem. The findings in this paper can help design RCTs that are more efficient than the conventional method of randomization.
I however had a very hard time understanding a lot of the concepts discussed in the paper. I wished the paper had been written with more clarity.

questions: - In assumption 2.1, it is stated that X is row-normalized. I can understand why we might want the design matrix to be standardized, but do not understand why it should be row-normalized.
- For the ITE estimation task, the samples are selected based on their leverage score (a sample with a higher leverage score has a higher chance of being selected). It is however not explained why these samples are more important than the other samples with lower leverage scores.
- In lines 158-160, it is stated that using a smoothed X would help mitigate sampling the same data point for both treatment arms. I don’t see how though. 
- It is stated that \pi is calculated based on the leverage scores; could you please provide the exact formula for calculating \pi?
- Why is S^0 sampled according to \pi and S^1 according to \pi(1-\pi)? Is it because we want to reduce the number of treated subjects as much as possible? Can the reverse yield a viable RCT?
- Why are there two separate algorithms for designing RCTs, one for ATE and another for ITE? We can calculate accurate ATE from a dataset with which we can estimate accurate ITEs (I understand the reverse may not be true).

limitations: N/A
review_title: null

Meta-review: metareview: The reviewers came to consensus that this paper makes a good progress on the study of treatment effect estimation. I agree with these opinions and please polish the manuscript so that the concerns raised by the reviewers become clear in the final version. In particular, I'm also interested in the concern on the constant $\delta$ and I expect that this point is addressed so that the dependence of the results on delta becomes clear.
confidence: Certain
Decision: Accept