Venue: CLeaR 2022 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary: This paper design an interactive test called i-bet. The paper deals with classic randomized experiments  The idea is to test by betting from a game-theory perspective, which improves the power under heterogeneous treatment effects. 
Main review: Strength:

- It follows a clear research direction from Wilcoxon rank test to CovAdj Wilcoxon test and to the proposed i-bet test.

- The idea to introduce reward and use a game-theoretic way to design the test is interesting. The intuition is explained clearly by "If an analyst can guess most treatment assignments correctly, then the treatment must have an effect".

- The cumulative products $M_t$ is well designed with good martingale properties under the null.

Room for improvements:

- Current simulation setting is relatively simple. The covariates have a dimension of 3 and follow simple distribution. It is better to evaluate in more challenging case.

- In the simulation, it is better to include more details such as how to compute the power in Figure 2 and 3 for the proposed method. 

- Why in Fig. 2 linear-CATE-test is better than i-bet but in Fig. 3 it is much worse? How to choose which one to use?

- One writing suggestion is to have a minimal introduction to the full algorithm (e.g., alg. 2) and with a demonstrative example first. Current methodology is not introduced until page 6, with many discussions and related works discussed before. 

In general, this paper presents an interesting idea and I recommend for acceptance. 

Post-rebuttal: thanks for the authors' feedback and I would maintain my recommendation.
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: The paper proposes a non-parametric tests for testing the global null hypothesis in a randomized experiment. It avoids the permutations employed in traditional nonparametric tests by interactive testing, that was recently introduced in the literature.
Main review: The key idea in this paper is summarized in Remark 2,  that whenever the test statistic is a nonnegative martingale (by design), permutations can be avoided. Based on this observation and Ville's (Doob's) maximal inequality, the authors construct a process M_t and show that it is a nonnegative martingale under the null. 

I am not very familiar with the interactive testing literature, but at least the application of this idea for testing the global null hypothesis appears to be new.

1. In equation (1), it is unclear what the notation Y_i | A_i=1, X_i refers to. According to the context, this should refer to the potential outcomes, but the notation seems to suggest observed outcomes.

2. The authors describe a specific way of choosing \pi_t in Section 2.2. I am wondering if they could comment on how would different choices of \pi_t would influence the power of the proposed test, and how should a practitioner make such choices in a real setting.

3. It is a pity that they did not apply this test to a real data set. Such data sets seem to be readily available, and a data application would help illustrate the method better, and help clarify such practice choices needed to made by practitioners; e.g. see my comment 2 above.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: Summary: This paper proposes a novel interactive hypothesis testing framework that possesses desirable statistical properties as well as computational advantages over existing permutation based tests. The theoretical properties are proved and performance is demonstrated on simulated data.
Main review: Originality: This paper provides a novel interactive betting hypothesis testing framework. Its novelty is justified in my opinion,and I'm not aware of existing work that already solves this problem. 

Significance: The problem of detecting the causal effect of a treatment is one that is important in many fields in the natural sciences, social sciences and engineering. This proposed method gives a game theoretic insight into the problem of hypothesis testing. 

Technical quality: To my best understanding, the claims are theoretically sound and substantiated by theorems put forth in the paper. 

Clarity: The submission is clearly written as well as a comparison to existing methods is made and the performance demonstrated on simulated data.
review_title: null

Meta-review: The paper proposes a new test for determining whether a treatment has an effect, rooted in a game-theoretic framework. 

The reviewers and area chair all liked the paper and the authors' rebuttal convincingly answers their questions. 

Please follow the suggestions when writing the revised version of the paper (computing the test power; choosing $\pi_t$ and $w_t$) and discussing in more length how the approach can be assessed on real-world data, when of course the ground truth is unknown.
Decision: Accept (Oral)