Venue: CLeaR 2022 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: review
Main review: The paper studies the hypothesis testing problem in learning
treatment-assignment policies. Specifically, they argue that we
should optimize for an assignment with maximal expected probability of
passing the test. The goal is to help a drug company to get a drug
approved, or help a researcher to find significantly positive
treatment effect. The post-selection issue is handled by sample
splitting.

This goal, though seemingly a little eye-brow raising at first, aligns
with active learning or Bayesian optimization, where we want to find
the data points that are most helpful for reducing the desired
prediction or optimization loss.

The optimization objective is straightforward, though optimizing for
it is challenging. In particular, the target of inference is a
function, which could be hard to handle if it relies on many or
high-dimensional covariates. Thanks to this concern, the paper
developed a solution based on submodular optimization, which is
interesting, though I am not an expert in submodular optimization to
judge its soundness.


The derivation of the paper seems sound. However, one thing that could
be better clarified is its connection to Bayesian optimization.
Bayesian optimization seems to be solving a related and possibly
harder problem. Striping away the sequential acquisition nature of
Bayesian optimization, I can think of Bayesian optimization as finding
the set of data points such that the resulting target (which is the
probability of passing the test here) is maximized. I can think of
finding treatment policy as a form of selection data points (e.g. we
have pool of data points with the counterfactual outcomes of all
possible treatment configurations.) In this framing, does the proposed
algorithm outperform standard Bayesian optimization algorithm (using
only one acquisition step of Bayesian optimization)? How is the
proposed algorithm better? I think the paper could benefit from
clarifying this point.
review_title: null

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: The paper proposes a heuristic method for finding subgroups of a population for which a treatment is most beneficial, using a decision tree with splits determined by statistical tests (e.g., t tests) applied to a hold-out sample.  
Main review: The originality seems to be in using the tests as split criteria. I am not familiar with the literature on competing methods but this seems well done.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: Summary: This paper works on the problem of using batch data to estimate a policy that on average has a high probability of being significantly better than a baseline policy. They provide asymptotic and finite sample guarantees for their approach.
Main review: High level overview of comments:
-	I would suggest a more through discussion of relevant literature, especially the literature on policy improvement (e.g. https://proceedings.mlr.press/v37/thomas15.html) and pure exploration (e.g. https://arxiv.org/abs/1507.04523), which are also concerned with ensuring estimation guarantees. Discussing how your work relates to this literature would help the reader better put your work in context.
-	Additionally, the writing can be clearer early on (i) what problem you are approaching (e.g., online vs offline setting, defining training set) and (ii) why an optimal policy for your problem is different from the optimal policy for minimizing regret.

More detailed comments:

-	Regarding the abstract: “Yet such algorithms typically optimize expected outcomes without taking into account that treatment assignments are frequently subject to hypothesis testing.” What does hypothesis testing of the treatment assignments mean? I thought in a randomized control trial with binary treatments the treatment assignments are most commonly selected with probability 0.5 each---why are there a hypothesis tests on these treatment assignments?
-	Whey you say “we assume that a policy obtained from training data only gets implemented when a hypothesis test…” what do you mean by implemented here? How is implementation related to the training data?
-	From reading your introduction I am confused about whether you are (i) using batch training data to estimate what policy you want to use to collect more data in order to maximize the probability of having a significant hypothesis test or (ii) you are in an online setting in which you are changing treatment assignment probabilities (in reference to the sentence “existing algorithms for the assignment of treatment typically optimize welfare without taking hypothesis testing into account”, which makes me think you in an online setting). It would be helpful for the reader to better understand at a high-level what problem setting you are in.
-	In your related work section, you do not mention literature on best arm identification and algorithms for pure exploration, which seem highly relevant. For example, see https://homes.cs.washington.edu/~jamieson/resources/bestArmSurvey.pdf, https://arxiv.org/abs/1507.04523, https://arxiv.org/abs/0802.2655, https://arxiv.org/pdf/2101.08534.pdf, https://maxkasy.github.io/home/files/papers/adaptiveexperimentspolicy.pdf . Is your method a non-online version of best arm identification? This literature is not concerned just with “optimizing welfare”, but very much take hypothesis testing into account.
-	In your introduction you say that you compare your algorithm to “algorithms that target maximal average outcome by assigning based on positive estimated treatment effects.” I wonder if there are better baselines to compare to because this is a bit like comparing a best arm identification algorithm to a regret minimizing bandit algorithm on the task of best arm identification. Would also comparing to Dwivedi et al. (2020) and/or Leqi and Kennedy (2021) make sense? You say that these works don’t directly target out-of-sample power as an objective but their methods could “lead to better out-of-sample power of statistical tests”, which makes me think they would be reasonable to compare to.
-	As I read into section 2, I want to confirm my understanding of the problem setup. I understand that you have n i.i.d. observations to train on and a hold-out set that you will eventually use to perform a hypothesis test. You are trying to estimate some optimal policy a^* using estimated policy \hat{a}, which is most likely to be significant if we compare treatment \hat{a} to the baseline treatment according to the hypothesis test on the held-out data. Based on my understanding, this is different from the problem of estimating an optimal policy regret minimizing policy from batch data, because the optimal policy for your problem will be a function of the noise variance in each state (which relates to the power of a test), while this does not matter for the problem of estimating optimal regret minimizing policies. If this is the case, it would be helpful to provide this intuition to the reader early on to prevent confusion.
-	The presentation of Algorithm2 is hard to read. It is not in the typical format of an algorithm, but instead one long paragraph.
review_title: null

Meta-review: Reviewers are largely positive about this paper. The author response has successfully removed some of the doubts and answered some of the questions. We encourage the authors to use the reviewer feedback to further improve the presentation and clarity, especially considering that their approach is somewhat non-standard.
Decision: Accept (Oral)