Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes several acquisition functions for performing active learning in a causal setting. The paper explores the limitations of more naive acquisition functions and motivates its own proposals. The performance of the proposed acquisition functions is empirically compared to that of more naive acquisition functions. 
main_review: **Originality**: The authors do acknowledge other similar methods; however, I think they fairly outline the differences between them and why they matter. Furthermore, in comparison to other acquisition functions, I think a good case is made for why the proposed functions are far superior. Overall, not wildly original, but original in ways that I was convinced do matter. 

**Quality**: I found the work to be technically sound, with careful consideration of the problems that can arise with naive acquisition functions and a logical progression towards more intelligently constructed ones. I found the experimental section convincing. I would have appreciated some comment on the limitations of the method / avenues for future work. 

**Clarity**: I found the work to be clearly written and organized. Figures 2 and 3 were very effective.

**Significance**: I believe the work is significant. Active learning is an important field of research, with undeniable potential for social impact. The proposed acquisition functions greatly outperform naive ones and the overall approach does differentiate itself in important ways from competing methods. The proposed acquisition functions can also be used in tandem with other BALD variants that may be developed in the future, furthering their applicability. 
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces "CausalBALD", a set of acquisition functions for use in Bayesian Active Learning when trying to estimate the Conditional Average Treatment Effect ("CATE") associated with an intervention, $\tau(\mathbf{x}) = \mathbb{E}[Y^{1}-Y^{0}|\mathbf{x}]$.  The authors introduce multiple information-theoretic acquisition functions, and then show the improved performance (relative to random sampling of examples) of their acquisition functions for estimating the true treatment effects present in three simulated/semi-synthetic datasets.  Through a toy problem (Figures 2/3), they also provide some intuition for why their acquisition functions outperform random sampling.
main_review: The paper was well-written and clear in its exposition.  The problem that the authors aim to tackle is interesting and important.  In many scenarios where you wish to estimate a treatment effect, you are also limited in the amount of data you can acquire - hence, applying Bayesian active learning here seems very natural. Additionally, as far as I know, there is not a lot of literature on using active learning when estimating treatment effects (aside from the two papers that the authors cite - namely Deng et al. (2011) and Sundin (2019)).  The authors' approach seems reasonable, and I liked the fact that the authors evaluated the performance of their method on not just 1, but 3 datasets.
limitations_and_societal_impact: The comments that I have relating to this paper are mostly minor:
- It would have been nice to have had a fuller description of the CMNIST dataset in text.  Could you even have shown a few examples of the labels, y, generated for a few example digits?  It was hard, if not impossible, to conceive of the elements of CMNIST from the description given in the main text.  The math in the supplement helped, but I am still having trouble visualizing exactly what some elements of this dataset look like.
- Do the authors have an explanation for why particular acquisition functions work better for some datasets than others?  E.g. it seems that $\mu\pi$-BALD works well on the synthetic dataset, but not so well on CMNIST
- It would also have been nice to have had (even in the supplement) a histogram of true treatment effects associated with each of the datasets used for evaluation.  I am wondering if your method works equally well for estimating all sizes of treatment effects, or if it is better at estimating larger treatment effects 
- Finally, it would have been nice to have seen some analysis of the examples selected by the authors' best performing acquisition functions. 
 This would have been particularly interesting in e.g. the CMNIST case, where one acquisition function vastly outperforms the other.  Which examples are being selected in that case, compared to for other acquisition functions?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a new acquisition function for applying active learning to observation data consisting of covariates, treatments and outcomes. The motivation of developing a new acquisition function is the existing ones could either query too many samples close to the distribution mode or query samples outside the overlapping region of the treatment and control group. Therefore, the key contribution of this work is to propose a acquisition function that 1) query samples in the overlapping region of case and control; and 2) reduce uncertainity of the model parameters.

The proposed function is built on the existing Bayesian active learning by disagreement (BALD). 

First, \tauBALD aims to maximize the conditional mutual information between CATE and model parameters \Omega. However, since the control outcome and treatment outcome cannot be observed simultaneously, leading to irreducible uncertainity. 

Second, to overcome this drawback, \muBALD was considered by maximizing the conditional mutual information between outcome Y^t and the model parameters \Omega. The proposed Causal-BALD improved \muBALD by considering both model uncertainty reduction and overlapping. Specifically, \mu\piBALD directly multiplies the propensity acquisition function and \muBALD. 

Third, \rhoBALD can also handle overlap by maximizing the mutual information between outcome Y^t and the CATE computed from model parameter \Omega. \rhoBALD can also be combined to \muBALD to form \mu\rhoBALD, which can handle both model uncertainity reduction and overlap.

Experimental results on synthetic dataset and semi-synthetic data showed the proposed \mu\rhoBALD outperforms existing acquisition function for the application active learning to individual treatment effect estimation.
main_review: This paper is well motivated. The problem setting of treatment effect estimation requires an adapted version of active learning algorithms that can focus on acquire samples from the overlap region. This work started from this intuition and proposed new acquisition function that can achieve both overlap and uncertainty reduction. The experimental results show the performance gain of the new acquisition function compared to existing baseline approaches.

I don't have major complains on the method of this work. A few minor comments are listed below.

1) A few variations of BALD are listed in this work. It could help to highlight the major contribution of this work in the abstract or introduction so that readers can clearly grasp this work's novelty.

2) In Table 2, are these results based on within-sample evaluation or out-of-sample evaluation?

3) Did the authors try to run experiments on other commonly used benchmark datasets for treatment effect estimation, such as Twins and Jobs?

4) The font size in the legend in Figure 2 should be increased to become more readable.
limitations_and_societal_impact: No
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Update (12th Aug 2021) - increased score to reflect that my comments were satisfactorily addressed by the authors in their rebuttal.

-----------
The authors propose new set of acquisition functions, dubbed Causal-BALD, based on Bayesian Active Learning by Disagreement (BALD) that consider overlap and information gain as a decision criterion for iterative improvement of a counterfactual estimator in settings in which direct treatment assignment by an algorithm is unethical or not feasible. The authors performed a clean evaluation in various settings and with multiple underlying models to show that their proposed approach outperforms the previous state-of-the-art.
main_review: **Originality**:
- The authors study the application of active learning methods for interactively querying a data set for the improvement of an underlying counterfactual estimator (deep ensembles and DUE). To the best of my knowledge, this is a non-trivial, relevant and understudied setting, and originality is therefore clearly a strength of the presented work.

**Quality**:
- The performed experiments and their descriptions are comprehensive. The authors evaluate multiple relevant settings, underlying models and benchmark datasets to substantiate their claims.

- The benefits of Causal-BALD over treatment-naive BALD (eq.s 1-2) are discussed at length in the manuscript. However, a treatment-naive BALD baseline is missing from the experimental comparison.

**Clarity**:
- The paper is well written and the text is easy to follow for a reader versed in the field of counterfactual inference. The exposition is clear and the paper follows a compelling narrative.

- How experimental hyper-parameters were systematically chosen for the models is not described in the manuscript. Only the final hyper-parameters used in the experiments are presented in the supplementary materials. On review of the source code, I saw that a systematic approach was likely performed (see tuning.py) - this information should be included in the manuscript.

- In terms of reproducibility, I particularly appreciate the comprehensive appendix that covers the details missing in the main body of the paper and the clear reference implementation provided by the authors.

**Significance**:
- Active learning for estimating individual treatment effects is an understudied, but relevant, problem with several potential practical applications. The proposed method seems to be generally applicable under the stated assumptions, and the experimental results support its use.

- The presented acquisition functions, crucially, rely on the availability of meaningful uncertainty estimates. An additional source of uncertainty in the counterfactual estimation setting is the potential presence of unobserved confounding that can not be ruled out in practical scenarios and will not be reflected in the models' uncertainty estimates. Claims towards "unbiasedness" should therefore be toned down to reflect that we can not test the unconfoundedness assumption, and the proposed method therefore will not have guarantees to be unbiased in  real-world applications.

limitations_and_societal_impact: 
**Societal Impact**: 
- As a general-purpose method for active learning in the counterfactual estimation setting with potentially broad applicability, I see no immediate reason for additional concern beyond that inherent in any active learning method. One risk is that the method could, e.g., lead to a biased, non-representative sampling in terms of ethnicity and other protected attributes - particularly, if the unconfoundedness assumption this work is based on is blindly trusted. The authors could briefly outline this risk and how one could preempt such a scenario with appropriate checks and balances.

**Questions/suggestions for the authors**:
- Please describe how the experimental hyper-parameters presented in the supplementary materials were chosen. In general, conducting experiments over multiple hyper-parameter sets would be preferable to ensure the results generalize to other choices of hyper-parameters for the underlying models.

- From the appendix, it appears the authors used a variant of TARNet [1] as constituents of the deep ensemble models (at least in some experiments). However, as far as I can see, [1] is not disclosed/referenced anywhere in the paper, nor is there a discussion of deep counterfactual representation learning methods in the paper. Please add the missing information on what models were the constituents of the deep ensembles and any related discussion to the related work section.

- Please include a short discussion on how the [standard in the field but relatively strict] assumptions (particularly the unconfoundedness assumption) and its potential impact - if violated - on the uncertainty estimates may affect your proposed acquisition function in practice so that readers are aware of this inherent risk.

- Please add treatment-naive BALD (eq.s 1-2) to the experimental comparison to highlight and quantify the benefit of adapting the acquisition function to the counterfactual setting.

**Conclusion**: A relevant and understudied setting, clear and compelling narrative and a comprehensive experimental evaluation. There are minor issues that I trust can be resolved during review. This paper ticks all the boxes for a clear accept.

Note: I have not verified the proofs in the supplementary materials.

[1] Shalit, Uri, Fredrik D. Johansson, and David Sontag. "Estimating individual treatment effect: generalization bounds and algorithms." International Conference on Machine Learning. PMLR, 2017.
review_title: null

Meta-review: The paper aims to provide a sample efficient strategy to train deep learning models that can effectively predict individual-level treatment effects from high dimensional observation data. A unique challenge in estimating personalized treatment effect is the necessity to choose data samples from regions with support from both treated and control populations so that the treatment effect can be properly identified. As a result, most existing data acquisition functions that primarily focus on uncertainty become less effective.  The proposed model, referred to as Causal-BALD, addresses this challenge by integrating Bayesian active learning and causal inference in novel ways that cover the uncertainty and overlapping criteria simultaneously. Effectiveness of the proposed acquisition functions is clearly demonstrated using both synthetic and semi-synthetic datasets. 

As acknowledged by all the reviewers, the proposed acquisition functions are clearly motivated, well explained, sufficiently novel, and further supported by convincing evaluation results. It has the potential to impact both the scientific community and multiple application domains. 
The authors adequately addressed most of the questions raised by reviewers in their original reviews during the discussion period. In particular, since multiple acquisition functions are proposed and some show better performance than others on certain datasets, it is important for the authors to offer some deeper insight to better understand the behaviors of different acquisition functions. It is also interesting to show some sampled data instances and connect them with the corresponding acquisition function. The authors promised to move some results to the appendix and add necessary discussions to better highlight the key properties of the acquisition functions and their potential limitations. 

Decision: Accept (Poster)