Potential Outcomes Estimation Under Hidden Confounders

Authors: Ahmed Aloui, Juncheng Dong, Ali Hasan, Vahid Tarokh
Primary area: causal reasoning
Keywords: Confounders, Causal Inference, Treatment Effects

Abstract:
One of the major challenges in estimating conditional potential outcomes and the conditional average treatment effects (CATE) is the presence of hidden confounders. Since testing for hidden confounders cannot be accomplished only with observational data, conditional unconfoundedness is commonly assumed in the literature of CATE estimation. Nevertheless, under this assumption, CATE estimation can be significantly biased due to the effects of unobserved confounders. In this work, we consider the case where in addition to a potentially large observational dataset, a small dataset from a randomized controlled trial (RCT) is available. 
Notably, we make no assumptions on the existence of any covariate information for the RCT dataset, only requiring the outcomes to be observed. We propose a CATE estimation method based on a pseudo-confounder generator and a CATE model that aligns the learned potential outcomes from the observational data with those observed from the RCT. Our method is applicable to many practical scenarios of interest, particularly when privacy is under concern (e.g., medical applications). Extensive numerical experiments are provided demonstrating the effectiveness of our approach for both synthetic and real-world datasets.

Decision: reject
Reviewer scores: 3, 6, 1, 3

Reviews and discussion:
<thread>
creation date:
1729109909125

thread_type:
official_review

<thread_object>
creator: reviewer_dgih

<content>
summary:
The paper introduces two novel methods: Marginals Balancing (MB) and Projections Balancing (PB), to address the challenge of estimating Conditional Average Treatment Effects (CATE) under hidden confounding. These methods leverage outcome-only Randomized Controlled Trial (RCT) data to mitigate bias from unobserved confounders, outperforming previous approaches. The combination of MB and PB (MB+PB) further enhances performance in both synthetic and real-world datasets, providing good results even with limited RCT data.

strengths:
- Empirically, the combined MB+PB method demonstrates superior performance compared to established baseline methods (albeit the experimental setting satisfies the strong assumptions imposed by the authors)
-  The idea of enforcing a balancing constraint to learn CATE is, to my knowledge, novel and interesting.

weaknesses:
- **Main concern**: A critical assumption of the paper is that potential outcomes share the same distribution across both RCT and observational study. This is (almost) never the case in real-world data where even assuming the two distributions have the same conditional mean ($E_{P^{rct}}[Y|X] \equiv E_{P^{os}}[Y|X] $, aka transportability) is considered too strong. Unfortunately, this limitation is never discussed in the paper and the methodology heavily relies on this assumption.

 - **Secondary concern**: The authors assume access to only the outcomes and treatments from RCT data, without the accompanying covariates. This scenario is rare and not typically observed in practical settings, and the authors never discuss the plausibility of their setting. Clarification and examples where such a condition might realistically occur would strengthen the motivation behind the proposed methodology.


- **Concerns on writing**
  - A significant portion of the paper is dedicated to explaining that hidden confounding introduces bias in treatment effect estimation. While this is fundamentally important, it is a well-acknowledged concept in existing literature, and thus, a more concise treatment might suffice.
  - The manuscript's extensive use of bold font could be seen as distracting, reducing the emphasis intended for the most critical points.

questions:
- Could the authors provide realistic scenarios where only the outcomes and treatments from an RCT are available, but not the covariates?
- Could the authors elaborate on their assumptions regarding the distribution of potential outcomes, specifically these distributions are identical for both randomized and observational data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729181833962

thread_type:
official_review

<thread_object>
creator: reviewer_aunr

<content>
summary:
The paper proposes a method for CATE estimation under unobserved confounding. It assumes the presence of a small RCT dataset to correct for the biased introduced due to the unobserved confounding.
Importantly, the paper does not reuqire the covariates of the RCT data to be observed.

strengths:
- The method does not require covariate information for the RCT dataset. This is convenient in practice and differentiates the method from existing works.

weaknesses:
- A discussion of methods for CATE estimation under hidden confounders (e.g. sensitivity analysis) is missing. Furthermore, a comparison to such methods is missing.
This prevents a fair assessment of the method's usefulness.
- The method is compared to a factual learner, which is not a proper baseline for comparison, as the biasedness of the learner is known and also has been discussed in the paper.
The povided empirical insights throughout Section 3 are misleading.
- The method is restricted to binary treatments. This is a disadvantage of the proposed method in comparison to existing methods for CATE estimation under hidden confounding.
- The presentation contains errors, hindering easy understanding of the line of thought. There is only limited flow in the paper.
- Limited referencing of existing works (e.g., line 70, 356)
- The paper only has limited theoretical justification.
- Evaluation: A proper evaluation of CATE estimatiors is only possible on synthetic or semi-synthetic datasets. However, the paper only considers one very low-dimensional
dataset with a very simple data generation mechanism. This is not sufficient to evaluate the general performance of the proposed method. An evaluation on further more complex
datasets with higher-dimensional unobserved confounding is necessary to properly evaluate the method.

questions:
- How would the method extend to non-binary treatments?
- Line 35: In my opinion, this definition implies the consistency assumption. However, the assumption is not stated.
- Line 58: The headline of this paragraph is "performance metric". However, this is not stated in the paragraph.
- Confounding degree: Why is this introduced here? If it is considered further on, a proper introduction of sensitivity models and sensitivity analysis is necessary.
- IMHO, Section 3.1 is unnecessary and only hinders the reader's concentration. Unobserved confounding is a known topic in causal inference. 
There is no need for a separate case study.
- Line 270: What is meant by pseudo-confounder?
- Line 296: Why is it reasonable to generate a pseudo-confounder? How is this done? What are the theoretical guarantees for the generated confounder?
- Proposition 3.2: What is g? Why is it necessary?
- Equation 8: This needs more mathematical explanation or at least references to works covering the respective theory
- What is the theoretical justification for the final method MB+PB?
- Figure 8: Although the combined MB+PB performs much better then the methods separately, the PEHE is still quite high (considering that the figure plots sqrt(PEHE) against log(Gamma)). 
The figure thus does not aid in assessing the usefulness of the proposed method. A fair comparison with other baselines (even partial identification) would be helpful.
- Figure 8: Why does the Obs-Oracle PEHE increase with log(Gamma) even if there is no unobserved confouding introduced in the data?
- Lines 459/460: The paper states that "MB+PB shows strong robustness". However, this is not shown in the figure. Considering the scale of figure 8, MB+PB is not robust with regard to Gamma.
- Influence of RCT sample size: Again, further evaluation is necessary to draw the general conclusions stated in the paper.
- Table 1 shows that MB+PB in some cases outperforms other methods which additionally consider covariate information from the RCT. This is counter-intuitive. How can this be explained?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730689819856

thread_type:
official_review

<thread_object>
creator: reviewer_7fgj

<content>
summary:
This paper studied the problem of conditional average treatment effects (CATE) estimation. The paper proposed a new CATE estimation method in the case where in addition to a potentially large observational dataset, a small dataset from a randomized controlled trial (RCT) is also available - in particular on the outcomes are required from RCT. The proposed method is based on a pseudo-confounder generator and a CATE model which aligns the learned potential outcomes from the observational data with the outcomes observed from the RCT dataset. Numerical experiments demonstrated the effectiveness of the proposed estimation approach.

strengths:
- this paper studied an important and practical problem of CATE estimation, where the estimator can be significantly biased due to the effects of unobserved confounders. The proposed method works in a setting where a large observational dataset and a small dataset from a randomized controlled trial are available - while only the outcomes in the RCT are required to be observed. I found this setting to be more realistic and applicable to many real-world cases.
- overall the paper is presented with a good clarity
- the two regularizations required by the estimation are model-agnostic, so the method is flexible to be applied to different CATE models, e.g. neural networks
- experiments with both simulation data and real-world case studies demonstrate that the proposed estimation algorithm outperform / comparable to other baselines

weaknesses:
There is a lack of theoretical analysis on the two regularizations, in particular quantifying the bias reduction from each of the regularization.

questions:
- How does the sample sizes of the potentially large observable data and the small RCT dataset affect the estimation outcome? 
- For ACTG case study, CorNet worked better than the proposed estimator. What are the potential causes for the bad performance on this particular dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730740889762

thread_type:
official_review

<thread_object>
creator: reviewer_hgg4

<content>
summary:
This paper uses RCT paired with observational data to help mitigate the impact of confounding in the observational data. The difference from other work in this literature is that it assumes no covariates are available in the RCT sample and instead tries to match the distribution of potential outcomes between the two populations.

strengths:
The issue is important, and the setting where covariate information is not available for the RCT does appear in unfortunately many applications.

weaknesses:
The identification strategy in the paper appears to hinge on the assumption that the potential outcome distribution is exactly the same between the two populations. This is very unlikely: RCT populations are notoriously different than observational populations in domains like health (where the privacy issues that the paper's motivation points to are most likely to arise) because trial recruitment is very far from just sampling the general population (e.g., sicker patients and patients from minority groups are typically underrepresented, along with a range of other issues) . Most work in the literature makes the weaker assumption that the CATEs are equal in the two populations and what differs is just the marginal distribution of covariates, which is much more plausible if the set of covariates available is sufficiently rich. I cannot think of any settings where imposing exact equality in the potential outcome distributions by themselves is a plausible assumption.

questions:
Are there application settings where the above assumption is justified?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731503108042

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Demanding Investigation of Reviewer aUnR for Ethical Violation

comment:
Dear TPC Chairs, Senior Area Chairs, Area Chairs

I am the senior author of this submission. I am writing to report a potential violation of ethics codes by reviewer aUnR.
As you can see the reviewer makes the allegation that the citations of our paper are non-existent, fake and machine generated and that the paper in part may be LLM generated. 

In the response that I just posted, we gave every single citation of the paper and provided links to every citation. This demonstrates the falseness of allegations made by the reviewer.

While we value constructive criticism, the reviewer makes several baseless claims, vague and non-sensial critiques, and the review has many frequent inaccuracies. We believe that these issues hinder the academic integrity of the review process. 

Note that I am a seasoned researcher and I and my students are used to low quality reviews and materially wrong reviews at ML conferences, but I find the reviewer aUnR baseless allegations very outrageous.

I am writing to you to demand an ethics investigation into reviewer aUnR  actions in this matter. From reading his/her/their review, I suspect that this review may be machine generated in part (as it is very non-coherent).

I will write to you directly (outside of the Openreview platform) to formally demand an investigation unless the reviewer aUnR retract his/her/their review.

Best regards, 
Senior Author
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes to account for unobserved confounding in observational causal inference by assuming the availability of outcome data from a small RCT, which is used to train a generator model to ensure that counterfactuals from observational data match the RCT outcomes. Several empirical demonstrations show improvement of CATE estimation compared to existing techniques. 

Reviewers have acknowledged the importance of the problem, particularly the fact that availability of covariate information from RCTs not being a requirement which contributes to the overall novelty of the contribution. 

However, reviewers have pointed out that exactly matching counterfactuals may not be ideal given that trial populations often vary significantly compared to observational data samples, lack of sufficient theoretical analysis, lack of sufficient discussion of sensitivity analysis literature, which is highly relevant, lack of justification of when the scenario that no covariates are available from RCTs holds in practice, clarity issues in writing in that a lot of space is dedicated to explaining the bias due to hidden confounding but insufficient justification of the proposed method. 

Author rebuttal has addressed some of the concerns but in general, the justification and responses are insufficient, the draft requires a considerable pass to improve clarity, provide additional justification for the setup, and justify the assumption of transportability of the CATE estimate to begin with. 

Overall based on these concerns, I recommend a reject.