Conformal prediction for causal effects of continuous treatments

Authors: Maresa Schröder, Dennis Frauen, Jonas Schweisthal, Konstantin Hess, Valentyn Melnychuk, Stefan Feuerriegel
Primary area: causal reasoning
Keywords: causality, dosage response curves, conformal prediction, uncertainty quantification

Abstract:
Uncertainty quantification of causal effects is crucial for safety-critical applications such as personalized medicine. A powerful approach for this is conformal prediction, which has several practical benefits due to model-agnostic finite-sample guarantees. Yet, existing methods for conformal prediction of causal effects are limited to binary/discrete treatments and make highly restrictive assumptions such as known propensity scores. In this work, we provide a novel conformal prediction method for potential outcomes of continuous treatments. We account for the additional uncertainty introduced through propensity estimation so that our conformal prediction intervals are valid even if the propensity score is unknown. Our contributions are three-fold: (1) We derive finite-sample prediction intervals for potential outcomes of continuous treatments. (2) We provide an algorithm for calculating the derived intervals. (3) We demonstrate the effectiveness of the conformal prediction intervals in experiments on synthetic and medical datasets. To the best of our knowledge, we are the first to propose conformal prediction for continuous treatments when the propensity score is unknown and must be estimated from data.

Decision: reject
Reviewer scores: 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1729690406999

thread_type:
official_review

<thread_object>
creator: reviewer_nayf

<content>
summary:
The paper applies the framework introduced by Gibbs et al. (2023) to produce prediction intervals, with a finite-sample coverage guarantee, for potential outcomes of continuous treatments. Gibbs et al. (2023) can provide certain types of conditional coverage guarantees by reformulating the coverage as coverage over a class of covariate shifts. Gibbs et al. (2023) already showed that given an appropriate choice of $\mathcal{F}$, that their prediction set achieves coverage under covariate shift, so long that the covariate shift can be represented as, $P_X$ titled by $f$, and $f \in \mathcal{F}$.

This work postulates the causal effect of continuous treatments as soft and hard interventions.  A soft intervention can be seen as a value shift in the generalized propensity score. A hard intervention can be seen as shifting the generalized propensity score to a Dirac delta function to the desired treatment value. In the paper, they define an appropriate function class for the different interventions and deal with their specific covariate shifts. For the hard intervention setting, they also ensure that the function class can deal with an estimated generalized propensity score with a bounded estimation error of the propensity function. 

They provide some experiments on synthetic datasets (from their own) and medical ICU datasets, benchmarking their prediciton intervals against an MC dropout and Ensemble approach.

[1] I. Gibbs, J. J. Cherian, and E. J. Candès, “Conformal Prediction With Conditional Guarantees,” Dec. 20, 2023, arXiv: arXiv:2305.12616. doi: 10.48550/arXiv.2305.12616.

strengths:
- **Clear Motivation:** The paper's objective is well-placed within causal machine learning, particularly in high-stakes areas like medicine, where uncertainty quantification is critical for decision-making.
- **Novel Contribution:** The authors propose a novel method for deriving conformal prediction intervals in the presence of continuous treatments and unknown propensity scores. This is a significant contribution, as most prior work focuses on binary or discrete treatments.

weaknesses:
- **Terminology Issues:** Terms such as "finite-sample predictive intervals" should be revised to "finite-sample validity guarantees" to align with established terminology in the field. Additionally, (line 090) “strong finite-sample mathematical guarantees” should be “finite-sample validity guarantees”. Similarly, the frequent use of the term "exact" in relation to conditional coverage is misleading, as exact conditional coverage is known to be unattainable. I also assume you do not provide exact coverage in the context of exact and conditional coverage.
- **Relation to Lei and Candes (2021):** In Figure 2, and the appendix, the paper claims that the work of Lei and Candes can not deal with an unknown propensity, this is not true it defines and proves a coverage gap based on the error in the estimation of the likelihood ratio’s (= propensity in the case of treatment effect) in weighted CP.
- **Relation to Gibbs et al. (2023): T**he paper mentions the paper of Gibbs et al. (2023) in that it reformulates the split CP as an augmented quantile regression and the use of Lemma 1. But throughout the work, several notions from this work are taken, such as the tilting of the propensity function (lines 210-215), Theorem 1 (lines 262-276) and Theorem 2 (lines 352-371) which is heavily inspired by this work as it can even be seen in the borrowed notation of Section 4 of Gibbs et al. (2023), Algorithm 1 similarly with Algorithm 1 of Gibbs et al. (2023).
- **Under-coverage in Empirical Results**: The experimental results show some under-coverage, particularly for specific significance levels (e.g., α = 0.1). This variability raises concerns about the method's robustness, and the number of runs (10) in some experiments is too low to draw reliable conclusions. More runs would improve the statistical robustness of the findings.
- **Computational Complexity**: While the paper introduces an algorithm, the appendix discusses the method's computational efficiency. It points out that complexity can be high due to the optimization solver. This should be included in the limitations. Additionally, you should include your experiments' fitting and inference times in the paper, as this could indicate if computational complexity is an issue in practice.
- **Too simple simulation experiments: T**he proposed synthetic experiments are way too simplistic, with only one discrete confounder, continuous treatment, and continuous outcome. This is not in line with other treatment effect literature; we cannot adequately evaluate the method with these experiments.
- **Baseline**: The choice of baseline methods, particularly MC dropout, may not be ideal for comparison. Since these approaches only quantify epistemic uncertainty, which is probably minimal due to the simple experiment settings, consider a benchmark approach that quantifies both aleatoric and epistemic uncertainty, or at least aleatoric such as the Gaussian process or maximum likelihood estimator. The MLE can be easily combined with MC dropout to get both aleatoric and epistemic to get a reasonable benchmark. Also, you did not evaluate vanilla CP without considering covariate shift as a benchmark.

questions:
- What is your added contribution to the work of Gibbs et al. (2023), which you leverage in this work?
- Can you explain the variability in empirical coverage across different significance levels, particularly for α = 0.1? Do you plan to increase the number of runs or experiments to solidify your claims?
- Could you indicate how long it takes to run these experiments?
- I think there is a typo on line 476 that should be Table 1, right?
- In your synthetic experiments, why are the hard interventions related to x and not just hard interventions?
- Related to Assumption 1 (line 295-299), does this assumption mean that the estimation error needs to be the same for each X given an intervention $a$, otherwise should it not be $c_a(x)$ instead of $c_a$.
- Related to Theorem 2, what is the meaning of resulting $\sigma$ and $c_a$ of the optimization problem.
- Related to Lemma 4 (lines 712-736):
    - I think you need to clarify that this is only relevant for absolute residual non-conformity score.
    - Next, I think the proof is complete. You do not prove the connection between the Bonferonni corrected significance level and the targeted significance level.
    - In my opinion, the proof for this lemma is even elementary since it is just an application of the Bonferonni correction.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730327402977

thread_type:
official_review

<thread_object>
creator: reviewer_wfcx

<content>
summary:
Conformal prediction for causal estimands is useful because it provides finite-sample coverage guarantees. Predicting potential outcomes is challenging due to the potentially unknown distribution shift between the training set and the test set, which is subject to intervention. This work extends conformal prediction under domain shift to address the challenges that are introduced when the treatment variable is continuous, and the causal estimand is a dose response.

strengths:
The problem of conformal prediction of dose responses is clearly significant. The authors proposed an original formulation that uses Gaussian smoothing over the treatments and finds an optimal bandwidth for each problem instance. It is also important to consider soft interventions (by the authors' definition) in addition to hard interventions, since it is common in medicine and other settings to study incremental changes.

weaknesses:
* I do not understand some core elements of the motivation. What is the intention behind the separation of challenge (a) and challenge (b)? Estimating the propensity score is more difficult with continuous treatments, but it is still a challenge with discrete treatments. By my understanding, the main challenge for bringing conformal prediction to continuous treatments is that you rarely see data points with the same treatment value, so you have to do some sort of smoothing. The motivating challenges do not mention this (unless I am misreading one of them).
 * I also found the roles of soft and hard interventions confusing. Under "Scenarios" in Section 4, the authors explain that soft interventions are relevant when the propensity is known, while hard interventions are reserved for when the propensity is unknown. This storyline is convenient for the mathematical derivations. However, I am not convinced: even if the propensity function is unknown, it can still be of interest to estimate the effect of increasing the treatment by a small amount on top of the existing policy. These cases have been studied before:
   * Many papers by Chernozhukov et al. that mention "average causal derivatives"
   * Marmarelis et al. "Policy Learning for Localized Interventions from Observational Data" in AISTATS 2024.
 * When the propensity / policy is unknown, the validity of the conformal prediction relies on a constant $M$ that bounds the error of the estimated propensity. This resembles a sensitivity analysis. It appears difficult to know which $M$ to use.
 * The results from the medical dataset do not seem convincing. It is good to see that intervals are wider in treatment regions that are less common in the data. However, the fact that intervals are wider than MC-dropout is not necessarily a good thing.

Minor notes.
 * "Quantile regression might yield non-unique solutions, so we later restrict the analysis to solvers invariant to the data ordering". Please expand on this sentence. How does ordering invariance help?
 * The algorithm text is small and quite dense.
 * An interpretation of the optimized $\sigma$ is provided in the supplement, but further discussion would be helpful. Please clarify its role in the optimization problem.

questions:
* Consider building semi-synthetic experiments for stronger results.
 * How can $M$ of Assumption 1 be calibrated in practice, for tightness?
 * Do you assume some kind of continuity in the dose response?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620078190

thread_type:
official_review

<thread_object>
creator: reviewer_xich

<content>
summary:
In the paper, the authors propose a Conformal-prediction (CP) based method to obtain the interval estimation of the potential outcomes under continuous treatments. 
To achieve this, the authors follows (Gibbs et al., 2023; Romano et al., 2019) and reformulated split CP as an augmented quantile regression.
The authors derive finite-sample prediction intervals for cases where the propensity score function is known and unknown.
Moreover, they provide finite-sample theoretical guarantees.

strengths:
1. They study a very important question in the causal inference and provide novel methods. 
2. They provide the finite-sample theorem for their methods. 
3. They clearly and accessibly present the research question and methods, using figures and concrete examples.

weaknesses:
The authors do not have a good evaluation of their method in terms of numerical studies. 

Firstly, the authors only focus on the empirical coverage, and claim that it should be the higher the better. 
In my opinion, it seems not very reasonable. 
If we simply choose the [-inf, inf] as the interval, it is always has the 100% empirical coverage. 
A more balanced evaluation should consider both the interval length and empirical coverage. Additionally, the empirical coverage should closely match the nominal coverage for optimal results.


When checking the numerical results, I do believe the method in the implementation is not very stable under some cases. 
For example, in Fig 4 ($\alpha=0.2$, 5), with only 50 repetitions, I can see many outliers in the boxplot, indicating there may be some issues there. Similar issue can be found in Fig 5 ($\alpha=0.1$, 1) and  ($\alpha=0.2$, 5). 
Anyway, I know such unstableness may be inevitable for a complicated method, I think the authors should have some discussions on this issue.

questions:
1. Major question is in **Weakness** part 
2. I think the real data in the paper can not justify the superiority of authors' method well as I can see at range [0.6, 1], it yields not very reasonable prediction (blood pressure can be less than 0). 
Also, the interval estimate is too wide to be meaningful.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711828406

thread_type:
official_review

<thread_object>
creator: reviewer_qnm2

<content>
summary:
This paper proposes a conformal prediction method for continuous treatments, addressing the challenge of unknown propensity scores. It derives finite-sample prediction intervals, provides an algorithm for calculating them, and demonstrates effectiveness on synthetic and real datasets. This is the first method for continuous treatments with unknown propensity scores.

strengths:
Clear writing & solid theoretical results

weaknesses:
The reviewer is not an expert in conformal prediction for causal inference but would assume there are working targeting the studies setting: uncertainty quantification of causal effects for continuous treatment.

Equation 14 seems to use the kernel function to approximate the indicator function. Does this imply some smoothness assumptions?

The optimal parameter $\sigma^*$ represents a trade-off between the uncertainty in the prediction and the uncertainty in the interval construction. It is similar to the bandwidth approximation of the indicator function using a kernel function. Are there any empirical values ​​here, such as \sigma = cn^{-2/5} to  trade-off between the prediction and interval.

The method is demonstrated on specific datasets and scenarios. While the authors claim that their method scales to high-dimensional settings, further validation on a broader range of datasets and treatment types would strengthen the generalizability of the findings.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733155885671

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Comment to all reviewers and AC

comment:
Dear reviewers, dear AC,

Thank you again very much for the constructive evaluation of our paper and your helpful comments! We addressed all of them in our rebuttal and the following discussion with the reviewers. However, we noticed that some reviewers have not yet responded to our latest updates and responses. If you have any remaining concerns, please do not hesitate to contact us. Otherwise, if you feel our responses have addressed your concerns satisfactorily, we would kindly appreciate it if you would consider updating the rating.
In summary, our main improvements are the following:

- We added **new empirical results** on a semi-synthetic medical dataset to show the applicability of our method to high-dimensional settings (see our **new Supplement C**). 

- We compare our method against **additional baselines** (vanilla CP and a Gaussian process prediction) to assess the importance of accounting for the propensity shift (see our **Section 5** and tables direct answers to the reviewers). We find that our method is clearly effective. 

- We provide **additional discussions** on the optimization parameters and the estimation error bound (see our **new Supplement E**). Furthermore, we improved the motivation for considering the different settings (hard and soft interventions; known and unknown propensity score) and elaborated on the differences to existing work for CP for binary or discrete treatments (see **Supplement E**).

- We added a **new Theorem**  for the special case when constructing prediction intervals for soft interventions under unknown propensity scores (see our **new Supplement A.3**).

As a result, we addressed all questions and comments from the review team. We incorporated all changes into the **updated version of our paper** or will upload all remaining changes to the camera-ready version. We highlighted all key changes in **blue and red color in our revised PDF**. Given these improvements, we are confident that our paper will be a valuable contribution to the literature on uncertainty quantification in causal effect estimation and a good fit for ICLR 2025.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed a conformal prediction method for continuous treatments, addressing the challenge of unknown propensity scores. The problem is well motivated and the extension of conformal inference to continuous treatments is important. However, some main concerns remain:

 - The novelty and contribution compared with Gibbs et al. (2023) is limited, and theoretical results largely follows this paper.

 - There are concerns raised about the method's robustness and the empirical performance. For example, the authors report zero coverage for vanilla CP and GP methods, which suggests the presence of purely epistemic uncertainty and extremely heteroscedastic uncertainty and seems inconsistent with the data-generating processes. Additionally, there is a critical issue with the reported average width of prediction intervals for V-CP (claimed to be 0.0003). This value is mathematically improbable and is several orders of magnitude smaller than these theoretical lower bounds, which raises serious questions about the experimental methodology or reporting.

 - About utilities. The time complexity of the proposed approach is not very practical. The reviewers also suggested to have semi-synthetic benchmarks in the main text.
 
Given the above reasons, after discussion with the reviewers, we agreed it is not quite ready for publication and we'd encourage the authors to take into consideration all the feedback provided by the reviewers to strengthen their manuscript for resubmission.