Sources of Gain: Decomposing Performance in Conditional Average Dose Response Estimation

## Abstract

Abstract Estimating conditional average dose responses (CADR) is an important but challenging problem. Estimators must correctly model the potentially complex relationships between covariates, interventions, doses, and outcomes. In recent years, the machine learning community has shown great interest in developing tailored CADR estimators that target specific challenges. Their performance is typically evaluated against other methods on (semi-) synthetic benchmark datasets. Our paper analyses this practice and shows that using popular benchmark datasets without further analysis is insufficient to judge model performance. Established benchmarks entail multiple challenges, whose impacts must be disentangled. Therefore, we propose a novel decomposition scheme that allows the evaluation of the impact of five distinct components contributing to CADR estimator performance. We apply this scheme to eight popular CADR estimators on four widely-used benchmark datasets, running nearly 1,500 individual experiments. Our results reveal that most established benchmarks are challenging for reasons different from their creators’ claims. Notably, confounding, the key challenge tackled by most estimators, is not an issue in any of the considered datasets. We discuss the major implications of our findings and present directions for future research.

## 1 Introduction

Despite the surge in machine learning (ML) methods for estimating heterogeneous treatment effects [, , , , , , , ], there is comparatively little research on estimating the heterogeneity of dose responses, i.e., the responses to interventions with a continuous component.
This is surprising as such interventions are ubiquitous and understanding a unit’s response to them is critical in several domains, e.g., for assigning optimal discounts in marketing [], or for administering an effective dose of a medication []. Estimating dose responses from observational data is distinct from estimating treatment effects: units can be exposed to one of several different interventions for which the associated dose can vary across units. This introduces several unique challenges and calls for tailored methodologies.

The literature proposing ML-estimators for conditional average dose responses (CADR), also referred to as “individual” [] or “heterogeneous” [] dose responses, is confined to only a few methods which were proposed in the past five years [, , , , , , , ], and that have not yet seen wide-spread usage in real-world applications.
While the state of the art has progressed significantly, research lacks alignment, focusing on different challenges and using different benchmarking datasets.
This becomes especially apparent when reviewing the established benchmarking practices in CADR estimation: to date, the field has relied on a selection of (semi-) synthetic benchmarking datasets created from

Figure: Figure 1: Selected components of our decomposition scheme. To disentangle the effects of confounding from the effects of non-uniform distributions of doses, we evaluate estimators in three scenarios: 1) When doses are randomly sampled from a uniform distribution, 2) when those distributions are not uniform, but also not specific to a certain unit, and 3) when the data is confounded, so when dose assignment is specific to a certain unit. The distribution of doses across the total population is the same in steps 2) and 3). Our complete scheme includes two additional steps related to the distribution of interventions when there are multiple intervention options (cf. Section 4).
Refer to caption: x1.png

manually defined data-generating processes (DGPs). These datasets claim to test estimators in the presence of certain challenges, most notably “confounding”.(^1^11Some ML papers refer to confounding as “selection bias” which also relates to the out-of-sample generalizability of response estimates []. To reduce ambiguity, we adopt the terminology traditionally used in the causal inference literature [, , , ].)
Yet, as our experiments show, those DGPs expose estimators to more than just one challenge, of which confounding is not the most important one.

We believe that further progress in the field requires a deeper understanding of the nature of CADR estimation and the challenges therein. To that end, we propose a problem formulation for CADR estimation that unifies existing research. We conceptualize DGPs along this formulation and identify five components contributing to CADR estimator performance. To facilitate future research, we propose a novel decomposition scheme that disentangles performance along these five components, allowing researchers to understand the sources of model performance (cf. Figure 1).

To that end, our paper makes three important contributions:
(1) We introduce a unifying problem formulation for CADR estimation and conceptualize synthetic DGPs for benchmarking dataset creation; (2) We propose a scheme for decomposing model performance along the mechanisms of a DGP and specific challenges in CADR estimation; (3) We test a selection of ML estimators and decompose their performance on four of the most popular benchmark datasets for CADR estimation. We elicit strengths and weaknesses and compare their performance against traditional supervised learning algorithms.
As such, we aim to establish a standardized approach that facilitates an effective evaluation and comparison of methods. Our ambition is for the proposed decomposition scheme to be adopted in future work on CADR estimation and to guide future research.

Outline.
Section 2 introduces related work on evaluating ML estimators, and previous practices in decomposing model performance. We conceptualize CADR estimation in Section 3. In Section 4, we conceptualize DGPs and introduce our novel decomposition scheme. We illustrate our approach for a prominent dataset from Bica et al. [] in Section 5, introducing established methods and discussing the findings. We provide results on different datasets in Section 6 and conclude in Section 7.

## 2 Context

Evaluating conditional average intervention response estimators is challenging.
Evaluating the performance of conditional average intervention response estimators is challenging, as per unit of interest we can only observe the response to a single “factual” intervention, and never to any other one “counterfactual” one []. In conditional average treatment effect (CATE) estimation, there is only one counterfactual intervention: units have received either the treatment or the control. In CADR estimation, this is further complicated. First, one can apply one of several distinct interventions, and second, a practically infinite number of doses. The full dose response curve hence cannot be observed. In consequence, using observational data limits evaluation to measuring accuracy in predicting factual responses as in, e.g., supervised learning [].

Lack of established benchmarking practices.
In ML research, CADR estimators are typically evaluated using semi- or fully synthetic datasets that are specified by a certain DGP. This DGP allows the calculation of counterfactual responses of any unit and to any intervention, to overcome the challenges in evaluating using observational data.
In CATE estimation researchers have relied predominantly on a single dataset for evaluating estimator performance []. For CADR estimation, there is no such established standard, and researchers have relied on several different datasets.
Moreover, the creators of these datasets typically do not clarify the challenges present that might complicate CADR estimation.
We will show in our experiments that a dataset often embodies several challenges (cf. Section 4), which further hinders the understanding of model performance. We aim to alleviate this issue, by proposing a scheme to decompose performance along different aspects of benchmarking data.

Decomposing model performance.
It is common practice in ML research to decompose (or to “ablate”) complex model architectures by systematically adding and removing components to measure their impact on performance. In CADR estimation, such studies have, e.g., been conducted by Schwab et al. [] and Bica et al. [].
While such ablations inform about which part of an architecture contributes to improvements in model performance, they do not allow us to understand the challenges in a certain dataset. This understanding is critical to effective methodology selection in real-life applications.
We attempt to close this gap in CADR estimator research by providing a scheme to decompose datasets, not models. Such a decomposition enables us to understand the scenarios in which an estimator might work well or fail.
This data-centric decomposition of model performance is in line with calls in other ML research domains [, ].

## 3 Problem formulation

Defining CADR estimation.   We find varying definitions of CADR estimation in the literature, typically differing in the availability of only one [, ] or several [, ] distinct interventions with an associated dose. In the following, we propose a unifying framework that subsumes any of these definitions.

We leverage the Neyman-Rubin potential outcomes (PO) framework [, ] and expect a unit of interest $i$ to be specified by a realization $\mathbf{x}_{i}$ of random variable $\mathbf{X}\in\mathcal{X}$ with $\mathcal{X}\subset\mathbb{R}^{m}$ being the $m$-dimensional feature space.
The unit is exposed to a single intervention $t_{i}$ sampled from $T\in\mathcal{T}$, with the intervention space $\mathcal{T}=\{\omega_{1},\dots,\omega_{k}\}$ being discrete with $k$ different intervention options.
Every unit is also exposed to a continuous intensity or dose $d_{i}$ sampled from $D\in\mathcal{D}$ with $\mathcal{D}\subset\mathbb{R}$.
For the remainder of our paper and without loss of generality, we set $\mathcal{D}=[0,1]$.
For any realization of intervention and dose variables, there is a potential outcome $Y(t,d)\in\mathcal{Y}\subset\mathbb{R}$. In line with Schwab et al. [] we define $Y(t,d)$ as the “dose response”.

We are interested in finding an estimate of the conditional average dose response (CADR), defined as

$$ $\mu(t,d,\mathbf{x})=\mathbb{E}[Y(t,d)|\mathbf{X}=\mathbf{x}]$ (1) $$

for every $t\in\mathcal{T}$, $d\in\mathcal{D}$ and $\mathbf{x}\in\mathcal{X}$, which is in line with the definition by Bica et al. []. A detailed overview of the notation is presented in Appendix A.

Figure: Figure 2: SWIG illustrating causal dependencies in observational data
Refer to caption: x2.png

Understanding the causal structure of dose responses.
The challenges in estimating dose responses arise from the causal relationships between variables $\mathbf{X}$, $D$, $T$, and $Y$, which we illustrate in a single-world intervention graph (SWIG) [] in Figure 2. Typically, an intervention $t$ is chosen based on the observed realization of the covariates $\mathbf{x}$. Given the
intervention, a dose $d$ is assigned.
The observed potential outcome is subsequently dependent on the realization of all those variables $\mathbf{x}$, $t$, and $d$. To find an unbiased estimate of $\mu(\cdot)$ we must hence use an estimator that is flexible enough to capture the relationship between outcome and intervention variables, while correctly adjusting for the effects of $\mathbf{X}$ on all $Y$, $T$, and $D$ [], but also for the effect of $T$ on $D$. The simultaneous influence of $\mathbf{X}$ on the intervention variables and the response is referred to as “confounding” [].

On the identifiability of CADR.   Estimating dose responses from observational data relies on a set of assumptions []. Specifically, the identifiability of intervention responses requires “strong ignorability” [], subsuming “consistency”, “no hidden confounders”, and “overlap”. We provide definitions of these assumptions in Appendix C. We assume these assumptions to hold for the remainder of our work, as most previously proposed methods require them. On top of that, different from estimating treatment effects [, ], there is little research on the impacts of violations of these assumptions when estimating dose responses.

## 4 A decomposition scheme for performance of CADR estimators

Abstracting DGPs.
When working with observational data, the underlying natural DGP is typically partially or fully unknown, adding to the challenge of evaluating CADR estimator performance (cf. Section 2). To counter this, ML research typically relies on synthetic DGPs for benchmarking intervention response estimators. Those DGPs assume a causal structure (in our case the SWIG in Figure 2) and define relationships between different variables, allowing for full control over challenges in the data, such as confounding.
A typical DGP for CADR consists of four components: (1) the definition of observed units through a covariate vector $\mathbf{X}$, (2) an intervention assignment function $a_{t}(\cdot)$ assigning an intervention to every unit, (3) a dose assignment function $a_{d}(\cdot)$ assigning a dose, and (4) an outcome function $\mu(\cdot)$. Each of those components influences the resulting benchmarking dataset. Functions can be defined to take variable inputs. By taking as input the covariate vector $\mathbf{X}$, $a_{t}(\cdot)$ and $a_{d}(\cdot)$ introduce confounding in the resulting data.

We provide a list of the established CADR benchmarking datasets in Appendix F along with an overview of established CADR estimators, and the datasets used for their evaluation. The most widely used datasets are those proposed in Bica et al. [] (TCGA-2) and Nie et al. [] (IHDP-1, News-3, and Synth-1).
Typically, assignment functions are non-deterministic to comply with the strong ignorability assumption. For interventions, this is done by assigning non-zero probabilities to the various possible interventions before sampling a factual intervention per unit. Doses are sampled from a distribution with a strictly positive probability mass over $\mathcal{D}$ and a mode conditional on the covariates of a unit, so, e.g., from a normal distribution [] or a beta distribution []. The outcome function $\mu(\cdot)$ specifies a CADR by taking as input the resulting vectors $\mathbf{X}$, $D$, and $T$. Individual responses are generated by adding random noise.

Figure: (a) No confounding ($\alpha~{}=~{}1$)
Refer to caption: x3.png

Disentangling impacts of synthetic DGPs on estimator performance.
Confounding is the presence of a covariate vector that influences intervention and dose assignment, as well as the response of a unit. Different mechanisms have been proposed to introduce confounding in a benchmarking dataset:
Bica et al. [] set a modal intervention and dose that would maximize a unit’s CADR, whereas Nie et al. [] use some polynomial non-linear function mapping a unit’s covariates to a modal dose.
The level and complexity of confounding in a (semi-) synthetic dataset may vary, depending on the confounding mechanism in the synthetic DGP. However, only some of the established DGPs allow to vary the level of confounding. Therefore, it may be unclear what exactly is driving model performance: the ability of a method to model a complex CADR, as specified by $\mu(\cdot)$, or its ability to adjust for confounding.

Moreover, the assignment functions $a_{t}(\cdot)$ and $a_{d}(\cdot)$ introduce more challenges to the dataset than just confounding.
As a certain assignment mechanism impacts the probability of being assigned different interventions and doses per unit, it simultaneously influences their distribution across the observed population (cf. Figure 1 and a detailed discussion in Appendix B). This is most evident upon analyzing benchmarking datasets with tunable levels of confounding, such as the TCGA-2 dataset proposed by Bica et al. []. When the level of dose confounding is set to $\alpha=1$, so no confounding, doses are uniformly distributed across $\mathcal{D}$. When the level of confounding increases, the distribution of doses changes (cf. Figure 3). This leads to significantly fewer observations for some dose intervals, potentially impacting the performance of ML estimators [].

Decomposing performance by randomizing intervention assignment.
The adjustment for confounding is a key challenge in CADR estimation [, ]. Yet, without further investigation, performance on a dataset could be attributed to any of the above-mentioned challenges in the data.
Standard ML benchmarking practices do not reveal whether the performance of an estimator is impacted by non-linearity of responses, confounding factors, or from the intervention and dose distributions across the population.
To overcome this limitation, we propose a novel decomposition scheme for CADR estimator performance.
Our scheme is performance metric-agnostic. The choice of performance metric is arbitrary and situation-specific.

We consider two boundary scenarios:
In the “randomized” scenario, interventions and doses are completely randomized and sampled uniformly. In the “non-randomized” scenario, the data aligns with its creators’ specifications, where interventions and doses adhere to assignment functions $a_{t}(\cdot)$ and $a_{d}(\cdot)$. We then explore three intermediary scenarios that progressively move from the “randomized” to the “non-randomized” setup.

First, we generate scenario “$t$ non-uniformity”. Per unit, an intervention is sampled from a joint distribution that is equivalent to the distribution of interventions in the “randomized” scenario. This conduct maintains the effects of $a_{t}(\cdot)$ on the distribution of interventions across the total population, yet removes confounding, as $\mathbb{P}(t|\mathbf{x})=\mathbb{P}(t)$ for all $t\in\mathcal{T}$ and $\mathbf{x}\in\mathbf{X}$. This scenario informs about the impact of population-level changes in intervention distributions.
Second, we generate scenario “$t$ confounding”, in which we operate under random dose assignment, but confounded interventions as generated by $a_{t}$.
This allows us to isolate the effects of intervention confounding from distributional effects investigated previously.
Third, and in addition to intervention confounding, we repeat the process for dose assignment, generating scenario “$d$ non-uniformity”, in which $\mathbb{P}(d|\mathbf{x})=\mathbb{P}(d)$ for all $d\in\mathcal{D}$ and $\mathbf{x}\in\mathbf{X}$..
The final scenario adds dose confounding, by generating doses according to $a_{d}$, yielding the “$d$ confounding” or “non-randomized” scenario, which aligns with the original specifications of the data.
To be in line with the causal dependencies in observational data as outlined in Section 3, we chose to first decompose along the treatment assignment, yet our scheme is flexible and would allow for alterations. The full decomposition scheme is summarized in Algorithm 1 in Appendix E. We refer to Appendix J for the technical implementation. We summarize the resulting five scenarios below:

- 1.
(Randomized) Random interventions and doses (sampled from uniform distributions)
- 2.
($t$ non-uniformity) Non-uniformly distributed interventions and random doses
- 3.
($t$ confounding) Confounded interventions and random doses
- 4.
($d$ non-uniformity) Confounded interventions and non-uniformly distributed doses
- 5.
(Non-randomized / $d$ confounding) Confounded interventions and confounded doses

By iteratively changing key characteristics of the data, our decomposition scheme represents an experimental design to test for the effects of various contributing factors on CADR estimator performance. As such, we attempt to further bridge between benchmarking practices in ML and experimental study design in the wider field of causal inference [, ].

## 5 Case study: decomposing performance on the TCGA-2 dataset

### 5.1 Experimental setup

Dataset.
To demonstrate the workings of our decomposition scheme and how it helps to understand model performance, we apply it to the TCGA-2 dataset proposed by Bica et al. [].(^2^22The level of intervention and dose confounding can be varied in the TCGA-2 dataset. We opt for the standard values proposed in the original paper [] with the level of intervention confounding set to $\kappa=2$ and the level of dose confounding set to $\alpha=2$.) Several other benchmarking datasets for CADR estimation have been proposed in previous studies, which we enlist in Appendix F.
We selected the TCGA-2 dataset for several reasons. First, it comprises the assignment of one of three distinct interventions per unit and an associated dose, whereas most other datasets consider a single intervention. Second, the covariate matrix used in the TCGA data [] is frequently used in other research. Third, the assignment mechanisms used in the DGP find use in several succeeding papers, such as in Nie et al. [], Schweisthal et al. [], Nie et al. [], and Vanderschueren et al. []. For technical details, we refer to the original paper [].

ML methods for CADR estimation.
We decompose the performance of selected established CADR estimators, as well as several supervised estimators, to provide a broad set of baseline methods. Hereby, we follow calls from related ML research to test novel algorithms against a wider array of methods []. Much of recent ML research has been devoted to developing neural network architectures to tackle analytical problems, with many papers ignoring classical methods such as regression or tree-based approaches.
First, we apply three prominent ML estimators for CADR estimation, namely DRNet [], SCIGAN [], and VCNet []. Each of these is a neural network [], providing several favorable characteristics to modeling data [].
The estimators have been the first tailored ML methods for dose response estimation, and have been used as benchmark methodologies for several later-proposed methods.
Each method uniquely tackles CADR estimation: SCIGAN uses generative adversarial networks (GANs) [] to generate additional counterfactual outcomes per observed unit and effectively randomize intervention and dose assignment, as such removing confounding.
DRNet trains separate models on a shared representation learner per combination of dose intervals and interventions, to reinforce the influence of the intervention variables in the model.
VCNet follows a similar motivation, yet leverages a varying-coefficient architecture [] to accomplish this.
Appendix D provides a detailed description of the three methods. A complete overview of ML dose response estimators is provided in Appendix F.
Following calls from other fields in ML to benchmark novel methodologies against a complete set of established methods [], we further apply five supervised learning methods, which have not been sufficiently benchmarked in prior work (for a list of benchmark methodologies per established ML dose response estimator see again Appendix F).
We apply a linear regression model, a regression tree [], a generalized additive model (GAM) [], xgboost [] as a state-of-the-art implementation of a gradient-boosted decision tree [], and a simple feed-forward multilayer perception (MLP). In comparing ML CADR estimators with traditional supervised learning methods we aim to understand both the complexity of benchmarking datasets and differences in the performance of methods concerning the challenges in dose response estimation. This facilitates practitioners and researchers in making a conscious tradeoff between interpretability and performance of estimators [], as some of the targeted estimators introduce significant complexity over established techniques.

Performance evaluation.
Our decomposition scheme is agnostic to the selection of a performance metric and could be used across scenarios and use cases. Next to CADR estimators, it could similarly be used to evaluate “average” dose response estimators. For evaluating CADR estimator performance, Schwab et al. [] propose three performance metrics, especially the “policy error” and “dose policy error” which evaluate the capability of an estimator to identify the most effective interventions and doses by the magnitude of the response, as well as the “mean integrated squared error” (MISE), which evaluates the mean accuracy of the CADR estimate over the intervention and dose spaces. Since the MISE makes minimal assumptions about the domain of application or use of a CADR estimator, we adopt it in the experiments reported in this paper. For a number of test units $N$, the true CADR $\mu(\cdot)$, and the estimated CADR $\hat{\mu}(\cdot)$ we calculate the MISE as

$$ $MISE=\dfrac{1}{N}\dfrac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}\sum_{i=1}^{N} \int_{d\in\mathcal{D}}(\mu(t,d,\mathbf{x}_{i})-\hat{\mu}(t,d,\mathbf{x}_{i}))^ {2}\mathop{}\!\mathrm{d}d$ (2) $$

We use 20% of the observations in the benchmarking dataset as a holdout test set to calculate the MISE.

Model selection.
Model selection in causal inference is challenging [, , ]. Several methodologies for tuning hyperparameters on observational data have been proposed, both stand-alone and accompanying an estimator. The choice of a selection procedure can have a large influence on model performance. To ensure each method performs (near-)optimally, models are selected based on the mean squared error in predicting the factual outcomes (factual selection criterion []) on a validation set (10% of the units in the covariate matrix), the remaining 70% of observations are used for training models.

### 5.2 Results

**Table 1: Performance decomposition on TCGA-2 dataset []**
|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | Scenario |  |  |  |  |
| \cdashline2-6 |  |  |  |  |  |
| Method | random.   $\rightarrow$ | $t$ non-unif.   $\rightarrow$ | $t$ conf.   $\rightarrow$ | $d$ non-unif.   $\rightarrow$ | $d$ conf. |
| Lin. reg. | 4.74 ± 0.02 | 4.84 ± 0.03 | 4.89 ± 0.02 | 5.41 ± 0.03 | 5.41 ± 0.03 |
| Reg. tree | 0.40 ± 0.01 | 0.39 ± 0.01 | 0.39 ± 0.01 | 0.55 ± 0.04 | 0.56 ± 0.04 |
| GAM | 3.17 ± 0.02 | 3.36 ± 0.04 | 3.30 ± 0.03 | 3.77 ± 0.23 | 3.77 ± 0.22 |
| xgboost | 0.98 ± 0.07 | 0.92 ± 0.10 | 0.88 ± 0.09 | 1.14 ± 0.07 | 1.13 ± 0.06 |
| MLP | 3.14 ± 0.04 | 3.22 ± 0.07 | 3.20 ± 0.07 | 5.33 ± 0.14 | 5.30 ± 0.17 |
| \hdashline |  |  |  |  |  |
| SCIGAN | 3.05 ± 1.17 | 2.34 ± 1.84 | 1.72 ± 0.48 | 4.40 ± 4.58 | 2.08 ± 0.77 |
| DRNet | 0.97 ± 0.03 | 1.00 ± 0.04 | 1.02 ± 0.04 | 1.17 ± 0.05 | 1.16 ± 0.03 |
| VCNet | 0.29 ± 0.03 | 0.38 ± 0.02 | 0.33 ± 0.02 | 0.60 ± 0.13 | 0.60 ± 0.13 |
| random.: randomized; non-unif.: non-uniformity; conf.: confounding |  |  |  |  |  |

Insights into the dataset.
Upon analyzing the decomposed performance of all estimators as presented in Table 1, we conclude that the TCGA-2 dataset is challenging due to dose non-uniformity, rather than dose or intervention confounding.
The non-uniformity of interventions only has a small detrimental effect on model performance. Confounding of interventions has seemingly no significant effect on the performance, both for CADR estimators and supervised learning methods.
The largest effect on model performance results from introducing dose non-uniformity, i.e., making some doses less likely to be observed across the population.
This increases the MISE for all methods significantly when compared with scenario “$t$ confounding”, in which doses are sampled from uniform distributions.
The introduction of dose confounding in scenario “$d$ confounding” again does not appear to significantly impact performance. This is surprising, given the proposition of the dataset
to test methods for robustness against confounding biases. Further, the non-uniformity of doses and interventions is not a causal issue, but rather related to ML challenges outside of causal inference and treatment effect modeling, such as learning from imbalanced datasets [, ]. We also observe this behavior in analyzing performance for other datasets, as discussed in Section 6.

Insights into model performance.
Comparing performance across the different estimators allows us to draw further conclusions. Estimators are only little affected by intervention non-uniformity, and not affected by their confounding. Conversely, performance might even be improved by confounding. This is counter-intuitive to the reasoning that confounding might adversely affect performance [, ]. A possible explanation for this result is that, under confounding, units have a higher probability of being assigned exactly those doses for which CADR heterogeneity is greatest. When comparing neural architectures, CADR estimators outperform the standard MLP. However, the best-performing model is a simple regression tree. We attribute these results to the overall low heterogeneity of the dose response space in the dataset (cf. Appendix H).

Figure: (a) $t=\omega_{1}$
Refer to caption: x5.png

Understanding sources of performance gain.
The presented experimental results indicate that a decomposition of performance is imperative to evaluate the capabilities of CADR estimators. The observation that model performance does not degrade due to confounding, but due to non-uniform distributions of interventions and doses indicates that the TCGA-2 dataset is not evaluating estimators for their robustness to confounding, but rather efficiency in learning from imbalanced or limited amounts of training data [, ]. This is surprising, as several ML papers use it to test estimators for robustness to confounding [, , ]. To confirm this hypothesis, we visualize the errors in CADR estimation made by an MLP per dose interval and intervention in Figure 4, next to a histogram plot of doses in the training data. The plots show that errors in predicting CADR increase with decreasing training observations for a specific dose.
This is most notable for interventions $\omega_{2}$ and $\omega_{3}$.

## 6 Insights from other datasets

Datasets proposed by Nie et al. [].
Other prominently used datasets for benchmarking CADR estimators were proposed by Nie et al. [] (IHDP-1, News-3, and Synth-1). We decomposed the performance of all methods from Section 5 on all three of these datasets and present results in Figure 5). Compared to the TCGA-2 dataset, the datasets only apply a single intervention, so our decomposition does not include Scenarios 2 and 3 ($t$ non-uniformity and $t$ confounding). The datasets are confounded as both the observed doses and outcomes are conditional on the covariate matrices. Yet, as in the TCGA-2 dataset, this confounding seems to have no additional effect on model performance. Moreover, also dose non-uniformity does not affect the models. We provide a population-level distribution of doses per dataset in Appendix H, which shows that distributions are less skewed compared to TCGA-2. This explains why supervised learning techniques, especially

Figure: (a) IHDP-1
Refer to caption: x8.png

xgboost, are performing competitively on these datasets.
Therefore, experiments using these data sets only enable limited insight into a method’s ability to tackle challenges inherent to CADR estimation.

Decomposing performance under high CADR heterogeneity.
Working with synthetic datasets also allows visualizing the dose responses of individual units in the data (see Appendix H). All datasets discussed previously show little heterogeneity in the dose responses across different units. This might be a reason why supervised learning methods perform competitively against CADR estimators. We test this hypothesis by proposing a new benchmarking dataset that leverages the IHDP covariates, the “IHDP-3” dataset. IHDP-3 is distinct from the other datasets discussed in this study. Most importantly, the DGP behind the dataset assumes that there are different archetypes of units that respond distinctly to an intervention, e.g., units of one archetype might respond positively to an increased dose, while units from another archetype might respond negatively. Archetype assignment is not known ex-ante, but deterministic to comply with strong ignorability. Confounding is introduced by sampling from a beta-distribution with a mode conditional on a unit’s archetype. For the technical details of the dataset, we refer to Appendix G. The increase in heterogeneity of CADR is visualized in Appendix H.

Figure: Figure 6: MISE per method on the IHDP-3 dataset. Strong increase in error per method attributed to confounding.
Refer to caption: x12.png

We present the decomposed performance of CADR estimators on the IHDP-3 dataset in Figure 6 and in Appendix I. Compared to the other datasets discussed in our study, we see a significant adverse effect of dose confounding on model performance, which is evident across all estimators. Similarly, results reveal the importance of model architecture, given the estimation problem. Training individual networks per dose strata, as implemented in DRNet, leads to errors that are only comparable to linear regression, indicating that the neural architecture is not capable of handling high degrees of heterogeneity in CADR across units. Some neural architectures, especially VCNet and the standard MLP, outperform other methods in both the randomized scenario and under dose non-uniformity. The results further indicate that VCNet’s varying coefficient structure aids successful confounding adjustment.

## 7 Conclusions and future research

This paper aims to understand the nature of conditional average dose response (CADR) estimation and to provide tools to decompose estimator performance along challenges inherent to the field. We have provided a unifying problem formulation using a single-world intervention graph (SWIG) and conceptualized synthetic data-generating processes (DGPs) typically used to evaluate estimator performance. We have proposed a novel decomposition scheme that allows us to attribute estimator performance to five challenges: The non-linearity of response surfaces, confounding of interventions and doses, and their non-uniform distributions.
From our experiments, we conclude that the inherent challenges are still poorly understood. Established benchmarks are challenging predominantly due to non-uniform distributions of interventions and doses, and non-linear response surfaces. Confounding, on the contrary, seems to have little effect on model performance on these datasets. Additionally, by proposing a novel DGP and benchmarking dataset (IHDP-3), we show that confounding is a challenge for estimators only when the heterogeneity of CADR is high.

Our results show critical limitations in existing ML research on CADR estimation and suggest that more research is needed to develop benchmarks that accurately test the capabilities of estimators. We hence encourage researchers to adopt our decomposition for any future research. Additionally, we provide three further takeaways for researchers and practitioners:

(1) Confounding can materialize in various ways.
There is no clear-cut definition of how confounding might materialize in a DGP. Our experiments show that the confounding in most previously established datasets does not pose a challenge. Further research is needed to understand and quantify when tailored methods are needed.

(2) Several challenges exist in CADR estimation.
In our experiments, the biggest impact on model performance was introduced by dose non-uniformity, not through confounding. This contrasts with claims from established datasets and is a novel finding, especially as dose non-uniformity is not a causal problem. A potential future research direction might hence be the adoption of methodologies such as data-efficient ML [, ].

(3) Supervised estimators might be appropriate to model CADR.
Finally, our results reveal that standard supervised learning methods might achieve competitive performance in estimating CADR. This supports takeaway (1) and calls for comparing any future CADR estimators against a complete set of established benchmarking methods, such as gradient-boosted trees. Improving transparency might help practitioners gain trust in ML CADR estimators and aid future adoption.

## Acknowledgments and Disclosure of Funding

This work was supported by the Research Foundation – Flanders (FWO) research projects G015020N and 11I7322N.

## Appendix A Notation

We summarize the relevant notation to our paper below. Random variables are denoted in capital letters, with realizations of such in lowercase. Matrices are denoted in boldface. Realizations of a variable at a certain position, are denoted with the position as subscript.

| $\mathcal{X}\in\mathbb{R}^{m}$ | Covariate space |
| --- | --- |
| $m$ | Size of $\mathcal{X}$ (number of covariates) |
| $\mathcal{T}\in\{\omega_{1},\dots,\omega_{k}\}$ | Intervention space |
| $k$ | Number of possible interventions |
| $\mathcal{D}\in\mathbb{R}$ | Dose space |
| $\mathcal{Y}\in\mathbb{R}$ | Outcome space |
| $\mathbf{X}\in\mathcal{X}$ | Covariates (random variable) |
| $T$ | Interventions (random variable) |
| $D:\mathcal{T}\rightarrow\mathcal{D}$ | Potential dose function (function-valued random variable) |
| $Y:(\mathcal{T},\mathcal{D})\rightarrow\mathcal{Y}$ | Potential outcome function (function-valued random variable) |
| $\mu(t,d,\mathbf{x})$ | Conditional average dose response ($\mathbb{E}[Y(t,d)|\mathbf{X}=\mathbf{x}]$) |

## Appendix B Methodology details

We illustrate the non-uniformity and confounding of doses in Figure 7, using a toy example with two individual units being assigned a dose, sampled from a probability distribution. This visualization is generalizable and serves to explain any effects on interventions as well.

In the base scenario, every dose in $\mathcal{D}=[0,1]$ is equally likely assigned to any of the units. Under non-uniformity, some doses are more likely to be assigned, specifically lower and higher ones, whereas medium doses around 0.5 are less likely. The individual distributions of the dose per unit are equivalent to the joint distribution. In the case of confounded doses, the joint distribution across the two units is the same as under non-uniformity but individual distributions differ. Specifically, Unit 1 is assigned lower doses on average, whereas Unit 2 is assigned higher doses.

Figure: (a) Base scenario
Refer to caption: x13.png

## Appendix C Strong ignorability in the dose response setting

The strong ignorability assumption [] is typically posed for estimating treatment effects, so in the setting of binary interventions. Below we provide definitions all all components of this assumption that match the continuous case as tackled in our paper:

###### Assumption 1 .

(Consistency) The observed outcome $Y_{i}$ for a unit $i$ that was assigned intervention $t_{i}$ and dose $d_{i}$ is the potential outcome $Y_{i}(t_{i},d_{i})$.

###### Assumption 2 .

(No hidden confounders) The assigned intervention $T$ and dose $D$ are conditionally independent of the potential outcome $Y(t,d)$ given the covariates $\mathbf{X}$, so $\{Y(t,d)|t\in\mathcal{T},d\in\mathcal{D}\}\perp\!\!\!\!\perp(T,D)|\mathbf{X}$

###### Assumption 3 .

(Overlap) Every unit has a greater-than-zero probability of receiving any possible combination of intervention and dose, so $\forall t\in\mathcal{T}:\forall d\in\mathcal{D}:\forall\mathbf{x}\in\mathcal{X
}\text{ with }\mathbb{P}(\mathbf{x})>0:0<\mathbb{P}((t,d)|\mathbf{x})<1$

###### Assumption 1 .

###### Assumption 2 .

###### Assumption 3 .

## Appendix D ML estimators for conditional average dose responses

We will introduce the three CADR estimators considered in our paper in detail:

DRNet [].
DRNet uses a multitask learning approach [] to estimate conditional average dose responses. The method trains a head network per individual intervention on a set of shared layers, as motivated by Shalit et al. [] in the binary intervention setting. Per intervention, the method partitions the dose space $\mathcal{D}$ into a set of strata. For each strata, another individual network is trained to infer the dose response. The architecture can further be combined with regularization terms during training, to overcome potential covariate shifts between different interventions.

SCIGAN [].
SCIGAN assumes that there is a shift in covariates between different levels of intervention and dose. This shift leads to non-adjusted estimators overfitting the training data. At the core of SCIGAN is a specialized generative adversarial network (GAN) structure, which attempts to generate the outcomes of counterfactual interventions and doses per unit. Creating those counterfactual observations removes the covariate shift, and allows any estimator to learn an unbiased dose response model.

VCNet [].
VCNet proposes a varying-coefficient architecture []. The method trains a neural network that has a network structure varying in the assigned dose per unit, reinforcing the influence of the dose on the predicted outcome. Nie et al. [] combine this architecture with estimating the generalized propensity score of the dose to calculate estimates of the average dose response through an approach proposed as “functional targeted regularization”, as motivated and inspired by Shi et al. []. In our experiments, we do not use this part of the architecture and focus on estimating the conditional average dose response.

## Appendix E Pseudocode

We provide pseudocode for replicating our approach for an arbitrary machine-learning method and data-generating process:

Figure: Algorithm 1 Performance decomposition

## Appendix F Overview of established estimators and benchmarking datasets

When studying previously established dose response estimators, we find that methods have typically been evaluated on a small selection of benchmark datasets (cf. Table 2). Similarly, each paper does not compare performance against a complete set of benchmarking methods.

Table: Table 2: Overview of dose response estimators

An overview of the proposed datasets for benchmarking dose response estimators can be found in Table 3 below, presenting the dimensionality of the covariate space, as well as the cardinality of the intervention space, and the presence of both intervention and dose confounding.

Table: Table 3: Benchmarking datasets for DR estimators

## Appendix G The IHDP-3 dataset

This proposal of a new dataset for benchmarking CADR estimators, the “IHDP-3” dataset, is motivated by the dose response heterogeneity (or the lack thereof) in previously established datasets. To our surprise, most estimators were not, or only a little affected by the presence of confounders in the DGP. Conversely, in those datasets, the biggest challenge in CADR estimation has been the non-uniform distribution of doses (cf. Section 5 and 6).

The IHDP-3 dataset leverages the same covariate matrix as the IHDP-1 dataset [], notably the covariates used in the study of Hill [], but the assignment of intervention variables, as well as the outcome calculation differ significantly.

We consider a scenario with only one distinct intervention. Yet, different from previous datasets, the response to this intervention has higher heterogeneity in the covariates of a unit. Specifically, there are four different archetypes of responses to the intervention. We assign every unit in the covariate matrix to one of these archetypes based on their realization of binary covariates b.marr and mom.lths. The CADR per archetype is given in Table 4. Per unit, we generate the individual responses by adding normally distributed random noise.

**Table 4: CADR per archetype of units in the IHDP-3 dataset**
| AT | Dose response curve |
| --- | --- |
| A1 | $f_{1}(\mathbf{x}_{i},d)=10*(\mathbf{x}_{i,0}+12*d*(d-\frac{3}{4}*(\mathbf{x}_{<br>i,1}+\mathbf{x}_{i,2}))^{2})$ |
| \hdashline |  |
| A2 | $f_{2}(\mathbf{x}_{i},d)=10*(\mathbf{x}_{i,1}+\sin(\pi*(\mathbf{x}_{i,2}+<br>\mathbf{x}_{i,3})*d))$ |
| \hdashline |  |
| A3 | $f_{3}(\mathbf{x}_{i},d)=10*(\mathbf{x}_{i,2}+12*(\mathbf{x}_{i,3}*d-\mathbf{x}<br>_{i,4}*d^{2}))$ |
| \hdashline |  |
| A4 | $f_{4}(\mathbf{x}_{i},d)=\mathbf{x}_{i,0}*3*\sin(20*\mathbf{x}_{i,2}*d)+20*<br>\mathbf{x}_{i,3}*d-20*\mathbf{x}_{i,4}*d^{2}+5$ |
| AT: Archetype, $\mathbf{x}_{i,j}$: Variable $j$ in the covariate vector of unit $i$ |  |

Next, we assume that doses are assigned to every observation based on their archetype, and assign every of the archetypes a modal dose in $\{\frac{1}{8},\frac{3}{8},\frac{5}{8},\frac{7}{8}\}$. We then sample for every unit a factual dose from a beta distribution with the respective mode and tunable variance, following the approach first introduced by Bica et al. []. For a detailed DGP and the technical implementation, we refer to our source code (cf. Section J).

The resulting data has a significantly higher heterogeneity in the dose responses across units as we visualize in Appendix H.

## Appendix H Deep dive into benchmarking datasets

We visualize the CADR per unit in the different benchmarking datasets in Figure 8. The heterogeneity in CADR varies along the datasets. For the previously established ones, per intervention, responses are generated from a single, but differently-parameterized function. This yields little heterogeneity in the CADR across units and might explain the good performance of supervised learning methods on these datasets. For IHDP-3, we can see higher degrees of heterogeneity with CADR depending on the archetype (cf. Appendix G). The confounding in the data significantly complicates the estimation of CADR, as seen in the detailed results (cf. Appendix I).

Figure: (a) IHDP-1
Refer to caption: x22.png

We further visualize confounding in the datasets by generating t-SNE plots [, ] of their covariate spaces and color coding observations by their factual dose (cf. Figure 9). Under confounding, we expect that units different in their covariates would be assigned different factual doses. Yet, not all plots indicate this behavior. Especially in the News-3 and IHDP-1 datasets units with different factual doses cannot be separated.

Figure: (a) IHDP-1
Refer to caption: x27.png

## Appendix I Results per dataset

Next to our detailed discussion of model performance on the TCGA-2 dataset (see Sectio 5), we provide results for every other available benchmarking dataset below:

**Table 5: Performance decomposition on IHDP-1 dataset**
|  |  |  |  |
| --- | --- | --- | --- |
|  | Scenario |  |  |
| \cdashline2-4 |  |  |  |
| Method | random.   $\rightarrow$ | $d$ non-unif.   $\rightarrow$ | $d$ conf. |
| Lin. reg. | 4.66 ± 0.03 | 4.76 ± 0.06 | 4.85 ± 0.07 |
| Reg. tree | 1.33 ± 0.11 | 1.35 ± 0.21 | 1.24 ± 0.10 |
| GAM | 1.67 ± 0.03 | 1.71 ± 0.03 | 1.95 ± 0.13 |
| xgboost | 1.04 ± 0.10 | 1.08 ± 0.10 | 1.14 ± 0.13 |
| MLP | 2.88 ± 0.26 | 3.10 ± 0.24 | 2.85 ± 0.24 |
| \hdashline |  |  |  |
| SCIGAN | 6.86 ± 1.21 | 6.54 ± 1.07 | 5.88 ± 0.47 |
| DRNet | 2.63 ± 0.15 | 2.39 ± 0.06 | 2.60 ± 0.11 |
| VCNet | 1.38 ± 0.20 | 1.18 ± 0.20 | 1.43 ± 0.21 |
| random.: randomized; non-unif.: non-uniformity; conf.: confounding |  |  |  |

**Table 6: Performance decomposition on IHDP-3 dataset**
|  |  |  |  |
| --- | --- | --- | --- |
|  | Scenario |  |  |
| \cdashline2-4 |  |  |  |
| Method | random.   $\rightarrow$ | $d$ non-unif.   $\rightarrow$ | $d$ conf. |
| Lin. reg. | 14.91 ± 0.21 | 14.79 ± 0.19 | 17.86 ± 0.52 |
| Reg. tree | 6.92 ± 0.77 | 8.89 ± 2.23 | 10.82 ± 4.19 |
| GAM | 15.22 ± 0.29 | 15.16 ± 0.32 | 17.30 ± 0.37 |
| xgboost | 7.06 ± 0.69 | 8.01 ± 0.57 | 10.92 ± 1.02 |
| MLP | 3.47 ± 0.23 | 3.97 ± 0.44 | 10.14 ± 0.45 |
| \hdashline |  |  |  |
| SCIGAN | 10.40 ± 2.24 | 12.33 ± 3.57 | 14.65 ± 6.24 |
| DRNet | 14.92 ± 0.21 | 15.08 ± 0.26 | 16.17 ± 0.16 |
| VCNet | 2.68 ± 0.35 | 3.76 ± 0.79 | 8.45 ± 0.76 |
| random.: randomized; non-unif.: non-uniformity; conf.: confounding |  |  |  |

**Table 7: Performance decomposition on News-3 dataset**
|  |  |  |  |
| --- | --- | --- | --- |
|  | Scenario |  |  |
| \cdashline2-4 |  |  |  |
| Method | random.   $\rightarrow$ | $d$ non-unif.   $\rightarrow$ | $d$ conf. |
| Lin. reg. | 1.07 ± 0.10 | 1.09 ± 0.11 | 1.08 ± 0.10 |
| Reg. tree | 1.26 ± 0.13 | 1.30 ± 0.10 | 1.29 ± 0.18 |
| GAM | 1.11 ± 0.08 | 1.16 ± 0.08 | 1.12 ± 0.05 |
| xgboost | 0.98 ± 0.06 | 0.98 ± 0.04 | 0.97 ± 0.05 |
| MLP | 1.04 ± 0.08 | 1.03 ± 0.12 | 1.01 ± 0.11 |
| \hdashline |  |  |  |
| SCIGAN | 1.57 ± 0.15 | 1.89 ± 0.22 | 2.32 ± 1.71 |
| DRNet | 1.01 ± 0.10 | 0.99 ± 0.09 | 1.00 ± 0.08 |
| VCNet | 0.91 ± 0.05 | 0.90 ± 0.04 | 0.78 ± 0.06 |
| random.: randomized; non-unif.: non-uniformity; conf.: confounding |  |  |  |

**Table 8: Performance decomposition on Synth-1 dataset**
|  |  |  |  |
| --- | --- | --- | --- |
|  | Scenario |  |  |
| \cdashline2-4 |  |  |  |
| Method | random.   $\rightarrow$ | $d$ non-unif.   $\rightarrow$ | $d$ conf. |
| Lin. reg. | 0.73 ± 0.03 | 0.73 ± 0.03 | 0.77 ± 0.03 |
| Reg. tree | 0.50 ± 0.05 | 0.53 ± 0.12 | 0.57 ± 0.11 |
| GAM | 0.44 ± 0.03 | 0.44 ± 0.03 | 0.48 ± 0.04 |
| xgboost | 0.41 ± 0.03 | 0.41 ± 0.02 | 0.49 ± 0.04 |
| MLP | 0.32 ± 0.02 | 0.32 ± 0.03 | 0.42 ± 0.05 |
| \hdashline |  |  |  |
| SCIGAN | 0.58 ± 0.11 | 0.62 ± 0.09 | 1.09 ± 0.13 |
| DRNet | 0.49 ± 0.03 | 0.49 ± 0.03 | 0.50 ± 0.03 |
| VCNet | 0.31 ± 0.03 | 0.31 ± 0.03 | 0.37 ± 0.04 |
| random.: randomized; non-unif.: non-uniformity; conf.: confounding |  |  |  |

## Appendix J Implementation and hyperparameter optimization

All experiments were written in Python 3.9 [] and run on an Apple M2 Pro SoC with 10 CPU cores, 16 GPU cores, and 16 GB of shared memory. The system needs approximately two days for the iterative execution of all experiments. The code to reproduce all experiments and figures in our paper can be found online via [https://github.com/christopher-br/CADR-performance-deco](https://github.com/christopher-br/CADR-performance-deco) including a reference to the necessary covariate matrices.

For SCIGAN and VCNet, we use the original implementations provided by Bica et al. [] ([https://github.com/ioanabica/SCIGAN](https://github.com/ioanabica/SCIGAN)) and Nie et al. [] ([https://github.com/lushleaf/varying-coefficient-net-with-functional-tr](https://github.com/lushleaf/varying-coefficient-net-with-functional-tr)). All remaining neural network architectures were implemented in PyTorch [] using Lightning []. Xgboost is implemented using the xgboost library []. GAMs were implemented using the PyGAM library []. All other methods were implemented using the Scikit-Learn library [] and the statsmodels library [].

For TCGA-based datasets, linear regression models and GAMs were trained using the first 50 principal components of the covariate matrix to reduce computational complexity.

Hyperparameter optimization.
For all methods, we used a validation set for hyperparameter optimization and chose the best model in terms of validation set mean squared errors (MSE). We do so to ensure fair model comparison and isolate model performance from parameter selection procedures, as presented accompanying some estimators [, ]. We ran a random search over the hyperparameter ranges as listed per the model below. If not specified differently, the remaining hyperparameters are set to match the specifications of the original authors. Results are not to be compared to the original papers, as the optimization scheme and parameter search ranges differ from the original records.

**Table 9: Hyperparameter search range for Linear Regression:**
| Parameter | Values |
| --- | --- |
| Penalty | $\{Elasticnet,None\}$ |

**Table 10: Hyperparameter search range for Regression Tree:**
| Parameter | Values |
| --- | --- |
| Max depth | $\{5,15,None\}$ |
| Min sample split | $\{2,5,20\}$ |
| Min samples per leaf | $\{1,5,10\}$ |
| Max features per split | $\{None,\sqrt{p(\mathbf{x})}\}$ |
| \hdashline |  |
| Splitting criterion | $\{Gini\}$ |

**Table 11: Hyperparameter search range for GAM:**
| Parameter | Values |
| --- | --- |
| Interaction type | $\{Univariate\}$ |
| Numb configurations | $\{20\}$ |

**Table 12: Hyperparameter search range for xgboost:**
| Parameter | Values |
| --- | --- |
| Learning rate | $\{0.01,0.1,0.2\}$ |
| Max depth | $\{3,5,7,9\}$ |
| Subsample | $\{0.5,0.7,1.0\}$ |
| Min child weight | $\{1,3,5\}$ |
| Gamma | $\{0.0,0.1,0.2\}$ |
| Columns sampled per tree | $\{0.3,0.5,0.7\}$ |

**Table 13: Hyperparameter search range for MLP:**
| Parameter | Values |
| --- | --- |
| Learning rate | $\{0.0001,0.001\}$ |
| L2 regularization | $\{0.0,0.1\}$ |
| Batch size | $\{64,128\}$ |
| Hidden size | $\{32,48\}$ |
| \hdashline |  |
| Num steps | $\{5000\}$ |
| Num layers | $\{2\}$ |
| Optimizer | $\{Adam\}$ |

**Table 14: Hyperparameter search range for SCIGAN:**
| Parameter | Values |
| --- | --- |
| Hidden size | $\{32,64,128\}$ |
| Batch size | $\{128,256\}$ |
| \hdashline |  |
| Num head layers | $\{2\}$ |
| Num dose samples | $\{5\}$ |
| $\lambda$ | $\{1\}$ |
| Optimizer | $\{Adam\}$ |

**Table 15: Hyperparameter search range for DRNet:**
| Parameter | Values |
| --- | --- |
| Learning rate | $\{0.0001,0.001\}$ |
| L2 regularization | $\{0.0,0.1\}$ |
| Batch size | $\{64,128\}$ |
| Hidden size | $\{32,48\}$ |
| \hdashline |  |
| Num dose strata | $\{10\}$ |
| Num steps | $\{5000\}$ |
| Num layers | $\{2\}$ |
| Optimizer | $\{Adam\}$ |

**Table 16: Hyperparameter search range for VCNet:**
| Parameter | Values |
| --- | --- |
| Learning rate | $\{0.001,0.01\}$ |
| Batch size | $\{128,256\}$ |
| \hdashline |  |
| Hidden size | $\{32\}$ |
| Num steps | $\{5000\}$ |
| Optimizer | $\{Adam\}$ |

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 10
presentation: 4
questions: As I mention above, I wondered whether the methods presented here could be extended to analyze the particular impact of weak overlap separately from the problem of confounding. Do you think that would be possible?
soundness: 4
strengths: I really like this paper. I think it is insightful, thorough, well-explained, and I find the authors' arguments convincing. I believe the proposed benchmarking approach is viable and I hope it catches on.
summary: Estimation of Conditional Average Dose Responses (CADR) is a challenging problem. The performance of a CADR estimator depends on its ability to cope with non-uniformity of intervention and/or dose assignment, and the presence of confounding variables that impact the outcome as well as the intervention and/or dose assignment. Which of these challenges is more or less important in a particular empirical setting depends on the nature of the underlying Data Generating Process (DGP). In ML, the performance of CADR estimators are typically assessed by benchmarking performance on synthetic or semi-synthetic datasets. These facts motivate a decomposition of the performance of CADR estimators on synthetic/semi-synthetic data by sequentially removing different challenges by altering the underlying DGP. In particular, one first assesses performance under the unaltered DGP with confounding of both dose and intervention. Then under confounded interventions but with unconfounded and non-uniform doses, then confounded and unconfounded uniform doses, unconfounded non-uniform interventions and unconfounded uniform doses, and finally uniform and unconfounded doses and interventions. 

By assessing performance under this sequence of altered DGPs, one can evaluate the impact of each of these distinct challenges (confoundedness and non-uniformity of intervention and dose) on estimator performance. If performance of an estimator improves substantially after one challenge is removed, then one can conclude that this challenge is responsible for a substantial degradation of performance for that estimator in that dataset. Thereby, one can assess a) whether some estimators are better able to deal with a particular one of the challenges, and b) how much difficulty each of these challenges poses in general in a particular dataset.

The authors then apply their methods to commonly used benchmarking datasets. Among other things, they demonstrate that in the TCGA-2 dataset confounding does not pose a significant challenge for the methodsthey apply, whereas non-uniformity of the dose has a much greater negative impact on performance.

The authors also find that modern neural-network methods do not exhibit superior performance in standard datasets, and they suggests that this may be because the degree of heterogeneity in the CADR is insufficiently high for the flexibility of these methods to be a substantial advantage. In resoponse, they develop a new synthetic dataset with greater heterogeneity.
weaknesses: Another feature of a DGP that can pose a substantial challenge for CADR estimation is weak covariate overlap. By `weak overlap' I mean cases in which $0<P((t,d)|x)<1$ but $P((t,d)|x)$ is close to zero or one for some values of $x$, $t$, and $d$. As with non-uniformity of dose and intervention, this problem can be hard to disentangle from the problem of confounding. I wondered whether the individual impact of weak overlap could be assessed by further subdividing the five scenarios on page 5.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: -	Why is the focus of the scheme on the treatment/dosage assignment and not the DR function?
soundness: 2
strengths: -	The paper tackles an important problem in causal inference by trying to propose a unifying approach for model evaluation (here for CADR estimation). In principle, I think this is a widely understudied issue as most papers just design their own semi-synthetic datasets, making it harder to evaluate the fairness of model comparisons.
-	The paper is nicely written and easy to follow
summary: The paper proposes a novel decomposition scheme for benchmarking CADR estimators. For that, the authors propose alternating different steps in the treatment and dosage assignment to investigate the effect of confounding on model performance. Further they provide experiments on widely-used benchmark datasets giving novel insights for existing CADR methods.
weaknesses: -	The contribution of the paper is only very limited. While I think it is nice to have a paper summarizing challenges in CATE/CADR evaluation and propose best practices, I think the added value is not sufficient for an ICLR paper. This is mainly a benchmarking paper and the setup of the scheme seems pretty arbitrary.
-	The interpretation of key parts of their framework could be clearly improved.
    -  The main goal is to “decompose the source of gain” in CADR estimation. However, the main focus of their DGP decomposition is only around the treatment and dosage assignment. However, if there is no heterogeneity in the dose-response curve, stronger dependence between assignment and X does not necessarily increase confounding since Y does not (strongly) depend on X. Thus, non-uniformity can be obviously even more challenging than the “confounded” setting because of increased data sparsity in some areas with low values of the marginal densities of the doses. While the authors acknowledge that and propose a dataset with increased heterogeneity in the DR curve, this finding should imply that for disentangling the performance drivers of CADR estimators such framework needs to jointly vary treatment/dosage assignment and heterogeneity of the DR curve and probably focus less on a distinction between non-uniformity and confounding.
    -	The evaluated benchmarks are also more dependent on the true DR curve than on the treatment/dosage assignment (e.g., DRNet and VCNet strongly depend on the “nodes” / “strata”, DRNet only does balancing on the treatment level so it’s not designed for dosage bias, VCNets functional targeted regularization is designed for optimizing ADR estimation instead of CADR). This could be more discussed and, thus, investigating the complexity and heterogeneity of the DR curve would probably give more insights into the advantages of such methods.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: Can these findings generalize more broadly? Do you have suggestions for future benchmarks to include forms of confounding that are more challenging?
soundness: 2
strengths: The weaknesses in the literature on estimating dose responses are clearly presented, and so is the distinction between confounding and treatment non-uniformity. Taking a closer look at the commonly used semi-synthetic benchmarks and their statistical characteristics is important especially because the community sometimes glosses over these details.
summary: This paper considers the practical implications of estimating dose responses with multiple intervention categories. The main argument is that confounding is different from non-uniform treatment distributions, and the latter seems to be the main source of difficulty in commonly used benchmark datasets. If so, this is important because the community generally considers confounding to be the main challenge.
weaknesses: While the goal of decomposing the statistical challenges of estimating dose responses is highly significant, I am uncertain if this contribution is substantive. My primary concern has to do with the narrow scope of the empirical evaluations. The authors focused on the TCGA-2 semi-synthetic benchmark proposed by Bica et al. (2020). While this is definitely one of the more commonly used datasets in the literature, there is little consensus on benchmarks in the field as a whole and many papers propose their own variants.

More broadly, the claim that confounding does not matter as much as non-uniformity of treatments is not strongly supported. This is evident in Section 6, where the results for IHDP-1, News-3, and Synth-1 are much more mixed. As for TCGA-2, I was expecting an analysis of different strengths of confounding to see the level at which confounding is or is not important, compared to the other "knobs". Presently, the paper considers just one specific version of confounding. Given the generality of what we can consider "confounding", these findings are more a critique of TCGA-2 than any generalizable insight. It would have been interesting if, at the very least, one of the hyperparameters for TCGA-2 were varied to allow exploration of different confounding landscapes.

Minor point: please explain in Table 1's caption that you are listing errors (MISE).

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: Q1: Why is the intervention space discrete if the interventions are supposed to be continuous? Why do you split t vs. d? (that's at least not the standard setting of DR; I like the flexibility but it may need a few additonal comments)

Q2: Table 1 reports +/-. What is this SD/SE/range? Over how many runs? (I may have overlooked this detail)
soundness: 2
strengths: * Benchmarking is relevant for causal ML and especially dose-response estimation (=continuous treatments) receive to little attention. 
* Clear writing
* Code is available (but I did not run the code)
summary: The paper performs an empirical study to better understand the performance in dose-response estimation (i.e., estimating causal effects from continuous treatments). The starting point for the paper is that benchmarking datasets for dose-response estimation missing. As such, the paper rightfully says that (1) continuous treatment is an important setting and that (2) benchmarking in causal effect estimation is overlooked (and should receive more attention). Similar benchmarking studies have been conducted by several papers (see below), however, for binary treatments and **not** continuous treatments. 

Curth, A., & Van Der Schaar, M. (2023, July). In search of insights, not magic bullets: Towards demystification of the model selection dilemma in heterogeneous treatment effect estimation. In International Conference on Machine Learning (pp. 6623-6642). PMLR.

Mahajan, D., Mitliagkas, I., Neal, B., & Syrgkanis, V. (2022). Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation. arXiv preprint arXiv:2211.01939.

Bach, P., Schacht, O., Chernozhukov, V., Klaassen, S., & Spindler, M. (2024, March). Hyperparameter Tuning for Causal Inference with Double Machine Learning: A Simulation Study. In Causal Learning and Reasoning (pp. 1065-1117). PMLR.
weaknesses: The depth of the implications is limited. Let me give you a few examples and also make suggestions for how to improve the contribution of the work. For example, the paper correctly points out that standardized benchmarks for dose-response estimation are missing. (For comparison, there is ACIC2016+2018 for binary treatment). The paper could make (for example) a much larger contribution if it would contribute new, standardized datasets. 

Likewise, the paper reports some results, but is fairly silent about meaningful implications for the field to evolve. I would love to see a paper that offers more 'implications' and 'learnings' that help researchers. A nice example is the paper from Alicia Curth, where very detailed implications are given. This might help to make the paper stronger and more meaningful (e.g., under which conditions should which estimator be used?). 

The findings would be stronger if more than one dataset were used; it is likely that other datasets may lead to different results. 

(Disclaimer: I used DR estimation myself but I would not have any real takeaway messages that I take from the paper. I think bechmarking studies are relevant and strongly recommend that the authors submit their work eg. to CLeaR)

Minor: Fig 2+3 imho have little value for the main paper.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: - The framework consists of 5 discrete steps which seem arbitrary. E.g., in the "non-uniformity" parts, the authors replace the treatment distribution but how is it generally specified? I expect the results to be highly dependent on that (e.g., setting the density to a highly concentrated distribution will result in strong overlap violations). The same holds for confounding: how should we decide on the "strength" of confounding, which is not a discrete quantity?

- How exactly are the datasets changed to fit into the framework? I assume the authors construct semi-synthetic datasets by using real covariates and creating synthetic treatments and outcomes?
soundness: 3
strengths: - The paper addresses an important and relevant topic: benchmarking practices in causal inference are not as streamlined as in other ML disciplines and it is not clear how to construct good benchmarks for causal inference methods. Steps towards a streamlined benchmarking process are necessary for the field to move towards a transparent evaluation of proposed methods.
- The paper is well-written and easy to follow.
summary: The paper proposes a new framework for constructing (semi-) synthetic benchmark datasets for causal inference, specifically the estimation of dose-response curves with continuous treatments. In their framework, the authors distinguish between distribution shifts and confounding on both the treatment type and the dosage given. Experiments are performed using a set of established estimators and datasets from the literature.
weaknesses: - Novelty: While I appreciate the general topic of the paper, I think that the novel ideas are relatively limited. As far as I understand, the proposed framework essentially disentangles the different confounding mechanisms in treatment and dosage.
- It is unclear why the proposed framework (i.e., steps 1-5 on page 5) is the best way to approach benchmarking for dose responses. It untangles the different sources of confounding but does not address other challenges for dose-response estimation. For example, lack of overlap is probably equally important when considering continuous treatments in causal inference. While there are certain connections between confounding strength and lack of overlap (e.g., a strong confounder will make overlap less likely), the framework does not address this in a principled manner. Another aspect that is missing is the complexity of the response function and treatment assignment, which can play important roles in estimator performance (see e.g., Kennedy 2023, Towards optimal doubly robust estimation of heterogeneous causal effects).
- The list of dose-response estimators considered (e.g., Table 2) is incomplete. For example, the causal forest can be used to obtain dose-response estimates. Additionally, all estimators considered are so-called "plugin" learners. In contrast, meta-learners are built upon semiparametric efficiency theory and Neyman orthogonal losses and can be combined with most regression-based approaches considered in this paper. For example, the R-learner ("Double ML") is implemented here for dose responses: https://econml.azurewebsites.net/spec/estimation/dml.html.
- I think results for the full framework on an additional dataset would strengthen the paper, given that the full framework does not apply to the datasets in Sec. 6.