AutoCATE: End-to-End, Automated Treatment Effect Estimation
Authors: Toon Vanderschueren
Tim Verdonck
Mihaela van der Schaar
Wouter Verbeke
Venue: ICML 2025 poster (2025)
Keywords: Treatment Effect Estimation
Causal Inference
AutoML
Primary area: general_machine_learning->causality
TL;DR: AutoCATE is the first end-to-end, automated framework for estimating conditional average treatment effects (CATE), unifying evaluation, estimation, and ensembling in a single framework.

Abstract: Estimating causal effects is crucial in domains like healthcare, economics, and education. Despite advances in machine learning (ML) for estimating conditional average treatment effects (CATE), the practical adoption of these methods remains limited, due to the complexities of implementing, tuning, and validating them. To address these challenges, we formalize the search for an optimal ML pipeline for CATE estimation as a counterfactual Combined Algorithm Selection and Hyperparameter (CASH) optimization. We introduce AutoCATE, the first end-to-end, automated solution for CATE estimation. Unlike prior approaches that address only parts of this problem, AutoCATE integrates evaluation, estimation, and ensembling in a unified framework. AutoCATE enables comprehensive comparisons of different protocols, yielding novel insights into CATE estimation and a final configuration that outperforms commonly used strategies. To facilitate broad adoption and further research, we release AutoCATE as an open-source software package.

AutoCATE: End-to-End, Automated Treatment Effect Estimation
Toon Vanderschueren 12
Tim Verdonck
Mihaela van der Schaar Wouter Verbeke
Abstract
marketers, these challenges may outweigh potential benefits, hindering the practical use of these techniques. To address this, advocate for automated, end-to-end solutions for learning ML pipelines for CATE estimation.
Estimating causal effects is crucial in domains like healthcare, economics, and education. Despite advances in machine learning (MI.) for estimat- ing conditional average treatment effects (CATE), the practical adoption of these methods remains limited, due to the complexities of implement- ing. tuning, and validating them. To address these challenges, we formalize the search for an optimal ML pipeline for CATE estimation as a counter- factual Combined Algorithm Selection and Hy- perparameter (CASH) optimization. We intro- duce AutoCATE, the first end-to-end, automated solution for CATE estimation. Unlike prior ap- proaches that address only parts of this problem. AutoCATE integrates evaluation, estimation, and ensembling in a unified framework. AutOCATE TE enables comprehensive comparisons of different protocols, yielding novel insights into CATE es- timation and final configuration that outper- forms commonly used strategies. To facilitate broad adoption and further research, we release AutoCATE oCATE as an open-source software package.
The challenge of automated CATE estimation. Despite significant progress in automated ML (AutoML) (see He et al., 2021), existing solutions do not address the unique challenges of CATE estimation. A key problem the lack of ground truth: the treatment effect the difference in out- comes with and without treatment, but only one outcome is observed per instance. Which outcome is observed depends confounding variables (e.g., older patients more often receive treatment), leading to covariate shift (Shalit et al., 2017). Finally, CATE estimation pipelines are highly com- plex: metalearners combine multiple baselearners, and often include both classification and regression models. Risk measures require predictions themselves and, thus, tuning ML pipelines. These challenges complicate training and validation of MI pipelines for CATE estimation.
Contributions. To address these challenges, we propose Aut oCATE, the first automated, end-to-end framework for constructing and validating an ML pipeline for CATE esti- mation. In doing so, we make the following contributions: COUNTERFACTUAL. CASH-V formalize the search for an ML pipeline for CATE estimation as counterfac- tual Combined Algorithm Selection and Hyperparameter (CASH) optimization. Our solution, outO,ATE, automati- cally searches across preprocessors, metalearners, evalua- tors, baselearners, and their hyperparameters. The process is orgamized into three ensembling-eade including several design choices.
1. Introduction
Accurately estimating causal effects is crucial in domains like healthcare, education, and economics. Despite advances in machine learning (ML. for estimating the conditional av- erage treatment effect (CATE), real-world adoption remains limited due to the complexity of developing ML pipelines for CATE estimation. Methods often involve numerous hyperpa- rameters, and their performance varies significantly across data sets and applications. Moreover, validating counterfac- tual predictions and tuning pipelines highly challenging, and the performance of evaluation criteria also varies with the data generating process (Curth & van der Schaar, 2023). For practitioners unfamiliar with MI., such as clinicians or
estimation, and
END-TO-END AUTOMATION- develop automated protocols that perform well across data sets and applica- tions. Our end-to-end approach includes ovten-overlooked aspects of CATE estimation, such as preprocessing, fea- ture selection, and ensembling. Our perspective uncovers novel insights (see Figure 14), questions (c.g.. the trade-off to use data for training or validation) and solutions (e.g., multi-objective optimization across evaluation criteria). SOFTWARE PACKAGE- release AutoCATE as an open-source tool to support future research on all aspects of CATE estimation. For practitioners unfamiliar with ML, AutoCATE makes automated CATE estimation with ad- vanced ML techniques accessible in few lines of code.
'K Leuven UUiiversity of Antwerp *University of Cambridge. Correspondence to: Toon Vanderschueren <toon vanderschuerene & caatiurrnn@mmmi.ccommmmm comi.
Proceedings of the 4rnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

AutOCATE: End-to-End, Automated Treatment Effect Estimation
2. Related Work
and the many design choices involved. Not only will ML algorithm be optimal in all possible settings, there is also no globally optimal metalcarner, as performance similarly de- pends on the (unknown) data generating process and sample size (Curth & van der Schaar, 2021). Finally, funing is more involved: for example, a DR-Learner combines four mod- els (to estimate the propensity, the outcome per treatment group, and the final treatment affect)-each of which can be different baselearner with separate hyperparameters.
Our work is related to two areas ML research: (1) Au-
toML, and (2) CATE estimation and model validation.
2.1. Automated Machine Learning (AutoML)
AutoML aims to efficiently and automatically construct per- formant ML pipelines. This involves a series of design choices regarding preprocessing, feature transformation and selection, ML algorithms, and hyperparameter tuning (Kar- maker et al., 2021). As the optimal choices depend on the data and task. AutoML is essentially search problem. Therefore, efficient search methods have been developed (Bergstra et al., 2011; Snoek et al., 2012; Alaa & van der Schaar, 2018) and meta-learning is used to incorporate in- formation across settings (Feurer et al., 2015). A key as- pect of AutoML is accessibility through low-code tools for practitioners unfamiliar with ML (LeDell & Poirier, 2020; Erickson et al., 2020; Jarrett et al., 2021; Wang et al., 2021).
Model validation. As the CATE is unobserved, evaluation criteria have been proposed to validate CATE estimators. A common approach is the error in predicting the observed outcome μl (the j--risk). However, this criterion has several limitations (Curth & van der Schaar, 2023; Doutreligne & Varoquaux, 2023): it does not account for confounding, may not accurately predict CATE error', and is not applicable to metalearners that directly predict the CATE, To mitigate the first issue, an inverse propensity weighted variant /aPw- risk, can be considered. Other evaluation criteria address all issues by constructing labels based on plug-in in estimates (e.g., S- or T-risk) or metalearner pseudo-outcomer (e.g.. R- and DR-risk), see Appendix B.2 for detailed overview.
AutoML solutions exist for a wide range of tasks, including reinforcement learning (Runge et al., 2019), time series fore- casting (Jarrett et al., 2021), semantic segmentation (Chen et al., 2018), and machine translation (So et al., 2019). For more comprehensive overviews, see (Elsken et al., 2019) and (He et al., 2021). However, to the best of our knowledge, AutoML has not yer been applied to CATE estimation. As discussed, estimating treatment effects presents unique chal- lenges, such as the absence of a ground truth, covariate shift due to confounding, and the need for intermediary models in metalearners and risk measures. These complexities render standard AutoMI. approaches ill-suited for CATE estima- tion and illustrate the need for specialized approaches.
There is no consensus the optimal validation criterion. Schuler et al. (2018) and Doutreligne & Varoquuux (2023) advocate for the R-risk, while Mahajan et al. (2023) favor the T- and DR-risk. Conversely, Curth & van der Schaar (2023) show that a risk measure's 's effectiveness varies with different factors, such as the metalearner and data generating process, with no single criterion being universally optimal. Additionally, Doutreligne & Varoquaux (2023) highlight the importance of using flexible estimators to construct pseudo- labels, with (Mahajan et al., 2023) advocating the use of AutoMI.. The lack of consensus and design choices involved stress the need for end-to-end, automated procedures.
Research gap-No No existing AutoML solutions address the unique and complex challenges of CATE estimation.
Research gap-Despite significant progress in using ML for CATE estimation and model validation, key questions remain unresolved: when use particular methods, how to tune them effectively, and how to optimize critical but overlooked aspects like preprocessing or ensembling.
2.2. Treatment Effect Estimation and Model Validation
Estimation. Various ML methods have been proposed for CATE estimation. Metalearners are general strategies for using supervised learning algorithms for CATE esti- mation (Künzel et al., 2019). Additionally, various ML algorithms have been adapted for CATE estimation, such as Gaussian processes (Alaa & van der Schaar, 2017), neural networks (Shalit et al., 2017: Yoon et al., 2018), decision trees (Rzepakowski & Jaroszewicz. 2012), or random forests (Wager& & Athey, 2018: Oprescu et al., 2019). Other com- ponents of the ML pipeline also present complexities when estimating treatment effects, such as missing value imputa- tion (Berrevoets et al., 2023), feature selection (Zhao et al., 2022), and ensemble selection (Mahajan et al., 2023).
3. Problem Formulation
Notation and assumptions. We represent an instance by a tuple (z, t, with covariates X E X c Rd, a treatment T T= 1). and an outcome Y TEYCR. R. The potential outcome Y associated with a treatment is denoted as Y ((). We aim to estimate the conditional average treatment effect (CATE): 7 = ([ [X] CATE estimation with observational data requires standard assumptions (see Ap-
For example, if both potential outcomes are overestimated by the same amount, the j--risk would indicate a poor model quality while the resulting CATE estimates would still be accurate.
Building an ML pipeline for CATE estimation presents sig- nificant challenges, related to the absence of ground truth
2

---

AutOCATE: End-to-End, Automated Treatment Effect Estimation
Treatment Effect Estimation - Core Functionalities
Evaluation
Estimation
>
Ensembling
KNN
DR
S
T
DR
X
R
Top
Top
Z
IF
RA
Lo
Z
U
F
Pareto
Stacking
Machine Learning Pipelines - Building Blocks
Preprocessing
Baselearners
Support Vector
k-Nearest Neighbours
Linear Regression
Machine
Feature
Feature Transformation
Scaling
Selection
Neural Network
Extra Trees
Decision Tree
Gradlent Boosting
Random Forest
Gausslan Process
Figure AutOCATE builds a pipeline in three stages. (1) Evaluation-leaming the appropriate risk measure(s), (2)
Estimation-tuning CATE estimation pipeline, and (3) Ensembling sereebbiinn-sllettnnng final model constructing ensemble. We build ML pipelines for evaluation and estimation based on a collection of preprocessing algorithms and ML baselearners.
pendix A.2). More background is provided in Appendix A. Goals and challenges. Given observational data Dirain- we aim to find the optimal ML pipeline for CATE estimation. This is counterfactual Combined Algorithm Selection and Hyperpurameter (CASH) optimization, involving search pipelines oд with algorithms E AA and hyperparam- eters h E Ha to minimize the error L on test data Drest:
(3) ENSEMBLING: Finally, we use the first stage's proxy risk to select and combine estimation pipelines from the second stage. The result can be a single pipeline or an ensemble. Figure shows a high-level overview of AutOCATE's functionalities per stage and the underlying building blocks.
4.1. Stage 1: Evesgnnionn-esigningg a Proxy Risk and Evaluation Protocol
arg min [(an) D.ea)). a,
(1)
The counterfactual CASH problem requires minimizing [(en) which involves two challenges: the lack of ground truth and the presence of covariate shift due confounding. To tackle these, we measure risk based validation data's predicted pseudo-labels-ice. proxies for
An algorithm can be an MI method tailored for CATE esti- mation or metalearner with one more baselearners. The counterfactual CASH problem involves unique challenges. pipeline's quality of on the train data L(a,, Duain) is unobserved, as there is no ground truth CATE. In addition, there is covariate shift between the observational training data and test data due confounding. Both points present challenges for both building and validating ML pipeline.
Risk measures. AutOCATE includes different possible risk measures, described Appendix B.2. We include pseudo- used in metalearners (DR-, R-, Z-, U-, and F), plug-in risks (7 and NN), and a risk approximation with influence functions (IF). We exclude the μe and μлpw- risks as they do not apply to all metalearners, and the S-risk due to poor results in prior work (e.g.. Mahajan et al., 2023). As constructing these risk measures requires estimating nuisance parameters, we search over preprocessing and ML algorithms to find good-performing ML pipelines.
4. AutoCATE: Estimation
Automated CATE
AutoCATE automatically finds the optimal ML pipeline in three stages: evaluation, estimation, and ensembling. (1) EVALUATION: In the first stage, construct proxy risk for L based on risk measure (e.g., R-risk) and evalua- tion metric (e.g., MSE). To accurately estimate this risk on the validation data, we perform an automated search over preprocessors, ML algorithms, and their hyperparameters.
There no ground truth and different measures may be preferable depending on the (unknown) data generating pro- To make evaluation more robust, allow for com- bining different measures. Similarly, as pseudo-outcomes are predictions, there is no "true" version, enabling us to construct multiple version of a single risk (e.g.. two R-risks). Using multiple risk measures results in a multi-objective search problem. To account for the varying scales of differ-
(2) ESTIMATION: The second stage searches over combina- tions of preprocessors, metalearners, baselearners, and their hyperparameters to obtain pipelines for CATE estimation.
3

---

AutOCATE: End-to-End, Automated Treatment Effect Estimation
risks, we normalize them by comparing each model's performance to average treatment effect (ATE) baseline.
in general. For good general performance, we can select the pipeline (or the top five) with the lowest average risk across objectives, Similarly, we can select based on each pipeline's Euclidean distance to the origin, or its average rank across objectives. Finally, we can apply the abovemen- tioned stacking procedure for each risk measure separately and averaging the weights a final stacked pipeline.
Metrics and implementation. Different metrics can com- pare the pseudo-labels and CATE predictions to evaluate their quality. We include general metrics of predictive accu- racy, like the mean squared error (MSE) or mean absolute percentage error (MAPE), and metries related to a down- stream application, like the Area Under the Qini Curve (AUQC) for ranking effects (Vanderschueren et al., 2024). The R-risk requires a metric that accommodates weights. Finally, we include a stratified training-validation stratified k-fold crossvvaliattion procedure. Figure 7 shows more information on the evaluation frameworks.
4.4. ML Pipeline Building Blocks
We construct ML pipelines in the evaluation and estima- tion stages. Pipelines consist of preprocessors and ML algorithms. built on top of scikit-learn (Pedregosa et al., 2011). For preprocessing, we provide different fea- ture selection and scaling algorithms. As baselearners we include different ML algorithms with classification and re- gression counterparts, ranging from linear regression to random forests. Appendix B.3 provides more information.
validation split and a
4.2. Stage 2: Essimation-Buildiii a CATE Estimation Pipeline
Different metaleamness can estimate the CATE. Metalearn- ers are general frameworks for using ML algorithms to estimate treatment effects. They are versatile, accommodate various ML algorithms, and can be efficiently trained us- ing existing ML packages. Common examples include the S-Learner (single model with the treatment as a feature), Lo-Learner (single model with treatment interaction terms), and T-Learner (separate models for each treatment group). Other metalearners use pseudo-outcomes that converge to the treatment effect, such as the DR-, X-, Rt, RA-, Z-, U-, and F F-Learners. Appendix B.1 provides more detailed information on each metalearner. Where available, we use the Causa MMI implementations (Chen et al., 2020).
The final search space includes a variety of preproces- sors, metalearners, baseleamners, and their hyperparameters. While efficient optimization strategies such as Bayesian ap- proaches could be used, we use random search throughout this work to focus on other design choices in Aut OCATE. Nevertheless, as the search is implemented with opt una (Akiba et al., 2019), we could use a range of optimizers.
4.5. AutoCATE Provides Low-Code CATE Estimation
AutoCATE is implemented in Python?: following scikit-learn's design principles (Pedregosa et al., 2011). The low-code API enables automated CATE es- timation with only four lines of code, as shown below:
4.3. Stage 3: Ensembling -eseemiing-Seleettnngg. and Ensembling Estimation Pipelines
from sre src. Aut.ATE import AutoCATE
The final ensembling stage evaluates the pipelines from the estimation stage with risk measures from the evaluation stage and select the best pipeline(s) for prediction. No estab- lished methods exist for ensembling CATE estimators and, due to the lack of ground truth, most standard ensembling methods are not applicable. We can select the best pipeline, or the best five for improved robustness and accuracy. We also include a novel stacking procedure that assigns weights (between zero and one) to each pipeline and optimizes these to minimize the squared error with the pseudo-outcomes. The weights are regularized, with tuning on a holdout set. Finally, we include stacking with softmax weights (Mahajan al., 2023)-to the best of our knowledge, this is the only ex- isting ensemble method for CATE estimation. Appendix B.5 provides more details each ensembling approach.
3 autocate AutOCATE ()
4 autocate. fit (X_train, t_train, yf_train)
s cate_pred = cetooctteprcdiie (X_test)
Different initialization arguments can be specified (e.g., the number of estimation trials; see Appendix B.6).
5. Empirical Results
This section analyzes AutoCATE'S design choices per stage: evaluation (5.2), estimation (5.3), and ensembling (5.4). We identify best practices and benchmark the result- ing configuration against common alternatives (5.5).
5.1. Experimental Setup: Data and Evaluation Metries
Our experiments compare various automated, end-to-end strategies for learning a CATE estimation pipeline. Using AutOCATE, we can evaluate a range of design choices. To
With multiple risk measures in a multi-objective search, model selection is more complex as there may not be one optimal pipeline, but rather a Pareto frontier. One strategy to select all Pareto optimal pipelines, though pipelines performing well on only one measure may not work well
"The software package and accompanying experimental code are publicly online at https://gthub..omm///ee hwtp:////w/.oooooooo///oCCCCCC.
4

---

AutOCATE: End-to-End, Automated Treatment Effeet Estimation
DR
F
IF
kNN
R
T
U
Z
IHDP
2.12 34
3.33 5s
3.13
2.22= 36
3.
2.15
3.58
5.40
ACIC
1.56
1.74 10
2.52=
1.74= 10
1.63
1.52
1.721.00 .0
2.40
Twins
.333+.00
.340+.00
.340+.o
.323-
.335+.00
.323
.359
.350+.01
News
2.42
2.48
2.73
2.43=
2.51
2.422.07
2.601.oa
3.02
(a) Comparing downstream performance for different risk measures
Combining risks
-risk Mull-plee versions Top 3
Best single
All
DR,T
DR,NNNNN
Top
Top
Top 5
IHDP
2,48+ +.36
2.19+
2.13+.a5
2.15
2.15+
2.17+.35
2.11 +.30
2.12+.34
ACIC
1.94 13
1.58
1.60
1.52
1.54 os
1.55 ooa
1.52 1.08
1.52 1.22.mm .o
Twins
.323 00
.324= 00
.323 00
0.323 +.00
+323+.00 .00
2.00
+323..00 i.o
News
2.52 07
2.41 06
2.41 07
2.42 07
2.41 o7
2.43 07
2.43 .7
2.42 .77
(b) Comparing downstream performance for different combinations of risk measures
Table 1: Comparing risk measures for model selection. Results in <PEHE (1). Bold highlights the best, underlined values fall within standard error. Results for 50 evaluation and 50 estimation trials with a T'LLarner and gradient boosting.
IHDP
ACIC
Twins
News
2:25
/PEHE
2.50
PEHE
PEEIE
0.328
/PEHE
2.00
2:25
0.528
2.46
1.75
0.324
2.40
1.75
0.322
2:35
10
2001
500
10
100
500
100
500
10
200
500
Evaluation trials
Evaluation trials
Evaluation trials
Evaluation trials
Figure 2: The importance of tuning validation models. We analyze the impact of tuning the models underlying the
evaluation more extensively, Results for a -risk and 50 estimation trials with a T T-Learner and gradient boosting.
obtain general insights, we leverage a collection of stan- dard benchmarks for CATE estimation: IHDP (Hill, 2011), ACIC (Dorie al., 2019), News (Johansson et al., 2016), and Twins (Louizos et al., 2017); Appendix C for details. These semi-synthetic benchmarks include 247 distinct data sets that vary in outcome (regression and classification), di- mensionality, size, and application area, allowing for com- prehensive analysis AutOCATE. Unless noted otherwise, results are reported precision in estimating heterogeneous treatment effects (PEHE): v PPEHE- v(r-
5.2.1. How TO MEASURE CATE PREDICTION QUALITY?
What risk measure works best? We compare performance of different risk measures for model selection in Table la. Three options consistently perform well: the DR-, kNN- and T-risk. These results largely correspond with exist- ing work. Curth & van der Schaar (2023); Mahajan et al. (2023) similarly found the DR-risk to work well, though the kNN-risk works comparatively better in our experiments. Although Curth & van der Schaar (2023) reported worse results for the T-risk, both our and Mahajan et al. (2023)'s findings show that it can give good results with proper tun- ing of the underlying models. We analyze the impact of tuning in Figure 2: tuning the evaluation models more in- deed results in better downstream performance. We test whether congeniality bias (Curth & van der Schaar, 2023) affects our results by repeating this experiment for different metalearners in Table 6, but find similar results.
For each experimental result, the caption describes the AutoCATE configuration that was used. For the evalua- tion and estimation stages, we describe the search strategy for automatically find the ML pipeline(s), including the base- and metalearners involved and the number of optimization trials per stage. Unless stated otherwise, AutoCATE, selects the best ML pipeline based on best average performance.
Is beneficial to use multiple risk measures? We can combine different risk measures in multi-objective search, leading to possibly more robust model selection, each risk offers a different proxy to the same ground truth. Table 1b shows both results for risk measure combinations and for multiple versions of a single measure based on different
5.2. Stage 1: Evaluation Design Choices
We examine design choices for each stage, while keeping the other stages fixed. For the evaluation stage. we compare risk measures, metrics, and evaluation procedures to analyze the impact model selection and downstream performance.
5

---

AutOCATE: End-to-End, Automated Treatment Effect Estimation
IHDP
ACIC
Twins
News
2.0
0.36
VPEHH
2.5
PEHE
VPEEE
1.8
0.31
2.0
1.6
2.4
0.32
0.1
0.3
0.5
0.7
0.9
12.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
Holdout ratio
Holdout ratio
Holdout ratio
Holdout ratio
Figure 3: How much data to use for evaluation? Results for varying holdout ratios, with fitted polynomial to gain insight
into the optimal ratio. Evaluation with a T-riik and 50 trials; estimation with 50 trials, a Teameer and gradient boosting.
IHDP
ACIC
109
10
PPHH
V
10a
10
10
10
20
50
100
200
10
20
50
100
200
Estimation trials
Estimation trials
Twins
News
VPEEH
10.0
AllBaae Best Meta- AllBase
VEHE
104
AllMeta BestBase ..... Mest Base
10a
10
50
100
200
2
10
20
50
100
200
Estimation trials
Estimation trials
Figure 4: What meta- and baselearners to include? We compare different search spaces for AutoCATE, either including all metalearners (AllMeta) only the best (BestMeta), as well as all baselearners (AllBase) or only the best (BestBase). Results for 50 evaluation trials with a -risk.
estimates. Combining different types or different versions of risk measures can indeed improve performance, but no strategy consistently improves upon the best single measure.
percentage error (MAPE) and area under the Qini curve (AUQC). As hypothesized, selecting models using a partic- ular metric generally improves performance for that metric.
5.2.2. WHAT EVALUATION PROCEDURE TO USE?
5.3. Stage 2: Estimation Design Choices
How to set the holdout ratio? As risk measures require learning estimates on validation data, there is a trade-off between using data for evaluation or estimation. Figure 3 presents results for different holdout ratios, illustrating this trade-off and showing that holdout ratio of 30-50% gener- ally works well. We use 30% in the rest of this work. Al- though more folds in cross- cross-validttion often improve model performance in supervised settings, we do not observe this effect for AutOCATE (see Table 5), likely due to the inter- action between the number of folds and the holdout ratio.
Given an evaluation protocol, we compare choices in the es- timation stage. We look at the impact of including different metaleamers and baselearners in AutoCATE's search.
Metalearners. Figure 4 compares different versions of AutOCATE with either all meta- and baselearners (see Fig- ure 1 for an overview), or only the selected best per cate- gory. The complete "AllMeta-AlIBe poorly. Performance generally improves with more trials, but poor results persist even after 100 trials on the News data. Further inspection reveals that bad iterations are due to instability of the R- and U U--Lerrers: while these perform well on the validation set, they can perform exceptionally poor on the test data after retraining on all data. Other metalearners (F and Z) are almost never chosen. There- fore, "BestMeta" excludes these metalearners (R, F, Z, and U) resulting in improved stability and performance. Appendix D.2 compares metalearners' precision and time
sometimes performs
What evaluation metric to use? All previous experiments used the mean squared error (MSE) to compare CATE pre- dictions and pseudo-outcome(s), corresponding the goal of minimizing PEHE. However, depending the down- stream application, alternative objectives might be more important. AutoCATe provides several metrics. Table 7 shows results for evaluating based on the mean absolute
6

---

AutOCATE: End-to-End, Automated Treatment Effect Estimation
Best model(s)
Stacking
Top
Top
COP
Softmax
IHDP
2.15
1.90
1.964.34
2.
ACIC
1.52+.00
1.34+.os
1.42+.09
1.33+.00
Twins
.323+.00
.325=.00
+344
.331+.a0
News
2.42
2.33=.06
2.33
2.32 1.22:.00
(a) Comparing ensemble strategics for single T-risk
Average
Distance
Ranking
Stacking
Top
Top 5
Top I
Top 5
Top
Top 5
COP
Softmax
Pareto
IHDP
2.19+.35
1.84
2.27+.37
2.99+.54
3.58+.00
2.99+.64
1.94+.a2
2.83+.61
2.19-.a0
ACIC
1.58
1.35
1.55
1.41
1.692.0x
1.41 2.0
1.43 09
1.33 09
1.50= os
Twins
.323..o0 o0
.325
..0
.341
.367..01 01
.341
.349 n0
.331
.326= 00
News
2.41+.00
2.32+.oa
2.42+.07
2.38+.or
2.58+.08
2.38+.07
2.34+.00
2.32
2.39+.or
(b) Comparing ensemble strategies when combining DR- and T'-risks
Table 2: Ensemble strategies. We compare ensembling strategies for single or multiple objectives in terms of v <PEEE, Bold highlights the best results, underlined values lie within 1 standard error. Results for 50 evaluation trials and 50 estimation trials with a T-Learner and gradient boosting.
efficiency, and shows how often metalearners are chosen.
5.5. Benchmarking AutoCATE
Baselearners. The "BestBase" versions in Figure only use baselearners that typically perform well with tabular data (random forests, extremely randomized trees, gradient boosting, and multilayer perceptrons), for both evaluation and estimation stages. Choosing the best baselearners im- proves performance, but less so than metalearner selection.
This section compares the optimized configuration of AutOCATE with common alternative approaches for tuning CATE estimation pipelines. The benchmarks select the best model using the error in predicting observed outcomes (μ- risk). We include both S- and T-Leamnrs. For T T-Learners, we tune models separately for the control and treatment groups. First, we compare a T-Learner with gradient boost- ing tuned based on the ju-risk against AutoCATE oCATE using only a T-Leamer and gradient boosting optimized for T-risk. While these strategies are similar, AUtOCATE evaluates the entire pipeline jointly and (potentially) adds preprocessing. Conversely, the traditional T-Learner's s search is more ef- ficient it tunes models separately per group. Figure 5 compares the two approaches: the j--risk strategy performs worse for Twins, but better for ACIC. Finally, Figure 6 com- pares AutOCATE with S- and T-Learners using random forests and gradient boosting. These approaches are concep- tually simple, but represent common and proven baselines. We observe that AutoCATE can obtain at least competitive performance to the best approach for each data set. These re- sults are due two factors. First, AutoCATE offers greater flexibility through a larger search space, including more meta- and baselearners and preprocessing (Table 10 illus- trates the value preprocessing). Second, model selection is better aligned with the goal of CATE estimation, using the T-ris,, and can include an ensemble of pipelines im- prove performance. Appendix D.4 shows more results on ranking treatment effects (i.e. uplift modeling) and validates AutoCATE's robustness to confounding with synthetic data.
5.4. Stage 3: Ensembling Design Choices
The ensemble stage selects CATE estimation pipelines using the risk(s) from the evaluation stage. Selected pipelines are re-trained on all training data and saved for inference.
Single objective. With one objective, we can select the best pipeline (Top 1), the best five (Top 5), or apply stacking to combine all pipelines in ensemble. Table 2a compares these strategies, showing that combining pipelines improves performance for all data except Twins. An ensemble also enables assessing predictive uncertainty, see Appendix D.3.
Multiple objectives. Model selection more complex with multiple objectives. We can select the top or top five pipelines based on the average risk, Euclidean distance the origin, or average rank. Alternatively, we can create stack- ing estimators for each objective and average their weights ("Stacking")), "), or select all Pareto optimal models ("Pareto""), Table 2b compares these strategies. Single pipelines typi- cally perform worse than the top five pipelines, the Pareto ensemble, or stacking. Selection using average risk per- forms well generally, but no strategy consistently optimal.
7

---

AutOCATE: End-to-End, Automated Treatment Effect Estimation
IHDP
ACIC
Twins
News
/PEHE
1.6
Auto
4
0.34
2.6
p-risk
1.4
0.30
2.4
1 20 50 100
200
20
50
100
200
20
200
20
50
100
200
Estimation trials
Estimation trials
Estimation trials
Estimation trials
Figure 5: Comparing AutOCATE with tuning based on j-risk. We compare tuning T-Leamnr with gradient boosting
using either AutOCATE (based T T-risk) or tuning based on the MSE the observed outcome. AutOCATE uses T-risk with 50 evaluation trials and top model selection.
ACIC
IHDP
S-RF
S-GB
AutoCATE-J 1 AutoCATE-5
2.5
T-RF
T.GB
2.0
1.5
10
20
100
200
500
1000
10
20
50
100
200
500
1000
Estimation trials
Estimation trials
Twins
News
3.00
0.34
2.75
0.33
2.50
0.32
2.25
5
100
200
500
1000
100
200
Estimation trials
Estimation trials
Figure 6: Benchmarking AutOCATE. We compare AutOCATE with common benchmarks using S- and L'Leearnrs with random forests and gradient boosting. AuTo uses a T-risk with 50 evaluation trials and BestMeta-BBstBaaae search spaces, with either Top 1 or Top 5 model selection.
6. Conclusion
While AutoCATE is an effective and versatile tool, no single method excels in all scenarios. It may be less suitable when data and compute are limited, when heavy customization or preprocessing are needed, and in settings violating our causal assumptions (e.g., instrumental variables). In such cases, alternative or tailored approaches may be preferable.
Despite the advances in ML for CATE estimation, adoption remains limited, due to the complexity of implementing, tuning, and validating them. We framed the problem of find- counterfuctual ing an MI. pipeline for CATE estimation CASH problem and proposed Aut oCAT: the first end-to- end, automated solution tailored to CATE estimation. Based on this solution, we analyzed design choices for evaluation, estimation, and ensembling, and identified best practices.
Aut OCATE enables a comprehensive analysis of existing methods (see Figure 14 and Appendix D.5), facilitating a better understanding of CATE estimation and guiding the development of new approaches. We envision opportunities for future research in all stages. For evaluation, advanced multi-objectinn strategies could improve performance and robustness. Novel methods for estimation could be automat- ically discovered using Neural Architecture Search. Gen- erally, efficiency can be improved with better search algo- rithms or strategies (c.g.. by re-using nuisance models across metalearner)). Related to this, the optimal time allocation between the stages remains an open question, where meta- learning could help by incorporating data set characteristics (Feurer et al., 2015). Finally, more advanced ensembling could be developed (c.g., combining different metaleamers).
The resulting configuration was validated empirically and outperformed widely used strategies for CATE estimation.
To maximize AutoCATE's practical impact, several limita- tions need to be addressed. Although AutoCATE relies standard assumptions for causal inference, it is crucial to assess its robustness against violations of these assumptions and to develop protocols for such cases. Additionally, most of the data used in this work is semi-synthetic (IHDP, ACIC, and News), which may not fully capture the complexities of real-world data. Although validating CATE estimates remains inherently challenging, approaches from related fields could offer inspiration (see e.g. Devriendt et al., 2020).
8

---

AutOCATE: End-to-End, Automated Treatment Effect Estimation
Acknowledgements
Chen, H., Harinen, T., Lee, J.-Y., Yung, M., and Zhao, Z. Causalml: Python package for causal machine learning. arXiv preprint arXiv: 1163..11..1,,,,, 2020.
We would like to thank Alicia Curth, Julianna Piskorz and Daan Caljon for their insightful input and feedback on ear- lier drafts of this paper. We also thank the anonymous reviewers for their helpful comments and suggestions. Toon Vanderschueren was supported by FWO PhD Fellowship 1117322N. The computational resources and services used in this work were provided by the VSC (Flemish Supercom- puter Center), funded by the Research Foundation Flanders (FWO) and the Flemish Government department EWI.
Chen, L.-C., Collins, M., Zhu, Y., Papandreou, G., Zoph. В., Schroff, F., Adam, H., and Shlens, Searching for efficient multi-scale architectures for dense image prediction. Advances in neural information processing systems, 31, 2018.
Curth, A. and van der Schaar, M. Nonparametric estimation of heterogeneous treatment effects: From theory learn- ing algorithms. In International Conference Artificial Intelligence and Statistics, pp. 1810-1818. PMLR, 2021.
Impact Statement
Facilitating the adoption of causal inference methods may heighten the risk of misuse in high-stakes applications. Transparency, interpretability, and fairness remain crucial, and we hope our work inspires more research in these areas.
Curth, A. and van der Schaar, M. In search of insights, not magic bullets: Towards demystification of the model se- lection dilemma in heterogeneous treatment effect estima- tion. In International Conference on Machine Learning, PMLR, 2023.
pp.
References
Devriendt, F,, Moldovan, D., and Verbeke, W. A literature survey and experimental evaluation of the state-of-the-are in uplift modeling: A stepping stone toward the devel- opment of prescriptive analytics. Big data, 6(1):13-4) 2018.
Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. Optuna: A next- gextt-eneratton hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery & data mining, pp. 2623-2631, 2019.
Devriendt, F., Van Belle, J., Guns, T., and Verbeke, W. Learning to rank for uplift modeling. IEEE Transactions Knowledge and Data Engineering, 34(10))4888-49) 2020.
Alaa, A. and van der Schaar, M. Autoprognosis: Automated clinical prognostic modeling via bayesian optimization with structured kernel learning. In International confer- ence on machine learning, pp. 139-148. PMLR, 2018.
Dorie, V., Hill, J., Shalit, U., Scott, M., and Cervone, D. Automated versus do-it-yoursell methods for causal infer- ence: Lessons learned from data analysis competition. Statistical science, 34(1):43-68, 68, 2019.
Alaa, A. and van der Schaar, M. Validating causal infer- ence models via influence functions. In International Conference on Machine Learning, pp. 191-201. PMLR, 2019.
Doutreligne, M. and Varoquaux, G. How select pre- dictive models for causal inference? arXiv preprint arXi::2302.00370 2023.
Alaa, A. M. and van der Schaar, M. Bayesian inference of individualized treatment effects using multi-task gaussian processes. Advances in neural information processing systems, 30, 2017.
Elsken, T., Metzen, J. H., and Hutter, F. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21, 2019.
Athey, S. and Imbens, G. W. Machine learning methods for estimating heterogeneous causal effects. stat, 1050(5): 1-26, 2015.
Erickson, N., Mueller, J., Shirkov, A., Zhang, H., Larroy, P., Li, M., and Smola, A. chrogluon-tabulaan Robust and accurate automl for structured data. arXiv preprint arXiv:2003.0653 2020.
Bergstra, J., Bardenet, R., Bengio, Y., and Kégl, B. Algo- rithms for hyper-parameter optimization. Advances in neural information processing systems, 24, 2011.
Feuerriegel, S., Frauen, D., Melnychuk, V., Schweisthal, J., Hess, K., Curth, A., Bauer, S., Kilbertus, N., Kohane, I. S., and van der Schaar, M. Causal machine learning for predicting treatment outcomes. Nature Medicine, 30(4): 958 958-968, 2024.
Berrevoets, J., Imrie, F., Kyono, Jordon, J., and van der Schaar, M. To impute or not to impute? missing data in treatment effect estimation. In International Conference on Artificial Intelligence and Statistics, pp. PMLR, 2023.
Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., and Hutter, Efficient and robust automated

---

AutOCATE: End-to-End, Automated Treatment Effect Estimation
machine learning. Advances in neural information pro- cessing systems, 28, 2015.
Karmaker, S. K., Hassan, M. M., Smith, M. J., Xu, L., Zhai C., and Veeramachaneni, K. Automl to date and beyond: Challenges and opportunities. ACM Computing Surveys (CSUR), 54(8):1-36, 2021.
Franks, A., D' Amour, A., and Feller, A. Flexible sensitiv- ity analysis for observational studies without observable implications. Journal of the American Statistical Associ- ation, 2020.
Kennedy, E. H. Towards optimal doubly robust estimation heterogeneous causal effects. Electronic Journal of Statistics, 17(2):3008 3049,, 2023.
Geffner, T., Antoran, J., Foster, A., Gong, W., Ma, C., Kici- man, E., Sharma, A., Lamb, A., Kukla, M., Pawlowski, N., et al. Deep end-to-end causal inference. arXiv preprint arXiv:2202.02199. 2022.
Künzel, S. R.,, Sekhon, S., Bickel, P. J., and Yu, B. Met- alearners for estimating heterogencous treatment effects using machine learning. Proceedings of the national
academy of sciences, 1 166110)4156-4165, 2019.
He, X., Zhao, K., and Chu, X. Automl: A survey of the state-of-te ttateoff-hhe-ar.. Knowledge-ber based systems, 212:106622, 2021.
Larsen, K. Information: Data exploration with informa- tion theory methods, 2023. URL https: /rran. r --project oerectoogge org/ wwwgppacagg--aaoo Information. R
Hill, J. L. Bayesian nonparametric modeling for causal infer- ence. Journal of Computational and Graphical Statistics, 20(1):217224,, 2011.
package version 0.2.1, Accessed: 2024-09-26.
LeDell, E. and Poirier, S. H2o automl: Scalable automatic machine learning. In Proceedings of the AutoML Work- shop at ICMI,, volume 2020. ICML San Diego, CA, USA, 2020.
Hillstrom, K.
Minethatdata: E-mail analytics and URL https:
data-driven marketing, 2008.
//blog.mineee www.mtattataaaaaammmaaaaaaaaaaaa cwm/ 2008/300/3/
minethatdata- eesee---mmat
html. Accessed: 2024-09-26.
Lo, V. S. The true lift model: a novel data mining ap- proach response modeling in database marketing. ACM SIGKDD Explorations Newsletter, 4(2):78-88 86, 2002.
Holland, P. W. Causal inference, path analysis and recursive structural equations models. ETS Research Report Series, 1988(1); ::0-00, 1988.
Louizos, C., Shalit, U., Mooij. J. M., Sontag, D., Zemel, R., and Welling, M. Causal effect inference with deep varialle models. Advances in neural information processing systems, 30, 2017.
Horvitz, D. G. and Thompson, D. J. A generalization sam- pling without replacement from a finite universe. Journal of the American statistical Association, 47(260);:63-685, 1952.
Mahajan, D., Mitliagkas, I., Neal, B., and Syrgkanis, Empirical analysis of model selection for heterogeneous causal effect estimation. In The Twelfth International Conference on Learning Representations, 2023.
Jarrett, D.. Yoon, J., Bica, I., Qian, Z., Ercole, A., and van der Schaar, M. Clairvoyance: A pipeline toolkit for medical time series. In International Conference Learning Representations, 2021.
Nie, X. and Wager, S. Quasi-oracle estimation of hetero- geneous treatment effects. Biometrika, 108(():299-319, 2021.
Jaskowski, M. and Jaroszewicz, S. Uplift modeling for clinical trial data. In ICML workshop clinical data analysis, volume 46, pp. 79-95. 2012.
Oberst, M., Johansson, F., Wei, D., Gao, T., Brat, G., Son- tag, D., and Varshney, K. Characterization of overlap in observational studies. In International Conference on Artificial Intelligence and Statistics, pp. 788-798. PMLR, 2020.
Jesson, A., Mindermann, S., Shalit, U., and Gal, Y. Identify- ing causal-effect inference failure with uncertainty-awant models. Advances in Neural Information Processing Sys- tems, 33:1 1637-11649, 2020.
Jesson, A., Mindermann, S., Gal, Y., and Shalit, U. Quanti- fying ignorance in individual-leved causal-effee estimates under hidden confounding. In International Conference PMLR, 2021.
Olaya, D., Vásquez, J., Maldonado, S., Miranda, J., and Verbeke, W. Uplift modeling for preventing student dropout in higher education. Decision support systems, 134:113320, 2020.
on Machine Learning, pp.
Johansson, F., Shalit, U., and Sontag, D. Learning repre- sentations for counterfactual inference. In International conference on machine learning, pp. 3020-3029. PMLR, 2016.
Oprescu, M., Syrgkanis, V., and Wu, Z. S. Orthogonal random forest for causal inference. In International Con- ference on Machine Learning. pp. 4932 4941. PMLR, 2019.
10

---

AutOCATE; End-to-End, Automated Treatment Effect Estimation
Oprescu, M., Dorn, Ghoummaid, M., Jesson, A., Kallus, N., and Shalit, U. B-learner: Quasi-oracle bounds heterogeneous causal effects under hidden confounding. In International Conference on Machine Learning, pp. 26599-26618, PMLR, 2023.
So, D., Le, Q. and Liang, C. The evolved transformer. In International conference on machine learning, pp. 5877- 5886. PMLR, 2019.
Teinemaa, I., Albert, J., and Pham, N. UpliftML:: A Python Package for Scalable Uplift Modeling. 2021. Version
Pedregosa, F,, Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2888 2830, 2011.
0.0.1.
Vanderschueren, T.,, Boute, R., Verdonck, T., Baesens, B., and Verbeke, W. Optimizing the preventive maintenance frequency with causal machine learning. International Journal of Production Economics, 258: 1087:8, 2023.
Powers, S., Qian, J., Jung, K., Schuler, A., Shah, N. H., Hastie, T., and Tibshirani, R. Some methods for hetero- geneous treatment effect estimation in high dimensions. Statistics in medicine, 37(1 176777777771 2018.
Vanderschueren, T., Verbeke, W., Moraes, F., and Proença, H. M. Metalcarners for ranking treatment effects. arXiv preprint arXiv:2405.0211 2024.
Robinson, P. M. Root-n-consistent semiparametric regres- sion. Econometrica: Journal of the Econometric Society, pp. 931-954, 1988.
Wager, S. and Athey, S. Estimation and inference of hetero- geneous treatment effects using random forests. Journal of the American Statistical Association, 113(523): 1228 1242, 2018.
Rolling, C. A. and Yang. Model selection for estimating treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76(4):749-70) 769, 2014.
Wang. C., Wu, Q., Weimer, M., and Zhu, E. Flaml: A fast and lightweight automl library. Proceedings of Machine Learning and Systems, 3:434 447, 2021.
Rubin, D. B. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of edu- cational Psychology, 66(5):688, 1974.
Yoon, J., Jordon, J., and van der Schaar, M. Ganite: Estima- tion of individualized treatment effects using generative adversarial nets. In International conference on learning representations, 2018
Rubin, D. B. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469):322-333 2005.
Zhang, W., Li, J., and Liu, L. A unified survey of treat- ment effect heterogeneity modelling and uplift modelling. ACM Computing Surveys (CSUR), 54(8):1-36, 2021.
Runge, F., Stoll, D., Falkner, S., and Hutter, F. Learning to design ma. In International Conference Learning Representations, 2019.
Zhao, Z., Zhang, Y., Harinen, T., and Yung, M. Feature selection methods for uplift modeling and heterogeneous treatment effect. In IFIP International Conference on Artificial Intellgencee Applications and Innovations, pp. 217-230. Springer, 2022.
Rzepakowski, P. and Jaroszewicz, S. Decision trees for up- lift modeling with single and multiple treatments. Knowl- edge and Information Systems, 32:303-327, 2012.
Schuler, A., Baiocchi, M., Tibshirani, R., and Shah, N. A comparison of methods for model selection when timating individual treatment effects. arXiv preprint arXiv:1804.05146, 2018.
Shalit, U., Johansson, F. D., and Sontag, D. Estimating individual treatment effect: generalization bounds and algorithms. In International conference on machine learn- ing, pp. 3076-3085. PMLR, 2017.
Sharma, A. and Kiciman, E. Dowhy: An end-to-end library for causal inference. arXiv preprint 1.X.v:2011.04216 2020.
1.04216,
Snoek, J., Larochelle, H., and Adams, R. P. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.

---

AutOCATE: End-to-End, Automated Treatment Effect Estimation
The appendix starts with more detailed introduction and background to CATE estimation in Appendix A. The next sections provide more details AutoCATE (Appendix B), describe the data used in this work (Appendix C), and present additional empirical results (Appendix D). Finally, we compare AutoCATE with other packages for CATE estimation in Appendix E,
A. Background on CATE Estimation
This section provides a more detailed introduction and background on treatment effect estimation. In accordance to the main body, we denote an instance by tuple (2,t,)), with covariates X X Rd, a treatment T 7 {0, 1}, and an outcome Y E C R. Following the potential outcomes framework (Rubin, 1974; 2005), we describe instance's potential outcome Y for a given treatment T' = t as (t). The Conditional Average Treatment Effect (CATE) is then defined as the expected difference in outcomes between treating and not treating:
E Y(1) - Y(O)x
(2)
Knowing this effect is crucial in variety of domains, such as education (Olaya et al., 2020), healthcare (Feuerriegel et al., 2024), and maintenance ( (Vanderschueren et al., 2023). Estimating the CATE from observational data involves significant challenges (Appendix A.1), requires standard assumptions (Appendix A.2), and tailored MI. methods (Appendix A.3). We explain these in the following.
A.1. Challenges: The Fundamental Problem and Confounding
The fundamental problem of causal inference (Holland, 1988) is that, for each instance, we only observe either Y()) or Y (1), depending on what treatment was administered. We refer to the observed outcome as the factual outcome and the unobserved outcome the counterfactual outcome. Because one outcome is always unobserved, we never know the true CATE which means that there is no ground Iruth CATE available for training or validation.
In observational data, the outcome that was observed is typically not random: some instances were more likely to be treated, while other instances were more likely not to receive treatment. For example, in healthcare, patients may be more likely to receive new treatment if they have access to better healthcare, have no pre-existing conditions, and are younger. The covariates that influence both the outcome and treatment assignment are called confounders, with the resulting non-random treatment assignment sometimes referred to as confounding.
Confounding presents an additional challenge for CATE estimation and validation as results in covariate shift. Some instance- traatteette pairs (the counterfactuals) will be absent in the observational training data compared to the hypothetical test data that contains all instance-treatment pairs (both factuals and counterfactuals) Because of this, an ML model may focus too much on the observed data points at the cost of worse predictions for the counterfactuals and, as such, the test data overall.
A.2. Assumptions For Identifiability
Identifying the causal effect from observational data requires making standard assumptions: consistency, overlap, and unconfoundedness.. This section explains these assumptions more detail.
Assumption A.1 (Consistency). The observed outcome given treatment is the potential outcome under that treatment: Y|x,t= Y
Assumption A.2 (Overlap). For each instance, there is a non-zero probability of receiving each treatment given their covariates: V x E X and t E T P(T = t|X = x) >> 0. This condition ensures that there is sufficient variability in the treatment assignment.
Assumption A.3 (Unconfoundedness). Given an instance's covariates, its potential outcomes are independent of the treatment assignment: Y (0), (1) 1 T X. This condition implies that all factors influencing both the treatment assignment and outcome are included in X. In other words, there are no unobserved confounders.
There has recently been much interest in CATE estimation under violation of these assumptions. For example, by quantifying the uncertainty or sensitivity of an estimate to a possible violation (Franks et al., 2020; Jesson et al., 2020; 2021), characterizing overlap violations (Oberst et al., 2020), or developing metalearners that can deal with unobserved confounders (Oprescu al., 2023). We believe that extending AutoCATE to deal with these settings and incorporate
12

---

AutOCATE; End-to-End, Automated Treatment Effect Estimation
these methods will improve its potential for real-world applicability even further. As such, consider it important direction for future versions.
A.3. CATE Estimation: Meta- and Baselearners
We briefly describe the approach of estimating the CATE with metalearner here. A straightforward way of estimating the CATE is using single ML model, where the treatment variable is considered ordinary imput variable. This metaleamer called the S-Learner and can be implemented with wide variety of baselearners (i.c., ML algorithms that predict an outcome based on data, such as decision tree neural network). An alternative metaleamerr the T-learner, fits two models-one model for each treatment group. Both models can use the same baselearner or a different one. More information on the metalearners in OCATE is provided in Appendix B.I. For more extensive overviews, we refer to (Devriendt et al., 2018), (Zhang et al., 2021), and (Feuerriegel et al., 2024).
B. AutoCATE: Additional Information
This section presents information metalearners (Appendix B.1), risk measures for evaluation (Appendix B.2), and AutoCATE's search spaces for preprocessors and baselearners (Appendix B.3).
B.1. Metalearners
We describe the metalearners implemented in AutOCATE in more detail below. We first define the estimates that make up the building blocks of these models: the estimated propensity score é(x) = E(t|z), the treatment-grout specific outcome (x())= = [2, t = 0) and in (x) = E(yla = 1), and the treamenttunawan outcome ((x)) = E(ylar). In the following, the function f describes a model that is learned with a base learner such as a neural network or gradient boosting.
S-Learner. The S-Learner, single learner, simply uses the treatment variable: fs(2, t) E(y| f). The CATE is then estimated as f = 01 go- = fs(z,t = - (z, 0).
Lo-Learner (Lo, 2002). The Lo-Learner is similar an S-Learner, in the sense that it uses the treatment as variable, but it adds interaction terms between the covariates x and treatment t: fio(1, t) TEyyar, t, t). The CATE 7 is then estimated as f
fio(1, t = 1) fto 0).
T-Learner. The T-Learner constructs two models one per treatment group: f2(x)) Elylr, = 0) and f+(x) E(y x, = 1), and predicts the CATE as f 91 9o (z) (x).
X-Learner (Künzel et al., 2019). The X-Learnr first learns two treatment-specific outcome models: (o(x) and 9 (x). It then uses these to impute the counterfactual outcome for each instance and, as such, obtain a pseudo-outcome Fx for the treatment effect: 7 = タ1 -y y if O, and 7 y Do (o()) else. For each treatment group, a model then learned on these
pseudo-outcome: f (z) 7 and (x) = - The final effect model then estimates fx(x) = g(x)f& + (1 g(z))ff
and predicts the treatment effect as 7 fx(x). g(x) E [0, is weighting function, typically the estimated propensity score g(x) ē(x).
RA-Learner (Curth & van der Schaar, 2021). The RA-Learner or regression-adjjuseed learner is similar to an X- Learner, but directly learns the final model on the pseudo-outcomes: fRA(2) E(fx|z), (z), predicting the treatment effect as f [R)(.).
The transformed outcome approach (Jaskowski & Jaroszewicz, 2012; Powers et al., 2018) or inverse propensity weighted estimator (Curth & van der Schaar, 2021) uses pseudo-outcome based on the Horvitz-Thompson transformation 1-t y- The Z-Learner then estimates fz(x)) E(fz (r)) and predicts the
Z-Learner.
(Horvitz & Thompson, 1952): řz [(w)
treatment effect as f fz(x).
U-Learner.
The U-Leamer is based on pseudo-outcome नU y p(x)
The final model fits fo IE(Ful ()) and
predicts the treatment effect (v))..
13

---

AutOCATE: End-to-End, Automated Treatment Effect Estimation
Single Split
Cross-Validation
Data
Data
holdout
holdout
/ number of folds
number of folds
1 number of folds
Training
Validation
Fold
Fold 2
Fold 3
holdout
holdout
Iteration 1
Training
Validation
Training
Validation
halddou ratio holdout
Train
Train validation models
Validate
estimation
validation
Training
Validation
models
models
Train
Train validation models
Validate models
models
Figure 7: Evaluation framework. We show two possible frameworks for validating pipelines based single split cross- cross-validation procedure. For each, the data is split in three groups to (1) train the estimation pipelines, (2) train the validation pipelines, and (3) validate the validation pipelines.
F-Learner (Athey & Imbens, 2015). The F-Learner uses the pseudo-outcome iF ((=)(1-e(x)) The final model fits fr(z) (F)l(z) and predicts the treatment effect as f fr(
t-6(a)
DR-Learner (Kennedy, 2023). The DR-Learner is robust version of the Z-Leamer, based the pseudo-outcome 1-t
Pz 1-t [
æ(σ)) (n(x)
-(x) (o(z). The final model is fDn(x) B(fpn(r)) and predicts
the treatment effect as = (D)
R-Learner (Nie & Wager, 2021). The R-Learner, based on Robinson's 's decomposition (Robinson, 1988), fits a model fa(œ)) using a weighted loss function with pseudo-outcomes ÿn t-A(m) effect can then directly be predicted as f fn(xx.
--((x) and weights = (t =2 The treatment
B.2. Evaluation and Risk Measures
The evaluation framework and data splitting underlying AutOCATE shown in Figure 7. Below, we describe the different types of risk measures included in framework.
Metalearner pseudo-outcomes. An instance's 's true CATE is unknown, but we can use the pseudo-outcomes F used by the T-, Z-, U- F-, DR-, and R-Leamers (see above) as ground truth.
Influence Function (IF) (Alaa & van der Schaar, 2019). The influence function criterion gives an estimate of an ML the treatment effect 7, estimated with T-Learner. This
pipeline's estimation error. It is based on a pseudo-outcome then debiased using the influence function. The final criterion is:
(1 - B) + By(F - f) D 7) +
with D - C é(x) and B - -
d-Nearest Neighbor (kNN) (Rolling & Yang, 2014). The nearest neighbor matching measure finds the nearest neighbor in the opposite group, defined using the Euclidean distance, and uses its outcome as the counterfactual outcome. As such, is essentially T-Leamer pseudo-outcome where the baselearner is restricted to a nearest neighbor model. We extend upon this by allowing alternative versions to constructed by increasing k.
14

---

AutOCATE: End-to-End, Automated Treatment Effect Estimation
Hyperparameter
Range
Hyperparameter
Range
VarianceThreshold
StandardScaler
threshold
[0, (0,0.04]
SelectPercentile
RobustScaler
[5, n.dim]
score.func
mutual info -roooereggession classif]
(b) Feature Scaling
(a) Feature Selection
Table Preprocessor search spaces. We describe the search spaces for the different preprocessors. If hyperparameter not mentioned, its default. All preprocessors implemented with scikit-learn (Pedregosa et al., 201 I); refer to their documentation for more information.
B.3. Preprocessor and Baselearner Search Spaces
Preprocessors. ML pipelines include three (optional) steps preprocess the data before being fed to model: feature selection, transformation, and scaling. For feature selection. include VarianceThreshold, SelectPercentile, no selection. For feature scaling, we include StandardScaler, RobustScaler, or no scaling. Finally, we include feature transformation algorithms our software package (SanineTransfoommr,,, PolynomialFeatures, KBBinDDscretizer), but do not include them in the experiments as they significantly slowed down training times. Other steps for feature selection and scaling from scikit-learn are similarly supported, but not included in the experiments, which is why do not discuss them here. Table provides detailed information on the search spaces.
Baselearners. We present the search spaces for all baselearners' hyperparameters Table 4. These are based largely upon existing AutoML packages (e.g., FLAML (Wang et al., 2021)) and some (limited) experimentation, so these may be improved in future versions.
AutoCATE's resulting search space of ML pipelines for CATE estimation is vast, with 2,187 possible pipelines even without considering hyperparameters:
3 feature selection scaling 27 metaleameer-baselearer configurations 9 baselearners
with 27 1 (S) + 2 4 (DR) +5 (X) + (R) + (RA) + ( +2) + (F), i.e., the sum of all
baselearners required per metalearner.
B.4. Example ML Pipeline
We give an example a pipeline built by AutoCATE, excluding baselearner hyperparameters. Evaluation using a T-Risk evaluation, with control outcomes estimated with gradient boosting and treatment outcomes estimated using a neural network. Estimation by first selecting top percentile of features based the F-value between the label and feature, followed DR-Leamen where propensity scores are estimated with a support vector machine, control outcomes with gradient boosting, treatment outcomes with a linear regression, and the final effect with random forest. This example illustrates the complexity of an ML pipeline for CATE estimation-in this case, there are six different ML models with several hyperparameters each. an ensemble is used for estimation, this complexity increases even more.
B.5. Ensembling and Multi-Objective Model Selection
This section describes the different approaches for ensembling and multi-objective model selection included in our framework.
With multiple objectives, no globally optimal ML pipeline may exist. We explore various strategies for ranking and selecting models in this context. We denote pipeline i's normalized score on objective j as 8ij- As different risk measures and metrics have different scales, we normalize each of these scores by dividing the raw score štg with the raw score of constant ATE baseline SATE:
15

---

AutoCATE: End-to-End tondn,,,,dd Automated Treatment Effect Estimation
Hyperparameter
Range
Hyperparameter
Range
Linear/Logistic Regression
Gradient Boosting
alpha
[c 6, 1c6]
n estimators
[50, 2000]
Gaussian Process
subsample
00.4, 10
min.samples.split
[2. 500]
n.restarts.optimm
[, 5]
learning.rate
00.05, 0.5]
normalize.y
True, False]
n.iter.no.change
55, 100]
alpha
l le2]
max_leaf.nodes
None
max.iter.predict
[100, 1000]
max.depth
None
Support Vector Machine
Random Forest
C
[1e-6, 6, 1c6]
n.estimators
50, 500]
kernel
[linear, poly, rbf, sigmoid] [1,
max_depth
None
degree
min samples split
[2, 100]
k-Nearest Neighbors
ure
00.4, 1.0]
n.neighbors
[1,30]
Extra Trees
weights
uniffrmm, distance]
nesstimators
50, 500]
Neural Network
max_depth
None
hiyden
[1,3]
min.samples.splic
[2, 100]
hiddensneurons
[8, 64]
max features
00.4, 1.0)
alpha
[ 6, lel]
Decision Tree
learning.rate.inie
[5- 4, le 2]
max.depth
[1, 2000]
batch size
[16 64
min.samples..ppit
[2, 500]
activation
tanh, relu
min_samples_lea.
[1, 500]
max.iter
200
max.features
[0.4,
solver
adam
early.stopping
True
Table 4: Baselearner search spaces. We describe the search spaces for each baselearner. If hyperparameter is not mentioned, we use its default. All baselearners are implemented with Miii---eernn (Pedregosa et al., 2011); we refer to their documentation for more information.
16

---

AutOCATE; End-to-End, Automated Treatment Effect Estimation
Average (normalized) score. For each pipeline i, compute the normalized average score objectives:
Sijr S = m 1=1
with m the number of objectives. We then select the pipeline(s) with the best St.
Euclidean distance to the origin. We compute each pipeline i's Euclidean distance to the origin:
Y
D;
Σ
f
m11
with mn the number of objectives. We then select the pipeline(s) with the lowest D..
Average rank. Rank all pipelines for each objective j. denoted as Taj. and compute the average rank:
m R3
Túj
m
j=1
Select the pipeline(s) with the lowest R..
Stacking-Constrained Optimization Problem. To combine multiple pipelines into stacked estimator, we introduce procedure that assigns weights Wij (where < w; S each pipeline i, optimizing these weights to minimize the squared error of the weighted prediction with respect those pscudo-outcomes of objective j. We additionally add an l2 regularization term, which can be tuned on a validation set. With multiple objectives, repeat this for each objective and then average the weights W; Σド=
m
Waj-
(Mahajan et al., 2023). An alternative stacking procedure is to determine the weight of each estimator with softmax function:
wi exp(xs:j)
Σド=1 exp(xs;k)
with x temperature parameter that can be tuned. With multiple objectives, we repeat this for each objective and then average the weights W; > Wiy-
Pareto. We select all pipelines that are Pareto optimal, meaning no other pipeline satisfies:
Sky sig Vj and Skj > sij for least one
B.6. AutoCATE's API: Additional Information
We give more information AutoCATE's initialization arguments in Listing 1.
class AutOCATE:
def init
self, evaluatioonmetriccs Risk measures to evaluate the performance evaluatiion_metziceennont
preprocessors: Preprocessors to try (defaults added later) prepzocessors=Nonee,
base base_learners: Baselearners try (defaults added later)
base_lcarners=None,
10
metalearners: Metalcarners to try (defaults added later)
11
metalearners=:None,
task: Type of task ('regression' or 'classificationn task="regression",
14
metric: Metric used to evaluate the model (e.g., (MSR")
17

---

AutoCATE: End-to-End, Automated Treatment Effect Estimation
metric="KsE",
16
ensenble_stratege: Strategy for selecting a final model
17
single_base_ learnerr Use only one base learner
single_base_
20
jeint_eptiimzaationn Same hyperparameters for baselearners
21
jolnt_opttimizationnee
22
n_folds: Number of folds for cross-validation
n_folds=1,
24
n_trials: How many trials to optimize the estimation pipeline n_trials=50,
25
26
n_eval_version Number of versions of each risk measure
n_eval_versions=
28
i__eall_tiiall Number of trials for evaluating the model n Leeal_ttriass-3
29
30
seed; Random seed for zeproducibility seed=42,
31
visualize: Whether to visualize results visualize=False,
33
34
max_time: time: Maximun time allowed for fitting the model max
35
max nax_time=None,
n_jobs: Number of parallel jobs to run n_jobs=-1,
38
Folds for cross-validated estimates
39
cress_vas wtt__aaa_peedtttttooooeoo
40
hol heldout_ratio: Ratio of data for vali (if single fold)
41
hol holdout_ratio-0.3
42
):
43
44
Initialization code (not included here)
Listing I: Arguments for the AutoCATE class initialization. We describe each argument and its default initialization.
C. Data: Additional Information
This section describes the data used in this work in more detail.
IHDP (Hill, 2011). The data come from the Infant Health and Development Program, describing the impact of child care and home visits on children's cognitive development. Treatments and outcomes were simulated for a total of 100 data sets. Each version contains 747 instances and d 25 covariates.
ACIC (Dorie et al., 2019). The data from the ACIC 2016 competition was based on data from the Collaborative Perinatal Project, studying drivers of developmental disorders in pregnant women and their children. 77 distinct data sets were created, each with n = 4,802 instances and 58 covariates. 100 iterations were originally created for each data set, but we use only the first one for each.
Twins (Louizos et al., 2017). The Twins data studies the effect of being the heavier twin mortaility. 11,984 pairs of twins are included, with d = 46 features each. Only one version of this data set exists, so we run 10 iterations of each experiment.
News (Johansson al., 2016). This data simulates reader's reading experience (y) based on the device they use for reading (!) and the news article (x). There 50 distinct data sets, each with 5,000 instances with and d = 3,477 covariates.
Below, we include results for two data sets on uplift modeling:
Hillstrom (Hillstrom, 2008). This data contains records of customers (n = 64,000) that were contacted by a marketing campaign over e-mail. Originally, customers received either no mail, a mail with men's merchandise, or one with women's
18

---

AutOCATE: End-to-End, Automated Treatment Effect Estimation
merchandise, but convert it contacted (7 0) contacted 0). For each customer, d 10 covariates available. As the outcome y, we consider whether the customer visited the website
Information (Larsen, 2023). The information data set comes from the Information package. It describes customers (n 10,000, d = 68) in the insurance industry, as well as whether they were contacted with marketing campaign and whether they made a purchase.
D. A Additional Results
D.1. Stage 1: Evaluation
Table 5 shows results for evaluating with k-fold cross validation for different values of k.
5
10
IHDP ACIC
2.15 as
2.162.as as
as
2.07 2.53
2. 2.299..4 42
2.25+.41
1.52= 09
1.58 08
1.48 os
1.5
1.50 os
1.53+.on on
Twins
.323 00
.324
.3222.00 oo
.3241.00
.3441.00
.3461.00
News
2.42+.07
2.40+.07
2.41+.06
2.41+.07
2.45+.07
2.45+.07
Table 5: The effect of k in k-fold cross validation. For each data set, we show result for varying number of cross cross- validatio folds. Results for 50 evaluation trials with a T'-iisk and 50 estimation trials with a 7-Learner and gradient boosting.
Risk measures may suffer from congeniality bias, by being predisposed to favor their related metalearners (Curth & van der Schaar, 2023). For example, a T-risk may pick T-Leamer more often, even when suboptimal. The results in our main body found that the T-risk works very well with T-Learner, but these results may not hold in general due congeniality bias. Therefore, again compare the different risk measures when estimating with either S-Learners only selected metalearners in Table 6.
Depending on the downstream application, there may be different objectives for estimating treatment effects. Corresponding to these objectives, different evaluation metrics may be important. Table 7 shows the results of using a different metric for AutoCATE's optimization: as hypothesized, selecting models based on a particular metric results better test time performance for that metric. These findings illustrate the importance of including diversity of metrics in our framework.
D.2. Stage Estimation
Figure 8 shows how often each metalearner gets picked in AutoCATE's BestMeta configuration. The difference in metalearner selection rates illustrates the importance of data-driven metalearner selection, as facilitated by AutoCATE. Interestingly, other metaleamers preferred for binary outcome (Twins) than for continuous outcomes (all others). This finding suggests that different BestMeta configurations may be optimal for different outcomes.
News
IHDP
ACIC
Twins
s SLoX
Lo
DR
RA
S
42
X
20
12
28
12
49
R.A
50
RA
RA
T
T 13
Lo
14
DR
T
X
Figure 8: Metalearner selection. We show how many times metalearner gets picked (in % of all data set iterations) for given data set. Results for oCAT'ssss BestMeta configuration, including the S-, T-, Lo- X-, RA-,, DR-, and U-Learners, with 50 evaluation and 500 estimation trials.
compare different metalearners in terms of PEHE in Table These results show that searching across metalearners
19

---

AutoCATE: End-to-End, Automated Treatment Effeet Estimation
DR
F
IF
kNN
R
T
U
Z
IHDP
3.21 1.5s
3.64 80
4.60
3.11=.a8 53
3.48 bs
3.10 .44
3.62
4.12
ACIC
1.61
1.79
2.07
1.88=
1.73
1.58
1.85
2.164
Twins
.328+.o0
.328+
.347+.02
.320-
.325+.00
.320
.321+.o0
.330+.on
News
2.47
2.51=.08
2.97
2.49
2.76
2.46:.0s
2.78
2.99
(a) Estimation with S-Learner
DR
kNN
IHDP
2.07
3.43
5.75
2.11=
3,45 56
2.17
3.18
4.38
ACIC
1,40+.09
1.87+.11
2.24+.14
1.97
1.57+.10
1.35+.00
1.79+.11
2.16+.11
Twins
.3282.o 00
.327
.384
.324=mm
.3282.00
.3262.00 .0
.344..01
.3481.01
News
2.42
2.60=.os .8
2.95 12
2.42 07
2.75
2.43
2.78
2.77
(b) Estimation with selected metalcarners (BestMeta configuration: S, DR, X, RA, Lo)
Table 6: Performance for validation based on different risk measures. Results in v /PEHE*SE (lower better). Bold highlights the best results, with underlined values falling within standard error. Results for 50 evaluation trials and 50 estimation trials with gradient boosting baselearner.
MSE
MAPE
AUQC
MSE
MAPE
AUQC
VPEHE
2.15+an +0.35
2.28
2.26+.41
/PEHE
1.52+.00
1.67+.09
1,50+.0s
MAРE
1.7621000
1.40 94
0.50
MАPE
1.10
1.03
1:21
AUQC
0.92+0.03
0.88
0.96
AUQC
0.91 1.01
0.90
0.91
(a) IHDP
(b) ACIC
MSE
MAPE
AUQC
MSE
MAPE
AUQC
УPEHE
.323..00
.323
2.44.00
PEHE
2.42
2.52
2.46
MAPE
MAPE
5.75+.14
5.83+
5,86+.85
AUQC
0.002.00
0.00
0.03
AUQC
0.66
0.64
0.65..o1
(c) Twins
News
Table 7: Comparing evaluation metrics. We compare model selection with different evaluation metrics. For the Twins data set, MAPE cannot be calculated, as the true CATE can be zero. Bold highlights the best results, with underlined values falling within standard error. Colored cells show the hypothesis that matching metries will yield the best performance. Results for 50 evaluation trials with T-risk and 50 estimation trials with T T-Learner and gradient boosting.
typically significantly improves precision compared using only metalearner. Moreover, some metalearners can result in very poor performance even after 200 optimization trials. Typically, these results are due to exceptionally poor performance in some iterations (e.g., the R- R-Learner). Additionally, we compare the performance trade-off in terms of time and precision for best metalearners in Figure 9. These results show that the S-, T-, and Lo-Learner are often the fastest to train and the most precise in terms of /PEHE. These results illustrate the potential of improving Aut outoCAAE's time efficiency by considering these trade-offs. To give a sense of AutoCATE's runtime, we include the required computation times run Aut oCATE on different data sets in Table 9, Although some time is required, running our framework locally feasible for small to moderate data sets.
A key innovation for AutoCATE is that it optimizes the entire ML pipeline, including preprocessing steps. In Table 10, we present an ablation study for our framework with and without preprocessing. For all data sets, AutoCATE, achieves the best performance with preprocessing, though the improvement is only significant for the IHDP and Twins data.
We can also apply explainability techniques to understand what drives a pipeline's predictions. Figure 10 illustrates this and shows how permutation feature importance can be used with AutOCATE.
20

---

AutOCATE: End-to-End, Automated Treatment Effect Estimation
T
DR
X
RA
Lo
F
AllMeta
IHDP 4.52
2.52
5.91
5.46
2752.36
5.80
2.47
20.09
7.45
9.58
1.54
(-37.5%)
ACIC 4.
4.26
3.61
1.
477525.02
3.27...1
3.
150829,
14
5.75
4.65
1.62
(-7.3%%
Twins .318 .345_
.320
.333
17.408ga1.07
.3231.00
.360g.00
.546..01
.418
.376..00
.321
(+ 0.9%)
News 2.8 2.53 3.38
2.93 36448.74.1449..
3.141.1a
2.57
16.06
2.74
3.41..1 2,40
(- 5.0%)
Table 8: Comparing metalearner precision. For each data set, we compare the different metalearner's performance in
terms of VPEHE, with the best result highlighted in bold. We also include comparison with searching over all metalearners (AllMeta) and, in brackets, show how much this outperforms the best single metalearner. For each result, AutoCATE uses a T-risk with 50 evaluation trials, 200 estimation trials, and top average model selection.
HIDP
ACIC
Twins d
News OR
U
7.5
WPEHE
RRA
RA
PR
3.0
5.0
0.4
DIt
Jo
2.5
RA
J.
5.U
7.5
10
Time [houre]
Time [hours]
Time hours]
Time [bours]
Figure 9: Comparing metalearner precision and time efficiency. We show each metaleamer's performance in precision (v (КPEНE) and time (excluding outliers, see Table 8). For each, AUtoCATE a T-risk with 50 evaluation trials, 200 estimation trials, and top I average model selection.
D.3. Stage Ensembling
The ensemble built by oCATE can be used to gauge the uncertainty regarding prediction, by highlighting the spread of predictions. We illustrate such an analysis in Figure 11.
D.4. Benchmarking AutOCATE
Table 11 presents results for additional benchmarks: S- and T-Learners based on linear or logistic models (without regularization).
A key challenge in CATE estimation is to deal with covariate shift due to confounding. To validate AutoCATES robustness to this phenomenon, we use synthetic experiment to precisely conrol selection into treatment and covariate shift and systematically evaluate performance of different methods. To this end, we use a synthetic data set in which we vary the degree of selection bias with the parameter y We generate 1, 000 instances with covariates X ~ (0, 1)5, treatment T ~ Bin r) where a (yru X) and ue ~ 21(11, (1) 1)5, and non-linear response surfaces- based on vectors w0, ul ~ 21(2((11 1)5-
Yo sin X) +
Yi Y, u, c2
where e <~ ~ (O, 0.1). y = 1, less than 1 1% of the propensities are extreme (< 0.01 or > 0.99). As gamma increases, this percentage grows to 72% (y 10) up to 99% (7 = 1,000). We repeat this experiment ten times. Following the experiments in the main body, we use a 70 30% train-test split. Our results in Figure Figure 12 indicate that while
Data set Size and dimensions
IHDP
ACIC
Twins
News
n = 747; d = 25
n = 4,802; d = 58
n = 11,984; 46 n = 5,000; d = 3,477
Time required
1'21"
6'00"
29'38"
6'49"
Table 9: AutoCATE time complexity. We show the average runtime required to run AutoCATE's complete, end-to-end optimization on a single iteration of different data sets. For each data set, we include the size (n) and dimensionality (d). AutoCATE uses evaluation trials and 50 estimation trials with the DestMeta-Besttaet configuration. These experiments were conducted locally, on a machine with an AMD Ryzen PRO 4750U processor (1.70 GHz), 32 GB of RAM, and 64-bit operating system.
21

---

AutOCATE: End-to-End, Automated Treatment Effect Estimation
Preprocessing
IHDP
1.25 18
1.69 27
ACIC
1.52+.09
1.58+.00
Twins
:315
.320=.00 0000
News
2.33 .06
2.38
Table 10: Analayzing the added value of preprocessing. We compare AutoCATE's performance with and without preprocessing included in the search space, in terms of VPEHE, with the best result highlighted in bold. Preprocessing includes feature scaling and selection. AutoCATE, results for a T-risk with 50 evaluation trials and 50 estimation trials with BestMeta-BestBase configuration.
Squared distance
0.4
0.3
0.2
0.1
0.0
18
16
11
12
15
24
22
20
21
17
23
Feature
Figure 10: Analyzing AutoCA feature importance. We can analyze how much each feature contributes to treatment effect heterogeneity. We illustrate this analysis for the first iteration of THDP using permutation feature importance, showing the squared distance the original prediction when permuting feature column.
AutoCATE's performance degrades selection bias increases, increasing the number of search trials helps mitigate this effect. Even under strong overlap violations (7 > 10), AutoCATE can result in good performance. Compare AutoCATE's to benchmark models across different bias levels. These results confirm that AutOCATE consistently performs competitive to each baseline in settings with moderate bias and remains relatively robust under extreme bias.
Figure 13 shows additional results for two data sets for uplift modeling (see Appendix for more information on the data). The effectiveness of Aut oCATE related least three factors. First, by using the AUQC metric, the search is aligned with the downstream task: prioritizing instances for treatment (Vanderschueren al., 2024). Second, the search space for AutoCATI TE includes more meta- and baselearners than the benchmarks. Third, the top five ensemble seems to improve the stability and accuracy of the predicted ranking.
D.5. Analyzing AutoCATE's Results
We analyze the results of AutoCATE's optimized pipelines in Figure 14. These results illustrate how AutoCATE facilitate higher-level, comprehensive analysis of methods for CATE estimation and model validation.
E. Comparing Software Packages for CATE Estimation
Table 12 lists software packages for CATE estimation, comparing their functionalities with AutoCATE, Notably, no other package is focused automated, end-to-end CATE estimation.
22

---

AutOCATE: End-to-End tond,, Automated Treatment Effect Estimation
6
4
2-
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Figure 11: Assessing uncertainty with AutOCATE. The ensemble returned by AutoCATE can be used to analyze uncertainty regarding the prediction. We illustrate this for the first 20 instances of the first iteration of the IHDP data. For each instance, the (usually unknown) ground truth is shown in green, while the predictions from the top five pipelines shown in blue and with a violinplot.
AutoCATE
Benchmarks
Top
Top 5
S-RF
T-RF
S-GB
T-GB
S-LR
T-LR
IHDP
1.25 18
1.38
3.30
2.61
3.02
1.86 20
5.73=.9 89
2.41 39
ACIC
1.52+.00
1.45+.10
1.67+.08
1.65+.00
1.48+.10
1.38
4.13+.26
3.08+.18
Twins
.315..00 00
.3141.00
.3181.00
.331 00
.3192.0
.334
.320
.3354.00
News
2.33#.06
2.29
2.46:.09
2.39
2.68
2.40=
3.68
2.93
Table 11: Comparing AutoCATE with common benchmarks on CATE estimation. We compare performance in terms of VPEHE, with the best result highlighted in bold. AutoCATE results for T-risk with 50 evaluation trials and 50 estimation trials with the BestMeta-BestBaan configuration.
AutoCATE AutoCATE--D
T LR
1.0
S-RF
01.00
AutoCATE-
0.5
T-RF
GB
0.25
T-GB
AutoCATE
Confounding strength
Confounding strength
Figure 12: Analyzing the robustness of AutoCATE to confounding with synthetic data. Using synthetic data, control the strength of confounding with parameter y. This setup allows compare the performance of AutoCATE for different number estimation and evaluation trials (left) and benchmark its performance to other methods (right).
Hillstrom
Information
0.06
LR
AUQC
Aroe
.... T-LR
S-RF
0102
S-GB
T-GB
Estimation tion trials
Estimation trials
30
100
100
500
1000
AutoCATE-5
Figure 13: Benchmarking AutoCATE for treatment prioritization. We present additional results in terms of AUQC for two uplift data sets, Hillstrom and Information. These show that AutoCATE is useful tool for prioritizing instances for treatment, and highlight that its optimization more effective at optimizing AUQC compared to the benchmarks based on p-risk. AutOCATE uses a T-riis with 50 evaluation trials and the AUQC metric, the BestMeta- Top or Top ensembling.
BestMeta-BestBase BestBase search space, and
23

---

AutOCATE: End-to-End to-End, Automated Treatment Effect Estimation
100%
40%
75%
30%
50%
Lo
20%
R
10%
25%
RA
0%
0%
ACIC IHDP Twins News
Unique metalearnens in top five
(a) How often is a particular metalearner optimal?
(b) How diverse are the best five metalearner?? (IHDP)
100%
100%
DT
GP
SVM
DT
75%
GP
75%
RF
ET
50%
GB
50%
GB
KNN
kNN
25%
25%
SVM
NN
NN
LR
0%
ACIC IHDP Twins News
Propensity Outcome
Eflect
(c) How often particular baseleamer chosen?
(d) What baseleamer best per model type? (IHDP)
Z
F
DR. T IF R
0.5
None
0.0
None
LR=
Percentile
RF
U
0.5
Robust
RA
KNN
INN
-1.0
Variance
Lo
NN-
D
TRT UINN
Selection
Scaling
Meta
Base
(c) Risk correlation (IHDP)
(f) Pipeline composition (Twins)
Figure 14: OCAT enables insights into CATE estimation. We analyze hundreds of pipelines optimized by AutoCATE
(see Section 5). Metalearners-(a) Different metalearners can be optimal for a data set, highlighting the need for searching across them. (b) The top five pipelines often feature a mix different metaleamers (c.g. (T, T, R.A, R.A, DR}: 3 unique types), showing that different metalearners can perform well and suggesting potential for combining them. Baselearners (c) The chosen baselearners
also diverse, and (d) different model types favor different ones. Using single baselearner is thus likely suboptimal, supporting choice tune submodels independently. Risk measures (e) The correlations between risk measures, shown here for single IHDP iteration, vary strongly. Surprisingly, risk measures can be strongly negatively correlated, suggesting potential for more advanced multi-objective approaches that adaptively learn which objectives are reliable for given data set. Optimal pipelines-(1 (f) There is variability in the optimal pipelines learned across ten iterations for the Twins data, suggesting that the data generating process is not the only relevant factor.
24

---

AutOCATE: End-to-End to-End, Automated Treatment Effect Estimation
PACKAGE FUNCTIONALITIES
GENERAL INFORMATION
Name
(1)
(2)
(3)
Language
Reference
Link
CausalML
X
Python
(Chen et al., 2020)
GitHub
EconML
/
V
X
Python
GitHub
DoWhy
xt
X
Python
(Sharma & Kiciman, 2020)
GitHub
Causica
X
J
X
Python
(Geffner al., 2022)
GitHub
UpliftML
X
V
X
X
Python
(Teinemaa et al., 2021)
GitHub
scikit-uplift
X
X
X
X
Python
GitHub
grf
X
/
/
X
R
(Wager & Athey, 2018)
CRAN
AutoCATE
V
Python
This work
GitHub
CausalML offers provides some tools for internal validity, such as comparing results across segments.
EEconML includes an R-risk can provide an ensemble based on this risk measure.
"DoWhy includes robustness checks for assumption violations.
"The grf package allows for evaluation based the Targeting Operating Characteristics curve.
Table 12: Software package comparison. We provide an overview commonly used packages for CATE estimation and compare their functionalities with AutOCATE, showing whether they support (1 evaluation, (2) estimation, (3) ensembling, and (4) automated, end-to-end optimization- as provided by AutOCATE or similar.
25


Reviews:
[{"review_id": "0G6rML9nD5", "reviewer": "Reviewer_NSJK", "summary": "The paper presents AutoCATE, an automated framework for CATE estimation, optimizing model selection, tuning, and validation via counterfactual Combined Algorithm Selection and Hyperparameter (CASH) optimization. It unifies evaluation, estimation, and ensembling, automating key design choices for improved generalization. Experiments on benchmarks show AutoCATE outperforms existing methods, and it is released as open-source software for broader adoption.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review!\n## Robustness to selection bias\nWe agree that synthetic data allows for precise control over selection bias and covariate shift, enabling a more systematic evaluation.\nWe have added a synthetic experiment where we vary the degree of selection bias (controlled by the parameter γ\\gammaγ). Our results indicate that while AutoCATE’s performance degrades as selection bias increases, increasing the number of search trials helps mitigate this effect. Even under strong overlap violations ($\\gamma > 10$), AutoCATE can result in good performance.\nWe also compare AutoCATE to benchmark models across different bias levels. The results confirm that AutoCATE consistently performs competitive to each baseline in settings with moderate bias and remains relatively robust under extreme bias. \nWe will update the paper to include a detailed explanation of the data-generating process (DGP) and expand on these findings.\nThank you for this valuable suggestion—we believe this addition strengthens our evaluation and provides clearer insights into AutoCATE’s generalization capabilities.\n#### _Synthetic data: setup_\n|Gamma|0|1|10|100|1000|\n|-:|:-:|:-:|:-:|:-:|:-:|\n|Extreme propensities ([0, 0.01) and (0.99, 1]) (%)|0.0|0.1|72.0|97.0|99.7|\n#### _Synthetic data: $\\sqrt{\\text{PEHE}}$ (SE) for AutoCATE with different number of evaluation and estimation trials_\n|Gamma|0|1|10|100|1000|\n|-:|:-:|:-:|:-:|:-:|:-:|\n|__5 trials__|0.49 (0.03)|0.53 (0.02)|0.88 (0.12)|0.93 (0.12)|0.95 (0.12)|\n|__10 trials__|0.20 (0.03)|0.25 (0.02)|0.46 (0.07)|0.54 (0.08)|0.58 (0.11)|\n|__50 trials__|0.14 (0.01)|0.15 (0.03)|0.35 (0.04)|0.52 (0.06)|0.44 (0.09)|\n#### _Synthetic data: $\\sqrt{\\text{PEHE}}$ (SE) for AutoCATE and benchmarks with 50 evaluation and estimation trials_\n|Gamma|0|1|10|100|1000|\n|-:|:-:|:-:|:-:|:-:|:-:|\n| __S-LR__|1.23 (0.06)|1.23 (0.06)|1.25 (0.07)|1.25 (0.07)|1.25 (0.07)|\n| __T-LR__|0.08 (0.01)|0.09 (0.01)|0.23 (0.07)|0.23 (0.08)|0.24 (0.08)| \n| __S-RF__|0.51 (0.03)|0.59 (0.03)|0.82 (0.04)|0.83 (0.04)|0.82 (0.04)|\n| __T-RF__|0.42 (0.02)|0.46 (0.03)|0.98 (0.06)|0.70 (0.07)|0.70 (0.06)|\n| __S-GB__|0.39 (0.02)|0.45 (0.03)|0.72 (0.03)|0.75 (0.04)|0.75 (0.05)|\n| __T-GB__|0.37 (0.02)|0.40 (0.03)|0.56 (0.05)|0.55 (0.04)|0.56 (0.05)|\n| __AutoCATE__|0.14 (0.01)|0.15 (0.03)|0.35 (0.04)|0.52 (0.06)|0.44 (0.09) |\n## Customizability\nAutoCATE is designed to be highly customizable yet easy to use. Users can specify the search space (preprocessors, metalearners, baselearners) and set design parameters (evaluation protocols, trial numbers, ensemble methods). Domain experts can also fine-tune risk measures, evaluation metrics, and validation procedures. The interface follows scikit-learn conventions for intuitive use, with configuration details available in Appendix B.6. Future versions will include an API for custom algorithms, enabling users to integrate their own CATE estimation and evaluation methods.\n## Limitations\nWhile AutoCATE is designed to be broadly applicable, we acknowledge that no method is universally optimal for all CATE estimation scenarios.\nAs such, there are certain settings where AutoCATE may be less suitable:\n- Very small datasets (n<50), where model selection based on pseudo-outcomes may be unreliable. For IHDP, we achieve good performance with only n=672 instances in the training set.\n- Large datasets with constrained compute, where an AutoML-based approach may be too computationally expensive.\n- Scenarios requiring strong domain knowledge integration, where a fully customized pipeline may be preferable.\n- Data that requires extensive preprocessing, such as raw image or text data.\n- Settings with fairness or regulatory constraints, where automated model selection may need additional safeguards.\n- Cases violating our causal assumptions, such as strong violations of overlap, hidden confounders.\n- Cases considering a different setting, such as instrumental variables or time series/panel data settings.\n\nWe agree that clarifying these limitations will strengthen the paper and will update the manuscript accordingly. Thank you for this valuable suggestion. If the reviewer thinks of other limitations that we have missed, we would be happy to also include those.\n## Theoretical contributions\nWhile our paper does indeed not present new formal proofs or theoretical guarantees, it aligns with ICML's emphasis on application-driven research, as highlighted in the call for papers. ICML encourages innovative techniques and problems motivated by real-world needs, with an emphasis on reproducible experiments and sound analysis, rather than mandatory theoretical components. Many influential ICML papers prioritize empirical insights and conceptual innovations, and we believe our work contributes to this tradition by addressing a significant practical challenge in causal inference.\n___\n\nThank you for your time and effort in reviewing our work! Please let us know if you have any remaining concerns.", "claims_and_evidence": "The authors claim that AutoCATE surpasses conventional CATE estimation methods and provide empirical evaluations on benchmark datasets that show AutoCATE achieving superior results compared to existing approaches.", "methods_and_evaluation_criteria": "Yes, the evaluation criteria make sense, however, it could be expanded to cover more scenarios (see section “Questions For Authors”).", "theoretical_claims": "The paper makes no theoretical claims.", "experimental_designs_or_analyses": "Yes, the experimental designs or analyses are valid.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "- The paper is well-written and easy to follow.", "other_comments_or_suggestions": "- Not sure if ICML is the right venue for this paper, as it lacks theoretical contributions (e.g., new proofs, formal guarantees, or novel optimization formulations).", "questions_for_authors": "- Besides these well-known datasets, why not use fully synthetic datasets to systematically control the degree of selection bias and covariate shift? This would allow for precise evaluation of AutoCATE's robustness across different levels of bias and distribution shift, ensuring clearer insights into its generalization capabilities.\n- To what extent can users intervene in the automated process? Are there provisions for customizing or overriding certain steps in the pipeline to tailor it to specific needs or preferences?\n- What are the known limitations or potential failure cases of AutoCATE? Are there scenarios where it might not be the optimal choice for CATE estimation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LCPJpL6Jwr", "reviewer": "Reviewer_NKef", "summary": "The authors propose a pipeline for automating the several design choices required for CATE estimation; from preprocessing datasets to different risk measures for model selection. The pipeline is divided into three stages corresponding to the following three questions; what risk measure should be used for model selection, what CATE estimators should be trained, and finally how should be select over the trained CATE estimators and combine them for better generalization. The authors conduct experiments on widely used benchmarks and present interesting insights regarding the numerous design choices in CATE estimation.\n\n## Update after rebuttal\n\nThanks for the rebuttal! My concerns have been addressed and I want to retain my rating for acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in reviewing our work!\n\n## Obtaining the best meta/base-learner and training nuisance models with AutoML\nPlease allow us to clarify our approach. AutoCATE follows an AutoML-based procedure to tune ML pipelines at multiple stages: first, to optimize risk measures for model selection, and second, to construct optimal CATE estimators by searching over meta- and base-learners. We obtain the best CATE estimator–i.e., a metalearner constructed with (multiple) baselearner(s)--in the second stage, based on the risk measure selected in the first stage. As the reviewer suggests, this indeed ensures that we pick the CATE estimator that best predicts the (observational) validation data. \n\nOur design ensures a fully automated, end-to-end approach that integrates evaluation, estimation, and ensembling within a single framework (see Section 4.5). Importantly, all components of a metalearner (i.e., each baselearner) are tuned automatically and simultaneously, ensuring the best possible downstream performance. We believe that this is one of the key strengths of our work compared to previous approaches!\n\nWe hope that these additions clarify our approach. We will update the paper to make this aspect more clear. Nevertheless, if there are any remaining questions, please do let us know.\n## Tuning the S- and T-Learner\nRegarding the tuning of S-/T-Learners, this is done based on a random search with the number of trails equal to AutoCATE, allowing for a fair comparison. As such, we do not use a manual grid search for either AutoCATE or the benchmarks. Mahajan et al. (2023) use AutoML separately for evaluation and for learning the nuisance models underlying CATE estimators. As neither S- and T-Learners use nuisance models, we believe that our approach is similar to the one in Mahajan et al. (2023). Additionally, they do not consider the impact of only using a limited number of optimization trials, which is an important consideration in practice. Nevertheless, if there are distinctions we have missed, we would be happy to look into them and adjust the experiments.\n\nWe will revise our manuscript to better clarify the reasoning behind and training procedure for the benchmarks.\n## Combining risk measures\nThank you for these thoughtful questions. We appreciate the opportunity to clarify our approach and conclusions.\n\n### How are risk measures combined?\nWe explore multiple strategies for combining risk measures (see also Appendix B.5), including:\n- Averaging normalized risk measures (as in Table 1b). This approach corresponds to the reviewer’s suggestion.\n- Averaging rankings of risk measures to improve robustness to outliers.\n- Euclidean distance to the origin (best possible performance).\n- Selecting all Pareto-optimal points.\n### What are the main conclusions?\nOur experiments serve as an initial exploration into combining risk measures. While our hypothesis was that relying on multiple risk measures could enhance robustness and performance, we find that no strategy consistently outperforms using a single T-risk. However, results indicate that this approach is promising, and further research—such as leveraging risk measure correlations (Figure 14a) or advanced ensembling—could improve performance. Therefore, we believe there is evidence that this novel approach provides a fruitful and promising direction for future research on CATE estimator validation. By highlighting the potential of this approach and providing a foundation for future research with our software package, we hope to encourage the community to further explore these ideas.\n\nWe will update our paper to clearly state the main conclusions. \n## Scale of the empirical study\nThank you for this great suggestion! We agree that summarizing the scale of our empirical study upfront helps highlight our contributions, and we will incorporate this more expliclity in the revised manuscript.\n\nIn the main body, we present experiments across a total of 247 distinct datasets, across four benchmark families, spanning binary and continuous outcomes, as well as different sizes and dimensionalities. Additionally, we include additional experiments on synthetic data for this rebuttal. In the appendix, we also explore AutoCATE on two uplift datasets.\n\nAutoCATE’s full search space consists of 2,187 possible pipelines (3 feature selection × 3 scaling × 27 meta/base-learner configurations × 9 base learners), excluding hyperparameters (Appendix B.3). It incorporates 8 different risk measures. A full overview of its configuration options can be found in Appendix B.6.\n\nWe appreciate the reviewer’s suggestion and will ensure this information is presented more explicitly in the paper.\n___\nOnce again, thank you for your detailed review! While we hope that these responses address your concerns, please let us know if there are any remaining points. We would be happy to engage further if needed.", "claims_and_evidence": "**Strengths**\n\n- The authors have done a really good job at empirical validating all the design choices involved with the proposed framework AutoCATE! The scale of the empirical study is quite comprehensive; experiments involve a variety of meta-learners, base-learners, and risk measures. This makes their findings interesting and significant for practitioners and future work, and their software package should also make it easy for practitioners to adopt the proposed pipeline.\n\n**Weaknesses**\n\n- My concerns are mostly regarding the claims and insights regarding the end-to-end automation part of AutoCATE. \n\n    - Regarding experiment in section 5.5, the authors should follow the procedure of AutoML to tune S/T Learner (Mahajan et al. 20023) instead of manual grid search. This would ensure stronger baselines and a fair comparison with them. Similarly, the authors can construct meta-learners with nuisance model trained via AutoML (Mahajan et al. 20023), and that could serve an alternative set of CATE estimators for experiment in section 5.3 (Estimation) as well. For example, the BestBase estimator currently involves a manual search over a grid of different algorithms and hyperparameters, but this could be automated via AutoML.\n\n     -   I am not sure what are the main conclusions from the experiments with combined risk measures? The authors did not experiment with many combinations, and only considered combining T & DR risk and different T risk. So the experiments are not exhaustive which makes it hard to interpret what the main trend should be and what recommendations can be made. Similar comment for ensembling with multiple risk measures; I think the strategy of combining risk measures is the most novel aspect of the work, so analyzing it in depth would make the paper strong.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense for the problem at hand. All the baselines and benchmarks used in this paper are widely used in causal inference.", "theoretical_claims": "There are no theoretical claims in the paper.", "experimental_designs_or_analyses": "Yes, I checked the soundness/validity of all the experiments in the paper, and the experiment design doesn't have any flaws.", "relation_to_broader_scientific_literature": "This work builds upon the prior literature [1, 2, 3]  on empirically analyzing  model selection strategies for CATE estimation. Model selection in causal inference is a challenging task due to the fundamental challenge of unobserved counterfactual potential outcomes, which led to several heuristics being proposed for this task without much clarity on when a certain strategy must be used. Hence, prior empirical studies [1, 2, 3] empirically analyzed these  model selection strategies, often finding that several strategies can be optimal. This paper further builds upon this, by proposing novel ways of combining multiple model selection strategies and also extensively analyzing the role of other factors like dataset preprocessing that was missed by earlier works.\n\nReferences\n\n- [1] Schuler, Alejandro, Michael Baiocchi, Robert Tibshirani, and Nigam Shah. \"A comparison of methods for model selection when estimating individual treatment effects. arxiv. 2018.\" arXiv preprint arXiv:1804.05146 (1804).\n\n- [2] Curth, Alicia, and Mihaela Van Der Schaar. \"In search of insights, not magic bullets: Towards demystification of the model selection dilemma in heterogeneous treatment effect estimation.\" In International conference on machine learning, pp. 6623-6642. PMLR, 2023.\n\n- [3] Mahajan, Divyat, Ioannis Mitliagkas, Brady Neal, and Vasilis Syrgkanis. \"Empirical analysis of model selection for heterogeneous causal effect estimation.\" arXiv preprint arXiv:2211.01939 (2022).", "essential_references_not_discussed": "No, I believe all essential references have been discussed to the best of my knowledge. The authors have written a detailed related works section.", "other_strengths_and_weaknesses": "**Strengths**\n\n- Authors introduce several novel components. To the best of my knowledge, analyzing the role of dataset preprocessing and dataset splits for training/evaluation in CATE estimation has not been done in prior works. Further, the authors experiment with novel strategies for model selection with multiple risk measures and ensembling of CATE estimators.\n\n- The paper overall is well written and organized which makes it easy to follow and understand main results. The experiment results are clearly presented with good discussion around them. I especially like their comparisons with the findings from prior benchmarking studies for CATE model selection.", "other_comments_or_suggestions": "- It would be nice to have statistics regarding the scale of the empirical study before section 5; like how many risk measures, how many meta-learners and base-learners for estimation, how many estimators are included for the model selection study, etc.", "questions_for_authors": "For major questions, please refer to the the \"claims and evidence\" sections above.\n\n- How are the multiple risk measures combined? Do we take the average of risk measures, like average of T and DR risk in the experiments?\n\n- How do the authors obtain the best meta-learner or best base-learner? Is it based on how well they fit the observational data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BeHMP5RzzB", "reviewer": "Reviewer_V9Fy", "summary": "This paper presents AutoCATE, an automated, end-to-end framework for estimating Conditional Average Treatment Effects (CATE). The core motivation is that while ML methods have made significant advancements in causal inference, their adoption remains limited due to the complexities in pipeline selection, hyperparameter tuning, and validation.\n\nTo address these issues, the authors propose framing the problem as a counterfactual Combined Algorithm Selection and Hyperparameter (CASH) optimization and develop AutoCATE, a framework that integrates evaluation, estimation, and ensembling into a single automated solution. The framework searches across various ML models, metalearners, and hyperparameters to optimize CATE estimation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review–this is highly appreciated!\n## Real-world data and violations\nWe agree that these are crucial considerations. While we use real-world data to validate AutoCATE as much as possible (e.g, Twins and the uplift data in Appendix D.4), we acknowledge that more real-world validation would be useful. Unfortunately, we are unfortunately limited by the fact that the CATE is unknown in real data.\n\nWe agree that considering violations of identifiability assumptions is important. We have added a synthetic experiment varying selection bias (γ) and found that while performance declines with higher bias, more optimization trials can partially mitigate the effect (we refer to the response to Reviewer NSJK). Even with strong overlap violations (γ > 10), AutoCATE remains competitive. We will update the paper to clarify the data-generating process and these findings. Finally, we also point out AutoCATE’s good performance on IHDP, which also contains overlap violations [1].\n\nMore generally, we recognize the need for improving robustness to such violations and will include this limitation in the revised version.\n## Domain knowledge and observational data diagnostics\nThank you for the insightful suggestion. We agree that incorporating domain knowledge and data diagnostics would be valuable, and we would love to support such features in future releases. AutoCATE allows for trimming extreme pseudo-outcomes (e.g., from very small propensity scores), though this was not included in our paper. We also agree that checking overlap/covariate balance and filtering propensity scores would be useful additions.\n\nWhile sensitivity analyses for hidden confounders are appealing, they may fall outside the scope of our automated approach. Instead, we aim for AutoCATE to be complementary to other packages (e.g. DoWhy), which address other aspects of causal inference. Our main goal is _automating_ CATE estimation, a gap we see in current tools. Nevertheless, we see many remaining challenges for truly supporting practical adoption of these methods.\n## Computational complexity and heuristics\nWe have designed AutoCATE with efficiency in mind, using parallelization via Optuna, efficient ML implementations via scikit-learn, and configurable constraints. Computational times for different datasets are summarized in Table 9.\n\nNevertheless, there are several opportunities to improve AutoCATE’s speed. We currently use a naive random search algorithm without heuristics. As our search is implemented with Optuna, more advanced search algorithms and pruning strategies can easily be used instead. Finally, different metalearners have widely varying time complexities (Figure 9). Future research could try to use these discrepancies to further optimize the search.\n\nIn the revised version, we will stress the complexity of AutoCATE more explicitly as a limitation.\n## Combining risk measures\nThank you for these insightful suggestions!\n\nWe explore various static strategies for combining risk measures (e.g., averaging, ranking) and compare them empirically in Table 3. While we analyse correlations between risk measures (Figure 13a), we do not yet use this information, though it seems promising.\nOur current work focuses on the feasibility of combining risk measures using simple approaches. While the suggested dynamic approaches are really exciting directions, their implementation is challenging–key issues include the lack of ground truth and the variance of risk measures. While tackling these challenges is beyond the scope of this work, we see them as important future research directions and hope our framework can serve as a foundation for further advancements.\n## Stacking\nWhy might stacking underperform? Without a ground truth CATE, validating and tuning stacking weights is challenging. Stacking may overfit pseudo-outcomes with high variance or outliers. The squared error loss is sensitive to outliers, and alternatives like Huber loss could improve robustness. Stacking also requires more training data, and with limited data, its complexity may hurt generalization.\n\nTo improve stacking, we could try to filter pseudo-outcomes based on reliability, possibly using risk measure agreement. Multi-objective stacking could create more generalizable models. Using loss functions like Huber loss could mitigate noisy outcomes. Further research into tailored stacking and ensembling for CATE estimation is needed. Limiting stacking to top models could also improve stability.\n___\nThank you again for your detailed and thoughtful review. We agree that there is much work ahead, and while some suggestions are beyond the scope of this paper, we are happy to consider any specific approaches you feel should be included for acceptance within the remaining time.\n___\n[1] Curth, A., Svensson, D., Weatherall, J., & van der Schaar, M. (2021). Really doing great at estimating CATE? A critical look at ML benchmarking practices in treatment effect estimation. NeurIPS.", "claims_and_evidence": "Claim 1: AutoCATE is “the first end-to-end, automated solution” tailored for CATE estimation. The authors provide an overview of existing libraries (e.g., CausalML, EconML) and show that, while these offer various metalearners or partial automation, they do not perform a comprehensive search across risk measures, model architectures, hyperparameters, and ensembling.\n\nEvidence: The paper includes a table comparing software packages, demonstrating that other libraries focus on some metalearners or on certain tuning aspects, whereas AutoCATE addresses the entire pipeline.\n\n\n\nClaim 2: Jointly optimizing evaluation methods and ML pipelines boosts performance over standard “predict the observed outcome” baselines and over conventional single metalearner approaches.\n\nEvidence: Empirical results indicate that simply optimizing based on the observed-outcome (µ) risk can lead to suboptimal CATE predictions, whereas using specialized risk measures (e.g., T-risk, DR-risk) aligns model selection with the actual causal objective. Experiments also show improved accuracy (in terms of √PEHE or Qini-based metrics) when comparing AutoCATE to typical T-/S-Learners that only tune on one group outcome at a time.\n\n\n\nClaim 3: Metalearners like T-, DR-, and RA-Learners tend to achieve competitive or best performance on average.\n\nEvidence: Through ablation studies, the paper shows consistent strong results for T-, DR-, and RA-Learners, whereas others (like U- or R-Learners) can produce outlier performance in some data sets. This is measured through extensive random-search trials and is illustrated with results in tables and plots.\n\n\nOverall, the major claims are around: (1) the novelty of a fully automated pipeline specifically for CATE, (2) the demonstrated importance of risk measures aligned with the causal objective, and (3) empirical benefits of ensembling. The evidence rests on thorough experiments across four well-known semi-synthetic causal benchmarks", "methods_and_evaluation_criteria": "- The paper includes multiple metalearners (S-, T-, Lo-, DR-, RA-, etc.) and a large suite of baselearners (random forests, gradient boosting, MLPs, etc.). Metalearners are combined with different pseudo-outcome-based risk measures (e.g., T-risk, DR-risk, kNN-risk) to evaluate the pipeline on held-out or cross-validation data.\n\n\n- Evaluation metrics for final performance include √PEHE (for measuring overall error on the true potential outcomes in synthetic data) and AUQC (for ranking-based tasks).\n\n\n- The approach is sensible for observational data with confounding: it respects standard assumptions (unconfoundedness, overlap). The use of pseudo-outcomes or IPW-based risk measures is standard in causal inference, but the novelty lies in systematically searching across them and ensembling.\n\n\nFrom a methodological perspective, the authors’ approach to separating the “evaluation pipeline” from the final “estimation pipeline” is carefully designed: it includes data splitting so that the pseudo-outcome (or other risk) is learned on a separate portion, then used to guide model selection. This is appropriate for CATE tasks where the ground truth is never observed for each individual’s counterfactual.", "theoretical_claims": "The work frames the search problem in a counterfactual CASH setting and cites known results about pseudo-outcomes converging to the true treatment effect under standard assumptions (e.g., DR-/R-Learners). However, the paper does not provide new proofs or theoretical derivations. Instead, it references known theoretical results (e.g., asymptotic unbiasedness of DR, properties of R-Learners).\n\n\nNo obvious errors appear in the sketches of theory or in the references to established results. The theoretical claims are primarily restatements of known properties (e.g., each metalearner’s consistency under standard conditions).", "experimental_designs_or_analyses": "- The experiments span four widely used semi-synthetic data sets in CATE research: IHDP, ACIC, Twins, and News. They vary in size, dimensionality, data-generating processes, and outcome type (binary vs. continuous).\n\n\n- The evaluation metrics (e.g., √PEHE, AUQC) are standard, and each data set has a known “ground truth” effect or can approximate it, so the experimental design is appropriate.\n\n\n\n- Model comparisons systematically vary key design choices, risk measures, baselearners, data splits, and provide results in tables and line plots.\n\n\n\n- An ablation study approach is used to highlight which parts of AutoCATE (like using T-risk vs. DR-risk or enabling feature selection vs. not) matter most.\n\n\n\n- Overall, the experimental setup is sound. The sample sizes and repeated runs (e.g., 50–200 trials, multiple random seeds) mitigate random variation.\n\n\nA potential limitation is that the paper relies heavily on semi-synthetic data, so real-world complexities (e.g., non-stationarity, unobserved confounders) may not always arise. The authors acknowledge this and suggest future applications to purely real-world data.", "relation_to_broader_scientific_literature": "- In AutoML, the authors draw parallels to general-purpose tools like AutoGluon, H2O AutoML, or FLAML, pointing out that standard AutoML focuses on conventional supervised tasks (e.g., classification, regression) and does not address the challenges of CATE estimation (lack of ground truth, confounding, etc.).\n\n\n\n- For CATE methods, they discuss standard metalearners such as the T-, S-, X-, DR-, R-Learners, and highlight that prior work compares them in isolation or tunes them partially, but has not combined them in one broad pipeline that includes “evaluation pipeline search.”\n\n\n\n- They position their approach as bridging these two areas: AutoML + CATE-specific validation.\n\n\nThis situates the paper well in an emerging area of interest, applying automated search for valid causal effect estimation.", "essential_references_not_discussed": "The paper cites many standard references (e.g., R-, DR-, X-Learners) and relevant AutoML works (FLAML, H2O, etc.). One might also compare with the EconML approach, which implements an “R-risk ensembling.” The authors do mention EconML, but they might elaborate on how exactly EconML’s ensemble compares with the new stacking approach. Nonetheless, the references in the paper are quite thorough, and there do not appear to be major missing lines of prior research.", "other_strengths_and_weaknesses": "Strengths:\n\n- The idea of decoupling an “evaluation pipeline” with multiple risk measures from the “estimation pipeline” and then ensembling is a strong conceptual contribution that addresses known challenges in model selection for causal inference.\n- The thorough ablation studies reveal which metalearners and which risk measures typically excel, giving the community new insights.\n- The open-source release (in Python) lowers the barrier to entry for robust causal effect estimation, potentially encouraging broader adoption.\n\nWeaknesses:\n\n- Real-world data sets that contain unobserved confounders or strong domain shifts are not tested extensively; thus, the framework’s performance under such violations remains open.\n- The search space can become large and computationally expensive (especially for high-dimensional data), though the authors do note ways to tune time or limit certain baselearners.\n- The paper does not propose new theoretical results for metalearners, it mainly integrates known approaches. However, the overall pipeline is novel, so this does not detract from the paper’s main contributions.", "other_comments_or_suggestions": "See the above weaknesses section.", "questions_for_authors": "Are there heuristics that shrink the space automatically based on intermediate findings?\n\n\n\nYou discuss combining multiple risk measures (e.g., DR + T + kNN). Did you consider dynamic weighting or meta-learning that adaptively emphasizes certain pseudo-outcomes depending on early performance signals?\n\n\n\nSome results show that simple top-k ensembles can outperform advanced stacking in certain data sets. Could there be reasons (e.g., overfitting to pseudo-outcomes) why stacking underperforms? Are there strategies to mitigate it?\n\n\n\nDo you have plans to incorporate domain knowledge or observational data diagnostics (e.g., checking positivity/overlap) more directly into AutoCATE’s workflow (e.g., automatic filtering of extreme propensity scores)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review."}}, {"review_id": "CDCWygN90X", "reviewer": "Reviewer_RR4p", "summary": "In this paper, authors have developed and released an AutoML library, called AutoCATE, for the automated selection and hyperparameter tuning of meta-learners for CATE estimation. They divide the CATE development pipeline into evaluation, estimation and ensembling phases, where evaluation corresponds to choosing a proxy risk, estimation corresponds to steps used for training CATE learners, including hyperparameter tuning, and ensembling uses proxy risk from evaluation stage to find the best learners from the estimation stage which can consist of ensemble of learners. They did extensive experiments on four benchmark datasets: IHDB, ACIC, Twins and News, to study different design choices for the three stages of AutoCATE.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review!\n___\n## Related work\nWhile prior work automates parts of CATE estimation, no approach–to our knowledge–provides an end-to-end automated framework. Existing work only automates part of our approach (e.g., learning pseudo-outcomes with AutoML). In contrast, AutoCATE extends automation across the entire pipeline, integrating model selection, preprocessing, and ensembling. In doing so, we address novel challenges like balancing data for training and validation or using multi-objective optimization. Our empirical study is also considerably more extensive, covering diverse datasets and revealing new insights into model selection.\n\nWe will revise our paper to better highlight these distinctions.\n## Metalearners\nAutoCATE is a general-purpose framework for CATE estimation, not limited to metalearners. While we use them for their flexibility, our key contribution is automating selection, tuning, and validation—critical for all CATE methods. Other approaches (e.g. TARNet) can easily be integrated in this framework. In future versions, we aim to support custom models through a user-friendly API.\n\nSimilar to general AutoML frameworks (e.g. FLAML), which do not incorporate each supervised ML algorithm, AutoCATE focuses on structured automation within an extensible search space. This remains a fundamental challenge, regardless of specific estimators.\n## Clarity: AllMeta and AllBase\nThese terms define search spaces in our automated ML pipeline for CATE validation and estimation. AllMeta includes all metalearners in our search (S, T, DR, X, R, RA, Lo, Z, U, F), covering all known techniques in the literature. AllBase consists of the nine base learners in our framework (see Figure 1). BestMeta and BestBase are selected subsets based on performance—metalearners are chosen from experiments and the baselearners are known to work well for tabular data.\n\nWe will clarify these in the revised paper—thank you for pointing this out! If anything else is unclear, we would be happy to clarify further.\n## AutoCATE stages\nIn Stage 1, AutoCATE constructs pseudo-outcomes as proxies for ground truth CATE. These are derived from metalearners, but not full CATE learners themselves: e.g., we use the DR-Learner’s pseudo-outcome directly, but without fitting the final model to predict it. This step is also performed on _validation_ data, not training data.\n\nStage 2 trains a CATE estimator on training data and evaluates it using pseudo-outcomes from Stage 1. As evaluation depends on a risk measure, it logically precedes estimation, avoiding a chicken-and-egg problem. Both stages involve choices not dictated by data (e.g., which risk measure to use), which we study systematically.\n\nRepeating stages until convergence is not straightforward, as estimation and evaluation are independent stages with distinct data splits. While evaluation informs estimator selection, estimators do not directly influence evaluation. Nevertheless, exploring these interactions (e.g., by sharing information in their optimization) could be an interesting avenue for future work.\n## Limited data\nWe agree that limited data can be a challenge for CATE validation. We emphasize the importance of having sufficient data for validation to mitigate such issues (as highlighted in Figure 3). We also study cross-validation (see Figure 7 and Table 5), which can help to stabilize the estimates for smaller datasets. Empirically, we observe good results for the IHDP data with only 672 instances. Some baselearners (e.g. linear regression) can perform reasonably well with very little data.\n\nThat said, we acknowledge that for very small datasets, other approaches may be more appropriate. We will include a more detailed discussion on scenarios where our framework may not be optimal in the conclusion. We refer to the response to reviewer NSJK, where we enlist more limitations.\n## Computational complexity\nEfficiency is an important consideration in our framework. AutoCATE runs fairly quickly on small to moderate-size datasets, often completing in minutes locally (see Table 9). AutoCATE also allows for trade-offs by limiting search space, reducing trials, or choosing faster learners (see Figure 9). Further improvements could be made based on more efficient search and pruning algorithms. Nevertheless, we agree that computational complexity is a potential limitation and will discuss it more in depth in the revision.\n## Multiple risk measures\nWe provide an initial exploration of combining risk measures. While no combination consistently outperforms the best single measure, we see significant potential for further development (see Figure 13e). Though not yet optimal, it provides a solid foundation for future improvements in both performance and efficiency. Although this approach can increase computational complexity, reusing components like propensity scores helps mitigate this impact. \n___\nThank you once again! Please let us know if there are any remaining concerns.", "claims_and_evidence": "1. They claimed AutoCATE as AutoML solution for CATE but it is limited only to meta-learner approaches and does not consider many other learners based on trees, neural networks, including causal forest, TARNet and SNet etc.", "methods_and_evaluation_criteria": "1. As discussed above, the analysis is limited to meta-learners only, and does not consider other CATE learners.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design looks okay to me as they have provided detaild experiments to study the three phases, i.e., evaluation, estimation and ensembling, of the the proposed framework, where they study the effect of using different approaches for risk, learners and baselearners. However, somethings were not clear  as I did not find there proper definition, e.g., AllMeta, and AllBase?", "relation_to_broader_scientific_literature": "1. As per my understanding, there already exists some work for automated CATE, as also referenced by the authors. However, the authors' discussion in the related work is focused on the conclusions of the existing work for auto CATE, which were dataset/setting specific. Authors should have directly focused on their approach and how authors' approach adds value to the literature? Is it just another paper/code for developing an auto-CATE pipeline?", "essential_references_not_discussed": "As discussed above, the discussion is focused only on meta-learners and their automated training which obviously does not cover all CATEs in general.", "other_strengths_and_weaknesses": "Strengths:  \n1. This paper discusses an important challenge of hyperparameter tuning and training, faced in the causal inference literature for CATE estimation. They develop auto-ml solution for CATE.\n2. The paper is generally clear and well-organised into clear heading and subheadings with relevant discussions.\n3. Overall experimental analysis covers all aspects of the proposed framework, considering 4 benchmark datasets.\n4. Authors have ensured the credibility of the work by releasing the code.\n\nWeaknesses:\n1. As discussed above, authors did not clearly placed their contribution against the literature as there exists some work on auto-CATE, and they did not clarify how their proposed is different from the existing ones rather they focused on findings of the existing works.\n2. Auto-CATE discussion is limted to meta-learners only which does not cover many other categories of CATE learners. So, in my opinion the contribution is not sufficient for this venue.\n\n## Update after the rebuttal and follow-up discussion\nDear authors, thanks for your response. I am convinced with your response to Related Work and Limited data points. So, I am raising my score from 2 to 3. All the best!", "other_comments_or_suggestions": "1. To find a risk measure, in Stage 1 Evaluation, you're training CATE learners by doing hyperparameter tuning etc which is required in a standard case for a CATE learner. So, it appears a chicken-and-egg problem to me. How do you make choices in the first stage? Moreover, it is possible to repeat the 3 phases until convergence to get a better estimates?\n\n2. Will your framework work in limited data settings? Because you make choices for risk estimators on the validation data which is smaller part of the given dataset and it is likely to lead to unstable estimates if the choices are made on a small dataset.\n\n3. Moreover, your AutoCATE framework seems to be computationally expensive to me. Comment on it and add as a limitation, if needed. Moreover, I did not notice significant improvement of combining multiple risks from the evaluation. So, is it useful as compared to the addtional computations/complexity it adds?", "questions_for_authors": "As discussed above, my concerns are related to the contributions of this work as compared with the literature, and second, AutoCATE is limited to meta-learners only.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a end-to-end framework for automatic estimation of the CATE alongside a detailed empirical study. Reviewers praised the inclusion of components of the pipeline not studied in previous work (e.g. various preprocessing and data splitting steps) and the comprehensive nature of the experiments. There were various questions about details of the experiments and the relationship to previous work which were largely resolved during the discussion period. This paper makes a valuable contribution to the area of heterogeneous treatment effect estimation and will be of interest to anyone working in the area.
Decision: Accept (poster)