Counterfactual Cross-Validation
Yuta Saito

Shota Yasui

Tokyo Institute of Technology
saito.y.bj@m.titech.ac.jp

CyberAgent, Inc.
yasui_shota@cyberagent.co.jp

ABSTRACT
There is intense attention in applying machine learning to makecausal inference in fields such as marketing, economics, and education. In particular, individual-level treatment effect (ITE) prediction
has important applications such as personalized recommendationand precision medicine. Most of the previous papers study prediction methods of the ITE. On the other hand, how to evaluate given
ITE predictors from observable data has not yet been thoroughly
investigated despite that it plays the critical part of causal inference.
In this paper, we propose a method to effectively select the best
ITE predictors from a set of candidates using only observational
validation set.
ACM Reference Format:
Yuta Saito and Shota Yasui. 2019. Counterfactual Cross-Validation. In Proceedings of the ACM Recsys Workshop on Reinforcement and Robust Estimators for Recommendation (REVEAL’19). ACM, New York, NY, USA, 2 pages.
https://doi.org/10.1145/nnnnnnn.nnnnnnn

1

INTRODUCTIONPredicting Individual-level Treatment Effects (ITE) of treatments
is essential to optimize the metric of interest in various domains.
For example, recommendation systems wish to recommend items
having positive causal effects on users preferences to maximize
user experiences.
Most of the previous studies propose machine-learning based
ITE prediction methods and achieve promising results on somebenchmark datasets. On the other hand, the evaluation of theseprediction methods is another essential step to conduct valid model
selection and hyperparameter tuning. However, the evaluation
problem of ITE prediction models has not much yet studied in spiteof its importance.
In this paper, we propose a model validation procedure called
CounterFactual Cross-Validation (CF-CV) that accurately ranks theperformance of ITE predictors with high confidence using only
factual validation set. Our proposed evaluation procedure satisfies
desirable theoretical properties to be used for model selection or
hyperparameter tuning of ITE predictors. Moreover, experimental
results show that our proposed method effectively ranks candidate
ITE predictors and select a better predictor among a set of candidates. Besides, the model selection performance of our method isPermission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
REVEAL’19, September 20th, 2019, Copenhagen, Denmark© 2019 Association for Computing Machinery.
ACM ISBN 978-x-xxxx-xxxx-x/YY/MM. . . $15.00
https://doi.org/10.1145/nnnnnnn.nnnnnnn

stable, and this is critical because we never know the true performance of any ITE predictors due to the existence of counterfactuals.

2PROBLEM FORMULATION

We denote X ∈ X as the feature vector and T ∈ {0, 1} as a binary treatment assignment indicator. Here, we follow the potential
outcome framework [4]that there exist two potential
 and assume
outcomes denoted as Y (0) , Y (1) ∈ Y × Y for each individual. Y (0)is a potential outcome associated with T = 0, on the other hand,Y (1) is associated with T = 1.
Now let us formally define the Individual-level Treatment
Effect (ITE) for an individual with a feature vector x ∈ X as:hiτ (x) = E Y (1) − Y (0) | X = x(1)
Next, we define the conditional probability of treatment assignment
as e (x) = P (T = 1 | X = x). This parameter is called propensity
score in causal inference and widely used to estimate treatment
effects from observational data [4]. Throughout this paper, we makethe standard assumptions in causal inference including Unconfoundedness, Overlap, and Consistency [4].
In previous studies [2, 5], the evaluation of an ITE predictor τ̂ (·)
is formulated as accurately estimating the following metric fromobservational validation dataset V = {X i ,Ti , Yiobs } as:Rt rue (τ̂ ) = EX (τ (X ) − τ̂ (X ))2(2)Here, Rt rue is the true performance metric of an ITE predictor τ̂ (·).
b (τ̂ )In this paper, we aim to construct a performance estimator Rsatisfying the following condition:b (τ̂ ) ≤ Rb τ̂ ′ , ∀ τ̂ , τ̂ ′ ∈ M.
Rt rue (τ̂ ) ≤ Rt rue τ̂ ′ ⇒ R(3)where M = {τ̂1 , ..., τ̂ |M | } is a set of candidate ITE predictors.
An estimator satisfying Eq. (3) gives accurate ranking of candidate predictors by the true metric values, and one can select thebest model among M.

3METHOD

To achieve our goal, we consider the following feasible estimatorof the performance metric:nÕb (τ̂ ) = 1(τ̃ (X i ) − τ̂ (X i ))2R
n i=1(4)where τ̃ (·) is called the oracle and is constructed from V. Under our formulation, we aim to answer the question: What is thebest plug-in oracle to rank the performance of given candidate ITE
predictors from observational validation dataset?

3.1

Proposed Oracle

Here we define our proposed oracle inspired by the doubly robust
estimator used to estimate average causal effects of treatments [1].

REVEAL’19, September 20th, 2019, Copenhagen, Denmark

Y.Saito and S.Yasui.

Definition 3.1. Let f (·, ·) : X×T → Y be a hypothesis predicting
potential outcomes and is defined as f (x, t) = h (Φ(x), t). Then, thedoubly robust oracle for a given data (X ,T , Y obs ) is defined as:T  obs
1 − T  obs
τ̃ (X ,T , Y obs ) =Y− f (X , 1) −Y− f (X , 0)e(X )1 − e(X )+ (f (X , 1) − f (X , 0))(5)where, the loss function to derive a hypothesis h and a representation function Φ is:n1 Õ (t )h, Φ = minw (x i ) · L (h (Φ (x i ) , ti ) , yi ) + λ · Ω(h)h,Φ ni=1+ α · IPMG {Φ (x i )}i:ti =0 , {Φ (x i )}i:ti =1(6)where, L(·, ·) is squared loss, Ω(h) is the regularization term formodel complexity, and IPMG {Φ (x i )}i:ti =0 , {Φ (x i )}i:ti =1 is calledIntegral Probability Metric (IPM) measuring distance between twot (1−2e(x ))+e(x )2

distributions [6, 7]. Finally, w (t ) (x) = e(x )(1−e(x ))function depending on the propensity score.

3.2

is a weighting

Summary of Theoretical Results

Candidate Predictors: We constructed a set of candidate predictors M by combining five machine learning algorithms3 and fivemeta-learners implemented in EconML package4 . Thus, |M | = 25.
Baselines: We compared CF-CV with the following baseline metrics. (1) IPW validation: This metric is proposed in Section 4.2 of [2]and Section 2.3 of [5]. (2) Plug-in validation: It uses predicted valuesby an arbitrary ITE prediction model as τ̃ (·) in Eq. (4). We used
Counterfactual Regression [6] for τ̃ (·) to ensure fair comparison. (3)τ -risk: This metric is proposed in [5]. We used Gradient BoostingRegressor to estimate observed outcomes (Y obs ).
Results: Table1 reports the averaged and the worst-case performance over 30 realizations. Rank Correlation is the Spearman rankcorrelation between the ranking by the true performance and theestimated metric values. Relative Root Mean Squared Error (RMSE)5is the true performance of the selected model in each metric relative to the best one in M. The results show the effective model
selection performance of our CF-CV. Moreover, ours significantly
outperformed with respect to the worst-case performance, and thisempirically suggests the stability of our metric.
Table 1: Experimental results on IHDP over 30 realizations.

Here, we state critical theoretical results for our proposed oracle
and the resulting performance estimator 1 .
IPWτ -riskPlug-inCF-CV

Current Theoretical Results
1. The performance estimator based on our oracle in Eq. (5)preserves the difference between the true metric values; thepredictor having the smallest expected value of our performance estimator among candidate predictors also has thesmallest value of Rt r ue among them.
2. Our oracle minimizes the upper bound of the finite sample
uncertainty term in the empirical version of the performance
estimator in Eq. (4).
Thus, our oracle is desirable because the performance estimatorusing our oracle is expected to preserve the difference of the true
performance metric and minimizes the upper bound of the finite
sample uncertainty; one can expect to select the best ITE predictor
among a set of candidates with high confidence.

4EARLY EXPERIMENTAL RESULT

Here we report the early experimental results on a semi-synthetic
dataset comparing the model selection performance of our CF-CV
with previous baselines.
Datasets and Setup: We used IHDP2 dataset provided by [3]. Thisis a semi-synthetic dataset containing 747 children with 25 features.
The detailed description of this dataset can be found in Section 5.1of [6]. We follow the experimental procedure in [5]; each metricevaluated and ranked pre-trained ITE predictors using only observational validation set, and the performance of each metric wasevaluated by the estimated performance of the candidate predictors.
We conducted the experimental procedure over 30 realizations with
35/35/30 train/validation/test splits.
1 Formal description of the theoretical results and proofs will be provided in the fullversion of the paper.
2 the Infant Health Development Program.

5Rank Correlation
Avg (± SE)
Worst
0.224 (±0.073) −0.659−0.399 (±0.051) −0.797
0.887 (±0.021)
0.385
0.929 (±0.008)
0.830

Relative RMSE
Avg (± SE)
Worst
2.027 (±0.242) 7.779
3.408 (±0.250) 8.884
1.123 (±0.039) 1.841
1.040 (±0.019) 1.515

CONCLUSION

This paper studies the evaluation problem of ITE prediction models.
We proposed the counterfactual cross-validation procedure, ensuring
accurate model selection with high confidence. The experimental
results on IHDP dataset show the effectiveness of our metric.

REFERENCE[1] Heejung Bang and James M Robins. 2005. Doubly robust estimation in missing
data and causal inference models. Biometrics 61, 4 (2005), 962–973.
[2] Pierre Gutierrez and Jean-Yves Gérardy. 2017. Causal inference and uplift modelling: A review of the literature. In International Conference on Predictive Applications and APIs. 1–13.
[3] Jennifer L Hill. 2011. Bayesian nonparametric modeling for causal inference.
Journal of Computational and Graphical Statistics 20, 1 (2011), 217–240.
[4] Guido W Imbens and Donald B Rubin. 2015. Causal inference in statistics, social,and biomedical sciences. Cambridge University Press.
[5] Alejandro Schuler, Michael Baiocchi, Robert Tibshirani, and Nigam Shah. 2018. A
comparison of methods for model selection when estimating individual treatment
effects. arXiv preprint arXiv:1804.05146 (2018).
[6] Uri Shalit, Fredrik D Johansson, and David Sontag. 2017. Estimating individual
treatment effect: generalization bounds and algorithms. In Proceedings of the 34th
International Conference on Machine Learning-Volume 70. JMLR. org, 3076–3085.
[7] Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf,
Gert RG Lanckriet, et al. 2012. On the empirical estimation of integral probabilitymetrics. Electronic Journal of Statistics 6 (2012), 1550–1599.

3 Decision Tree, Random Forest, Gradient Boosting Tree, Ridge Regressor, and SupportVector Regressor with RBF kernel.
4 https://github.com/microsoft/EconML/blob/master/econml/meta learners.py
5 We used Relative RMSE because potential outcomes of IHDP dataset have different
scales among realizations.