Package: econml
EconML: A Python Package for ML-Based Heterogeneous Treatment Effects EstimationEconMLis a Python package for estimating heterogeneous treatment effects from observational data via machine learning. This package was designed and built as part of theALICE projectat Microsoft Research with the goal to combine state-of-the-art machine learning
techniques with econometrics to bring automation to complex causal inference problems. The promise of EconML:Implement recent techniques in the literature at the intersection of econometrics and machine learningMaintain flexibility in modeling the effect heterogeneity (via techniques such as random forests, boosting, lasso and neural nets), while preserving the causal interpretation of the learned model and often offering valid confidence intervalsUse a unified APIBuild on standard Python packages for Machine Learning and Data AnalysisOne of the biggest promises of machine learning is to automate decision making in a multitude of domains. At the core of many data-driven personalized decision scenarios is the estimation of heterogeneous treatment effects: what is the causal effect of an intervention on an outcome of interest for a sample with a particular set of features? In a nutshell, this toolkit is designed to measure the causal effect of some treatment variable(s)Ton an outcome
variableY, controlling for a set of featuresX, Wand how does that effect vary as a function ofX. The methods implemented are applicable even with observational (non-experimental or historical) datasets. For the estimation results to have a causal interpretation, some methods assume no unobserved confounders (i.e. there is no unobserved variable not included inX, Wthat simultaneously has an effect on bothTandY), while others assume access to an instrumentZ(i.e. an observed variableZthat has an effect on the treatmentTbut no direct effect on the outcomeY). Most methods provide confidence intervals and inference results.For detailed information about the package, consult the documentation athttps://econml.azurewebsites.net/.For information on use cases and background material on causal inference and heterogeneous treatment effects see our webpage athttps://www.microsoft.com/en-us/research/project/econml/Table of ContentsNewsGetting StartedInstallationUsage ExamplesEstimation MethodsInterpretabilityCausal Model Selection and Cross-ValidationInferencePolicy LearningFor DevelopersRunning the testsGenerating the documentationBlogs and PublicationsCitationContributing and FeedbackCommunityReferencesNewsFebruary 12, 2024:Release v0.15.0, see release notesherePrevious releasesNovember 11, 2023:Release v0.15.0b1, see release noteshereMay 19, 2023:Release v0.14.1, see release noteshereNovember 16, 2022:Release v0.14.0, see release noteshereJune 17, 2022:Release v0.13.1, see release noteshereJanuary 31, 2022:Release v0.13.0, see release noteshereAugust 13, 2021:Release v0.12.0, see release noteshereAugust 5, 2021:Release v0.12.0b6, see release noteshereAugust 3, 2021:Release v0.12.0b5, see release noteshereJuly 9, 2021:Release v0.12.0b4, see release noteshereJune 25, 2021:Release v0.12.0b3, see release noteshereJune 18, 2021:Release v0.12.0b2, see release noteshereJune 7, 2021:Release v0.12.0b1, see release noteshereMay 18, 2021:Release v0.11.1, see release noteshereMay 8, 2021:Release v0.11.0, see release noteshereMarch 22, 2021:Release v0.10.0, see release noteshereMarch 11, 2021:Release v0.9.2, see release noteshereMarch 3, 2021:Release v0.9.1, see release noteshereFebruary 20, 2021:Release v0.9.0, see release noteshereJanuary 20, 2021:Release v0.9.0b1, see release noteshereNovember 20, 2020:Release v0.8.1, see release noteshereNovember 18, 2020:Release v0.8.0, see release noteshereSeptember 4, 2020:Release v0.8.0b1, see release noteshereMarch 6, 2020:Release v0.7.0, see release noteshereFebruary 18, 2020:Release v0.7.0b1, see release noteshereJanuary 10, 2020:Release v0.6.1, see release noteshereDecember 6, 2019:Release v0.6, see release noteshereNovember 21, 2019:Release v0.5, see release noteshere.June 3, 2019:Release v0.4, see release noteshere.May 3, 2019:Release v0.3, see release noteshere.April 10, 2019:Release v0.2, see release noteshere.March 6, 2019:Release v0.1, welcome to have a try and provide feedback.Getting StartedInstallationInstall the latest release fromPyPI:pip install econmlTo install from source, seeFor Developerssection below.Usage ExamplesEstimation MethodsDouble Machine Learning (aka RLearner) (click to expand)Linear final stagefromeconml.dmlimportLinearDMLfromsklearn.linear_modelimportLassoCVfromeconml.inferenceimportBootstrapInferenceest=LinearDML(model_y=LassoCV(),model_t=LassoCV())### Estimate with OLS confidence intervalsest.fit(Y,T,X=X,W=W)# W -> high-dimensional confounders, X -> featurestreatment_effects=est.effect(X_test)lb,ub=est.effect_interval(X_test,alpha=0.05)# OLS confidence intervals### Estimate with bootstrap confidence intervalsest.fit(Y,T,X=X,W=W,inference='bootstrap')# with default bootstrap parametersest.fit(Y,T,X=X,W=W,inference=BootstrapInference(n_bootstrap_samples=100))# or customizedlb,ub=est.effect_interval(X_test,alpha=0.05)# Bootstrap confidence intervalsSparse linear final stagefromeconml.dmlimportSparseLinearDMLfromsklearn.linear_modelimportLassoCVest=SparseLinearDML(model_y=LassoCV(),model_t=LassoCV())est.fit(Y,T,X=X,W=W)# X -> high dimensional featurestreatment_effects=est.effect(X_test)lb,ub=est.effect_interval(X_test,alpha=0.05)# Confidence intervals via debiased lassoGeneric Machine Learning last stagefromeconml.dmlimportNonParamDMLfromsklearn.ensembleimportRandomForestRegressor,RandomForestClassifierest=NonParamDML(model_y=RandomForestRegressor(),model_t=RandomForestClassifier(),model_final=RandomForestRegressor(),discrete_treatment=True)est.fit(Y,T,X=X,W=W)treatment_effects=est.effect(X_test)Dynamic Double Machine Learning (click to expand)fromeconml.panel.dmlimportDynamicDML# Use defaultsest=DynamicDML()# Or specify hyperparametersest=DynamicDML(model_y=LassoCV(cv=3),model_t=LassoCV(cv=3),cv=3)est.fit(Y,T,X=X,W=None,groups=groups,inference="auto")# Effectstreatment_effects=est.effect(X_test)# Confidence intervalslb,ub=est.effect_interval(X_test,alpha=0.05)Causal Forests (click to expand)fromeconml.dmlimportCausalForestDMLfromsklearn.linear_modelimportLassoCV# Use defaultsest=CausalForestDML()# Or specify hyperparametersest=CausalForestDML(criterion='het',n_estimators=500,min_samples_leaf=10,max_depth=10,max_samples=0.5,discrete_treatment=False,model_t=LassoCV(),model_y=LassoCV())est.fit(Y,T,X=X,W=W)treatment_effects=est.effect(X_test)# Confidence intervals via Bootstrap-of-Little-Bags for forestslb,ub=est.effect_interval(X_test,alpha=0.05)Orthogonal Random Forests (click to expand)fromeconml.orfimportDMLOrthoForest,DROrthoForestfromeconml.sklearn_extensions.linear_modelimportWeightedLasso,WeightedLassoCV# Use defaultsest=DMLOrthoForest()est=DROrthoForest()# Or specify hyperparametersest=DMLOrthoForest(n_trees=500,min_leaf_size=10,max_depth=10,subsample_ratio=0.7,lambda_reg=0.01,discrete_treatment=False,model_T=WeightedLasso(alpha=0.01),model_Y=WeightedLasso(alpha=0.01),model_T_final=WeightedLassoCV(cv=3),model_Y_final=WeightedLassoCV(cv=3))est.fit(Y,T,X=X,W=W)treatment_effects=est.effect(X_test)# Confidence intervals via Bootstrap-of-Little-Bags for forestslb,ub=est.effect_interval(X_test,alpha=0.05)Meta-Learners (click to expand)XLearnerfromeconml.metalearnersimportXLearnerfromsklearn.ensembleimportGradientBoostingClassifier,GradientBoostingRegressorest=XLearner(models=GradientBoostingRegressor(),propensity_model=GradientBoostingClassifier(),cate_models=GradientBoostingRegressor())est.fit(Y,T,X=np.hstack([X,W]))treatment_effects=est.effect(np.hstack([X_test,W_test]))# Fit with bootstrap confidence interval construction enabledest.fit(Y,T,X=np.hstack([X,W]),inference='bootstrap')treatment_effects=est.effect(np.hstack([X_test,W_test]))lb,ub=est.effect_interval(np.hstack([X_test,W_test]),alpha=0.05)# Bootstrap CIsSLearnerfromeconml.metalearnersimportSLearnerfromsklearn.ensembleimportGradientBoostingRegressorest=SLearner(overall_model=GradientBoostingRegressor())est.fit(Y,T,X=np.hstack([X,W]))treatment_effects=est.effect(np.hstack([X_test,W_test]))TLearnerfromeconml.metalearnersimportTLearnerfromsklearn.ensembleimportGradientBoostingRegressorest=TLearner(models=GradientBoostingRegressor())est.fit(Y,T,X=np.hstack([X,W]))treatment_effects=est.effect(np.hstack([X_test,W_test]))Doubly Robust Learners (click to expand)Linear final stagefromeconml.drimportLinearDRLearnerfromsklearn.ensembleimportGradientBoostingRegressor,GradientBoostingClassifierest=LinearDRLearner(model_propensity=GradientBoostingClassifier(),model_regression=GradientBoostingRegressor())est.fit(Y,T,X=X,W=W)treatment_effects=est.effect(X_test)lb,ub=est.effect_interval(X_test,alpha=0.05)Sparse linear final stagefromeconml.drimportSparseLinearDRLearnerfromsklearn.ensembleimportGradientBoostingRegressor,GradientBoostingClassifierest=SparseLinearDRLearner(model_propensity=GradientBoostingClassifier(),model_regression=GradientBoostingRegressor())est.fit(Y,T,X=X,W=W)treatment_effects=est.effect(X_test)lb,ub=est.effect_interval(X_test,alpha=0.05)Nonparametric final stagefromeconml.drimportForestDRLearnerfromsklearn.ensembleimportGradientBoostingRegressor,GradientBoostingClassifierest=ForestDRLearner(model_propensity=GradientBoostingClassifier(),model_regression=GradientBoostingRegressor())est.fit(Y,T,X=X,W=W)treatment_effects=est.effect(X_test)lb,ub=est.effect_interval(X_test,alpha=0.05)Double Machine Learning with Instrumental Variables (click to expand)Orthogonal instrumental variable learnerfromeconml.iv.dmlimportOrthoIVest=OrthoIV(projection=False,discrete_treatment=True,discrete_instrument=True)est.fit(Y,T,Z=Z,X=X,W=W)treatment_effects=est.effect(X_test)lb,ub=est.effect_interval(X_test,alpha=0.05)# OLS confidence intervalsNonparametric double machine learning with instrumental variablefromeconml.iv.dmlimportNonParamDMLIVest=NonParamDMLIV(projection=False,discrete_treatment=True,discrete_instrument=True)est.fit(Y,T,Z=Z,X=X,W=W)# no analytical confidence interval availabletreatment_effects=est.effect(X_test)Doubly Robust Machine Learning with Instrumental Variables (click to expand)Linear final stagefromeconml.iv.drimportLinearDRIVest=LinearDRIV(discrete_instrument=True,discrete_treatment=True)est.fit(Y,T,Z=Z,X=X,W=W)treatment_effects=est.effect(X_test)lb,ub=est.effect_interval(X_test,alpha=0.05)# OLS confidence intervalsSparse linear final stagefromeconml.iv.drimportSparseLinearDRIVest=SparseLinearDRIV(discrete_instrument=True,discrete_treatment=True)est.fit(Y,T,Z=Z,X=X,W=W)treatment_effects=est.effect(X_test)lb,ub=est.effect_interval(X_test,alpha=0.05)# Debiased lasso confidence intervalsNonparametric final stagefromeconml.iv.drimportForestDRIVest=ForestDRIV(discrete_instrument=True,discrete_treatment=True)est.fit(Y,T,Z=Z,X=X,W=W)treatment_effects=est.effect(X_test)# Confidence intervals via Bootstrap-of-Little-Bags for forestslb,ub=est.effect_interval(X_test,alpha=0.05)Linear intent-to-treat (discrete instrument, discrete treatment)fromeconml.iv.drimportLinearIntentToTreatDRIVfromsklearn.ensembleimportGradientBoostingRegressor,GradientBoostingClassifierest=LinearIntentToTreatDRIV(model_y_xw=GradientBoostingRegressor(),model_t_xwz=GradientBoostingClassifier(),flexible_model_effect=GradientBoostingRegressor())est.fit(Y,T,Z=Z,X=X,W=W)treatment_effects=est.effect(X_test)lb,ub=est.effect_interval(X_test,alpha=0.05)# OLS confidence intervalsDeep Instrumental Variables (click to expand)importkerasfromeconml.iv.nnetimportDeepIVtreatment_model=keras.Sequential([keras.layers.Dense(128,activation='relu',input_shape=(2,)),keras.layers.Dropout(0.17),keras.layers.Dense(64,activation='relu'),keras.layers.Dropout(0.17),keras.layers.Dense(32,activation='relu'),keras.layers.Dropout(0.17)])response_model=keras.Sequential([keras.layers.Dense(128,activation='relu',input_shape=(2,)),keras.layers.Dropout(0.17),keras.layers.Dense(64,activation='relu'),keras.layers.Dropout(0.17),keras.layers.Dense(32,activation='relu'),keras.layers.Dropout(0.17),keras.layers.Dense(1)])est=DeepIV(n_components=10,# Number of gaussians in the mixture density networks)m=lambdaz,x:treatment_model(keras.layers.concatenate([z,x])),# Treatment modelh=lambdat,x:response_model(keras.layers.concatenate([t,x])),# Response modeln_samples=1# Number of samples used to estimate the response)est.fit(Y,T,X=X,Z=Z)# Z -> instrumental variablestreatment_effects=est.effect(X_test)See theReferencessection for more details.InterpretabilityTree Interpreter of the CATE model (click to expand)fromeconml.cate_interpreterimportSingleTreeCateInterpreterintrp=SingleTreeCateInterpreter(include_model_uncertainty=True,max_depth=2,min_samples_leaf=10)# We interpret the CATE model's behavior based on the features used for heterogeneityintrp.interpret(est,X)# Plot the treeplt.figure(figsize=(25,5))intrp.plot(feature_names=['A','B','C','D'],fontsize=12)plt.show()Policy Interpreter of the CATE model (click to expand)fromeconml.cate_interpreterimportSingleTreePolicyInterpreter# We find a tree-based treatment policy based on the CATE modelintrp=SingleTreePolicyInterpreter(risk_level=0.05,max_depth=2,min_samples_leaf=1,min_impurity_decrease=.001)intrp.interpret(est,X,sample_treatment_costs=0.2)# Plot the treeplt.figure(figsize=(25,5))intrp.plot(feature_names=['A','B','C','D'],fontsize=12)plt.show()SHAP values for the CATE model (click to expand)importshapfromeconml.dmlimportCausalForestDMLest=CausalForestDML()est.fit(Y,T,X=X,W=W)shap_values=est.shap_values(X)shap.summary_plot(shap_values['Y0']['T0'])Causal Model Selection and Cross-ValidationCausal model selection with the `RScorer` (click to expand)fromeconml.scoreimportRScorer# split data in train-validationX_train,X_val,T_train,T_val,Y_train,Y_val=train_test_split(X,T,y,test_size=.4)# define list of CATE estimators to select amongreg=lambda:RandomForestRegressor(min_samples_leaf=20)clf=lambda:RandomForestClassifier(min_samples_leaf=20)models=[('ldml',LinearDML(model_y=reg(),model_t=clf(),discrete_treatment=True,cv=3)),('xlearner',XLearner(models=reg(),cate_models=reg(),propensity_model=clf())),('dalearner',DomainAdaptationLearner(models=reg(),final_models=reg(),propensity_model=clf())),('slearner',SLearner(overall_model=reg())),('drlearner',DRLearner(model_propensity=clf(),model_regression=reg(),model_final=reg(),cv=3)),('rlearner',NonParamDML(model_y=reg(),model_t=clf(),model_final=reg(),discrete_treatment=True,cv=3)),('dml3dlasso',DML(model_y=reg(),model_t=clf(),model_final=LassoCV(cv=3,fit_intercept=False),discrete_treatment=True,featurizer=PolynomialFeatures(degree=3),cv=3))]# fit cate models on train datamodels=[(name,mdl.fit(Y_train,T_train,X=X_train))forname,mdlinmodels]# score cate models on validation datascorer=RScorer(model_y=reg(),model_t=clf(),discrete_treatment=True,cv=3,mc_iters=2,mc_agg='median')scorer.fit(Y_val,T_val,X=X_val)rscore=[scorer.score(mdl)for_,mdlinmodels]# select the best modelmdl,_=scorer.best_model([mdlfor_,mdlinmodels])# create weighted ensemble model based on score performancemdl,_=scorer.ensemble([mdlfor_,mdlinmodels])First Stage Model Selection (click to expand)First stage models can be selected either by passing in cross-validated models (e.g.sklearn.linear_model.LassoCV) to EconML's estimators or perform the first stage model selection outside of EconML and pass in the selected model. Unless selecting among a large set of hyperparameters, choosing first stage models externally is the preferred method due to statistical and computational advantages.fromeconml.dmlimportLinearDMLfromsklearnimportclonefromsklearn.ensembleimportRandomForestRegressorfromsklearn.model_selectionimportGridSearchCVcv_model=GridSearchCV(estimator=RandomForestRegressor(),param_grid={"max_depth":[3,None],"n_estimators":(10,30,50,100,200),"max_features":(2,4,6),},cv=5,)# First stage model selection within EconML# This is more direct, but computationally and statistically less efficientest=LinearDML(model_y=cv_model,model_t=cv_model)# First stage model selection ouside of EconML# This is the most efficient, but requires boilerplate codemodel_t=clone(cv_model).fit(W,T).best_estimator_model_y=clone(cv_model).fit(W,Y).best_estimator_est=LinearDML(model_y=model_t,model_t=model_y)InferenceWhenever inference is enabled, then one can get a more structureInferenceResultsobject with more elaborate inference information, such
as p-values and z-statistics. When the CATE model is linear and parametric, then asummary()method is also enabled. For instance:fromeconml.dmlimportLinearDML# Use defaultsest=LinearDML()est.fit(Y,T,X=X,W=W)# Get the effect inference summary, which includes the standard error, z test score, p value, and confidence interval given each sample X[i]est.effect_inference(X_test).summary_frame(alpha=0.05,value=0,decimals=3)# Get the population summary for the entire sample Xest.effect_inference(X_test).population_summary(alpha=0.1,value=0,decimals=3,tol=0.001)#  Get the parameter inference summary for the final modelest.summary()Example Output (click to expand)# Get the effect inference summary, which includes the standard error, z test score, p value, and confidence interval given each sample X[i]est.effect_inference(X_test).summary_frame(alpha=0.05,value=0,decimals=3)# Get the population summary for the entire sample Xest.effect_inference(X_test).population_summary(alpha=0.1,value=0,decimals=3,tol=0.001)#  Get the parameter inference summary for the final modelest.summary()Policy LearningYou can also perform direct policy learning from observational data, using the doubly robust method for offline
policy learning. These methods directly predict a recommended treatment, without internally fitting an explicit
model of the conditional average treatment effect.Doubly Robust Policy Learning (click to expand)fromeconml.policyimportDRPolicyTree,DRPolicyForestfromsklearn.ensembleimportRandomForestRegressor# fit a single binary decision tree policypolicy=DRPolicyTree(max_depth=1,min_impurity_decrease=0.01,honest=True)policy.fit(y,T,X=X,W=W)# predict the recommended treatmentrecommended_T=policy.predict(X)# plot the binary decision treeplt.figure(figsize=(10,5))policy.plot()# get feature importancesimportances=policy.feature_importances_# fit a binary decision forestpolicy=DRPolicyForest(max_depth=1,min_impurity_decrease=0.01,honest=True)policy.fit(y,T,X=X,W=W)# predict the recommended treatmentrecommended_T=policy.predict(X)# plot the first tree in the ensembleplt.figure(figsize=(10,5))policy.plot(0)# get feature importancesimportances=policy.feature_importances_To see more complex examples, go to thenotebookssection of the repository. For a more detailed description of the treatment effect estimation algorithms, see the EconMLdocumentation.For DevelopersYou can get started by cloning this repository. We usesetuptoolsfor building and distributing our package.
We rely on some recent features of setuptools, so make sure to upgrade to a recent version withpip install setuptools --upgrade.  Then from your local copy of the repository you can runpip install -e .to get started (but depending on what you're doing you might want to install with extras instead, likepip install -e .[plt]if you want to use matplotlib integration, or you can usepip install -e .[all]to include all extras).Running the testsThis project usespytestfor testing.  To run tests locally after installing the package, you can usepip install pytest-runnerfollowed bypython setup.py pytest.We have added pytest marks to some tests to make it easier to run a subset, and you can set the PYTEST_ADDOPTS environment variable to take advantage of this.  For instance, you can set it to-m "not (notebook or automl)"to skip notebook and automl tests that have some additional dependencies.Generating the documentationThis project's documentation is generated viaSphinx.  Note that we usegraphviz'sdotapplication to produce some of the images in our documentation, so you should make sure thatdotis installed and in your path.To generate a local copy of the documentation from a clone of this repository, just runpython setup.py build_sphinx -W -E -a, which will build the documentation and place it under thebuild/sphinx/htmlpath.The reStructuredText files that make up the documentation are stored in thedocs directory; module documentation is automatically generated by the Sphinx build process.Release processWe use GitHub Actions to build and publish the package and documentation.  To create a new release, an admin should perform the following steps:Update the version number ineconml/_version.pyand add a mention of the new version in the news section of this file and commit the changes.Manually run the publish_package.yml workflow to build and publish the package to PyPI.Manually run the publish_docs.yml workflow to build and publish the documentation.Underhttps://github.com/py-why/EconML/releases, create a new release with a corresponding tag, and update the release notes.Blogs and PublicationsJune 2019:Treatment Effects with Instruments paperMay 2019:Open Data Science Conference Workshop2018:Orthogonal Random Forests paper2017:DeepIV paperCitationIf you use EconML in your research, please cite us as follows:Keith Battocchi, Eleanor Dillon, Maggie Hei, Greg Lewis, Paul Oka, Miruna Oprescu, Vasilis Syrgkanis.EconML: A Python Package for ML-Based Heterogeneous Treatment Effects Estimation.https://github.com/py-why/EconML, 2019. Version 0.x.BibTex:@misc{econml,
  author={Keith Battocchi, Eleanor Dillon, Maggie Hei, Greg Lewis, Paul Oka, Miruna Oprescu, Vasilis Syrgkanis},
  title={{EconML}: {A Python Package for ML-Based Heterogeneous Treatment Effects Estimation}},
  howpublished={https://github.com/py-why/EconML},
  note={Version 0.x},
  year={2019}
}Contributing and FeedbackThis project welcomes contributions and suggestions.  We use theDCO botto enforce aDeveloper Certificate of Originwhich requires users to sign-off on their commits.  This is a simple way to certify that you wrote or otherwise have the right to submit the code you are contributing to the project.  Git provides a-scommand line option to include this automatically when you commit viagit commit.When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.This project has adopted thePyWhy Code of Conduct.CommunityEconML is a part ofPyWhy, an organization with a mission to build an open-source ecosystem for causal machine learning.PyWhy also has aDiscord, which serves as a space for like-minded casual machine learning researchers and practitioners of all experience levels to come together to ask and answer questions, discuss new features, and share ideas.We invite you to join us at regular office hours and community calls in the Discord.ReferencesAthey, Susan, and Stefan Wager.Policy learning with observational data.Econometrica 89.1, 133-161, 2021.X Nie, S Wager.Quasi-Oracle Estimation of Heterogeneous Treatment Effects.Biometrika 108.2, 299-319, 2021.V. Syrgkanis, V. Lei, M. Oprescu, M. Hei, K. Battocchi, G. Lewis.Machine Learning Estimation of Heterogeneous Treatment Effects with Instruments.Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS), 2019.(Spotlight Presentation)D. Foster, V. Syrgkanis.Orthogonal Statistical Learning.Proceedings of the 32nd Annual Conference on Learning Theory (COLT), 2019.(Best Paper Award)M. Oprescu, V. Syrgkanis and Z. S. Wu.Orthogonal Random Forest for Causal Inference.Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.S. Künzel, J. Sekhon, J. Bickel and B. Yu.Metalearners for estimating heterogeneous treatment effects using machine learning.Proceedings of the national academy of sciences, 116(10), 4156-4165, 2019.S. Athey, J. Tibshirani, S. Wager.Generalized random forests.Annals of Statistics, 47, no. 2, 1148--1178, 2019.V. Chernozhukov, D. Nekipelov, V. Semenova, V. Syrgkanis.Plug-in Regularized Estimation of High-Dimensional Parameters in Nonlinear Semiparametric Models.Arxiv preprint arxiv:1806.04823, 2018.S. Wager, S. Athey.Estimation and Inference of Heterogeneous Treatment Effects using Random Forests.Journal of the American Statistical Association, 113:523, 1228-1242, 2018.Jason Hartford, Greg Lewis, Kevin Leyton-Brown, and Matt Taddy.Deep IV: A flexible approach for counterfactual prediction.Proceedings of the 34th International Conference on Machine Learning, ICML'17, 2017.V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, and a. W. Newey.Double Machine Learning for Treatment and Causal Parameters.ArXiv preprint arXiv:1608.00060, 2016.Dudik, M., Erhan, D., Langford, J., & Li, L.Doubly robust policy evaluation and optimization.Statistical Science, 29(4), 485-511, 2014.