Generic Machine Learning Inference on Heterogeneous Treatment Effects in Randomized Experiments, with an Application to Immunization in India

We propose strategies to estimate and make inference on key features of heterogeneous effects in randomized experiments. These key features include best linear predictors of the effects on machine learning proxies, average effects sorted by impact groups, and average characteristics of most and least impacted units. The approach is valid in high dimensional settings, where the effects are proxied by machine learning methods. We post-process these proxies into the estimates of the key features. Our approach is generic, it can be used in conjunction with penalized methods, deep and shallowneural networks, canonical and newrandom forests, boosted trees, and ensemble methods. Estimation and inference are based on repeated data splitting to avoid overfitting and achieve validity. For inference, we take medians of p-values and medians of confidence intervals, resulting from many different data splits, and then adjust their nominal level to guarantee uniform validity. This variational inference method, which quantifies the uncertainty coming from both parameter estimation and data splitting, is shown to be uniformly valid for a large class of data generating processes. We illustrate the use of the approach with a randomized field experiment that evaluated a combination of nudges to stimulate demand for immunization in India. Victor Chernozhukov Department of Economics Massachusetts Institute of Technology 77 Massachusetts Avenue Cambridge, Mass. 02139 vchern@mit.edu Mert Demirer Department of Economics Massachusetts Institute of Technology 77 Massachusetts Avenue E52-300 Cambridge, MA 02139 mdemirer@mit.edu Esther Duflo Department of Economics, E52-544 MIT 77 Massachusetts Avenue Cambridge, MA 02139 and NBER eduflo@mit.edu Iván Fernández-Val Department of Economics Boston University 270 Bay State Rd Boston, MA 02215 ivanf@bu.edu GENERIC MACHINE LEARNING INFERENCE ON HETEROGENOUS TREATMENT EFFECTS IN RANDOMIZED EXPERIMENTS, WITH AN APPLICATION TO IMMUNIZATION IN INDIA VICTOR CHERNOZHUKOV, MERT DEMIRER, ESTHER DUFLO, AND IVÁN FERNÁNDEZ-VAL Abstract. We propose strategies to estimate and make inference on key features of heterogeneous effects in randomized experiments. These key features include best linear predictors of the effects on machine learning proxies, average effects sorted by impact groups, and average characteristics of most and least impacted units. The approach is valid in high dimensional settings, where the effects are proxied by machine learning methods. We post-process these proxies into the estimates of the key features. Our approach is generic, it can be used in conjunction with penalized methods, deep and shallow neural networks, canonical and new random forests, boosted trees, and ensemble methods. Estimation and inference are based on repeated data splitting to avoid overfitting and achieve validity. For inference, we take medians of p-values and medians of confidence intervals, resulting from many different data splits, and then adjust their nominal level to guarantee uniform validity. This variational inference method, which quantifies the uncertainty coming from both parameter estimation and data splitting, is shown to be uniformly valid for a large class of data generating processes. We illustrate the use of the approach with a randomized field experiment that evaluated a combination of nudges to stimulate demand for immunization in India.