Counterfactual Self-Training

Unlike traditional supervised learning , in many settings only partial feedback is available . We may only observe outcomes for the chosen actions , but not the counterfactual outcomes associated with other alternatives . Such settings encompass a wide variety of applications including pricing , online marketing and precision medicine . A key challenge is that observational data are influenced by historical policies deployed in the system , yielding a biased data distribution . We approach this task as a domain adaptation problem and propose a self-training algorithm which imputes outcomes with finite discrete values for finite unseen actions in the observational data to simulate a randomized trial . We offer a theoretical motivation for this approach by providing an upper bound on the generalization error defined on a randomized trial under the self-training objective . We empirically demonstrate the effectiveness of the proposed algorithms on both synthetic and real datasets . 1 INTRODUCTION . Counterfactual inference ( Pearl et al. , 2000 ) attempts to address a question central to many applications - What would be the outcome had an alternative action was chosen ? It may be selecting relevant ads to engage with users in online marketing ( Li et al. , 2010 ) , determining prices that maximize profit in revenue management ( Bertsimas & Kallus , 2016 ) , or designing the most effective personalized treatment for a patient in precision medicine ( Xu et al. , 2016 ) . With observational data , we have access to past actions , their outcomes , and possibly some context , but in many cases not the complete knowledge of the historical policy which gave rise to the action ( Shalit et al. , 2017 ) . Consider a pricing setting in the form targeted promotion . We might record information of a customer ( context ) , promotion offered ( action ) and whether an item was purchased ( outcome ) , but we do not know why a particular promotion was selected . Unlike traditional supervised learning , we only observe feedback for the chosen action in observational data , but not the outcomes associated with other alternatives ( i.e. , in the pricing example , we do not observe what would occur if a different promotion was offered ) . In contrast to the gold standard of a randomized controlled trial , observational data are influenced by historical policy deployed in the system which may over or under represent certain actions , yielding a biased data distribution . A naive but widely used approach is to learn a machine learning algorithm directly from observational data and use it for prediction . This is often referred to as direct method ( DM ) ( Dudı́k et al. , 2014 ) . Failure to account for the bias introduced by historical policy often results in an algorithm which has high accuracy on the data it was trained on , but performs considerably worse under a different policy . For example in the pricing setting , if historically most customers who received high promotion offers bear a certain profile , then a model based on direct method may fail to produce reliable predictions on these customers when low offers are given . To overcome the limitations of direct method , Shalit et al . ( 2017 ) ; Johansson et al . ( 2016 ) ; Lopez et al . ( 2020 ) cast counterfactual learning as a domain adaptation problem , where the source domain is observational data and the target domain is a randomized trial whose assignment of actions follows a uniform distribution for a given context . The key idea is to map contextual features to an embedding space and jointly learn a representation that encourages similarity between these two domains , leading to better counterfactual inference . The embedding is generally learned by a neural network and the estimation of the domain gap is usually slow to compute . Update 𝑓 on target data . In this paper , while we also view counterfactual inference as a domain adaptation problem between observational data and an ideal randomized trial , we take a different approach - instead of estimating the domain gap between the two distributions via an embedding , we explicitly simulate a randomized trial by imputing pseudo-labels for the unobserved actions in the observational data . The optimization process is done by iteratively updating the pseudo-labels and a model that is trained on both the factual and the counterfactual data , as illustrated in Figure 1 . As this method works in a selfsupervised fashion ( Zou et al. , 2018 ; Amini & Gallinari , 2002 ) , we refer to our proposed framework as Counterfactual Self-Training ( CST ) . The contribution of our paper is as follows . First , we propose a novel self-training algorithm for counterfactual inference . To the best of our knowledge , this is the first application of self-training algorithm for learning from observational data . Moreover , in contrast to the existing methods from domain adaption on counterfactual inference , CST is flexible and can work with a wide range of machine learning algorithms , not limited to neural networks . Second , we offer a theoretical motivation of our approach by providing an upper bound on the generalization error defined on a randomized trial under the self-training objective . In other words , we show that the counterfactual self-training algorithm helps minimizing the risk on the target domain . Our theoretical bounds suggest generating pseudo-labels with random imputation , which is a methodological departure from traditional self-training algorithms which impute hard labels . Third , we present comprehensive experiments on several synthetic datasets and three counterfactual learning datasets converted from multi-label classification tasks to evaluate our method against state-of-the-art baselines . In all experiments , CST shows competitive or superior performance against all the baselines . Moreover , our algorithm is easy to optimize with a much faster training time than other baselines . 2 RELATED WORK . Counterfactual policy optimization has received a lot of attention in the machine learning community in the recent years ( Swaminathan & Joachims , 2015a ; Joachims et al. , 2018 ; Shalit et al. , 2017 ; Lopez et al. , 2020 ; Kallus , 2019 ; Kallus & Zhou , 2018 ; Wang et al. , 2019 ) . Most of the proposed algorithms can be divided into two categories : counterfactual risk minimization ( CRM ) and direct method ( DM ) . Both can be used together to construct doubly robust estimators ( Dudı́k et al. , 2014 ) to further improve efficiency . CRM , also known as off-policy learning or batch learning from bandit feedback , typically utilizes inverse propensity weighting ( IPW ) ( Rosenbaum , 1987 ; Rosenbaum & Rubin , 1983 ) to account for the bias in the data . Swaminathan & Joachims ( 2015a ) introduces the CRM principle with a variance regularization term derived from an empirical Bernstein bound ( Maurer & Pontil , 2009 ) for finite samples . In order to reduce the variance of the IPW estimator , Swaminathan & Joachims ( 2015b ) proposes a self-normalized estimator , while BanditNet ( Joachims et al. , 2018 ) utilizes the baseline technique ( Greensmith et al. , 2004 ) in deep nets . As pointed out by Lefortier et al . ( 2016 ) , CRM-based methods tend to struggle with medium to large action spaces in practice . Morever , CRM-based methods generally require a known and stochastic logging policy , along with full support on the action space . When either one of the requirements is violated , Sachdeva et al . ( 2020 ) ; Kang et al . ( 2007 ) observe direct method often demonstrates a more robust performance . When the logging policy is not available , the counterfactual learning problem is often referred to as learning from observational data , which is the setting we focus on . In addition to select the optimal actions , direct method can also be used to identify causal treatment effect ( Künzel et al. , 2019 ) , CST can be viewed as an extention to direct method . Learning from observational data is also closely related to estimating Individualized Treatment Effects ( ITE ) ( Shpitser & Pearl , 2012 ) or conditional average treatment effect ( CATE ) , which is defined as the difference of expected outcomes between two actions , with respect to a given context . The main challenge of identifying ITE is that unlike an ideal randomized trial , observational data is biased and we do not have the access to the counterfactuals . Hill ( 2011 ) uses a bayesian nonparametric algorithm to address this issue . Yoon et al . ( 2018 ) proposes using generative adversarial nets to capture the uncertainty in the counterfactual distributions to facilitate ITE estimation . Johansson et al . ( 2016 ) ; Shalit et al . ( 2017 ) approach counterfactual inference with representation learning and domain adaptation . Their key idea is to learn a representation between observational data and a randomized trial that encourages better generalization on all possible actions . It is achieved by minimizing a weighted sum of the factual loss on the observational data ( loss for direct method ) plus an estimated domain gap measured by integral probability metrics . Lopez et al . ( 2020 ) further extends this framework to multiple treatments using Hilbert-Schmidt Independence Criterion ( HSIC ) ( Gretton et al. , 2008 ) and achieves state-of-the-art performance . The HSIC proposed in Lopez et al . ( 2020 ) has a computation time of at least O ( N2 ) , making its training process slow . While the aforementioned methods and our approach can be viewed as extensions to direct method , we tackle the domain adaptation problem differently by explicitly augmenting the observational data to create a simulated randomized trial via self-training . Different counterfactual estimation algorithms are classified as X- , T- , S-learner in Künzel et al . ( 2019 ) , for example Hill ( 2011 ) is an instance of S-learner . Our approach is similar to X-learner which uses pseudo-label to create counterfactuals , but CST considers multiple instead of binary actions and is trained in an iterative fashion . Self-training algorithms have been widely studied in semi-supervised learning and domain adaptation problems ( Nigam et al. , 2000 ; Amini & Gallinari , 2002 ; Grandvalet & Bengio , 2005 ; Zou et al. , 2019 ; Han et al. , 2019 ) . Grandvalet & Bengio ( 2005 ) proposes to use entropy regularization for semi-supervised learning as a class-overlapping measure to utilize unlabeled data . Nigam et al . ( 2000 ) ; Amini & Gallinari ( 2002 ) ; Zou et al . ( 2019 ) formulate the pseudo-label imputation as classification EM algorithm and show its convergence under proper initialization . Han et al . ( 2019 ) points out that pseudo-label imputation can be viewed as minimizing min-entropy as a type of Rényi entropy 11−α log ( ∑n i=1 p α i ) when α→∞ , and Shannon entropy in Grandvalet & Bengio ( 2005 ) is the case when α → 1 . Self-training is also shown to be effective in semi-supervised learning for many other machine learning models besides neural networks ( Tanha et al. , 2017 ; Li et al. , 2008 ) . It is worthy to mention that unlike traditional self-training where the target domain is given by the problem , we propose to construct a target domain by imputing pseudo-labels on all unseen actions to simulate a pseudo-randomized trial . Moreover , instead of hard labels used in traditional self-training , we propose to use random imputation to create pseudo-labels which have a theoretical motivation tailored for counterfactual inference and are shown to be more effective based on the experiments results .

Summary: The paper proposes to use self-training to tackle the fundamental problem of causal inference where only one potential outcome is seen. The proposed self-training method is iterative: after training a model on the observational dataset, they run points with different actions (treatments) through the trained model and collect the predictions, which are the pseudo-labels. They then continue the training of the model, including the pseudo-labels, until convergence. The paper experiments with two versions of the method -- one with deterministic pseudo-labels (CST-AI) and another with soft pseudo-labels sampled from a probability distribution (CST-RI). It is assumed that there are no unobserved confounders.