Counterfactual Self-Training
Authors: Ruijiang Gao
Max Biggs
Wei Sun
Ligong Han
Venue:  (2021)
Keywords: self-training
counterfactual inference

Abstract: Unlike traditional supervised learning, in many settings only partial feedback is available. We may only observe outcomes for the chosen actions, but not the counterfactual outcomes associated with other  alternatives. Such settings encompass a wide variety of applications including pricing, online marketing and precision medicine. A key challenge is that observational data are influenced by historical policies deployed in the system, yielding a biased data distribution. We approach this task as a domain adaptation problem and propose a self-training algorithm which imputes outcomes for the unseen actions in the observational data to simulate a randomized trial. We offer a theoretical motivation for this approach by providing an upper bound on the generalization error defined on a randomized trial under the self-training objective. We empirically demonstrate the effectiveness of the proposed algorithms on both synthetic and real datasets.

Under review as a conference paper at ICLR 2021
COUNTERFACTUAL SELF-TRAININ
Anonymous authors Paper under double-blind review
ABSTRACT
Unlike traditional supervised learning, in many settings only partial feedback is available. We may only observe outcomes for the chosen actions, but not the coun- terfactual outcomes associated with other alternatives. Such settings encompass a wide variety of applications including pricing, online marketing and precision medicine. A key challenge is that observational data are influenced by historical policies deployed in the system, yielding a biased data distribution. We approach this task domain adaptation problem and propose self-training algorithm which imputes outcomes with finite discrete values for finite unseen actions in the observational data to simulate randomized trial. We offer theoretical motiva- tion for this approach by providing an upper bound on the generalization error defined on randomized trial under the self-training objective. We empirically demonstrate the effectiveness of the proposed algorithms on both synthetic and real datasets.
INTRODUCTION
Counterfactual inference (Pearl al. 2000) attempts to address question central many appli- cations What would be the outcome had an alternative action was chosen? It may be selecting relevant ads to engage with in online marketing (Li al., 22000), determining prices that max- imize profit in revenue management (Bertsimas & Kallus 2016), or designing the most effective personalized treatment for a patient in precision medicine (Xu et al. 2016). With observational data, we have access to past actions, their outcomes, and possibly some context, but in many cases not the complete knowledge the historical policy which gave rise to the action (Shalit al., 2017). Consider a pricing setting in the form targeted promotion. We might record information of cus- tomer (context), promotion offered (action) and whether an item was purchased (outcome), but we do not know why particular promotion was selected.
Unlike traditional supervised learning, we only observe feedback for chosen action in observa- tional data, but not the outcomes associated with other alternatives (i.e., in the pricing example, do not observe what would occur if different promotion was offered). In contrast to the gold stan- dard of a randomized controlled trial, observational data influenced by historical policy deployed in the system which may over or under represent certain actions, yielding a biased data distribution. naive but widely used approach is learn a machine learning algorithm directly from observa- tional data and use it for prediction. This is often referred to as direct method (DM) (Dudik et al.) 2014). Failure account for the bias introduced by historical policy often results an algorithm which has high accuracy on the data it was trained on, but performs considerably worse under different policy. For example the pricing setting, if historically most customers who received high promotion offers bear certain profile, then a model based on direct method may fail to produce reliable predictions on these customers when low offers are given.
To overcome the limitations of direct method, (Shalit et al. (2077); Johansson et al. (2016); Lopez ct al. 2020)) cast counterfactual learning as a domain adaptation problem, where the source domain observational data and the target domain is randomized trial whose assignment of actions fol- lows a uniform distribution for a given context. The key idea is to map contextual features to an embedding space and jointly learn representation that encourages similarity between these two domains, leading to better counterfactual inference. The embedding is generally learned by a neural network and the estimation of the domain gap is usually slow compute.

---

Under review as a conference paper at ICLR 2021
Update on target data
Customer
$1
$2
Customer
$2
$3
A
?
A
1
(x, n/
f f(x,p;0)
px
Pz Pipp
Observational Data
Randomized
Trial
Impute pseudo-labels
Figure 1: Illustration of the proposed Counterfactual Self-Training (CST) framework. There two sales records (observational data) shown in the table, i.e., Customer was offered $2 and bought an item; Customer B was offered $1 and did not buy. The question marks in the tables represent the counterfactual outcome which we do not observe. For all these unseen counterfactual outcomes, pseudo-lahels which colored in red in the tables imputed by model and used to augment the observational data. The model is subsequently updated by training on both the imputed counterfactual data and the factual data. This iterative training procedure continues until converges.
In this paper, while we also view counterfactual inference as a domain adaptation problem between observational data and an ideal randomized trial, we take different approach instead of estimating the domain gap between the two distributions via an embedding, we explicitly simulate a randomized trial by imputing pseudo-labels for the unobserved actions in the observational data. The optimiza- tion process is done by iteratively updating the pseudo-labels and model that is trained on both As this method works in a self-
the factual and the counterfactual data, as illustrated in Figure
[
supervised fashion (Zou et al.) 2018 Amini & Gallinari 2002). we refer to our proposed framework as Counterfactual Trainirainnn (CST).
The contribution of our paper is as follows. First, we propose novel self-training algorithm for counterfactual inference. To the best of our knowledge, this is the first application of self-training algorithm for learning from observational data. Moreover, in contrast to the existing methods from domain adaption on counterfactual inference, CST is flexible and can work with a wide range of ma- chine learning algorithms, not limited to neural networks. Second, offer theoretical motivation of our approach by providing an upper bound on the generalization error defined on randomized trial under the self-training objective. In other words, show that the counterfactual self-training algorithm helps minimizing the risk on the target domain. Our theoretical bounds suggest gener- ating pscudo-labels with random imputation, which methodological departure from traditional self-training algorithms which impute hard labels. Third, we present comprehensive experiments on several synthetic datasets and three counterfactual learning datasets converted from multi-label clas- sification tasks to evaluate our method against sate-of-theear baselines. In all experiments, CST shows competitive or superior performance against all the baselines. Moreover, our algorithm is easy to optimize with much faster training time than other baselines.
RELATED WORK
Counterfactual policy optimization has received a lot of attention in the machine learning com-
munity in the recent years (Swaminathan & Joachims, [201aa] Joachims et al.) 2018) Shalit et al.
2017 Lopez et al.) 2020 Kallus, 2019] Kallus & Zhou, 2018 Wang et al. 2019). Most of the proposed algorithms can be divided into two categories: counterfactual risk minimization (CRM) and direct method (DM). Both can be used together to construct doubly robust estimators (Dudik et al. 2014) to further improve efficiency. CRM, also known as off-policy learning or batch learn- ing from bandit feedback, typically utilizes inverse propensity weighting (IPW) (Rosenbaum 1987
Rooenbaum & Rubin 1983) to account for the bias in the data. Swaminathan & Joachims (2015a)
introduces the CRM principle with variance regularization term derived from an empirical Bemn- stein bound (Maurer & Pontil 2009) for finite samples. In order to reduce the variance of the IPW
2

---

Under review as a conference paper at ICLR 2021
estimator, Swaminathan & Joachims (2015b) proposes self-normalized estimator, while Bandit- Net (Joachims et all.[2018) utilizes the baseline technique (Greensmith et al. (004) in deep nets. As pointed out by Lefortier et al. (2016), CRM-based methods tend to struggle with medium to large action spaces in practice. Morever, CRM-based methods generally require a known and stochastic logging policy, along with full support on the action space. When either one of the requirements is violated, [Sachdeva et al.) (2020); Kang et al.] (2007) observe direct method often demonstrates more robust performance. When the logging policy is not available, the counterfactual learning problem is often referred to as learning from observational data, which is the setting we focus on. In addition to select the optimal actions, direct method can also be used identify causal treatment effect Künzel et al. 2019). CST can be viewed as extention to direct method.
Learning from observational data is also closely related to estimating Individualized Treatment Ef- fects (ITH) (Shpitser & Pearl 2012) or conditional average treatment effect (CATE), which is de- fined as the difference of expected outcomes between two actions, with respect to given context. The main challenge of identifying ITE is that unlike an ideal randomized trial, observational data is biased and we do not have the access to the counterfactuals. Hill 201T) uses a bayesian nonpara- metric algorithm to address this issue. Yoonn et al. (2018) proposes using generative adversarial nets to capture the uncertainty in the counterfactual distributions to facilitate ITE estimation. Dohansson et al. (2016); Shalit et al. (2017) approach counterfactual inference with representation learning and domain adaptation. Their key idea is to learn a representation between observational data and a randomized trial that encourages better generalization on all possible actions. is achieved by min- imizing a weighted sum of the factual loss on the observational data (loss for direct method) plus an estimated domain gap measured by integral probability metrics, Lopez al. (2020) further extends this framework to multiple treatments using Hilbert-Schmin Independence Criterion (HSIC) (Gret- ton et al. 2008) and achieves state-of-the-an performance. The HSIC proposed in Lopez et al. (2200) has computation time of at least O(N2), making its training process slow. While the aforementioned methods and our approach can be viewed as extensions to direct method, we tackle the domain adaptation problem differently by explicitly augmenting the observational data to create a simulated randomized trial via self- tolf-nraining. Different counterfactual estimation algorithms are classified as X-, T-, S-learnr in Künzel al. (2019), for example [Hill (201 1) is instance of S-learner. Our approach is similar to X-learner which uses pseudo-label to create counterfactuals, but CST considers multiple instead of binary actions and is trained in an iterative fashion.
learning and domain adap- tation problems (Nigam et al. 2000 Amini & Gallinari 2002; Grandvalet & Bengio 2005 Zou et al. 2019 Han et al. 2019). Grandvalet & Bengio (2005) proposes to use entropy regularization for sumi-supervised learning as a baesllivera measure utilize unlabeled data. Nigam et al. (2000); Amini & Gallinari (2002); Zou et al. (2019) formulate the pseudo-label imputation as classi- fication EM algorithm and show its convergence under proper initialization. Haan al.) (2019) points out that pseudo-label imputation can be viewed as minimizing min-entropy as a type of Rényi en- tropy I-a log [ogg>> D pf¹) when a + oo, and Shannon entropy in Grandvalet & Bengio 20055 is the case when a + 1. Self-training is also shown to be effective in semi semi-supervised learning for many other machine learning models besides neural networks (Tanha et 2017 et al. 20088. ) It is wor- thy to mention that unlike traditional self-training where the target domain given by the problem, we propose to construct target domain by imputing pseudo-labels on all unseen actions to simulate a pseudo-randomized trial. Moreover, instead of hard labels used in traditional self-training, we pro- pose to use random imputation to create pseudo-labels which have a theoretical motivation tailored for counterfactual inference and are shown to be more effective based on the experiments results.
Self-training algorithms have been widely studied in semi-supervinn
PROBLEM STATEMENT
Following the notation in Lopez et al. 2020), we use X' to represent an abstract space and P(r)) is a probability distribution on X. Each sample x = Z1, In, E X is drawn independently from P(x). P is the discrete action space that central agent can select for each sample, after which discrete reward r with finite possible values is revealed to the agent. In precision medicine, X' may represent patient cohort, P refers to feasible treatment for a disease, and can be the indicator of whether a patient survives after the treatment. Similarly, X, P, r can represent visitors, ads shown and whether visitor clicks in online marketing.
3

---

Under review as a conference paper at ICLR 2021
We focus example of pricing to illustrate method. We use E d P(z) to denote customer. Let P represent finite price options a central agent can offer to customers. After offering price p E P. the agent observes the response from the customer r P(r|æ, p). i.e., either a (buy) or a 0 (no-buy). As a direct method, the task is to learn a function f(x,p) p) by minimizing the loss E Rg= (wn C( (x. r)) where (p|2z) is randomized assignment policy (Shalit al., 2017 Lopez et al., (2200). The estimation task is often referred to as demand estimation (Wales & Woodland 1983). which critical for many downstream decisions such as inventory optimization and revenue management (Kök & Fisher 2007; McGill & Van Ryzin 1999). This is in contrast to CRM- methods which use the final reward as objective learn policy (plzz) that maximizes (===-((a).p><<p(ple)) [[r| p] (Swaminathan & Joachims 2015a).
With observational data, the individualized treatment effect not always identifiable. We use Ru- bin's & potential outcome framework and assume consistency and strong ignorability which is suf- ficient condition for identifying ITE (Imbens Wooldridge 2009 Pearl 2017). Here we formally
present the ignorability assumption (Rubin) 2005) Shalit et al. 2017))
Assumption 3.1 (Ignorability). Let P be action set, x is context (feature), r(p) [r is observed reward for action p P given context x, rp)) 1L pla, Yp E P.
In other words, assume there is unobserved confounders. This condition generally cannot be made purely based on data and requires some domain knowledge.
4 ALGORITHM
In this section, we introduce Counterfactual Self-Trainin (CST) algorithm, which can viewed as an extension of the direct method via domain adaptation. Unlike existing methods using represen- tation learning, we propose a novel self-training style algorithm to account the bias inherent in the observational data.
4.1 SELF-TRAINE
Self-training has recently been used in unsupervised domain adaptation (UDA) and semi-supervised
learning (SSL) and achieved great success (Zou et al. 2019 Han al. 2019 Zou et al. 2018 Amini & Gallinari 2002 Nigam et al.) 2000) Grandvalet & Bengio 2005 The self-training algorithm works in an iterative fashion: First, after training a classifier (z, p) on a source dataset, pscudo- labels are created by the best guess of f. Next, the model is trained on target dataset, and the trained model is used to generate new pseudo labels. This idea is illustrated in Figure
To formulate the counterfactual learning problem as domain adaptation problem, observational data is viewed as data sampled from a source distribution Ds = (plz)) The target domain is randomized trial on the same feature distribution ensure uniformly good approximation actions. Our goal is to transfer observational data from the source domain to a simulated pseudo- randomized trial via self-training. To accomplish this, we first train an initial classifier fo fo p) on observational data, then impute pseudo-labels on all unseen actions from the observation data with isp f(x, p). The model is then updated by training with the following objective:
all
1 min NIP
Σ
Σ Σ
((oo(E,)))Fi.)))
PEPiP
Cave
The first term Lore in Equation corresponds to the loss used in direct method, defined over the factual data alone. Meanwhile, the second term refers to the loss defined over the imputed counter- factual data. In other words, in order get good model across all actions, population induced from imputation which represents a simulated randomized trial. We iteratively train the model and impute pseudo-labels until it converges. The algorithm is stated in Algorithm [
utilize the pseudo-
Note that a key difference between our CST algorithm and traditional self- eeff-training (ST) methods for unsupervised domain adaptation (such in Zou et al. (2018)) Pseudo-labels in traditional ST
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Counterfactual Self- Self-Training
1:
while NOT converged do for each i {1....MM N do for each p E P \ Ps do
Main training loop
2:
3:
4:
Impute pseudo-label fu, ~ fo(r pr)...).......).)))))))))))))(
Pseudo-label imputation
5: end for end for
6:
7: Update 0 by minimizing Lst defined in Equation
Self-training
8:
end while
generated from hard imputation while ours are sampled from a probability distribution as illustrated in Algorithm line Not only this randomized imputation has theoretical motivation presented
in Section 4.2 it also demonstrates superior performance over hard imputation in our experiments
in Section
4.2 THEORETICAL MOTIVATION
As our objective is augment observational data to randomized trial such that the learnt model is able to perform better on all feasible actions, we focus on bounding the generalization error defined on randomized trial. We use D to represent the distribution of true randomized trial where the assignment policy is uniform probability over P given context, and D is the distribution of pseudo- label generated by the current model output fo(1|2, p). Define R. (f) as the risk of function f with respect to loss function ((-,)) as RD( Ex ((([f p)], and (f) empirical risk on Đ. Assume our classifier outputs a probability estimation [r( p) for a feature and action pair (r, p). and we use random imputation & ~ (r(r p) to generate outcomes for the unseen actions. We have the following theorem on the generalization bound:
Theorem 1. Assume fr(r)z, M where Mo > 1 constant, x,p e.ptl is defined on a compact, discrete set respectively, let M min(maxe,p F. 1), Mo). f = argminses Ro(f). D is the dataset generated by random imputation of current model output fe. and f minimizes the empirical risk on D. For any loss function U(:,' )), we have:
Ro(f) - RD(f") (00o(())c((()))))))))))))))))( / V + V log(1 (0g(1/0) + (M - 1) -fo()
(2)
V is the VC dimension of hypothesis class F, and C' is universal constant. The proof is in Ap- pendix A.I By replacing M with My and minimizing the right hand side of Equation 2 over 0, recover Equation Equation [ which is the objective that
optimizing in the training procedure. This complete optimization involves optimizing over f and f, and can be solved via classification EM (CEM) (Amini & Gallinari 2002) and traditional self-training is an instance of CEM (Zou al., 2019). These methods use a hard label as a classification step to impute the pseudo-labels but not clear how it relates the risk that we are interested in. To establish Theorem [ we require a random imputation of labels based on the probability output of the classifier to upper bound the risk under true randomized trial using this objective. Therefore, we use a random sampling to generate pseudo-labels in our algorithm, and it is shown to be more robust than hard labels in our experiments. We would like to note this bound is relatively loose when P is very different from fe- thus we only use it as motivation of our proposed algorithm. Since in the source domain, the data generated by the true data distribution, it is possible to get a tighter upper bound, which we leave for future work.
Mo+1 is also proposed in Zou et al. (0119) in the form of entropy regularization to prevent the model converging too fast and getting too confident in the early stages of training. Since cross-validation is biased in the counterfactual learning due to the logging pol- icy (Lopez et al., 2020), we avoid using hyperparameters and do not use this regularization in our experiments. Note that in traditional semi-supervised learning, deterministic pseudo-labed are com- monly used by argmax operation f = argmax, (r| p). which we refer to as CST-AI, and we refer the with random imputation as CST-RI.
Our assumption fe(r|z.p) >
5

---

Under review as a conference paper at ICLR 2021
In Proposition 1 we show the convergence result of CST-AI. Theoretical analysis convergence of CST-RI is more challenging. However, we empirically observe CST-RI converges without any additional techniques in all of experiments. We show empirical loss curves and discuss the connection between CST-RI and entropy regularization (Grandvalet & Bengio 2005 in Section A.3 in Appendix.
Proposition 1. CST with argmax imputation is convergent under certain conditions.
Proof. Please refer to Section A.2 of Appendix.
5 EXPERIMENTS
We construct synthetic datasets for a pricing example and utilize three real datasets to demonstrate the efficacy of our proposed algorithm. Implementationwis;, we use three layer neural network with 128 nodes as our model and binary entropy loss as the loss function. We avoid using early stop- ping and train each method until convergence ensure fair comparison. The following baselines are considered in our experiments:
Direct Method (DM)): This baseline directly trains model observational data.
HSIC (Lopez et al. 2020): We use the last layer as an embedding and calculate HSIC between the embedding and the actions. The training objective is binary cross entropy loss + X-HSIC, where À is the hyperparameter which we choose from grid search over 10.01, 0,1, 10, 100).
BanditNet (Joachims et al. 2018): BanditNet is CRM-based method developed for deep nets. For the baseline required BanditNet, we normalize the reward as [Swami- nathan & Joachims (2015a) and choose the hyperparameter using a grid search over 0,0.2, 0.8] and cross validation. We fit an additional logging policy model on historical data for BanditNet.
Uniform DM (Wang et al. 2019): Uniform DM (UDM) also estimates the logging policy using historical data and importance sampling simulate a randomize trial.
Since BanditNet is designed for reward maximization, evaluation of the accuracy (i.e., hamming loss) is not appropriate under problem. In each experiment, we only evaluate BanditNet in the reward comparison. We also experiment with two versions of CST, CST-AI and CST-RI. Unlike CST, HSIC and BanditNet require a hyperparameter Joachims et al. (2018) 8): Lopez et al. (2000) we use a 5-fold cross-validation and grid search to select the hyperparameter for all experiments. All experiments are conducted using one NVidia GTX 1080-Ti GPU with five repetitions. Mean and standard error are reported for each metric.
an input their algorithms. Following
5.1 SYNTHETIC DATASETS
In synthetic experiments, we use a pricing example similar to the experiment in Lopez et al. (2020) Let (-)-) be uniform distribution. Assume customer features are a 50-dimensional vector X drawn from (0, 50 and there are 10 price options from $1 to $10. The logging policy set as (pp i|.) = > a; Five types of demand functions are simulated, and the complete data generation process is detailed Appendix A.5
We generate 1000 samples for each demand function and report hamming loss which relies on the hard labels generated by the algorithm in Table [ In addition, as we are interested in probability estimation, report the multi-label soft margin loss in Table 2 Lastly, pricing application, also evaluate the revenue generated on the test set by solving the revenue maximization problem:
p1 argmax Perr 1|x,,p)
For each dataset, the test set has 5000 samples from the corresponding demand distribution. The results are shown in Table 3
Among all datasets, CST-RI has the best performance in terms of both hamming loss and soft margin loss. HSIC outperforms DM baseline by a significant margin and comes as close second to CST- RI. In 4 out of 5 demand functions (with the exception of D1), CST-RI achieves comparable

---

Under review as a conference paper at ICLR 2021
superior performance reward shown in Table 3 Hence, while CST-RI results in the best demand model in terms of the losses, it does not guarantee the highest revenue in all cases. This is because the downstream optimization task is independent from demand estimation (Elmachtoub) & Grigas] 2017). Nevertheless, CST-RI significantly outperforms BanditNet which is designed for reward maximization due unknown logging policy (Kang al. (007). We also want to point that CST-AI performs worse than DM which is a naive baseline, demonstrating the importance of random imputation in our algorithm.
DI
D2
D3
D4
D5
Direct Method
0.2001=000002
0.2418=00006
0.3858+0000036
0.261420.0051
0.1753=000024
CST.RI
0.1066:00.0200
0.1833-0.0030
0.2716--0.0049
0.2006
0.1049=000009
CST-AT
0.4442-00.699 0.5588=0.0498
045444+000522
0.5514+0.0507
0.4769=000053
HSIC
0.1940=0.0036 0.1947=0.0023
0.2821
0.1969=0.0033
0.1447=0.0029
UDM
0.2246:000053
0.2687=0.0114
0.2951+0.0059
0.2059=0.0049
0.1892=#0.0155
Table 1: Hamming Loss on Synthetic Dataset.
D4
D5
Direct Method
18
2.2517:10.174
1.,733
1.2841110.6444
1.9047
CST-RI CIT-AI HSIC UDM
0.3729-40.0047
0.4134=0.0020
0.5230+000004
0.4319:000019
0.3206-=000026
1.2618.10.15
1.5679L001181
1.389610.09888
1.553410.1216
1.3196.10.129
0.44074400134
0.4393=000088
0.5578-40.0077
0.4405-+0.0064
0.3284=0.0050
0.9057
1.012410.011
1.0082-10.0317
0.8481100.0299
0..152.0..482
Table 2: Multi-Label Soft-Margin Loss on Synthetic Datasets
D1
D2
D3
D3
Direst Methxd
4861.6 149.3
4240
108
780666611413
4442.6:124 24.3
323866::133
CST-I
3894.4-118.2
4626.8-166
265.
76220+218.0
4692.6+298.
3381.6+-299.8
CST-AI
3800.6=1133 113.6
1312
697522442176
2084/42228.2
1082.6=:0.7
7692.0176.3
436144-200.
7416.012764
465394
3050.4-76.2
BanditNet
101044106..4
4334.6-114. 144.7
6739.211127
4582.01335.9
2907.2 2907.24-222.2
UDM
3580.8
4430.0=264.4
6014.6:::35.2
4208 24-452.5
1844.8::16.8
6505.2 158.8
5082.010
9527.61133.0
1909,41129,2
423,4111.8
Table 3: Total Reward on Synthetic Dataset. Oracle represents the best possible reward with perfect knowledge of the demand function.
5.2 MULTI-LABEI. DATASETS
We use three multi-label datasets from LIBSVM repository (Elisseeff & Weston 2002 Boutell
al. 2004) Chang & Lin 201T), which are used for semantic scene, text and gene classification.
We convert the supervised learning datasets to bandit feedback by creating a logging policy using 5% of the data following Swaminathan & Joachims (2015a); Lopez et al. (2020). More specifically, each feature has label y 10, where p is the number of labels. After the logging policy selects a label (action) i, a reward yi is revealed as bandit feedback (x. i, y:). i.e., for each data point, if the policy selects one of the correct labels of that data point, it gets reward of 1 and otherwise. By doing so, we have the full knowledge of counterfactual outcomes for evaluation. Data statistics are summarized in Section A. Appendix.
Hamming loss, multi-label soft margin loss and reward are reported in Table 4 5 respectiell... CST-RI generally achieves comparable datasets. Since we assume we do not know the logging policy, BanditNet performs poorly in dataset like Scene, which consistent with Kang al. (2007). HSIC has a comparable performance with CST-RI on TMC and Yeast, but performs poorly on Scene. We suspect this is due to the bias introduced in coss-validation which in turn results in sub-optimal hyperparameter selection. UDM can improve over DM effectively but CST-RI still outperforms it significantly. Overall, CST- RI shows the most robust performance across all three metries being studied.
superior performance against all baselines in all three
5.3 RUNNING TIME ANALYSIS
We compare the average running time for one repetition for each experiment under same number of epochs. The results are summarized in Bection[A.4]pp in Appendix. Unsurprisingly DM is the fastest algorithm. While our method is almost twice as slow DM, it is still relatively fast compared to
7

---

Under review as a conference paper at ICLR 2021
TMC
Yeast
Scene
Direct Method
0.122410.0194
0..669
0.2028:00.0035
CST-RI
0.0775:10.0022
0.2796:10.0011
0.2306::000372
CST-AI HSIC UDM
0.5589410 0336
0.3179::005500
1.2288::022555
0.0797::00016
0.3024::000447
0.4504=t..054
0.0905+-0.0039
0.3547=000007
0.2293+000009
Table 4: Hamming Loss on Real Datasets
TMC
Yeas!
Scene
Direct Method
2.0578+0.20,
9.2483=0.0137
5.
CST-RI
0,23254:.0022
0.5573=0.0048
0,491810.0512
CST-AI HSIC
3.0167+0.0682
0.9718=0.1691
1.2288::0.2255
0.22599+0.0037 0.5898=000019
0.9403++02616
UDM
1.3178+001877
0.0044
3.8074 +0.0444
Table 5: Multi-Labol Soft- Moft-Margin Loss on Real Datasets
TMC
Direct Method
3926,51106.2
457.814.7
587.013.66
CST
5035-+
6.7.8+1.6
60..0-555.3
CST-AI
194.04-99.1
226.6+300.
26490=17.99
HSIC BanditNer UDM
4874.0=241.3
675.421.9
277,8
4302.3.44842
574.8+8.6
243.4-452.0
4408.3+19.3
480.6#2.7
584.6+33.2
Table 6: Reward on Real Datasets
the other baselines. BanditNet is relatively slow due to the cross validation selection. Note that the time efficiency of HSIC is bottlenecked by its high computational complexity (Lopez et al. (020). We thus observe HSIC is approximately 30 to 100 times slower than CST across all datasets. Since CST offers a competitive performance against HSIC with a much faster running time, it is potentially
suitable for large-scale applications which require frequent model update, such as a daily updated pricing system. For example, HSIC may take days for model re- re-training but CST can be updated day-to-day.
CONCLUSION AND FUTURE WORK
In this paper, we proposed novel counterfactual self-training algorithm for learning from obser- vational data. Comparing to existing approaches, our method is easy to compute and optimize. also does not have the need for hyperparameter selection through cross validation, which is biased in nature for observational data. We provided a theoretical analysis showing self-training objective serves as an upper bound of the true risk of randomized trial. However, our CST framework has several limitations. First, CST requires finite discrete action set. In order to augment observation data, CST will augment every action not observed. For continuous action, discretization or joint kernel embedding proposed in Zenati et al. (2020) might be used we leave for future work. Second, CST in this paper can only work with discrete outcomes. the outcome is continuous, is also possible extend framework continuous valued problems by: (1) discretize continuous value into discrete categories; (2) the pseudo-labels can be defined as self-ensemble (French et al.) 2017) predictions, e.g. dropout (Bayesian neural networks) ensemble or temporal ensembling (Laine & Aila 2016).
extension CST, which
While this analysis is tailored for counterfactual learning, hope it shed light broader range of problems such as unsupervised domain adaptation and semi-supervised learning. It may also open doors for solving counterfactual learning with a model-based extrapolation for direct method. As shown in our pricing example, good demand model may not necessarily lead to the highest revenue because the downstream revenue maximization optimization (Elmachtoub & Gri- gas 2017). A different formulation of target domain may help address this problem, which we leave for future work. Moreover, we believe our counterfactual self-training framework can be adapted to yield many specific algorithms for tasks such as learning from observational data with structured reward (Lopez et al. 20200 Kallus, 2019) and deficient historical logging policy (Sachdeva et al.
2020).
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Massih-Reza Amini and Patrick Gallinari. Semi-supervised logistic regression. 2002.
Dimitris Bertsimas and Nathan Kallus. The power and limits of predictive approaches to observational-date-de optimization. arXiv preprint arXiv:/605.02344 2016.
Matthew Boutell, Jiebo Luo, Xipeng Shen, and Christopher M Brown. Learning multi-label scene classification. Pattern recognition, 3709)115771171, 2004.
Chih-Chung Chang and Chih-Jen Lin. Libsvm: library for support vector machines. ACM trans- actions on intelligent systems and technology (TIST), 2(3):1-27, 2011.
Miroslav Dudik, Dumitru Erhan, John Langford. Lihong Li, al. Doubly robust policy evaluation and optimization. Statistical Science, 29(4):485-511, 2014.
André Elisseeff and Jason Weston. A kernel method for multi-labelled classification. In Advances in neural information processing systems, pp. 68 681-687, 2002.
Adam N Elmachtoub and Paul Grigas. arXiv:1710.08005. 2017.
Smart" predict, then optimize".
arXiv preprint
Geoffrey French, Michal Mackiewiez, and Mark Fisher. Self-ensembling for visual domain adapta- tion. arXiv preprint arXiv:1706.05200 2017.
Yves Grandvalet and Yoshua Bengio.
Semi-supervised learning by entropy minimization. In Ad- vances in neural information processing systems, pp. 529-536, 2005.
Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(Nov):1471-1530, 2004.
Arthur Gretton, Kenji Fukumizu, Choon H Teo, Le Song, Bernhard Schölkopf, and Alex Smola. A kernel statistical test of independence. In Advances in neural information processing systems, pp. 585-592, 2008.
Ligong Han, Yang Zou, Ruijiang Gao, Lezi Wang, and Dimitris Metaxas. Unsupervised domain adaptation via calibrating uncertainties. In CVPR Workshops, volume 9, 2019.
Jennifer I. Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217-240, 2011.
Guido W Imbens and Jeffrey M Wooldridge. Recent developments in the econometrics of program evaluation. Journal of economic literature, 47(1)55-86, 2009.
Thorsten Joachims, Adith Swaminathan, and Maarten de Rijke. Deep learning with logged bandit feedback. International Conference on Learning Representations, 2018.
Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual infer- ence. In International conference on machine learning, pp. 3020-3029, 2016.
Nathan Kallus. Classifying treatment responders under causal effect monotonicity. arXiv preprint arXiv: 1902.00902. 2019.
Nathan Kallus and Angela Zhou. Confounding robust policy improvement. In Advances in neural information processing systems, pp. 9269-9279, 2018.
Joseph DY Kang, Joseph L Schafer, et al. Demystifying double robustness: A comparison of alter- native strategies for estimating a population mean from incomplete data. Statistical science, 22 (4):523-539, 2007.
Gürhan Kök and Marshall L Fisher. Demand estimation and assortment optimization under sub- stitution: Methodology and application. Operations Research, 1006)110 2007.

---

Under review as a conference paper at ICLR 2021
Sören R Künzel, Jasjeet S Sekhon, Peter J Bickel, and Bin Yu. Metaleamnrs for estimating heteroge- neous treatment effects using machine learning. Proceedings of the national academy of sciences, (o<110):415644163 2019.
Samuli Laine and Timo Aila. Temporal ensembling for sumi-super learning. arXiv preprint arXiv:1610.02242. 2016.
Damien Lefortier, Adith Swaminathan, Xiaotao Gu, Thorsten Joachims, and Maarten de Ri- jke. Large-scale validation of counterfactual learning methods: A test-bed. arXiv preprint arXiv:1612.00367 2016.
Lihong Li. Wei Chu, John Langford, and Robert E Schapire. A contextual- approach to personalized news article recommendation. Proceedings of the 19th international conference on World wide web, pp. 661-670, 2010.
Yuanqing Li, Cuntai Guan, Huiqi Li, and Zhengyang Chin. A self-training semi-supervised svm algorithm and its application in an eeg-based brain computer interface speller system. Pattern Recognition Letters, 29(9):
2008.
Romain Lopez, Chenchen Li, Xiang Yan, and Junwu Xiong. Cost-effective incentive allocation via structured counterfactual inference. 2020.
Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample variance penal- ization. arXiv preprint arXivv0907.3740, 2009.
Jeffrey I McGill and Garrett J Van Ryzin. Revenue management: Research overview and prospects. Transportation science, 33(2):233-256, 1999.
Kamal Nigam, Andrew Kachites McCallum, Sebastian Thrun, and Tom Mitchell. Text classification from labeled and unlabeled documents using em. Machine learning. 39(2-3):103-113 134, 2000.
Judea Pearl. Detecting latent heterogeneity. Sociological Methods & Research, 46(3):370-389, 2017.
Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CombriiggUUniveesityyPeess, 2000.
Kenneth Rose, Eitan Gurewitz, and Geoffrey Fox. deterministic annealing approach to clustering. Pattern Recognition Letters, (1955589,, 1990.
Paul R Rosenbaum. Model-based direct adjustment. Journal of the American Statistical Association, 82(398):387-394 1987.
Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-555 1983.
Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469):322-3311 2005.
Sebastian Ruder. overview of gradient descent optimization algorithms. arXiv preprint 1609.004777 2016.
Noveen Sachdeva, Yi Su, and Thorsten Joachims. Off-policy bandits with deficient support. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 965-975, 2020.
Uri Shalit, Fredrik D Johansson, and David Sontag, Estimating individual treatment effect: gener- alization bounds and algorithms. In International Conference on Machine Learning, pp. 3076- 3085, 2017.
Ilya Shpitser and Judea Pearl. Identification of conditional interventional distributions. arXiv preprint arXiv:/206.6877 2012.
Adith Swaminathan and Thorsten Joachims, Counterfactual risk minimization: Learning from logged bandit feedback. In International Conference on Machine Learning, pp. 814-823, 2015a.
10

---

Under review as a conference paper at ICLR 2021
Adith Swaminathan and Thorsten Joachims. The self- normalzzdd estimator for counterfactual learn- ing. In advances in neural information processing systems, pp. 3231-3239, 2015b.
Jafar Tanha, Maarten van Someren, and Hamideh Afsarmanesh. Semi-supervised self self-training for decision tree classifiers. International Journal of Machine Learning and Cybernetics, 8(1):355- 370, 2017.
Terence J Wales and Alan Donald Woodland. Estimation of consumer demand systems with binding non-negativity constraints. Journal of Econometrics, 21(3):263-22 285, 1983.
Lequn Wang, Yiwei Bai, A. Bhalla, and T. Joachims. Batch learning from bandit feedback through bias corrected reward imputation. 2019.
Yanbo Xu, Yanxun Xu, and Suchi Saria. A bayesian nonparametric approach for estimating in- dividualized treatment-response curves. In Machine Learning for Healthcare Conference, pp. 282 382-300,220 2016.
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Ganite: Estimation of individualized treatment effects using generative adversarial nets. In International Conference Learning Rep- resentations, 2018.
Alan L Yuille, Paul Stolorz, and Joachim Utans. Statistical physics, mixtures of distributions, and the em algorithm. Neural Computation, 6(2):334 0444,440000 1994.
Houssam Zenati, Alberto Bietti, Matthieu Martin, Eustache Diemert, and Julien Mairal. Counter- factual learning of continuous stochastic policies. 2020.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Domain adaptation for semantic segmen- tation via class-balancee. self-training, arXiv preprint (rXiv:1810.07911 2018.
Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In Proceedings of the IEEE International Conference on Computer Vision, pp- 5982-5991, 2019.
A APPENDIX
A.1 PROOF OF THEOREM 1
Theorem 1. Assume
2 Mo+I. where Mo > 1 is constant, x, is defined on a compact, discrete set respectively, let M = min(maxx.,, - 1), Mo). f = argminfer Ro(f), D is the dataset generated by random imputation of current model output fe. and f minimizes the empirical risk on D. For any loss function 1(,, -), we have:
( + - SCL
log log(1/6) (1/8) + (M =1) RRD(f*)
(2)
Proof.
Rols) - Ro(l")
=Ro(f) RD(f*) + Ro(f) - (Ro(f)-
(4)
=Ro(f) Rb(F') + P(r|z,p) -
sRo(f) - RT( + -
(5)
To(r|e,))
RD(
(6)
-(M + +1)(R((F) - + (M + -1)
(7)
<C(V V +
log (1/5)
+ (M + 1) -
(8)
11

---

Under review as a conference paper at ICLR 2021
Equation 4 comes from adding and subtracting the risk defined D and D. Since D is imputed with probability Po. we can use importance sampling to get Equation | We get Equation 7 by adding and subtracting the empirical risk. Equation X from the basic excess-risk bound. V the VC dimension of hypothesis class F, and C' is universal constant.
A.2 PROOF OF PROPOSITION
Proposition CST with argmax imputation is convergent under certain conditions.
Proof. Our CST objective is defined
N
N E Σ (f
min C1 = NIPI (((fo(o.p))).)) 6R
(9)
PEP
where r; is the factual data observed and P. is imputed by trained classifier fe. We show our proof using binary cross entropy loss which we use the paper, it can be generalized cross entropy easily. We show the convergence of CST-AI defined in Section [ which imputes pseudo-label using argmax operation. The objective is optimized via the following two steps:
1) Pseudo-I Imputation: Fix 0 and impute R to solve:
min
- fog + n.) log(1 - fo(2,, ff(xx,)))
fo(z.,)))))))))
(10)
PEP
s.t. i,p E A, Pi,
where A is the possible discrete values of the outcome.
2) Model Retraining: Fix R and solve the following optimization using gradient descent, where
-(--)) is binary cross entropy loss:
min Σ ((((rep)))))
+
Σ Σ 1=1 PEP
((o(a, Ptpp)
(11)
For CST-AI, we have:
Step non-increasing: (10) has an optimal solution which is given by pseudo-labels imputed by argmax operation with feasible set being all possible outcomes. As a result, (10) is non- non-increasing.
Step is non-increasing: If use gradient descent minimize the loss defined in Equation I] The loss is guaranteed to decrease monotonically with a proper learning rate (Zou et al. 2019). For
mini-batcl gradient descent commonly used in practice, the loss not guaranteed to decrease but also almost certainly converge to lower value.
Since loss Equation is lower bounded, CST-AI is convergent.
A.3 EMPIRICAL CONVERGENCE ANALYSIS OF CST-RI
For CST-RI, since the convergence analysis more challenging, we show empirically CST-RI con- verges in all of our experiments without any additional techniques. We show loss curves in all experiments for our synthetic and multi-label datasets in Figure 7 and [1 respectielly. CST-RI is trained with gradient descent with momentum (Ruder 2016 For syntheic datasets, set lear- ing rate as le-3 and momentum as 0.9. For multi-label datasets, we set learning rate as le-1 and momentum 0.9.
Next, we share some intuition on CST-RI and its connection with entropy regularization (Grandvalet & Bengio 2005). Consider the second term in Equation 9 with stochastic imputation:
12

---

Under review as a conference paper at ICLR 2021
C41
010
Figure 2: D1
Figure D2
Figure 4: D3
Figure 5: D4
Figure 6: D5
Figure 7: Loss Curves for Synthetic Datasets
Figure 8: TMC
Figure 9: Yeast
Figure 10: Scene
Figure 11: Loss Curves for Multi-Label Datasets
N
Σ
Σ
(f:loo + (1 f:) log(1 - So -o(xx,
(12)
t=1
PEP
f. ~ Bern(Jo((r,,))) Vi,p
Since r E 1). by taking expectation over f,
N
E
-(n (o log
fo(x..)) (og + (1 - A. log(1 -f
(13)
PEP
=
Σ PEP
(fo(zr,)) log (z + (1 - fo p)) log(1 fo(2,,)))
(14)
which equals to the entropy term defined fe. thus Our CSI-RI framework can be viewed variant of entropy regularization in semi-supervised learning (Grandvalet & Bengio 2005). Since aim to simulate randomized trail, the hyper-paramedd in Grandvalet & Bengio (2005) is set
to in CST. Instead taking the argmax imputation which is commonly used in classification EM (CEM) Amini & Gallinari 2002) that minimizes the objective, impute a randomly assigned
13

---

Under review as a conference paper at ICLR 2021
label with larger probability to be the CEM solution. This step is very similar to deterministic annealing EM (Grandvalet & Bengio 2005 Yuille et al. 1994] Rose et al. 1990) where pseudo- label is generated by the output probability with annealing temperature instead of CEM solution, which aims to find the global minimum more efficiently.
A.4 EXPERIMENT RESULTS FOR RUNNING TIME ANALYSIS
DI
D4
DS
Direct Method
21.64=1.46
21.3411811
21.70:1..71
1..4411.66
21.67:11.18
CST-RI
18.28-1.223
35.30+0.02
36.65++0.27
17.19+0..3
37.71+0.800
CST-AT
38.00=1.144
35.90+00.44
36.99+00.26
38. 10++044
37.89++0.33
HSIC
4038.35-14.65
4029,02:10.51
4087.17:1.366
4060.0110.799
1055.30=1.11
BanditNet
657.82-10.50
614.91+1.99
632.93+2.777
639.83++1553 25.18+0..99
659.10+3.11
UDM
24.99=0.34
25.85++0911
24.23++049
25.57+40.8
Table Running Time on Synthetic Datasets (measured in seconds)
TMC
Yeast
Scene
Direct Method
570.63
33.4312.95
26.8810.29
CST-RI
1428.65
93.60::4.14
71.98:10.99
CST-AI HSIC BanditNet
1441.3++3.00
95.11+44.54
72.74=10.99
43806.3+211.005
4276.82::44..25
18.9
10342.65+:35.83
488.71+35.84
271.22:18.79
UDM
958.49+1.999
39.03
32.20-10.04
Table 8: Rumning Time on Real Datasets (measured in seconds)
A.5 DATA GENERATION FOR SYNTHETIC DATASET
In the synthetic experiments, we use a pricing example similar to the experiment in Lopez et al. 2020). Let (t,,)) be uniform distribution. Assume customer features 50-dimensional vector X drawn from UO, 1y 1 and there are 10 price options from $1 to $10. The logging policy is set as denotes sigmoid function. We simulated five types of demand functions,
(p i|z) E z
with h(x) Σ a Σ
exp by ||
ell), b, UO, 1)50. .. {0,1)
<h((x) 2ι0 p)
r~ o~ (Io 0.5) 0.4 p)
[h((( stepwise (xx) ) p)
Th(((( stepwise2 z1) p)
(h(x)) (ar + +1) p)
where the stepwise function is defined as:
0.7, if i 0.1
0.5, if 5.3
stepwisel(z)
(15)
0.3, 0.3 <..60
0.1, 0f <
0.65, if 5 0.1 and >>>>>>> 0.5
0.45,
if a.1 y 0.5
0.55,
If0.1 0.3 and 0.5
0.35,
100.1.0 0.1 and y 5 0.5
stepwise2(r.. y)
(16)
0.45, if 0.3<x5< 0.6 and y > 0.5
0.25, if 11.3x55 0.6 and y / 0.5
0.35,
if0.6
and y 0.5
0.15,
if0.6 < < and 0.5
14

---

Under review as a conference paper at ICLR 2021
A.6 MULTI-LABEL DATASETS STATISTICS
# features
labels
train size
test size
Yeast
103
14
1500
917
Scene
294
1211
1196
TMC
30438
22
21519
7077
Table 9; Dataset Statistics
15


Reviews:
[{"review_id": "_MWWeSJAMU2", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After Reading Authors' response.\n\nThanks the authors for the response. The additional discussion regarding convergence is appreciated. The explanation of how BanditNet is trained and the additional baseline makes the empirical comparison more clear. I would like to increase my evaluation to 6.\n\nI still have the following concerns.\n\nThe generalization error bound is too loose to be informative.\n\nThere is no guarantee that the algorithm will continue to improve the performance.\n\n------------------------------------------------------Original Comments----------------------------------------------------------\n\nSummary\n\nThe paper proposed to use self-training, a semi-supervised learning technique to solve batch learning from biased and partial bandit feedback problem. The self-training process is as follows:\n\n(1) Train a reward prediction model from observed data. \n(2) Impute rewards for unobserved actions given a context with the reward prediction model. \n(3) Train a reward prediction from the imputed rewards and go to step 2. \n\nThe paper provides generalization error analysis that connects performance of a policy under the uniform distribution to the objective they use for self-training. \n\nThe paper presents empirical analysis on both simulated data and multi-label classification dataset. \n\nStrengths:\n\n1. Using semi-supervised learning methods for batch learning from bandit feedback is interesting.\n\n2. The paper is clearly written and easy to follow. \n\n\nWeaknesses:\n\n1. The proposed algorithm does not take the selection bias into account. This leads to a generalization error that depends on M, which can be very large in practice. \n\n2. There is no guarantee that the algorithm will converge or continue to improve. \n\n3. It would be great to compare with methods that model the selection bias, e.g. \\sum\\frac{1}{N}\\frac{1}{\\mu(p_i|x_i)} l( prediction_i ,r_i ) in Wang et al. Batch Learning from Bandit Feedback through Bias Corrected Reward Imputation. I understand that the paper assumes the logging policy is not available, but there are ways to estimate the logging policy, e.g. Hanna et.al. Importance Sampling Policy Evaluation with an Estimated Behavior Policy. Similarly, it seems not fair to use BanditNet but without modeling the selection bias. \n\n\n\n\nAdditional feedback\n\nIt would be great to mention that V is the VC dimension in the main paper.\n\nIn the generalization error bound, minimizing the right hand side does not equation 1 which uses the observed reward instead of imputed reward for actions selected by the logging policy. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ALenG5Wy2_H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post-Rebuttal:\n\nI would like to thank the authors for their rebuttal.\nThe updated version of the paper has addressed some of my comments.\nHowever, I still fail to understand why the method should 1) converge in general, and 2) converge to a good solution. \nI have updated my score accordingly.\n\n=====================================================================================================\n\nOriginal Review:\n\nSummary:\nThis paper proposes a method for causal inference given observational data. The authors formulate this task as a domain adaptation problem and propose a self-training algorithm to address it. The algorithm imputes the counterfactual outcomes (to simulate a randomized controlled trial) in an iterative process that supposedly gets better and better at estimating treatment outcomes. \n\nI have many concerns about the paper; I touch on some of them in the following:\n- There seems to be a misunderstanding in the literature review. There are two main approaches for *Policy Optimization*: Direct Method (DM) and Utility Maximization (UM). Counterfactual Risk Minimization (CRM) is a method in the UM category. It is the *Causal Inference* task that this paper addresses; not *Policy Optimization*.\n- It is unclear why the proposed algorithm, namely Counterfactual Self-Training, should perform well; i.e., why a model trained on a new dataset with randomly imputed counterfactuals should do any better … \n- It seems that the second term in the objective function in Eq. (1) is always zero; since, $\\hat{r_{i, p}}$ is equal to $f_{\\theta}(x_i, p)$ according to the line just above Eq. (1). Therefore, the objective function reduces to an unweighted factual loss, which we know would not result in a good model, since it does not account for selection bias.\n- Algorithm 1, on the other hand, says that $\\hat{r_{i, p}}$ is sampled from $P_{\\theta}( r | x_i, p )$. The authors should comment on whether this is equivalent to $f_{\\theta}(x_i, p)$...?\n- The iterative process of imputing counterfactuals and then training a new model on the new RCT-like dataset is said to continue until the optimization converges. However, the authors do not discuss what the convergence criteria are.\n- Definition for $M$ in Theorem 1 is vague: which values for $P$ and $P_{\\theta}$ are used? Shouldn’t $M$ be a constant?\n\n\nMinor comment(s):\n- There are many typos in the paper, some of which are in the equations. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "9aKz3JGkI6z", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use self-training to tackle the fundamental problem of causal inference where only one potential outcome is seen. The proposed self-training method is iterative: after training a model on the observational dataset, they run points with different actions (treatments) through the trained model and collect the predictions, which are the pseudo-labels. They then continue the training of the model, including the pseudo-labels, until convergence. The paper experiments with two versions of the method -- one with deterministic pseudo-labels (CST-AI) and another with soft pseudo-labels sampled from a probability distribution (CST-RI). It is assumed that there are no unobserved confounders.\n\nAlthough I liked the paper's proposal to bring self-training ideas to causal inference, the results don't seem fully convincing yet. I would have liked to see more baselines ( e.g. inverse propensity weighting), and perhaps experiments on more classic synthetic causal inference datasets (e.g. IHDP). \n\nResults: \n- Synthetic data: why the choice of hamming loss? RMSE and PEHE are more typical metrics here. The rather bad (and consistently bad) performance of CST-AI on the synthetic data is a little surprising. Do the authors have any explanation for why this is the case? Perhaps it was mentioned in the paper but I missed it.\n- The run time analysis is interesting but its meaningfulness really depends on the setting in which the authors envision that this work could be used. Slow training time may be acceptable if test time inference is fast. As the authors pointed out, BanditNet is slow due to cross-validation, which other methods don't really do, so the comparison doesn't seem fair. Can the authors comment more on the practical significance of the run-time results?\n\nReferences: in the classic causal inference literature, there are learners such as S, T, and X learners (this terminology is recent, but the models besides X learner have been around for a while; https://www.pnas.org/content/116/10/4156.abstract has some good explanations. The S learner learns a single model on observational data, then takes the data, switches the actions (treatment), and runs it through the model to collect the pseudo-labels. Training is not continued; treatment effects are simply estimated as a difference of the labels and pseudo-labels. But it also assume that pseudo-labels collected by training a model on observational data and switching actions are valid, so it might be interesting to cite some of this literature. Hill 2011 cited in the paper is actually a S-learner.\n\nMinor points:\n- I would suggest renaming \"skyline\" (e.g. in Table 3) which \"represents the best possible reward with perfect knowledge of the demand function\" (authors' words) to oracle, which is a more common term. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an intriguing approach for "individual treatment effect" estimation from an observational dataset.  The approach is developed for multiple discrete actions (beyond binary treatments as typically studied in ITE literature) and discrete outcomes (a special case compared to related literature). The idea is to use the "direct method" (i.e. learn a probabilistic classifier using the observed dataset) and sample imputed outcomes for all unobserved action-outcomes. Then, learn a probabilistic classifier that fits the observed+imputed dataset well, and iterate the procedure. This intriguing idea seems to converge empirically on a few different problems, and sampling the imputations rather than using deterministic imputations seems to be an important detail.
Proof of convergence is however shown for deterministic imputations. The generalization error bound (Theorem 1) also does not show adequate motivation for the proposed method -- even with infinite data (n->infinity), the excess risk could scale with the empirical risk of the returned model on the imputed dataset. Without an additional step proving that empirical risk on hat{D} (the imputed dataset) converges to 0 during successive iterations of the procedure, the generalization error bound is incomplete. 
Consider the example of Figure 1, but where customer A has arrived to the system twice. So, the dataset contains {x1, $2, 1} and {x1, $3, 0}. When constructing the imputed dataset, the first data-point would create 2 regression examples {x1, $1, ..} and {x1, $3, ..} while the second data-point would create 2 regression examples {x1, $1, ..} and {x1, $2, ..}.
Now, if the two {x1, $1, ..} examples have different imputation labels sampled from the model, this sets up an unrealizable learning problem and the empirical risk on hat{D} cannot be 0 for any predictor.
In this toy example, we might know that we should "collapse" the two data-points (e.g., de-duplicate the dataset to only have unique x's with aggregated action-outcomes across all observations) in the original data-set and only create one set of imputed labels -- but similar unrealizability can happen for x's that are "close" to each other that no model has capacity to label them differently.

The strength of the paper is its intriguing approach to ITE estimation. It is a form of an iterative S-learner (vanilla S-learners have been widely used in ITE estimation).
The low-point of the paper is this weakness in theory and analysis -- it is unclear if the proposed procedure with sampling imputations (which seems to be important for empirical performance) is even a consistent algorithm.
The paper would be much stronger with a more rigorous analysis of when the method will reliably work, and importantly, its limitations -- such a study will help practitioners know when to use self-training over direct method, targeted max likelihood, S-learners, etc. 
Decision: Reject