Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes to use self-training to tackle the fundamental problem of causal inference where only one potential outcome is seen. The proposed self-training method is iterative: after training a model on the observational dataset, they run points with different actions (treatments) through the trained model and collect the predictions, which are the pseudo-labels. They then continue the training of the model, including the pseudo-labels, until convergence. The paper experiments with two versions of the method -- one with deterministic pseudo-labels (CST-AI) and another with soft pseudo-labels sampled from a probability distribution (CST-RI). It is assumed that there are no unobserved confounders.

Although I liked the paper's proposal to bring self-training ideas to causal inference, the results don't seem fully convincing yet. I would have liked to see more baselines ( e.g. inverse propensity weighting), and perhaps experiments on more classic synthetic causal inference datasets (e.g. IHDP). 

Results: 
- Synthetic data: why the choice of hamming loss? RMSE and PEHE are more typical metrics here. The rather bad (and consistently bad) performance of CST-AI on the synthetic data is a little surprising. Do the authors have any explanation for why this is the case? Perhaps it was mentioned in the paper but I missed it.
- The run time analysis is interesting but its meaningfulness really depends on the setting in which the authors envision that this work could be used. Slow training time may be acceptable if test time inference is fast. As the authors pointed out, BanditNet is slow due to cross-validation, which other methods don't really do, so the comparison doesn't seem fair. Can the authors comment more on the practical significance of the run-time results?

References: in the classic causal inference literature, there are learners such as S, T, and X learners (this terminology is recent, but the models besides X learner have been around for a while; https://www.pnas.org/content/116/10/4156.abstract has some good explanations. The S learner learns a single model on observational data, then takes the data, switches the actions (treatment), and runs it through the model to collect the pseudo-labels. Training is not continued; treatment effects are simply estimated as a difference of the labels and pseudo-labels. But it also assume that pseudo-labels collected by training a model on observational data and switching actions are valid, so it might be interesting to cite some of this literature. Hill 2011 cited in the paper is actually a S-learner.

Minor points:
- I would suggest renaming "skyline" (e.g. in Table 3) which "represents the best possible reward with perfect knowledge of the demand function" (authors' words) to oracle, which is a more common term. 
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: After Reading Authors' response.

Thanks the authors for the response. The additional discussion regarding convergence is appreciated. The explanation of how BanditNet is trained and the additional baseline makes the empirical comparison more clear. I would like to increase my evaluation to 6.

I still have the following concerns.

The generalization error bound is too loose to be informative.

There is no guarantee that the algorithm will continue to improve the performance.

------------------------------------------------------Original Comments----------------------------------------------------------

Summary

The paper proposed to use self-training, a semi-supervised learning technique to solve batch learning from biased and partial bandit feedback problem. The self-training process is as follows:

(1) Train a reward prediction model from observed data. 
(2) Impute rewards for unobserved actions given a context with the reward prediction model. 
(3) Train a reward prediction from the imputed rewards and go to step 2. 

The paper provides generalization error analysis that connects performance of a policy under the uniform distribution to the objective they use for self-training. 

The paper presents empirical analysis on both simulated data and multi-label classification dataset. 

Strengths:

1. Using semi-supervised learning methods for batch learning from bandit feedback is interesting.

2. The paper is clearly written and easy to follow. 


Weaknesses:

1. The proposed algorithm does not take the selection bias into account. This leads to a generalization error that depends on M, which can be very large in practice. 

2. There is no guarantee that the algorithm will converge or continue to improve. 

3. It would be great to compare with methods that model the selection bias, e.g. \sum\frac{1}{N}\frac{1}{\mu(p_i|x_i)} l( prediction_i ,r_i ) in Wang et al. Batch Learning from Bandit Feedback through Bias Corrected Reward Imputation. I understand that the paper assumes the logging policy is not available, but there are ways to estimate the logging policy, e.g. Hanna et.al. Importance Sampling Policy Evaluation with an Estimated Behavior Policy. Similarly, it seems not fair to use BanditNet but without modeling the selection bias. 




Additional feedback

It would be great to mention that V is the VC dimension in the main paper.

In the generalization error bound, minimizing the right hand side does not equation 1 which uses the observed reward instead of imputed reward for actions selected by the logging policy. 


review_title: Counterfactual Self-Training

final_score: rating: 4: Ok but not good enough - rejection
review_content: Post-Rebuttal:

I would like to thank the authors for their rebuttal.
The updated version of the paper has addressed some of my comments.
However, I still fail to understand why the method should 1) converge in general, and 2) converge to a good solution. 
I have updated my score accordingly.

=====================================================================================================

Original Review:

Summary:
This paper proposes a method for causal inference given observational data. The authors formulate this task as a domain adaptation problem and propose a self-training algorithm to address it. The algorithm imputes the counterfactual outcomes (to simulate a randomized controlled trial) in an iterative process that supposedly gets better and better at estimating treatment outcomes. 

I have many concerns about the paper; I touch on some of them in the following:
- There seems to be a misunderstanding in the literature review. There are two main approaches for *Policy Optimization*: Direct Method (DM) and Utility Maximization (UM). Counterfactual Risk Minimization (CRM) is a method in the UM category. It is the *Causal Inference* task that this paper addresses; not *Policy Optimization*.
- It is unclear why the proposed algorithm, namely Counterfactual Self-Training, should perform well; i.e., why a model trained on a new dataset with randomly imputed counterfactuals should do any better … 
- It seems that the second term in the objective function in Eq. (1) is always zero; since, $\hat{r_{i, p}}$ is equal to $f_{\theta}(x_i, p)$ according to the line just above Eq. (1). Therefore, the objective function reduces to an unweighted factual loss, which we know would not result in a good model, since it does not account for selection bias.
- Algorithm 1, on the other hand, says that $\hat{r_{i, p}}$ is sampled from $P_{\theta}( r | x_i, p )$. The authors should comment on whether this is equivalent to $f_{\theta}(x_i, p)$...?
- The iterative process of imputing counterfactuals and then training a new model on the new RCT-like dataset is said to continue until the optimization converges. However, the authors do not discuss what the convergence criteria are.
- Definition for $M$ in Theorem 1 is vague: which values for $P$ and $P_{\theta}$ are used? Shouldn’t $M$ be a constant?


Minor comment(s):
- There are many typos in the paper, some of which are in the equations. 
review_title: Many details have not been discussed; I think the current version requires a major revision.

Meta-review: The paper proposes an intriguing approach for "individual treatment effect" estimation from an observational dataset.  The approach is developed for multiple discrete actions (beyond binary treatments as typically studied in ITE literature) and discrete outcomes (a special case compared to related literature). The idea is to use the "direct method" (i.e. learn a probabilistic classifier using the observed dataset) and sample imputed outcomes for all unobserved action-outcomes. Then, learn a probabilistic classifier that fits the observed+imputed dataset well, and iterate the procedure. This intriguing idea seems to converge empirically on a few different problems, and sampling the imputations rather than using deterministic imputations seems to be an important detail.
Proof of convergence is however shown for deterministic imputations. The generalization error bound (Theorem 1) also does not show adequate motivation for the proposed method -- even with infinite data (n->infinity), the excess risk could scale with the empirical risk of the returned model on the imputed dataset. Without an additional step proving that empirical risk on hat{D} (the imputed dataset) converges to 0 during successive iterations of the procedure, the generalization error bound is incomplete. 
Consider the example of Figure 1, but where customer A has arrived to the system twice. So, the dataset contains {x1, $2, 1} and {x1, $3, 0}. When constructing the imputed dataset, the first data-point would create 2 regression examples {x1, $1, ..} and {x1, $3, ..} while the second data-point would create 2 regression examples {x1, $1, ..} and {x1, $2, ..}.
Now, if the two {x1, $1, ..} examples have different imputation labels sampled from the model, this sets up an unrealizable learning problem and the empirical risk on hat{D} cannot be 0 for any predictor.
In this toy example, we might know that we should "collapse" the two data-points (e.g., de-duplicate the dataset to only have unique x's with aggregated action-outcomes across all observations) in the original data-set and only create one set of imputed labels -- but similar unrealizability can happen for x's that are "close" to each other that no model has capacity to label them differently.

The strength of the paper is its intriguing approach to ITE estimation. It is a form of an iterative S-learner (vanilla S-learners have been widely used in ITE estimation).
The low-point of the paper is this weakness in theory and analysis -- it is unclear if the proposed procedure with sampling imputations (which seems to be important for empirical performance) is even a consistent algorithm.
The paper would be much stronger with a more rigorous analysis of when the method will reliably work, and importantly, its limitations -- such a study will help practitioners know when to use self-training over direct method, targeted max likelihood, S-learners, etc. 
Decision: Reject