Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper the authors studied the problem of off-policy learning, in the bandit setting when a batch log of data generated by the baseline policy is given. Here they first summarize the surrogate objective functions derived by existing approaches such as importance sampling and variance regularization (Swaminathan et. al). Then they extend the results in Theorem 2 of the paper by Cortes et. al (which also uses the empirical Bernstein inequality by Maurer and Pontil), and derive a new surrogate objective function that involves the chi-square divergence. Furthermore, the authors also show that the lower bound of this objective function can be iteratively approximated by variational f-GAN techniques, which could potentially be more numerically stable and empirically has lower variance. 

In general, I think the problem studied in this paper is very interesting, and the topic of counterfactual learning, especially policy optimization with the use of offline and off-policy log data, is important. However, I think the theoretical contribution in this paper on off-policy learning is quite incremental. Also the parts that involve f-GAN is still questionable to me.

Detailed comments:
In these variance regularization formulations (for example the one proposed in this paper, or the one derived in Swaminathan's paper), \lambda can be seen as a regularization parameter that trades-off bias and variance of the off-policy value estimator R(h) (for example the RHS of equation 6). To exactly calculate \lambda either requires the size of the policy class (when the policy class is finite), or the complexity constants (which exists in C_1 and C_2 in equation 7, but it is not clearly defined in this paper). Then the main question is on how to choose \lambda such that the surrogate objective function is reasonable. For example in the safety setting (off-policy policy learning with baseline performance guarantees, for example see the problem setting in the paper by P. Thomas 2015: High Confidence off-policy improvement), one always needs the upper-bound in 6) to hold. This makes the choice of \lambda crucial and challenging. Unfortunately I don't see much discussions in this paper about choosing \lambda, even in the context of bias-variance trade-offs. This makes me uncomfortable in believing that the results in experiments hold for other (reasonable) choices of \lambda.

The contribution of this paper is of two-fold: 1) the authors extend the results from Cortes's paper to derive a new surrogate objective function, and 2) they show how this objective can be approximated by f-GAN techniques. The first contribution is rather incremental as it's just a direct application of Theorem 2 in Cortes's paper. Regarding the second contribution, I am a bit concerned about the derivations of Equation 9, especially the first inequality and the second equality. I see that the first inequality is potentially an application of the conjugate function inequality, but more details are needed (f^* is not even defined). For the second equality, it's unclear to me how one can swap the sup and the E_x operators. More explanations are definitely needed to show their mathematical correctness, especially when this part is a main contribution.  Even if the derivations are right, the f-GAN surrogate objective is a lower bound of the surrogate objective function, while the surrogate function is an upper bound of the true objective function (which is inaccessible). How does one guarantees that the f-GAN surrogate objective is a reasonable one? 

Numerical comparisons between the proposed approach, and the approach from Swaminathan's paper are required to demonstrate the superiority of the proposed approach. Are there comparisons in performance between the approach from the original chi-square surrogate function and the one from the f-GAN objective (in order to showcase the need of using f-GAN) as well?

Minor comments:
In experimental section, method POEM is not defined.
The paper is in an okay status. But there are several minor typos, for example \hat{R}_{(} in page 3, and several typos in Algorithm 1 and Algorithm 2.

In general, I think this paper is studying an interesting topic, but the aforementioned issues make me feel that the paper's current status is still unsuitable for publication. 
review_title: Variance Regularized Counterfactual Risk Minimization via Variational Divergence Minimization

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes an interesting alternative to recent approaches to learning from logged bandit feedback, and validates their contribution in a reasonable experimental comparison. The clarity of writing can be improved (several typos in the manuscript, notation used before defining, missing words, poorly formatted citations, etc.).
Implementing the approach using recent f-GANs is an interesting contribution and may spur follow-up work. There are several lingering concerns about the approach (detailed below) that detract from the quality of their contributions.

[Major] In Lemma 1, L(z) is used before defining it. Crucially, additional assumptions on L(z) are necessary (e.g. |L(z)| <= 1 for all z. If not, a trivial counter-example is: set L(z) >> 1 for all z and Lemma 1 is violated). It is unclear how crucially this additional assumption is required in practice (their expts with Hamming losses clearly do not satisfy such an assumption).

[Minor] Typo: Section 3.2, first equation; the integral equals D_f(...) + 1 (not -1).

[Crucial!] Eqn10: Expected some justification on why it is fruitful to *lower-bound* the divergence term, which contributes to an *upper-bound* on the true risk.

[Crucial!] Algorithm1: How is the condition of the while loop checked in a tractable manner?

[Minor] Typos: Initilization -> Initialization, Varitional -> Variational

[Major] Expected an additional "baseline" in the expts -- Supervised but with the neural net policy architecture (NN approaches outperforming Supervised on LYRL dataset was baffling before realizing that Supervised is implemented using a linear CRF).

[Major] Is there any guidance for picking the new regularization hyper-parameters (or at least, a sensible range for them)?

[Minor] The derived bounds depend on M, an a priori upper bound on the Renyi divergence between the logging policy and any new policy. It's unclear that such a bound  can be tractably guessed (in contrast, prior work uses an upper bound on the importance weight -- which is simply 1/(Min action selection prob. by logging policy) ).
review_title: Interesting combination of off-policy learning from bandit feedback and f-GANs, with some weaknesses in theory and experiment

final_score: rating: 7: Good paper, accept
review_content: This paper studies off-policy learning in the bandit setting. It develops a new learning objective where the empirical risk is regularized by the squared Chi-2 divergence between the new and old policy. This objective is motivated by a bound on the empirical risk, where this divergence appears. The authors propose to solve this objective by using generative adversarial networks for variational divergence minimization (f-GAN). The algorithm is then evaluated on settings derived from supervised learning tasks and compared to other algorithms.

I find the paper well written and clear. I like that the proposed method is both supported by theory and empirical results. 

Minor point: I do not really agree with the discussion on the impact of the stochasticity of the logging policy in section 5.6. Based on Figure 5 a and b, it seems that the learned policy is performing equally well no matter how stochastic the logging policy is. So I find it a bit misleading to suggest that the learned policy are not being improved when the logging policy is more deterministic. Rather, the gap reduces between the two policies because the logging policy gets better. In order to better showcase this mechanism, perhaps you could try using a logging policy that does not favor the best action.

quality and clarity:
++ code made available
+ well written and clear
- The proof of theorem 2 is not in the paper nor appendix (the authors say it is similar to another work).


originality
+ good extension of the work by Swaminathan & Joachims (2015a): derivation of an alternative objective and use of a deep networks
. This paper leverages a set of diverse results

significance
- The proposed method can only be applied if propensity scores were recorded when the data was generated.
- no test on a real setting
++ The proposed method is supported both by theoretical insights and empirical experiments.
+ empirical improvement with respect to previous methods


details/typos:

3.1, p3: R^(h) has an indexed parenthesis
5.2; and we more details
5.3: so that results more comparable
review_title: Well written paper, good contribution by leveraging several diverse work but average/limited applicability

Meta-review: The reviewers agree that the paper studies and interesting problem with an interesting approach. The reviewers raised some concerns regarding the theoretical and empirical results. The authors have made changes to the paper, but given the theoretical nature of the paper and the extent of changes, another review is needed before publication.
Decision: Reject