Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies the problem of distributionally robust batch policy optimization in contextual bandits, where the distributional shift occurs over both the context and reward distribution. Under bounded error in KL-divergence, the authors show that the value of a policy can be optimized by considering the dual objective under both distribution shifts. The authors deviate from prior work by considering the two distribution shifts separately in a factored optimization objective.

The authors theoretically show that that the estimated value of a policy converges to the true one asymptotically, and empirically demonstrate its effective against existing baselines in several real-world domains; in doing so, the authors propose how their proposed objective can be estimated outside of tabular domains (i.e. when context is a continuous vector). 
strengths_and_weaknesses: Strengths:
- The paper tackles an important problem of robust batch policy optimization. The authors provide a compelling motivation for their work via voting and healthcare applications. 
- The proposed algorithm is sound and provides substantive improvements over existing baselines, including one that considers joint distribution shift rather than the context and reward shifts separately. The authors also provide an interesting approach to extend their approach beyond tabular environments via estimating the moments of the distribution.
- The empirical evaluation is comprehensive. The authors compare their proposed approach against strong, well-tuned baselines in multiple domains that rely on real-world classification and voting data. 

Weaknesses:
- The exact algorithm somewhat lacks in novelty, as it is a hierarchical application of the dual objective trick used in prior works, notably Si et al. (2020). This also applies to the analysis, as Lemma 1 matches Theorem 1 of Si et al. (2020).  
- To my knowledge, the theoretical guarantees (Theorem 2 and 3)  argue mostly about asymptotic convergence. The results would be stronger if there were sample complexity bounds and showed the dependence on the number of samples n. There also is a lack of guarantees on lower-bounds.
- Though the experiments use real-world data, the actual distribution shifts are simulated. It would be interesting to use a real-world example of a distribution shift. 
questions: I was wondering if sample complexity bounds can be derived for Theorem 2 and 3 akin to Lemma 1? Specifically, is there something that prevents the authors from applying the same analysis techniques on the context distribution shift as they did on the reward distribution? 
limitations: The authors discuss limitations of their work in a separate Discussion section. An important one that the authors mention is how to choose the maximum divergence, which would affect how conservative the optimized policy will be.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work considers the contextual bandit problem with shifts on the distributions of both contexts and reward generators. An analysis on asymptotical optimality is provided. A practical algorithm as well as the empirical evaluations are presented. 
strengths_and_weaknesses: Strength:
This work is well-motivated. The problem of distribution shift is significant in practice.
The theoretical result is satisfying, even though it analyses the asymptotic behavior of the algorithm.
This work attempts to provide practical version of the algorithm.

Weakness:
The proposed algorithm seems not very practical as it relies on Taylor expansion, especially when the underlying functions have many parameters.

questions: Sometimes the distribution shift on the context does not matter under some weak assumptions, e.g., LinUCB works on linear contextual bandit with any context distribution. Therefore, a relevant question is when we should take context distribution shift into consideration?
limitations: no obvious limitation.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an improvement of distributionally robust offline learning for contextual Bandit problem. The authors formulates the robust learning problem by proposing a nested robustness criteria involving context drfit and drift in reward distribution. It builds upon the recent work but expands to the 'factored' form to make the robustness specification more fine-grained. 
strengths_and_weaknesses: To me this is a good piece of work that builds upon the existing work and make it more fine-grained. The motivation of the problem has been clearly articulated and the mathematical super-structure has been nicely laid out thorugh both the main text and the appendix. This is not in my field of expertise, but I think the amount of novelty and the rigor is enough for an acceptance.

At some points it feels like a jump, especially from eq 2 to eq 4 and 5. It would be nice to provide more detailed steps. 
questions: Can you concretely show how your algorithm can fix the issue highlighted in line159-161?
limitations: I think the authors have done a good job in listing the limitations and I have nothing further to add.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper provides a method (Factored-DRO) that learns a distributionally robust policy over separately handled shifts in context and reward distributions.
strengths_and_weaknesses: **Strengths** 

Overall, I find the motivations to be convincing, the methodology to be reasonable, and the paper to be generally well-written. 


**Weaknesses** 

I would be interested in a more in-depth conceptual, theoretical, and practical comparison between FDRO and its closest point of comparison baseline-DRO (Si et al). For example, while FDRO has the advantage of being able to separate out shifts in context and reward distributions, by requiring more parameters to be set, is it more sensitive to improperly set parameters, i.e. does the error compound? Does this imply that FDRO may be less advantageous to use for users who have less information about the distribution shifts in their application of interest? 

Please see my questions below.
questions: - Why is it the baseline-DRO can be overly conservative compared to FDRO? Are there situations where FDRO can also be overly conservative (or overly optimistic)?

- Is FDRO guaranteed to be always better than baseline-DRO in test, and why? If not, is there a situation where we can expect baseline-DRO to do better?

- Is it possible to set $\delta$ and $\delta_x, \delta_c$ such that baseline-DRO and FDRO will have equivalent performance?

- As the authors mention, one limitation common to DRO methods is setting the $\delta$ parameter. Since FDRO has two parameters to set, is it more sensitive to improperly set values? E.g. if $\delta_x$ and $\delta_c$ in FDRO are set too large, does the error compound, leading to worse performance, compared to baseline-DRO when $\delta$ is similarly set too large?

- In this light, does this imply that FDRO may be less advantageous to use for users who have less information about the distribution shifts in their application of interest? Should the user be more or less conservative in setting the parameter $\delta$ in FDRO vs baseline-DRO?

- Why are finite sample guarantees given for reward shift but asymptotic for context shift? Is it possible to obtain finite sample guarantees for policy evaluation and learning (Theorems 2 and 3) versus convergence of reward shift estimates, and if not, what is the barrier?
limitations: Aside from the points addressed in my comments above, the authors have adequately addressed the limitations and impacts of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an algorithm to learn a contextual bandit policy in the batch setting when there is a distributional shift between training and deployment. The main motivation of the proposed approach is the benefit of constraining the factors of the distribution of the context and the reward. 

Based on the existing theoretical work on the minimax approach to DRO, the paper proposes a procedure for policy evaluation and improvement with convergence analysis. With other estimation details filled, FDRO is empirically evaluated against other baselines highlighting the strength of the proposed approach.
strengths_and_weaknesses: Strengths

- The proposed policy iteration is well-motivated by a previously established theoretical work.
- Also, the convergence analysis of the proposed method is provided.
- Empirically it is demonstrated that the factor constraining generally does not harm the policy iteration compared to the joint constraining except for a slim edge case (the same train/test).

Weaknesses

- Rather than calling them weakness, I put relevant questions in below box.
- There are some typos
  * in eq.(6) $\frac{-R}{\alpha_c}$ subscript $j$ of $R$ missing
  * at line 290, $V^* = \min_{\pi} V(\pi)$ which seems to have to be $\max_{\pi}$, 
  * in line 102 'We assume we the', 
  * in eq.(4), closing curly/square parenthesis are swapped, etc. 

questions: - It seems that MGF approximation seems working well in practice. However, (considering a bit of theoretical flavor of this paper) in theory, MGF is not always guaranteed to exist. Then if the nominal (true) distribution has the MGF, could it be possible to say that a distribution of $\epsilon$ KL-ball of the nominal distribution had MFG?
- For continuous contexts, binning approach has been proposed. I was wondering if there is any other approach the author tried, for example, k-NN like smoothing, estimating reward distribution for a given context using (R,C,A) triplets from nearby contexts.
- When there is a joint shift in the context and the reward, how can FDRO be compared with the joint constraining, e.g. baseline-DRO?
- Does theorem 2 hold uniformly for $\pi$? Since the uniform convergence is mentioned on line 208, it would be better to clarify this.
- Factor constraining is basically considering KL-ball for each factor from some factorization of the original full distribution. In the full KL-ball of the same factorization, it is KL-distance on context distribution and expected KL-distance on context-action conditioned reward distribution. Then this can motivate a variation of FDRO, in which not setting the uniform (minimax) bound of KL-distance on context-action conditioned reward distribution, expected (Bayesian) bound of KL-distance on reward distribution. How would this be compared with the FDRO, any thought on this?
- In contrast to context shift, reward shift $\delta_c$ is fixed to $0.001$. How is the sensitivity to this hyperparameter?
limitations: The main motivation seems that the training tractability of factored constraining over joint constraining. However, the benefit of the current approach does not seem to be easily translated into continuous contexts making the estimation of reward distribution difficult. Still, the benefit of FDRO in discrete contexts is noticeable. Maybe its extension of the currently proposed binning approach for continuous contexts can be future work. In relation to this limitation, I am curious about the extent of the FDRO in discrete reward settings, for example, in the case where there are a larger number of discrete contexts and discrete actions, how FDRO would perform. Rather than asking to perform additional experiments for this, I would be happy if the authors can share their thought and previous analyses on this.
review_title: null

Meta-review: metareview: This paper studies off-policy learning with an environment shift, where the distributions of both contexts and rewards can change. The authors address both challenges in a factored form, and derive error bounds for both off-policy evaluation and optimization. The proposed approach is evaluated on real-world datasets. The original ratings of the paper were 6, 6, 6, 6, and 5; and they did not change after the rebuttal. The reviewers generally praised the paper and their main concerns were addressed in the rebuttal:

* Asymptotic errors bounds were replaced with finite-sample guarantees.

* Synthetic shifts in the distributions in experiments were complemented with actual shifts in data.

This is a good paper to accept and I believe that the authors will improve the paper further based on the feedback of the reviewers.
confidence: Certain
Decision: Accept