Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a distributionally robust optimization (DRO) framework for learning models that can generalize well under distribution shifts, when partial knowledge about the nature of the shift is available. As opposed to classical domain adaptation methods, the proposed method does not require data from the target distribution. For an uncertainty set of distributions defined to satisfy a set of moment conditions (the partial knowledge about the shift), the authors propose to train a model to minimize risk with respect to the distribution in the uncertainty set which minimizes the relative entropy from the training distribution. Using the empirical distribution of the training data as the source distribution, the authors show that the solution to this minimax problem upper bounds the optimal risk and provide finite sample guarantees. The authors then develop a two step learning procedure: First, they estimate the I-projection (using a fast gradient-like method, Algorithm 1). Then, they can efficiently compute the model parameters. They also consider simple extensions to the case of off policy evaluation in reinforcement learning. Synthetic and real data experiments demonstrate the effectiveness of the proposed method given a good choice of the relative entropy radius hyperparameter.
main_review: The distribution shift problem addressed in this paper is very relevant in the community, and a distinct advantage of DRO methods (like the proposed method) is that they do not require samples from the test distribution. The structure for expressing knowledge about the distribution shift seems useful and interpretable. I thought the paper was well-written and I really appreciated the examples---they helped me understand how the proposed framework would work on practical examples. The experiments were presented in a way in which it was easy to draw conclusions about how the method performs as sample size and hyperparameters vary.

I had a couple questions/concerns that I hope the authors can address:
- Can the authors provide any guidance on how to choose the $r$ hyperparameter (defining the radius of the relative entropy ball)? From a practical perspective, this seems like one of the primary challenges that would make the method hard to use. As the experiments demonstrate, the proposed method can achieve very good performance, but this is highly sensitive to the choice of $r$.
- The authors claim that, e.g., the principle of maximum entropy has not been used to model general distribution shifts. I would like to point the authors to Chen et al. (2018), which considers learning a regression model under covariate shift with moment constraints. Their setting is a little different from the current paper (the current paper considers a more general problem), but I think it deserves to be mentioned.

Chen, X., Monfort, M., Liu, A., & Ziebart, B. D. (2016, May). Robust covariate shift regression. In Artificial Intelligence and Statistics (pp. 1270-1279). PMLR.

As a suggestion, I think that the discussion of related work could be broadened to include work at the intersection of distribution shift, causality, and distributional robustness. Tools from causality have been used to represent structural knowledge about distribution shifts, and recent works have established connections between causal predictors and distributional robustness under shifts arising from interventions.

Causal inference & distributional robustness:
- Rothenhäusler, D., Meinshausen, N., Bühlmann, P., & Peters, J. (2021). Anchor regression: Heterogeneous data meet causality. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 83(2), 215-246.
- Meinshausen, N. (2018, June). Causality from a distributional robustness point of view. In 2018 IEEE Data Science Workshop (DSW) (pp. 6-10). IEEE.
- Rojas-Carulla, M., Schölkopf, B., Turner, R., & Peters, J. (2018). Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1), 1309-1342.
- Subbaswamy, A., Schulam, P., & Saria, S. (2019, April). Preventing failures due to dataset shift: Learning predictive models that transport. In The 22nd International Conference on Artificial Intelligence and Statistics (pp. 3118-3127). PMLR.

Using causal graphs to represent knowledge about shifts.
- Subbaswamy, A., Schulam, P., & Saria, S. (2019, April). Preventing failures due to dataset shift: Learning predictive models that transport. In The 22nd International Conference on Artificial Intelligence and Statistics (pp. 3118-3127). PMLR.
- Pearl, J., & Bareinboim, E. (2011, August). Transportability of causal and statistical relations: A formal approach. In Twenty-fifth AAAI conference on artificial intelligence.

As a very minor note: On line 16, reference [47] actually coins the term "dataset shift" for the "distribution shift" problem the authors refer to.

### UPDATES:
Thanks to the authors for their response. I think the practical discussion of the radius (i.e., its interpretation and general guidance on how to select it) should be included in the main paper. Even better would be to include or modify an experiment to demonstrate the (subjective) process by which a user would select a radius. Contingent on the inclusion of this practical discussion (as well as the expansion of the related work), I am raising my score 1 point.
limitations_and_societal_impact: Yes, though I wonder if there might be any implications for performance on subgroups created by the choice of shift constraints.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a new MDI-DRO approach that is the combination of an I-projection (to adjust for distribution shift) and DRO over both a KL-divergence constraint and a moment constraint. The major contribution is the generalization / efficiency theory results, and a convex optimization procedure for solving the DRO problem reasonably efficiently.
main_review: The introduction was mostly clear, but the discussion of P^f around line 97 stuck out, because there's is an abrupt (and very important) statement 'we identify P^f with an I-projection of P onto Pi' . But this is actually a really tricky and somewhat subtle point. You can either view this as an assumption on P^f, where you get to pick an arbitrary phi, or a statement that phi is fixed (as in prop 3.5) and P^f is arbitrary. In either case, this seems like in many ways the key assumption and structure, so I'd be alot more careful about the writing here. 

As a related work note, I believe 'Robust Classification Under Sample Selection Bias' and the line of work by Brian Ziebart may be relevant. There's some similarity to this work in that they consider minimax games with moment constraints on the distribution. Their setup is similar to the DRO problem in eq (5), but they don't have a I-projection step as in eq (4). I think it would add to the paper to carefully discuss the differences between these approaches and MDI-DRO.

The writing in section 5 can also be made much clearer. I wasn't sure exactly what 'z' was until Thm 5.2, where it became clear that it was going to be used to construct \mu_{k,\eta} which was the solution. I'd suggest a slight rewrite that brings the high level plan before the details in the 'computation of i projection' section. 

I also think the slater point discussion could be improved a bit - my understanding is that this is a fairly mild assumption, equivalent to the existence of an interior point, but the writing doesnt make it clear if this is something that's a regularity condition that can safely be treated as a detail, or something that has substantial implications for the applicability of this optimization approach. 

Experiment wise, the second, 'real world data' experiment has several points that could use further discussion. The left panel shows out of sample cost that increases substantially as r goes to zero. This seems fairly counterintuitive, esp given the discussion in the earlier experiment that there is a 'known tradeoff stating that small regularization parameter r leads to small out of sample risk'. It's also pretty surprising to me that the model doesn't degenerate as r grows large. at r=10.0, I'm guessing that the KL constraint is sufficiently loose that the KL constraint does very little, and almost all of the gains come from the moment constraint. Is this true?

As a framing thing, I think it may also be helpful to mention that the goal of the paper is in many ways the upper bound on the risk, rather than the out of distribution risk itself. This is because it seems likely that if your goal is only out of distribution performance, then you might as well take r=0, which would just be the I-projection.

Minor notes — I think the arguments in line 127 about the conditional limit theorem don't necessarily justify min. discriminating information (which is an argument made in line 129). My understanding was that the this is a type of consistency result (as n→infty, I-projections converge to the population) and I fail to see how this alone provides an intuitive justification for modeling distribution shifts via I-projections. 

As a style note- the large red boxes around the legends make it harder to read the figures and the legend.
limitations_and_societal_impact: No issues expected.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, they aim at test and training distribution shift problem where partial knowledge of shifted test distribution is known. With the principle of minimum discriminating information, they introduce a new modeling framework catching prior structural information, and a specific MDI-DRO with theoretical generalization bound, satisfying the statistical guarantee of out-of-sample.  For computational tractability, they leverage convex duality and accelerated first-order methods to efficiently solve the DRO program. Experiments on two distinct problem classes prove the availability of the proposed approach.

========
After reading the authors' response and other reviews, I would like to raise my score to 5, but still think the paper in its current form is not self-contained due to the lack of ablation study and important baselines.
main_review: 1. This work assumes and takes advantage of prior structure knowledge of tests in their modeling framework. So in their experiments, approaches that do not employ such information should be compared to verify the validity of leveraging partial test information. For example, there are a series of works facing an arbitrary and unknown distribution shift based on training data (Stable Prediction across Unknown Environments. KDD, 2018). 
2. The baselines and experimental settings are inadequate. Other DRO algorithms and other types of methods that use or do not use the test information should be discussed and compared, for example, self-normalized IPS (reweighing approach) and double robust methods for inventory control example. And the OOD generalization problem has developed to cover complicated data structures. More real data should be applied. 
3. The presentation of theory and method is confusing. Some propositions and examples can be moved to the appendix and they can highlight the theorem and related conclusion in writing.

limitations_and_societal_impact: I wonder if their approach can be applied to large-scale complex data structures, for example, image data that has been fully studied for the OOD problems.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes to solve a distribution shift problem by minimizing risk over an uncertainty ball around the empirical training distribution, subject to a shift with a known structure. This structure is encoded as knowing the expectation of some function of the random observation falls within a compact convex set E. The paper uses the principle of minimum discriminating to justify encoding *only* these structural requirements, being the least prejudiced approximation to the original distribution given only the structural knowledge.

The authors gives statistical guarantees and a tractable algorithm for finding the I-projection (the closest distribution in relative entropy which satisfies the structural requirement) of the empirical training distribution and then finding an approximate DRO solution around this projection.
main_review: I think this is a nice paper with a well-defined focus and useful results. My background is not in this area so I did not check the math in detail but I am in favor of acceptance. A couple notes:

* The paper is quite dense, particularly the examples. At a minimum, it would be helpful to indicate earlier on which of the examples will be persistent, so that the reader can focus on understanding these and perhaps not so much on the ones which will not appear again. This would also help justify the lengthy descriptions and encourage the reader not to skip them. I'd recommend even dropping all but the OPE example, or at least drastically shortening them.
* I'm struggling to understand the equation between lines 133 and 134. We're assuming N samples (an empirical expectation) but then also evaluating the true expectation? I've considered interpreting the LHS as $\mathbb{E}_{\mathbb{P}}[L(\theta, \xi) \mid \psi(\xi) \in E]$ but I think this is not correct. What is this meant to say?
* Line 137, $\ll$ is typically "absolutely continuous w.r.t." but I don't think it's defined anywhere. I think this term is probably worth explicitly defining.
limitations_and_societal_impact: Sure
review_title: null

Meta-review: Authors propose a DRO formulation that incorporates prior information in the form of moments of functionals, using minimum discriminating information. I agree with the reviewers that the new formulation is a meaningful contribution.

One remaining concern is that prior information studied in the experiments is somewhat contrived. The narrative of the paper will be substantially stronger if authors can demonstrate their method on a realistic example where the prior information presents naturally.
Decision: Accept (Poster)