Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work introduces R-PDRO, a distributionally robust optimization framework based on parametric likelihood ratios. Experimental results show that training models with R-PDRO are consistently more robust to distribution shifts when compared to other DRO approaches. 

main_review: Strengths:
- The idea is simple and yields better results than other methods. 
- The empirical evaluation is thorough. R-PDRO is studied from quite a few perspectives. In particular, the ablation study does a good job at demonstrating the effect and importance of each part of the algorithm. 
- The paper is well written and quite pleasant to read. Ideas are presented clearly.

Weaknesses:
- The likelihood ratio is known to be easy to estimate numerically when the denominator distribution $p$ covers the numerator distribution $q$ well enough but is otherwise difficult to estimate accurately. What happens when the training distribution $p$ is distributionally far from the worst-case distribution $q$?
- In Section 5.2, it is observed that simultaneous updates to both model and adversary parameters yield better results than when the adversary is allowed to take more gradient steps. I believe this is a major observation that the authors should investigate further since it shows that if Eq. 6 is solved exactly, as initially proposed in this work, then this leads to suboptimal results. As a matter of act, shouldn't this call for a new formulation of the objective? [A study of the training dynamics, similar to 1705.10461, is certainly worth carrying out, albeit well outside the scope of this paper.]

Questions:
- Batch-level normalization is said to "not introduce any additional hyper-parameters", but Eq. 9 includes the hyper-parameter $\tau$. Can you clarify the first statement?
- The evaluation is carried out on four benchmark datasets: Biased SST, FDCL18, Waterbirds, and CelabA. Sagawa et al (2020) evaluate on MultiNLI while Michel et al (2021) evaluate on DWMW17. Is there a reason why these two benchmarks were not considered in the experimental evaluation of R-PDRO?
summary_of_the_review: This paper proposes a simple and effective algorithm for training models with DRO. Experimental results are convincing. My main issue concerns the training objective, which leads to weaker results when solved more accurately. The preliminary observations discussed in Section 5.2 are a first step in this direction, but the contribution could be much stronger if the authors had investigated this more deeply. Nevertheless, I believe the overall contribution to be worthy of acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a modficiation to the DRO/PDRO method based on a parameterization of the various distributions that allows the problem to be recast in an optimization directly over likelihood ratios which belong to the exponential family. The paper additionally proposes two methods for controlling the normalizations and then tests the method against several datasets with under-represented subpopulations.
main_review: The paper does a good job of outlining the problem of interest and steps through the rationale for the design choices. Parts of the method seem a bit heuristic but the paper acknowledges this and the choices are largely justified by previous literature and empirical validation. The experiments support the claim that D-PDRO improves the performance for under-represented subpopulations but does so at the expense of the better-represented populations. This seems like a reasonable trade-off and is consistent with robust/standard accuracies in the adversarial setting. The problem of reducing bias against under-represented populations is a relevant issue and this method takes a positive step in that direction.
Overall, the paper is well-organized and the development makes sense but the method has many parts that build on one another which makes things seem somewhat convoluted. Intermediate figures/cartoons or an algorithm block may help to clarify/streamline the method. Additionally, the front-matter focuses heavily on the general distribution shift problem but the experiments focus entirely on subpopulation under-representation within a larger dataset. The issues are clearly related and I do not have any misgivings with this focus in the experiment section, however, it would feel less disjoint if the paper alluded to this focus earlier.

Minor issues:
Some of your figures have gridlines and some do not. I would recommend adding the to all the figures.
summary_of_the_review: The paper addresses an important, modern issue (subpopulation bias in large datasets) and does a good job explaining, developing, and exercising the idea on appropriate datasets with relevant baselines. I recommend accepting the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method to robustify any risk minimization learning to data set shift in the test set. It sets up an adversarial min-max framework where the learner needs to minimize a loss function but under a max for perturbations of reweighting the training distribution.
main_review: This paper tackles an important problem of data set shift. Most methods in industrial application will need to have some robustness to data set shift since there is always the potential for temporal drifts in the data distribution from when the training data was collected to when a system is deployed.

The setup to reparameterize from a constraint on the shift distribution q to a reweighting function r in (5) is clever and useful.

Comments:
- It would be interesting to see this method benchmarked against simpler baselines like randomly reweighting the training points in a non-adversarial way. If that weighting was done in a way that was tuned across many problems it might be quite competitive and simpler.

- In order to find more "natural" examples of data set shift, it would be good to find benchmark data sets that are in principle iid but have a timestamp associated with them. If the train test split were done temporally instead of randomly it would give some natural example of more subtle data set shift.

- I think the paper should make a clearer distinction between the case that p(x) shifts and p(y|x) remains the same, and the much more difficult case where p(y|x) changes as well. Which methods work best will likely be a function of which scenario one is in.

Questions:
- The paper considers bounding the data set shift in terms of KL. However, would it make more sense to consider limiting the data set shift in terms of JSD? That might be more interpretable because it could phrased in terms of the number of data points required to distinguish the new distribution from the original training distribution.

- The paper considers a batch level normalization constraint to make sure the adversarial distribution is actually normalized. It would be interesting to see a study into how effective that actually is at enforcing normalization. Are those distributions actually normalized if we use heavier duty methods (e.g., HMC) to assess it?

- The study of label noise is interesting in that proposes a theory on why the nonparametric robust optimization methods don't work as well. It would be interesting to further follow up on this ablation using a semi-synthetic data set that is homoscedastic by construction. Because in that case, the nonparametric method should not be able to hyper focus on the high noise regions of the space.
summary_of_the_review: Overall good paper with potential impact. Some things should have been evaluated better.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: As far as I can see, this work is about practical methods for distributionally robust optimization (DRO) with a special focus on trying to overcome the limitations of previous methods that have been proposed for this kind of problem. This work builds on the idea of instance-reweighting of the loss function via a re-weighting function that plays the role of the likelihood ratios (between the distributions in the class used for enforcing robustness and the training distribution). The re-weighting functions considered here are the well-known "exponential weights" and this work proposes and explores (i) using mini-batch level renormalizations and (ii) ensuring the KL-constraint by adding a KL penalty term to the training objective. The framework and corresponding objectives are described, and they are tested in experiments on benchmark image and text classification data sets, and compared to other existing DRO methods and to plain ERM.
main_review: PROS

The paper deals with an interesting problem of timely relevance.

The proposed practical method for DRO looks promising and may deserve the attention of the community.

The paper is relatively well written (some editorial feedback is given below).


CONS

Some unclarity in the presentation of ERM and DRO methods (distinction between "ideal" and "empirical" objectives).

Missed some pointers to work on "DRO with guarantees" e.g. what's been done on the theory literature for DRO. 

The previous is suggesting that some pointers could be helpful just to inform the reader (maybe in the section on related literature).

A number of editorial changes (minor mostly) need to be made for the paper to be in publishable form.


EDITORIAL FEEDBACK

Section 1 (Introduction):

Replace "neural-network" with "neural network" (no hyphen)

First line of second paragraph: replace "explained by" with "attributed to"

Regarding this second paragraph: The ERM principle consists of using the empirical risk (average loss over the finite sample) as optimization objective. Ideally the optimization objective should be the risk, i.e. the expected loss on a random example, but the latter is inaccessible because the distribution is unknown. This passage as currently written might a bit misleading.

Suggesting a rewrite:

"In ERM, models are trained to minimize the average loss over a finite sample from a fixed training distribution (Vapnik, 1992). This is because the empirical risk is taken as a proxy for the risk $\mathbb{E}_{(x,y) \sim p}[\ell_\theta(x,y)]$, namely the expected loss on a random example chosen from the training distribution $p$, which is assumed to be a fixed distribution, albeit unknown."

Next sentence: "This favors models which perform well \emph{on average} on samples from a fixed distribution, as opposed to models which would perform equally well on a variety of distributions that may better reflect the diverse set of subpopulations that can be encountered at test time, including those that may not have been represented adequately by the training distribution."

Next sentence: "On the other hand, distributionally robust optimization (DRO) proposes an appealing alternative to risk minimization."

Might be worth mentioning that the "uncertainty set" ($\mathcal{Q}$) is also called "ambiguity set" in some literature.

The objective $\mathcal{L}_\mathrm{DRO}(\theta)$ defined in Eq. (1) corresponds to the "ideal objective" I think, while in practice the implemented objective will be an empirical version of this objective? Say $\hat{\mathcal{L}}_\mathrm{DRO}(\theta)$, where the hat indicates the empirical version where the expectation is replaced with average over the finite sample? 

Alternatively, clarify how the expectations $\mathbb{E}_{(x,y) \sim q}[\ell_\theta(x,y)]$ are to be implemented.

Next page: Maybe use the name RP-DRO for your approach? 
To indicate the ratio-based (R) parametric (P) distributionally robust optimization (DRO).

"In this paper, we propose a new approach for DRO, called RP-DRO, based on a key modification
of the P-DRO algorithm: instead of modeling the worst-case distributions directly, we reparametrize 
the likelihood ratio between the training distribution and the worst-case distribution. This [...]"
 
Replace "likelihood-ratio" with "likelihood ratio" (no hyphen)

Replace "neural-network" with "neural network" (no hyphen)

This looks like a broken sentence: "a penalty-form to the KL divergence uncertainty set" -- rectify as per the intended meaning.

Choose one of "mini-batch" or "minibatch" to use consistently throughout the paper.

Same comment for "sub-population(s)" versus "subpopulation(s)"

Section 2:

Suggesting to rewrite what comes after Eq. (2):

"Notice that the DRO loss in Eq. (1) is the inner maximum in Eq. (2), it provides an upper bound on the expected loss of the model under any distribution in the uncertainty set Q, which motivates the use the minimizer of the min-max game in Eq. (2) as a robust model. We refer to the solution of the inner maximum as the “adversary” from now on."

Next sentence:

"However this objective is only useful insofar that (1) $\mathcal{Q}$ covers test distributions of interest (corresponding to different domains, demographics, etc.) and (2) $\mathcal{Q}$ is not overly pessimistic. To fulfil [...]"

Section 2.1:

Last line of first paragraph: replace $\chi_2$ with $\chi^2$ ?

Section 2.2:

Write "Equation (3)" or "Eq. (3)" for cross-references to equation numbers.

Choose one of "non-parametric" or "nonparametric" and use consistently throughout the paper.

Section 3:

Insert some text describing the section: "In this section we [...]"

Section 3.1:

"In the situation that all distributions in $\mathcal{Q}$ are absolutely continuous with respect to the training distribution $p$"

This "definition" of absolute continuity is restricted to discrete distributions. In case you consider more general distributions, the corresponding general definition of absolute continuity should be given instead.

Section 3.2:

Replace "likelihood-ratio" with "likelihood ratio" (no hyphen)

Replace "limit the choice" with "restrict the choice"

Top of next page: make the expression for the KL into a displayed math expression (no number, just display)

Two lines after Eq. (6): the method name in the subscript needs to be corrected (twice)

Next paragraph, the right hand side of the equation for $r-\psi$ is missing the normalizing constant.

"and instead we relax it" (insert "we")

"in the form of a term $\tau\mathbb{E}_p[..]$" (delete "KL" here)

Section 3.3:

In the second paragraph, I did not understand why this set is called a "simplex" -- can this be justified?

Or else replace "simplex" with "set"

Line after Eq. (7): "this penalizes adversaries that [..]"

Section 4:

Insert some text describing the section: "In this section [...]"

Section 4.1:

Second line of first paragraph: insert a comma after FDCL18

Next paragraph: replace "doesn't" with "does not"

Next paragraph: "off-the-shelf"

Section 4.2:

"BERT-based model (Devlin et al., 2018)"

In the captions of Table 1 and Table 2: "($p$-value < 0.05)"

Paragraph after Eq. (10): Using the zero-one loss makes complete sense for classification problems. Please write this part to make it very clear for what purpose the 0-1 loss is used (e.g. only for hyperparameter selection, or also for optimization?)

Replace $\chi_2$ with $\chi^2$ (twice)

Top of page 7: "We will release code for our experiments in a future revision" -- does this mean that the code will be released upon de-anonymization? (i.e. will the code be made available with the camera ready paper if accepted?)

Section 4.3:

Should "higher robust accuracies" be replaced with "more robust accuracies"? -- rectify as per intended meaning.

Next paragraph: replace "doesn't" with "does not"

Section 5:

Choose one of "Biased SST" or "BiasedSST" and use consistently throughout the paper.

Section 5.1:

Line 3 of first paragraph: "optimal weights generally only depend on"

Next paragraph: $\chi^2$

Section 5.2:

Line 2 of first paragraph: Replace "R-PDRO" with the method to which the comparison is made.

"Eq. (5)"

Section 5.3:

"matches still matches" has one too many matches:)

Section 5.4:

"the effective weight of each sample then depends on the weights of the other samples within the minibatch" ?

Choose one of "hyper-parameter(s)" or "hyperparameter(s)" and use consistently throughout the paper.





summary_of_the_review: Overall I enjoyed reading this paper. I think the main strength of this paper is in the experiments sections, demonstrating the advantages of their proposed DRO method with respect to previously proposed methods. I am being conservative with my evaluation mainly because I am not familiar with the whole literature in this area (notice my confidence of 3) and I look forward to see other reviewers' reactions to this work. I am willing to reconsider my evaluation if my feedback is addressed adequately and provided that no other flaws are found.
review_title: null

Meta-review: The paper builds upon parametric distributionally robust optimization (PDRO) and proposes ratio PDRO (R-PDRO) where the ratio of the worst case distribution and training distribution is parameterized by a discriminative network. This has a benefit over PDRO which needs to do generative modeling of worst case distribution. The paper empirically demonstrates R-PDRO improves over existing methods on group robustness problems. Reviewer are overall positive about the paper, and have appreciated the significance of the problem, writing clarity, and thorough empirical evaluation. There were some minor questions which have been adequately addressed by the authors.
Decision: Accept (Poster)