Venue: ICLR 2022 Track Blog

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This blog post discusses results presented in the following paper:
- Sagawa *et al.*, '19



The introduction is well developed: it provides a good description of the main challenges and limitations of ERM for certain applications (e.g. medical applications) and it justifies the need for a different approach (DRO) that keeps into account that small subsets of the dataset may not show features that are displayed by the majority of the data (spurious correlations).



The description of the dataset is too long and it can be shortened, without losing its message. In particular the overall description of the three datasets (the bulleted list) can be shortened and integrated into the bold paragraph "For all the datasets, [...] breaks this correlation."

The main body of the blog post reproduces the arguments that are presented in the paper under study. However, not much insight is given over certain crucial passages, e.g.
- why a natural solution to reduce the generalization gap is to introduce regularization? (some motivation is given at page 5 of Sagawa *et al*, '19)
- why does $\ell^2$ regularization fails with BERT?
- why is the group adjustment term an additive power-like term $Cn_g^{-\frac{1}{2}}$? Could it have a different expression? (some motivation is given at page 6 of Sagawa *et al*, '19)

Additionally, some paragraphs are copy-paste from the paper ("maintaining a distribution $q$ over groups, with high masses [...] to the mass on its group.", "the algorithm provide stability [...] average loss at each iteration"), without much elaboration from the author(s) of the blog post.



I do recommend this blog post for publication, but I strongly advice for some changes and improvements to be implemented before acceptance.


### Style

- "The paper studies" is repeated several times within two consecutive paragraph (below Figure 1)

- there is a lot of swinging between the informal "we" ("we have", "we need", etc.) and the formal "the paper" ("the paper utilizes", "the paper finds", etc.)



### Major comments

- It looks from the outcomes at Table 4 that the choice of regularization is data-dependent ("It is also important to choose the best regularization technique for each dataset/model"). Could the author(s) elaborate on this statement? 
This is an important remark, but there is no follow up on what "best" means.

- The experiment on Figure 1 is missing any description on how it was conducted (what architecture? what minimization algorithm? what is the size of the training and test datasets? etc.). The numbers are also too suspiciously round.

- The major experiments presented in the blog post are simply drawn from Sagawa *et al.*, '19. No new experiments have been conducted to confirm (or not) the claims in the paper. 

-  "The algorithm provides stability": it is not explained what kind of stability this sentence is referring to.

- The section on the comparison between DRO and Importance Weighting feels rushed and it doesn't give much insight on why DRO works better.

- In the take-away messages, it should be stressed out that the stochastic algorithm proposed in the paper has theoretical convergence guarantees: see Proposition 2 in Sagawa *et al.*, '19, which is *not* discussed in the blog post. This is remarkable, since in the existing literature some of the (successful) proposed algorithms do not have convergence guarantees.


### Minor comments and typos

- bullet list under Formula (1): $L$ is the loss function $\mapsto$ $\ell$ is the loss function
- a group is sub-distribution  $\mapsto$ a group is a sub-distribution
- Even with DRO, models perform similarly to ERM models $\mapsto$ Even with DRO, models perform similarly to ERM
- it could be useful for the reader to explain what do the author(s) mean by "heavy" or "strong" regularization
- regularization changed the results (see Table 3) $\mapsto$ regularization changed the results (see Table 4)
- See Table 3 below for detailed results $\mapsto$ See Table 4 below for detailed results
- low worst-case loss also on both training and test data $\mapsto $  low worst-case loss on both training and test data
- Formula (3): what is $C$?
- it could be beneficial to the reader to explain that the algorithm is using two learning rates $\eta_\theta$ and $\eta_q$

- slightly different from equation () $\mapsto$ slightly different from equation (4)

- The algorithm provides stability and obtaining convergence guarantees [...] $\mapsto$ some English grammar is lacking
- And also theoretically by proving [...] $\mapsto$ this sentence has no subject


- Table 6 is the same as Table 5. In the caption of Table 6 it is mentioned upweighting (UW), which is never described in the blog post.

- group DRO can achieve a good worst-group generalization gap $\mapsto$ group DRO can achieve a small worst-group generalization gap


review_title: Referee Report: Distributionally Robust Neural Networks for Group Shifts: On The Importance of Regularization for Worst-case Generalization

Meta-review: As mentioned in the review, "The main body of the blog post reproduces the arguments that are presented in the paper under study". We considered that the added value of this blog post was too limited to be accepted. Also, in the future, we strongly recommend the author avoid copy pasting sentences from the initial paper.
Decision: Reject