Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: To the best of my knowledge, this is the first paper to carefully address and propose an algorithm (with guarantees) for distributionally robust learning in the overparametrized regime, which is typical of modern large deep neural networks. Following other work, the paper formalizes distributionally robust learning as the minimization of a worst-case loss over a set of possible distributions. The main message of the paper is that for distributionally robust learning, regularization plays an important role by avoiding perfect fitting to the training data, at the cost of poor generalization (thus lack of robustness) in some of the possible distributions. Furthermore, the paper proposes a new stochastic optimization algorithm to minimize the loss that corresponds to distributionally robust learning and gives convergence guarantees for the proposed algorithm.

I think this is an interesting and solid paper, with a clear presentation style, and well-supported contributions. To the best of my knowledge, this is novel work and, in my opinion, it is relevant work, both in terms of applicability as well as in terms of contribution to the understanding of the generalization behavior of overparameterized deep neural networks.

review_title: Official Blind Review #5

final_score: rating: 8: Accept
review_content: This paper describes a method of training neural networks to be robust to a worse case mixture of a set of predefined example attributes. This is done with a loss in accuracy in the average case but improvements in the worse case. The proposed algorithm is relatively simple and convergence rates are also given for this new algorithm.

Building neural networks that perform well in the face of group-level worse case test-set distributions is a very important problem particularly in areas such as health and safety-critical applications as past work points out. This paper shows good results in the worse case and additionally shows that the common technique of importance reweighting cannot arrive at the same solution. The convergence analyses also yield additional insight into this new algorithm. The paper is well written and relatively easy to understand with good details on the experimental setup. The algorithm has a downside in that the groups must be known a priori, is it possible for these groups to be learnt? Also, can using a hinge loss also improve robustness to the worse case examples?

However, there are some unanswered questions in the paper. What is the effect on the training time of this algorithm? Is it just the time for an additional forward prop? What is the effect on the worse-case examples of weight decay on the Bert model? Even though it hurts the average performance does it improve the worse case at all? In the last line of algorithm 1 why is q_G used instead of q_g?

Other comments:
At the bottom of P5: The ordering of 93.4% and 97.1% seem to be reversed.
Above eq. 5 , \delta_g seems to be overloaded. In the paragraph, it first refers to the generalization gap and then later to a heuristic.
Table 2: The drop in average accuracy for waterbirds does not seem 'small'.
Bottom of P8: 'background is more unique', it seems this is supposed to mean the background appears less often?

======================================================================================================
Update after rebuttal:

Thanks for the detailed answers to my comments and the additional experiments done with a hinge loss. I will keep my rating.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a novel training method based on group DRO that makes it possible for overparameterized neural networks to achieve uniform accuracy over different groups of the data, where the data is required to obey a mixture distribution composed of these groups. In addition, the authors identify that increased regularization plays a critical role for worst-group performance in the overparameterized regime by a series of empirical studies. The authors also compare  their method with the traditional importance weighting method. Finally, they introduce a stochastic optimizer for their group DRO method.

Pros: 

* The experiments are well designed and the results show the effectiveness of the approach.
* Most parts of the paper are well written and easy to follow.

 Cons:
* The title is a little bit vague and may overstate the paper's contribution. In fact, the problem addressed in the paper is not as general as the title suggests and has little to do with the generic DRO framework.
* In the group DRO setting the data is distributed as a mixture of different groups. However, the groups need to be chosen using prior knowledge of "spurious associations", which may arouse doubts about the algorithm's actual effectiveness and undermines the value of the algorithm on the application side.
* Some terms used in the paper are ambiguous. For example the concept of  "group" is not defined properly. Sometimes it refers to a subset of the dataset while sometimes it denotes a sub-distribution in the mixture distribution. The authors should use more precise terminology.
* There is a typo in Proposition 1, where the uncertainty set Q should be a set of probability distributions,  not a subset of R^m.
review_title: Official Blind Review #2

Meta-review: This paper proposes distributionally robust optimization (DRO) to learn robust models that minimize worst-case training loss over a set of pre-defined groups. They find that increased regularization is necessary for worst-group performance in the overparametrized regime (something that is not needed for non-robust average performance).

This is an interesting paper and I recommend acceptance. The discussion phase suggested a change in the title which slightly overstated the paper's contributions (a comment which I agree with). The authors agreed to change the title in the final version. 

Decision: Accept (Poster)