Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors develop a framework named BREEDS for studying population shift, putting it in their words, they address the problem of how well do models generalize to data subpopulations they have seen during training, in the specific domain of images, without altering the inputs or requiring new data. They propose to create large scale subpopulation shift benchmarks  to assess how models generalize beyond the diversity of the training examples.  The underlying idea is to identify superclasses from the dataset. 
Breeds can be used as a tool to test and improve models and increase their generalization capabilities to distribution shift.
The problem is very important and the paper well written. However,  the applicability is limited to domains with class structure. The main application presented is images although the authors claim that the approach could generalize to other ares like nlp. Grouping similar classes may not be possible in many application and a discussion about it could have been helpful.
The lack of formalization of the framework makes the paper content hard to read and overall vague. However, with some effort, it is possible to get a clear idea of their contribution. Still a formalization would have been very desirable. 
Figure 1 claims it illustrates the pipeline but it is not clear what the figure represents, there is no pipeline per se. 
review_title: Important problem addressed, lack of formalization and distinction from the related work. 

final_score: rating: 7: Good paper, accept
review_content: ##########################################################################

Summary:


The paper introduces a new benchmark, based on a subset of Image-Net, to evaluate how well a model generalizes to data subpopulations which are not observed during training. The key concept is to create breeds, which is done in a semi-automatic fashion -- it starts from a modified version of the WordNet, and then calibrated by human annotators.

##########################################################################

Reasons for score: 


I tend to accept this work, because this benchmark is a good way to evaluate the model's generalization in terms of subpopulation. Though there were several datasets for domain generalization, I think the view of this benchmark is different and it can be a starting point for another direction.

##########################################################################

Pros: 


1. The benchmark introduced in this paper is very interesting. Roughly speaking, it tries to investigate: if I train a model with the images of "British shorthair cat" only (plus some other classes' images of course), and now present it an image of "American curl", will the model at least tell me it is a "cat".

2. The effort on constructing this dataset and evaluating existing methods is significant.

##########################################################################

Cons: 


I may disagree this benchmark can be viewed as an instance of domain generalization (DG). DG most commonly refers to evaluating a model trained with photo cat only can generalize to carton cat. This study still focus on one domain (one style of images). Therefore, the chosen interventions in Sec 5.2 may not be the best choices.

##########################################################################
review_title: A new benchmark for evaluating model's generalization on subpopulation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper addresses the problem of model robustness to subpopulation shift. Authors propose building large-scale subpopulation shift benchmarks wherein the data subpopulations present during model training and evaluation differ. In this regard, their approach is based on leveraging existing dataset labels and use them to identify superclasses to construct classification tasks over such superclasses and repurpose the original dataset classes to be the subpopulations of interest. They train some learning models over the generated benchmarks to evaluate model robustness to subpopulation shift and, finally, they try various learning interventions (from the literature) to decrease model sensitivity to this sort of data perturbations.


Strengths:

- Paper is very well-written.
- The problem addressed is very important (learning model generalisation to data shift) and of interest for the majority of ML/AI research community.
- The methodology followed is well defined and correct.
- The authors have performed an excellent work with the comprehensive experimental setting proposed in the paper.

Weaknesses:

- It is difficult to characterize what new scientific understanding or knowledge was presented in this paper. The presented approach for identifying superclasses and subpopulations of interests is somehow straightforward. Also, I doubt that manual procedure for hierarchy creation / restructuring process is a trivial task (with relatively little effort) for most benchmarks with no structured organisation. Results in sections 4.3 and 5.1 appear entirely unsurprising (though admittedly, this could be hindsight bias). Results in 5.2 appear to present more interesting insights (e.g.  little effect of train-time interventions on model robustness to subpopulation shift). However, one is left wondering whether this insight generalizes beyond the specifics of this experiments/dataset or whether this will create an isolated ImageNet sub-community for addressing image-classification robustness tasks.


Comments after author rebuttal:

Looking at the author's comments (as well as the other reviewer's feedback), I think that the authors have made a good job with the responses and I'm now more convinced about the usefulness of this work. I'm increasing my original recommendation to 6 "Marginally above threshold".




review_title: Straightforward approach for evaluating model robustness to subpopulation shift

Meta-review: I agree with the reviewers' positive comments about the paper. The BREEDS approach to generating benchmarks seems to be a useful one and addresses an important problem in the space. This approach could be the start of a nice direction of inquiry that will give us new insights into subpopulation shift. And most of the reviewers' negative concerns were addressed by the revision.
Decision: Accept (Poster)