Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper provides a benchmark dataset that can be used for training & evaluation of Machine Learning models. Their contribution is that they have a large collection of 12,868 sets of natural images across 410 classes obtained from visual genome data and its metadata for annotations.  This helps in accounting for large natural shifts in the data. They also provide a way to measure distance between two subsets to quantize the distribution shift between any two of its data sets. 
The paper provides a good justification for the need of such a dataset for computer vision tasks and motivate the idea well. It also talks in detail about the steps taken to generate MetaShift from Visual Genome, it also provides a generalization of their 4-step process of dataset creation on any dataset with multilabel, results presented for COCO dataset. The paper further discusses the use of this dataset for two major cases- Evaluating distribution shifts & Assessing training conflicts. They provide the impact of shift distance on the domain generalization by keeping test set same and varying training subsets randomly. Further, it talks about subpopulation shifts where the train and test distribution of same domain with different mixture weights. They show that no algorithm consistently performs better than other algorithms for larger shifts. It provides a detailed understanding of training conflict by analyzing the contribution of each training subset to the change of the validation loss of each validation set during the training process. 
Overall, it’s a well written paper about the motivation, use cases, applicability, and generalizability of their proposed data set. 

main_review: Strengths:
Paper has a strong motivation for building the dataset. The authors also present a detailed understanding of major applications of their dataset across ML models. It provides a good quantization for the population shift and show with experiments how it impacts domain generalizability. The generalizability of the dataset creation for any dataset with multilabel is another strong point of the paper. 

Weakness:
The paper only talks about all advantages for MetaShift that’s derived from Visual Genome data but don’t have any similar comparison or quality analysis for other datasets generated using their 4-step dataset creation, for example, analysis could have been provided on COCO dataset too. The paper also doesn’t address the dependency of dataset performance on metadata, what if there are inconsistencies in the metadata of dataset but images are perfect, how will the dataset creation and performance be impacted for a dataset with such metadata. It would also have been interesting to see the analysis of performance of certain model around underrepresented subsets vs over represented subset, set/characteristics of datum leading to training conflict in general, if there is any pattern in images. Other obvious underlying data bias issue has already been acknowledged in the paper too and I hope the authors will research more into solving it. 

summary_of_the_review: I feel the authors have done a good job highlighting the motivation of such a dataset, steps of creation of the dataset from Visual Genome, paying attention to the generalizability of the approach, discussing the major applications of the dataset in detail. The github link also provides holistic understanding of the work. This dataset will help the research in the field of CV. Once they overcome and address the current weaknesses of the dataset, it will become even better dataset asset.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a collection called MetaShift to study the impact of dataset distribution. The major advantage of MetaShift is that it provides annotation/information to measure the amount of distribution shift between any two of its data sets. In the experiment, this work constructs two applications, 1) evaluating distribution shifts, assessing training conflicts. 
main_review: **[Strengths]** Good idea to study the impact of distribution shift.
+ This paper is well written and easy to understand. Section 3 gives a good introduction to the step-by-step construction of MetaShift.
+ The main advantage of MetaShift is it contains systematic annotation about the differences between different shifts. This further helps to study the effects of distribution shifts (e.g., subpopulation shifts)
+ The idea of generating a large number of real-world distribution shifts that are well-annotated and controlled is attractive. The proposed MetaShift is well illustrated and the figures are helpful to know the information of MetaShift. 
+ Section 4.3 is interesting and the results give some insights. 

**[Weaknesses]** The major concern is about the experimental evaluation where the constructed tasks are only binary classification. 
- In Section 4.1, this work constructs four binary classification tasks to study the impact of the shift under the generalization setting. One question is, how about constructing more challenging tasks which involve more classes? 
- When evaluating subpopulation shifts, the tasks are also binary which contain spurious correlation. The same question is multi-class classification tasks might be needed. 
- MetaShift is a collection of 12,868 sets of natural images from 410 classes. Why do the experiments only focus on binary classification (e.g., cat vs. dog, bus vs. truck)? In another word, it seems the same settings in Sec. 4 can be constructed based on other classes. It would be helpful if this work could discuss this. Again, more challenging multi-classification setting would be very useful.
summary_of_the_review: This work introduces a good way to study the effects of distribution shifts. Specifically, this work proposes a framework called Metashift, which contains systematic annotation about the differences between different shifts. However, the major concern is about the constructed tasks in the experiment. More explanation/discussion can be included to eliminate the question.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors introduce a new dataset (actully, a collection of datasets) called MetaShift. MetaShift is built on top of Visual Genome and leverages its metadata to cluster images, thus providing a context for each image (labels are of the form class+context, eg,  ‘cat in grass’, ‘dog in bathroom’). This context is then used to generate dataset shifts. Besides been much larger than similar (openly available) datasets, MetaShift explicitly provides the context, which can be used to compute a “distance score” pf distribution shift between any two datasets.
main_review: ### Pros
+ The paper is well written and easy to follow.
+ The proposed approach to leverage metadata (form previously published large-scale dataset) to create datasets of domain shifts is simple and well motivated. Splitting a large dataset (with multiple labels) in a meaningful ways to study dataset shift is not trivial. However the authors came up with an intuitive (and relatively simple) approach.
+ The problem of studying shifts in dataset distribution is very relevant and important to machine learning. This dataset can benefit the community by allowing a more system evaluation of dataset shifts.

### Cons
- It would be nice to have more descriptions of the methods used to benchmark the dataset (ERM, IRM, DRO, CORAL and CDANN), architectures used (which model? pretrained or from scratch? what is the model capacity?) and training details (which loss, which optimization, learning rate, batch size, etc). If not enough space on the main text, these informations could be added on appendix.
- It would be nice if the authors would give more detail on how the embeddings of meta-graphs are computed. For example what is the matrix A and how are the embeddings computed? Why using spectral embeddings specifically rather than other approaches (eg, using the word embedding (pretrained on large language corpus) of each context)?
- It could be nice to show some more quantitative or qualitative (eg t-SNE) for the meta-graph embeddings (used to compute the shift between datasets).
- The paper states it generates >12K datasets (across 410 classes), however experiments are done only on a very tiny number of datasets (cat/dog, bus/truck. elephant/horse and bowl/cup). The proposed dataset would be much more useful to the community if the authors would provide a much larger subset of “pre-made” datasets for easy experimentation.

summary_of_the_review: I am inclined to accept this paper because of (i) the simplicity of the approach to generate the datasets and (ii) the usefulness to the community. However, I would be more confident with acceptance if the authors would address the weaknesses of the paper (se above).
review_title: null

Meta-review: This work studies the impact of distribution shift via a collection of datasets-MetaShift. Reviewers all agreed that this work is simple, effective, and well-motivated, and has key implications, and will be quite useful to the community. There were some concerns about the lack of analysis of MetaShift, and the binary classification setting, which was addressed by the authors’ responses. Thus, I recommend an acceptance.
Decision: Accept (Poster)