MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts
Authors: Weixin Liang
James Zou
Venue: ICLR 2022 Poster (2022)
Keywords: benchmark dataset
distribution shift
out-of-domain generalization

Abstract: Understanding the performance of machine learning models across diverse data distributions is critically important for reliable applications. Motivated by this, there is a growing focus on curating benchmark datasets that capture distribution shifts. While valuable, the existing benchmarks are limited in that many of them only contain a small number of shifts and they lack systematic annotation about what is different across different shifts. We present MetaShift—a collection of 12,868 sets of natural images across 410 classes—to address this challenge. We leverage the natural heterogeneity of Visual Genome and its annotations to construct MetaShift. The key construction idea is to cluster images using its metadata, which provides context for each image (e.g. “cats with cars” or “cats in bathroom”) that represent distinct data distributions. MetaShift has two important benefits: first, it contains orders of magnitude more natural data shifts than previously available. Second, it provides explicit explanations of what is unique about each of its data sets and a distance score that measures the amount of distribution shift between any two of its data sets. We demonstrate the utility of MetaShift in benchmarking several recent proposals for training models to be robust to data shifts. We find that the simple empirical risk minimization performs the best when shifts are moderate and no method had a systematic advantage for large shifts. We also show how MetaShift can help to visualize conflicts between data subsets during model training. 

Published as a conference paper at ICLR 2022
METASHIFT: A DATASET OF DATASETS FOR EVALUAT- ING CONTEXTUAL DISTRIBUTION SHIFTS AND TRAIN- ING CONFLICTS
Weixin Liang
James Zou Stanford University j www.d:etttanrdddeeeeoooo edu
Stanford University wxLiangest
ABSTRACT
Understanding the performance of machine learning models across diverse data distributions is critically important for reliable applications. Motivated by this, there is growing focus on curating benchmark datasets that capture distribution shifts. While valuable, the existing benchmarks are limited in that many of them only contain a small number of shifts and they lack systematic annotation about what is different across different shifts, We present MetaShift-a collection of 12,868 sets natural images across 410 classes-to to address this challenge. We leverage the natural heterogeneity of Visual Genome and its annotations to construct MetaShift. The key construction idea is to cluster images using its metadata, which provides context for each image (e.g. cats with cars cats in bathroom) that represent distinet data distributions. MetaShift has two important benefits: first, it contains orders of magnitude more natural data shifts than previously available. Second, it provides explicit explanations of what is unique about each of its data sets and a distance score that measures the amount of distribution shift between any of its data sets. We demonstrate the utility of MetaShift in benchmarking several recent proposals for training models be robust to data shifts. We find that the simple empirical risk minimization performs the best when shifts are moderate and no method had a systematic advantage for large shifts. We also show how MetaShift can help to visualize conflicts between data subsets during model training
INTRODUCTION
major challenge machine learning (ML) is that model can have very different performances and behaviors when it's applied to different types of natural data (Koh et al. 2020 Izzo al.) 1021 2022). For example, if the user data have different contexts compared to the model's training data (e.g. users have outdoor dog photos and the model's 's training
mostly on indoor images), then the
model's 'ss accuracy can greatly suffer (Yao et al., (022). A model can have disparate performances even within different subsets within its training and evaluation data (Daneshjou al., 2021 Eyuboglu let al. 2022). In order to assess the reliability and fairness
model, we therefore need to evaluate its performance and training behavior across heterogeneous types of data. However, the lack of well-structured datasets representing diverse data distributions makes systematic evaluation difficult.
In this paper, we present MetaShift tackle this challenge. MetaShift is a collection of 12,868 of natural images from 410 classes. Each set corresponds to images in a similar context and represents a coherent real-world data distribution, as shown Figure [ The construction MetaShift is different from and complementary to other efforts to curate benchmarks for data shifts by pulling together data across different experiments or sources. MetaShift leverages heterogeneity within the large sets of images from the Visual Genome project (Krishna et al.) 2017) by clustering the images using metadata that describes the context each image. The advantage of this approach is that MetaShift contains many more coherent sets of data compared to other benchmarks. Importantly, have explicit annotations of what makes each subset unique (e.g. cats with cars or dogs next to a bench) as
Dataset and code available fictps://metase ft. readthedocs i//

---

Published as a conference paper at ICLR 2022
Dog (Frisbee)
Dog (Surfboard)
Dog (Car)
Dog (Boat)
Cat (Keyboard)
Cat (Sink)
Cat (Box)
Cat (Remote Control)
Figure I: Example subsets of natural images from MetaShift. MctaShift leverages the natural heterogencity within each class (e.g., "cat", "dog") to provide many subsets of images. Each subset corresponds to images in similar context (the context stated in parenthesis) and represents coherent real-world data distribution. Here, we only show 2 out of 410 classes and out 12,868 subsets images from MetaShift.
well as a score that measures the distance between any two subsets, which is not available in previous benchmarks of natural data.
We demonstrate the utility of MetaShift in two applications. First, MetaShift supports evaluation on both domain generalization and subpopulation shifts settings. Using the score between subsets provided by MetaShift, we study ML models' behavior under different carefully modulated amounts of distribution shift. Second, MetaShift can also shed light on the training dynamics of ML models. Since we have the subset membership information for each training datum, we could attribute the contribution of each gradient step back to the training subsets, and then analyze how different data subsets provide conflicting training signals.
Our contributions: We present MetaShift as an important resource for studying the behavior of ML algorithms and training dynamics across data with heterogeneous contexts. Our methodology for constructing MetaShift can also be applied to other domains where metadata is available. We empirically evaluate the performance of different robust learning algorithms, showing that ERM performs well for modest shifts while no method is the clear winner for larger shifts. This finding suggests that domain generalization is an important and challenging task and that there's still a lot of room for new methods.
2 RELATED WORK
Existing Benchmarks for Distribution Shift Distribution shifts have been a longstanding challenge in machine learning. Early benchmarks focus on distribution shifts induced by synthetic pixel transformations. Examples include rotated and translated versions of MNIST and CIFAR (Worrall et al. (017);; surface variations such as texture, color, and corruptions like blur in Colored MNIST (Gulrajani & Lopez-Pazi 2020), ImageNet-C (Hendrycks Dietterich] 2209)). Although the synthetic pixel transformations are well-defined, they generally do not represent realistic shifts in real-world images that we capture in MetaShift.
Other benchmarks do not rely on transformations but instead pull together data across different experiments or sources. Office-31 (Sæenko et al., 2010) and Office-home (Venkateswara et al.) 2017)
contain images collected from different domains like Amazon, clipart. These benchmarks typically
have only a handful of data distributions. The benchmarks collected in WILDS (Koh et al. 2020)
combine data from different sources (e.g., medical images from different hospitals, animal images
2

---

Published as a conference paper at ICLR 2022
from different camera traps). Similarly, some meta-leaming benchmarks (Triantafillou al. 2019
Guo et al. 2020) focuses on dataset-level shift by combining different existing datasets like ImageNet, Omniglot. While valuable, they lack systematic annotation about what is different across different shifts. Santurkar et al. (2020); Ren et al. (0188) utilize the hierarchical structure of ImageNet to construct training and test sets with disjoint subclasses. For example, the "tableware" class uses "beer glass" and "plate" for training and testing respectively. Different from their work, we study the shifts where the core object remains the same while the context changes. NICO (He et al. (020) query different manually-curatee phrases on search engines to collect images of objects in different contexts. key difference is the scale of MetaShift: NICO contains 190 sets of images across 19 classes while MetaShift has 12,868 sets of natural images across 410 classes.
To sum up, the advantages of our MetaShift are:
Existing benchmark datasets for distribution shifts typically have only a handful of data distribu- tions. In contrast, our MctaShift has over 12,868 data distributions, thus enabling a much more comprehensive assessment of distribution shifts.
Distribution shifts in existing benchmarks are not annotated (i.e. we don't know what drives the shift) and are not well-controlled (i.e. we can't easily adjust the magnitude the shift). The MetaShift provides explicit annotations of the differences between any two sub-datasets, and it quantifies the distance of the shift.
Evaluating conflicts on training data Recent work has shown that different training data points play heterogeneous role in training. To quantify this, Data Shapley (Ghorbani & Zou 2019 Kwon
& Zou 2022) provides mathematical framework for quantifying the contribution of each training datum. Data Cartography (Swayamdipta et al.] 2020) leverages a model's training confidence to discover hard-to-learn training data points. Such understanding has provided actionable insights that benefit the ML workflow. For example, removing noisy wwwcootriiuttion training data points improves the model's final performance (Liang et al.) [2200a] 2021). Furthermore, active learning identifies the most informative data points for humans to annotate (Liang et al. 2020b). Complementary to their work, our analysis sheds light on not only which but also why a certain portion of the training data are hard-to-leame because different subsets are providing conflicting training signals.
3 THE METASHIFT CONSTRUCTION METHODOLOGY
What is MetaShift? The MetaShift is collection of subsets of data together with an annotation graph that explains the similarity/distance between two subsets (edge weight) well as what is unique about each subset (node metadata). For each class, say "cat", we have many subsets of cats, and we can think of each subset as node in the graph, as shown in Figure 2 Each subset corresponds to "cat" in a different context: e.g. "cat with sink" or "cat with fence". The context of each subset is the node metadata. The "cat with sink" subset is more similar to "cat with faucet" subset because there are many images that contain both sink and faucet. This similarity is the weight of the edge; a higher weight means the contexts of the two nodes tend to co-occur in the same data.
How can we use MetaShift? It is flexible framework to generate a large number of real-world distribution shifts that are well-annotated and controlled. For each class of interest, say "cats", we can use the meta-graph of cats to identify a collection of cats nodes for training (e.g. cats with bathroom-related contexts) and a collection of cats nodes for out-of-domain evaluation (e.g. cats in outdoor contexts). Our meta-graph tells us exactly what is different between the train and test domains (c.g. bathroom vs. outdoor contexts), and it also specifies the similarity between the two contexts via graph distance. That makes it easy to carefully modulate the amount of distribution shift. For example, if we use cats-in-living-roon as the test set, then this is a smaller distribution shift.
Visual Genome We leverage the natural heterogeneity of Visual Genome and its annotations to construct MetaShift. Visual Genome contains over 100k images across 1.702 object classes. For each image, Visual Genome annotates the class labels of all objects that occur in the image. Formally, for each image z(0,, we have a list of meta-data tags (+()) = (t(() t ..... +(0 }. each indicating the presence of an object in the context. We denote the vocabulary of the meta-data tags as M- = {ma.... mp J. MetaShift is constructed on a class- blass-by-class basis: For each class, say "cat", we pull out all cat images and proceed with the following steps.
Base Dataset:
3

---

Published as a conference paper at ICLR 2022
cet (blanket).
cBt (sheet)
CCF
(comforter)
(bed)
cat (towell
(cortains
cat [sink]
Fooo
controll
(couch)
(pillow)
(doat)
icounter)
CBB (fauset)
(window))
(tailett
(person)
el
(collar)
cat (picture)
(blinds)
(cabineti
(drawer)
(bag))
(tance)
book
(bottle)
cat (man)
(lamp)
(wall)
(mat)
Mespkshelly
(vase)
cal
(ground)
cat
1g Tglasses
Containery
(table]
(pooks)
call (carpet)
(biind)
(paper)
(chair)
cat (rug)
memera
(bak)
(vomen)
graass)
(water)
(banch)
cat (speakeer)
(telexision)
ttoy)
(mon
(DOw))
cDt (plate)
ppppersi
(keyboard)
(Tood)
cak
ipptopl
(comgute (computer mquselen)
the diverse data distributions within the "cat" class. Each node represents subset of the cat images. Each subset corresponds to "cat" in different context: e.g. "cat with sink" "cat with fence". Each edge indicates the similarity between the connecting subsets. Node colors indicate the communities automatically detected by graph-based algorithms. Inter-community edges are colored and intra-community edges are grayed out for better visualization. The border color of each example image indicates its community in the graph. We have one such meta-grap meta-graph for each of the 410 classes in the MetaShift. Beyond visualization, meta-graph also provides natural and systematic way quantify the distance between any two subsets (i.e., nodes), which not available in previous benchmarks natural data.
Figure 2:
Step 1: Generate Candidate Subsets We first generate candidate subsets by enumerating possible meta-data tags. We construct [M] candidate subsets where the ith subset contains all images of the class of interest (i.c., "cat"" that has meta-tag m,, We then remove subsets whose sizes are less than threshold (c.g..2))
Step 2: Construct Meta-graphs Since the meta-data not necessarily disentangled, the candidate subsets might contain significant overlaps (e.g., "cat with sink" and "cat with faucet"). To capture this phenomenon, construct meta-graph model the relationships among all subsets of each class. Specifically, for each class j E Y, we construct meta-graph, a weighted undirected graph denotes a candidate subset, and the weight of each edge the
(V,E) where each node overlap coefficient between two subsets:
XNYI min([X], [Y])
overlap
(X (verlap(XXY)
(1)
We remove the edges whose weights are less than threshold (e.g., 0.1) to sparsify the graph. As shown Figure ? the meta-graph G captures meaningful semanties of the multi-modal data distribution of the class of interest.
Step 3: Quantify Distances of Distribution Shifts The geometry of meta-graphs provides natural and systematic way to quantify the distances of shifts across different data distributions: Intuitively, if two subsets far away from each other in the MetaGraph, then the shift between them tend to be large. Following this intuition, we leverage spectral embeddings (Belkin & Niyogi 20033; Chung & Graham 1997) to assign embedding for each node based the graph geometry. Spectral
4

---

Published as a conference paper at ICLR 2022
410 classes
Avg. 31.4 subsets/class,
1,853 distinct contexts
Avg. 200.4 images/subset
clock, bridge, sign, water, carriage, policeman,
bus
1,702 object presence
w/ clock
w/ bridge
w/ sign
w/ water
horse
indoor, outdoor, bedroom, ocean,
37 general contexts
carriage
policeman
w/ fence
dog
red, plastic, triangular,
cup
114 object attributes
candle
toothbrush
cat
w/ clock
Figure 3: Summary of MetaShift. MetaShift covers wide range of 410 classes and 12,868 sets of natural images total. For each class, we have 31.4 subsets average together with annotation graph (i.e., meta- graph) that explains the similarity/distance between two subsets (edge weight) well what unique about each subset (node metadata), More concretely, the subsets are characterized by diverse collection of 1,853 distinct contexts, which covers 1,702 object presence, 37 general contexts and 114 object attributes.
embedding minimizes the expected square distance between nodes that are connected:
min OXT = &JEV
Σ A1, X1
where X; is the embedding for node i E V and K is the dimension of the embedding, and A is the adjacency matrix. We denote by X the matrix of dimension
K whose i-th row X, corresponds to the embedding of node i. The constraint X = O forces the embedding to be centered and X X IK ensures that we do not get trivial solution like node embeddings located at the origin (i.e., X 0). Denoting by L D - A the Laplacian matrix of the graph, we have:
tx(x = Σ
Au |X, - xx,1
(3)
'EEV
K+1
min -XX
tn x LX) = Σ A
(4)
2=2
The minimum is reached for X equal to the matrix of eigenvectors of the Laplacian matrix associated with the cigenvalues .... Äx+ +1+ After calculating the spectral embeddings, we use the euclidean distance between the embeddings of two nodes as their distance. Other off tff-the-hhelf node embedding methods like Node2Ves Vec (Grover Leskovec. 2016) can be readily plugged into MetaShift. We delay exploring them as future work.
Although the subset overlap (i.e., edge weight) can also be used as similarity metric, does not incorporate the structural information from neighboring nodes. Our node embedding-ben distance captures not only such overlap, but also broader similarities.
Step 4: Simulating Distribution Shifts MetaShifts allows users benchmark both (1) domain generalization and (2) subpopulation shifts in a well-annotated (explicit annotation of what drives the shift) and well-controlled (easy control of the amount of distribution shift) fashion.
In domain generalization, the train and test distributions comprise data from related but distinct domains. This arises in many real-world scenarios since is often infeasible to construct a comprehensive training set that spans all domains. To simulate this setting, sample two

---

Published as a conference paper at ICLR 2022
distinct collections of subsets the train domains and the test domains respectively (e.g. bathroom vs. outdoor contexts). To adjust the magnitude of the shift, we can fix the test domains and change the train domains with different distances. For example, if we use cats -n- in-living-room as the test set, then this is a smaller distribution shift.
In subpopulation shifts, the train and test distributions are mixtures of the same domains, but the mixture weights change between train and test. It is also an important setting since ML models are often reported to perform poorly on under-represented groups. To simulate this setting. we can sample the training set and test set from the same subsets but with different mixture weights. To adjust the magnitude of the shift, we can use different mixture weights for the training set while keeping the test set unchanged.
We demonstrate both settings in the experiment section. It is also worth noting that different subsets may share common images-eee dog with grass and dog with frisbee. Therefore, a post-processing step is needed to remove the training images that also occur in the test set to ensure no data leakage.
dog image can have both grass and frisbee would occur both
MetaShift Statistics Figure 3 shows the statistics of MetaShift across all tasks. We start from the pre-processed and cleaned version of Visual Genome (Hudson & Manning. 2019), which contains 1 113,018 distinct images across 1,702 object classes. After the dataset construction, we have 12,868 sets of natural images from 410 classes. Concretely, each class has 31.4 subsets, and each subset has 200.4 images on average.
The subsets are characterized by a diverse vocabulary of 1,853 distinct contexts. Beyond 1,702 contexts defined by object presence, MetaShift also leverages the 37 distinct general contexts and 1 14 object attributes from Visual Genome. The general contexts typically describe locations (i.e., indoor, outdoor). weather (e.g., rainy, cloudless), and places (e.g.. bathroom, ocean). The object attributes include color (e.g., white, red), material (e.g., wood, plastic), shape (e.g., round, square), and other object-specific properties (e.g., empty/full for plates). See Appendix A for examples and more information.
Generalizability: Case Study on COCO The MetaShift construction methodology is quite simple, and can be extended any dataset with metadata tags (i.e., multilabel). To demonstrate this, we apply the construction methodology on MS-COCO dataset (Lin et al. 2014). which provides object detection labels for each image. Applying our construction methodology we are able construct 1321 subsets, with an average size of 389 and median size of 124, along with 80 meta peta-graphs that help to quantify the amount of distribution shift among MS-COCO subsets. Experiments on MetaShift from COCO can be found in Appendix D
Even though the data that we used here (e.g. Visual Genome, COCO) not new, the idea of turning one dataset into a structured collection of datasets for assessing distribution shift is less explored. COCO (Lin et al. 2014) and many other datasets provide meta-data that could be naturally used for generating candidate subsets. The main contribution of our methodology is that after generating the candidate subsets, our method provides a meta-graph that could be used to determine the train and test domains, and also specifies the similarity between the two subsets via graph distance. MetaShift opens up new window for systematically evaluating domain shifts.
EXPERIMENT
We demonstrate the utility of MetaShift two applications:
Evaluating distribution shifts: MetaShift supports evaluating both domain generalization (Sec- tion 4 and subpopulation shiftss (Section 4.2p. We perform controlled studies on ML models' behavior under different amounts of distribution shift. We also evaluate several recent proposals for training models to be robust to data shifts.
Assessing training confficts: The subset information in MetaShift also sheds light on the training dynamics of ML models (Section 4.3). Since we have the subset membership information for each training datum, we could attribute the contribution of each gradient step back to the training subsets, and then analyze the heterogeneity of the contributions made by different training subsets.

---

Published as a conference paper at ICLR 2022
Task 1: Cat vs. Dog
Task Bus vs. Truck
Task Elephant Horse
Task 4: Bowl vs. Cup
Algorithm
1=0.444=:0771
d=1.43
d=0.81
d=1.20 d=1.42
d=1.52
d=0.44 d=0.63
d=0.89
d=1.44
d=0.16
d=1.03 d=1.31
ERM
0.844 0.605 0.357
0.240
0.950
0.863
0.702
0.609
0.964
0.821
0.793
0.729
0,888
0.768
0,401
0.276
IRM
0.814 0.628
0.380
0,341
0.863
0.901
0.752
0.634
0.943
0.886
0.764
0.750
0.883
0.793
0,426
0.404
GroupDRO
0.837
0.597 0.434
0.264
0,901
0.857
0.770
0.665
0.936
0.864
0.829
0.743
0.829
0.765
0.444
0.303
CORAL
0.798
0.589
0.481
0.302
0.925
0.801
0.783
0.640
0.929
0.900
0.814
0.771
0,850
0.734
0.482
0.274
CDANN
0.729 0.620 0.380 0.326
0.944
0.888
0.789
0.584
0.921
0.857
0.836
0.779
0.879
0.752
0.432 0.280
Table Evaluating domain generalization: out-of-domain test accuracy with different amounts distribution shift Higher dl indicates more challenging problem. Test data are fixed and only training data are changed.
4.1 EVALUATING DOMAIN GENERALIZATION
In domain generalization, the train and test distributions comprise data from related but distinct do- mains. To benchmark domain generalization using MetaShift, we can sample two distinct collections of subsets as the train domains and the test domains respectively. We also showcase adjusting the magnitude of the shift by sampling different training subsets while keeping the test set unchanged.
Setup As shown in figure 2, subsets like "cat with sink" and "cat with faucet" are quite similar to each other. To make our evaluation settings more challenging, we merge similar subsets by running Louvain community detection algorithm (Blondel et al., 20088 on each meta-graph. Node color in Figure 2 indicates the community detection result. The embeddings of the merged subset are calculated as the average embeddings of subsets being merged, weighted by the subset size. After merging similar subsets, we construct four binary classification tasks:
1. Cat vs. Dog:
Test on dog(shelf) with 129 images. The cat training data is cat(sofa + bed) (i.e., cat(sofa) unions cat(bed)). We keep the test set and the training data unchanged. We keep the total size of training data as 400 images unchanged. We explore the effect of using 4 different sets of dog training data:
(a) dog(cabiner + bed) as dog training data. Its distance dog(shelf) is d=0.44
(b) dog(bag box) as dog training data. Its distance dog(shelf) is d=0.71
(c) dog(bench + bike) as dog training data. Its distance to dog(shelf)) is d=1.12
(d) dog(hoat- + surfboard) as dog training data. Its distance to dog(shelf) is d=1.43
2. Bus vs. Truck:
Test on truck(airplane) with 161 images. The bus training data is bus(clock + traffic light). The 4 different sets of truck training data are truck(cone + ffncc), truck(bike + mirror), truck(flag + tower), truck(traffic light + dog). Their distances to truck(airplane) are 1=0.81,d=1.22 d=1.42, d=1.52, respectively, shown in Table T
Elephant vs. Horse: Test on horse(barn) with 140 images. The elephant training data is ele- phantifence + rock). The 4 different sets of horse training data are horse(dirt + trees), horse(fence + helmet), horse(car+ wagon), horse(statue + carr), with distances d=0.44/0/ 4...0.44..663....... 89/1.44.
4. Bowl vs. Cup: Test on cup(coffee) with 375 images. The bowl training data is bowl(friits + tray). The 4 different sets of horse training data are cup(knife tray), cup(water + cabiner), cup(computer + lamp), cup(toilet + box), with distances d=0.
respectively.
Domain generalization algorithms Following recent benchmarking efforts (Koh et al. 2020
Gulrajani & Lopez-Paz 2020), we evaluated several recent methods for training models be robust to data shifts: IRM (Arjovsky et al., 2019), GroupDRO (Sagawa et al., 2020). CORAL (Sun & Saenko 2016), CDANN (Long et al. 2018). To ensure fair comparison, we use the implementation and the default hyperparameters from DomainBed (Gulrajani & Lopez-Paz, (022)). We discuss more
experiment setup details in Appendix B.1 We also extend the binary classification experiment here
to 10-class classification experiment in Appendix B.2.1
Results Our experiments show that the standard empirical risk minimization performs the best when shifts are moderate(i.e., when d is small). For example, ERM consistently achieves the best test accuracy on the smallest d of all tasks. This finding is aligned with previous research, as the domain generalization methods typically impose strong algorithmic regularization in various forms (Koh et al.] 22200) Santurkar et al. 2020). However, when the amount of distribution shift increases, the domain generalization algorithms typically outperform the ERM baseline, though algorithm
7

---

Published as a conference paper at ICLR 2022
Average Acc.
Worst Group Acc.
Algorithm
p=12% p=6% p=1% p=12% p=6% p= %
ERM
0.840 0,823 0.759 0.715 0.701 0.562
IRM
0.825 0.830 0.773 0.729
0.715 0,528
GroupDRO CORAL
0.837 0.818 0.762 0.701 0.715 0.576
0.835 0.816 0.755 0.667
0.694 0.597
CDANN
0.836 0,810 0.774 0.750 0.701 0.625
Table 2: Evaluating subpopulation shift: y is the percentage of minority groups in the training data. Lower indicates more challenging setting. Evaluated on balanced test set (p=50%). The classification task is "Cat vs. Dog" with "indoor/outdoor" as the spurious correlation.
a consistent winner compared to other algorithms for large shifts. This finding suggests that domain generalization is an important and challenging task and that there's s still lot of room for new methods.
4.2 EVALUATING SUBPOPULATION SHIFTS
In subpopulation shifts, the train and test distributions mixtures of the same domains with different mixture weights. This is a more frequently-encounteer problem since real-world datasets often have minority groups, while standard models are often reported to perform poorly on under-represented demographics (Buolamwini & Gebru 2018 Koenecke et al. (020).
Setup To benchmark subpopulation shifts using MetaShift, we can sample two distinct collections of subsets as the minority groups and majority groups respectively. We then use different mixture weights construct the training set and test set. For "Cat vs. Dog", we leverage the general contexts "indoor/outdoor" which have natural spurious correlation with the class labels. Concretely, in the training data, cat(ourdoor) and dog(indoor) subsets are the minority groups, while cat(indoor) and dog(outdoor) are majority groups. We keep the total size of training data as 1700 images unchanged and only vary the portion of minority groups. We use balanced test with 576 images report both average accuracy and worst group accuracy. We extend the binary classification experiment here to 10-class classification experiment Appendix B.2.2
Results Table 2 shows the evaluation results on subpopulation shifts. In terms of the average accuracy, ERM performs the best when p is large (i.e., the minority group less underrepresented) as expected. However, in terms of the worst group accuracy, the algorithms typically outperform ERM, showcasing their effectiveness in subpopulation shifts. Furthermore, it worth noting that no algorithm is a consistent winner compared to other algorithms for large shifts. This finding is aligned with our finding the previous section, suggesting that subpopulation shift is an important and challenging problem and there's still lot of room for new methods.
4.3 ASSESSING TRAINING CONFLICTS
Next, we demonstrate how MetaShift can shed light on the training dynamics of ML models. An important feature of MetaShift is that each training datum is not only associated with class label, but also the annotations of subset membership. Such annotations open a window for a systematic evaluation of how training each subset would affect the evaluation performance other subsets.
Setup Concretely, we evaluate the MI model after each gradient step. We record the change the validation loss of each validation subset. This gives us multi-axes description of the effect of the current gradient step. Meanwhile, we also record the subset membership information for each of the training data in the current batch. We then training subset to the change the validation loss of each validation subset.
linear regression analyze the contributions of each
We sampled 48 subsets from cat & dog classes. We use 22 of them as out-of-domain evaluation sets and the other 26 as training and in-domain evaluation. We sample 50 images from each subset to form the training set. Figure 4 shows the abridged results for brevity and better visualization.
8

---

Published as a conference paper at ICLR 2022
Evaluation
Data
cat(computed)
cat(keyboard)
catidrawen)
cat(cabinet)
cat(faucet)
cat(sink)
don(carper)
dog(sheet)
dog(blanket)
dogi dogi (errtai)))
dog(truck)
(fg(fris
cat(fence)
peeteeeevision
tog(television
dog(animar)
Training Data
cat(computer)
1000
Normalized Performance Contribution
cat(keyboord)
0.75
cat(drawer)
cat(cabinet)
3
car(faucet)
cat(sink)
0.0
dogícarpet)
dug(sheel)
dng(blanket))
(cg(ruatin))
5
dog(truck)
dog(frisbee)
-1.00
4
In-Domain Evaluation Out-of-Domain
Figure 4: A Visualization of Training Confficts. Each row represents training subset, and each column represents evaluation subset. A positive value (green) indicates that training the training subset would improve performance on evaluation subset. negative value (red) indicates that the training would hurt the performance on the evaluation subset. See text for discussions.
Results Region (1) and (2) in Figure 4 highlighted the "conflict zones": indoor cat subsets and indoor dog subsets are having severe training confficts with each other. This makes sense since they share the same spurious feature "indoor", making the model confused. However, such training con- flicts important for out-of-domain generalization. As highlighted in Region (6), the dog(animal) subset (last column, which comprises primarily outdoor images) benefits most from indoor subsets like dog(sheet), rather than outdoor dog subsets like dog(truck). These results might indicate that training conflicts force the ML model to learn generalizable features.
Furthermore, Figure also also reveals the subsets that have low training contributions. As highlighted in Region (3) and (4), training on cat(faucet) and dog(frisbee) has little effect on other subsets. We hypothesize that this is because these subsets are too easy and thus provide little insight for the model. To verify this hypothesis, we remove cat(fauce(), and dog(frisbee) from training and use them for out-of-domain validation. We found that the model achieves a near perfect accuracy on both subsets: cat(faucer): 98.1%, dog(frisbee): 96.6%. This shows that subsets with low training contributions tend to be the subsets that are too easy.
Columns of cat(television) and dog(television) in Region (6) look opposite to each other. In addition, dog(Iruck) as highlighted in (5) exhibits a very different behaviour compared to other dog subsets: training dog(truck) conflicts with all other dog subsets.
Other observations include:
CONCLUSION
We present MetaShift- collection of 12,868 sets of natural images from 410 classes an important resource for studying the behavior or ML algorithms and training dynamics across data with heterogeneous contexts. MetaShift contains orders of magnitude more natural data shifts than previously available. also provides explicit explanations of what is unique about each of its data sets and distance score that measures the amount of distribution shift between any two of its data sets. We demonstrate the utility of MetaShift in both evaluating distribution shifts, and visualizing conflicts between data subsets during model training. Even though the data that we used here (e.g. Visual Genome) is not new, the idea of turning one dataset into a structured collection of datasets for assessing distribution shift is less explored. Our methodology for constructing MetaShift is simple and powerful, and can be readily extended to create many new resources for evaluating distribution shifts, which is very much needed.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
Our project website httts::/Metashiftt readthedocs. io/ provides detailed documen- tation and installation guidelines. The dataset and code are also available at /tthttt:://ggtt thub com/ Www.eixx meeeiiin-iee -iinnni Li ang/ a The implementations will enable researchers to download MetaShift and reproduce the results described here as well as run their own evaluations on additional datasets. In the experiments of evaluating domain generalization and subpopulation shifts, we use the implementation and the default hyperparameters from Domainbed (Gulrajani Lopez-Paz (020) to ensure a fair comparison. Our MetaShift dataset and code are available in the supplementary material, and would use the Creative Commons Attribution 4.0 International License.
ETHICS STATEMENT
One limitation that our MetaShift might inherit existing biases in Visual Genome, which is the base dataset of our MetaShift. Potential concerns include minority groups being under-represented in certain classes (e.g., women with snowboard), or annotation bias where people in images are by default labeled as male when gender unlikely be identifiable. Existing work in analyzing, quantifying, and mitigating biases in general computer vision datasets can help with addressing this potential negative societal impact. We can also use MetaShift understand how certain subsets of training data can lead to model biases in other contexts, as done in our experiments in Figure
REFERENCES
Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. CoRR, hts/1907.0289, 2019.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15())113331139, 2003.
Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks, Journal of statistical mechanics: theory and experiment, 2008 (10):P10008, 2008.
Ulrik Brandes, Daniel Delling, Marco Gaertler, Robert Gorke, Martin Hoefer, Zoran Nikoloski. and Dorothea Wagner. On modularity clustering. IEEE transactions on knowledge and data engineering, 1722)172-1888 2007.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pp. 77-91.
PMLR, 2018.
Fan RK Chung and Fan Chung Graham. Spectral graph theory. American Mathematical Soc., 1997.
Roxana Daneshjou, Kailas Vodrahalli, Weixin Liang, Roberto A. Novoa, Melissa Jenkins, Veron- ica M Rotemberg, Justin M. Ko, Susan M. Swetter, Elizabeth E Bailey, Olivier Gevaert, Pritam Mukherjee, Michelle Phung. Kiana Yekrang, Bradley Fong, Rachna Sahasrabudhe, James Zou, and Albert Sean Chiou. Disparities in dermatology ai: Assessments using diverse clinical images. ArXiv, abs/21 11 11.11..0000 080006, 2021.
Sabri Eyuboglu, Maya Varma, Khaled Kamal Saab, Jean-Benoit Delbrouck, Christopher Lee-Messer, Jared Dunamon, James Zou, and Christopher Re. Domino: Discovering systematic errors with cross-modal embeddings. In International Conference on Learning Representations, 2022. URI htttp::/ net/ Forumiid-FPMM
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In ICML, volume 97 of Proceedings of Machine Learning Research, pp. 2242-2251. PMLR, 2019.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 655-864,77 2016.
10

---

Published as a conference paper at ICLR 2022
1. Gulrajani and D. Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study of cross-domain few-shot learning. In European Conference on Computer Vision, pp. 124-141. Springer, 2020.
Y. He, 7.. Shen, and Cui. Towards non-Ill) image classification: A dataset and baselines. Pattern Recognition, 110, 2020.
D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019.
Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pp. 6700-6709, Computer Vision Foundation / IEEE, 2019.
Zachary 1zzo, Lexing Ying, and James Y. Zou. How to learn when data reacts to your model:
Performative gradient descent. In ICML, 2021.
Zachary Izzo, James Zou, and Lexing Ying. How to learn when data gradually reacts to your model. In AISTATS, 2022.
Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14)7684--763) 2020.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, Jure Leskovee, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. CoRR, abs/2012.07421, 2020.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis., 123(1 333-73, 2017.
Yongchan Kwon and James Zou. Beta shapley: unified and noise-reduced data valuation framework for machine learning. In AISTATS, 2022.
Weixin Liang, James Zou, and Zhou Yu. Beyond user self-reported likent scale ratings: comparison model for automatic dialog evaluation. In ACL, pp. 1363-1374. Association for Computational Linguistics, 2020a.
Weixin Liang, James Zou, and Zhou Yu. Alice: Active learning with contrastive natural language explanations. In EMNLP. 2020b.
Weixin Liang, Kaihui Liang, and Zhou Yu. HERALD: an annotation efficient method to detect user disengagement in social conversations. In ACL/JCNNP (1), pp. 3652-3665. Association for Computational Linguistics, 2021.
Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zimick. Microsoft COCO: common objects in context. ECCV (5), volume 8693 of Lecture Notes in Computer Science, pp. 740-755. Springer, 2014.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Conditional adversarial domain adaptation. In NeurIPS, pp. 1647-1657, 2018.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. In International Conference on Learning Representations, 2018.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV (4), volume 14 of Lecture Notes Computer Science, pp. 213-226. Springer, 2010.
11

---

Published as a conference paper at ICLR 2022
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. ICLR. OpenReview.net, 2020.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. In International Conference on Learming Representations, 2020.
Baochen Sun and Kate Saenko. Deep CORAL: correlation alignment for deep domain adaptation. In
ECCV Workshops (3), volume 9915 of Lecture Notes in Computer Science, pp. 443-440, 2016.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannanch Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In EMNLP (1), pp. 9275-9293. Association for Computational Linguistics, 2020.
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: the new data in multimedia research. Commun. ACM, 59(2):64-73, 2016.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint arXiv: 000..90330000 903,03096, 2019.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, pp. 5385-5394, IEEE Computer Society, 2017.
D. E. Worrall, S. J. Garbin, D. Turmukhambetov, and G. J. Brostow. Harmonic networks: Deep translation and rotation equivariance. In Computer Vision and Pattern Recognition (CVPR), pp- 5028-5037, 2017.
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Y. Zou, and Chelsea Finn. Improving out-of-distribttion robustness via selective augmentation. ArXiv, abs/2201.000 .00299, 2022.
12

---

Published as a conference paper at ICLR 2022
A ADDITIONAL DATASET INFORMATION
For each image class (e.g. Dogs), the MetaShift dataset contains different sets of dogs under different contexts represent diverse data distributions. The contexts include presence/absence of other objects (e.g, dog with frisbee). Contexts can also reflect attributes (e.g. black dogs) and general settings (e.g. dogs sunmy weather). These concepts thus capture diverse and real-world distribution shifts. We list the attribute and general location contexts below.
A.1 GENERAL LOCATION AND ATTRIBUTE CONTEXTS
A.1.1 GENERAL LOCATION CONTEXTS
Dog (Ocean)
Dog (Grass)
Dog (Street)
Dog (Cloudy)
Dog (Snow)
Dog (Bedroom)
Figure 5: Example subsets based on general contexts (the global context is stated in parenthesis). MetaShift covers global contexts including location (c.g., indoor, outdoor) and weather (c.g., sunny, rainy).
GENERAL CONTEXT ONTOLLGYY - =
'indoer/outdoor" [' l'indoor'', 'eutdoors'),
'wetter'; ['clear', 'overcast', 'cloudless', 'cloudy', 'sunny', 'oogy', 'rainy"), 'room's ('bedroom', 'kitchen', 'bathroom', 'living room'l,
"place': ('road', 'sidewalk', 'field', 'beach', 'park', 'grass', 'farm', 'ocean', 'pavement',
'lake', 'street', 'train station', 'hotel room', 'church', 'restaurant', 'forest', 'path', 'display', 'store', 'river', 'yard', 'snow', 'airport', 'parking lot')
Figure 6: The general contexts and their ontology in MetaShift. MetaShift covers 37 general contexts including location (e.g., indoor, outdoor, ocean, snow) and weather (e.g., couldy, sunny, rainy).
13

---

Published as a conference paper at ICLR 2022
A.1.2
ATTRIBUTE CONTEXTS
Dog (Sitting)
Dog (Jumping)
Cat (Orange)
Cat (White)
Bench (Wooden)
Bench (Metallic)
Plate (Round)
Plate (Square)
Figure 7: Example Subsets based on object attribute contexts (the attribute is stated in parenthesis). MetaShift covers attributes including activity (e.g., sitting, jumping), color (e.g.. orange, white), material (e.g.. wooden, metallic), shape (e.g., round, square), and on.
AFTRIBOTR_CCo COw.EEET__CCTTTCCCCC CONTEXTTOONCLLOOY
"dacknasst; [Tdack", ,, "beighe!), "dryness": [faet', 'dry"],
"colordul" ('colccfu',, "shiay'l, "leaf": ('loady', "bare"),
lenotion": ['aaapy', "maln'l, 'sporca's ('eseballl' 'Lemiaa),
"Elatnessta p'flat', "curved"), "ichtnasa's p'licht', "heavy",, "gerdents Pmole', "femalell, twetth's ('ide" "rarrou"), "deplltts ("deupt, 'shellow'l, "hardoese's is [hhrdd', bsatl", "eleanlinass's p'clean', 'dissy",, 'suitah's praa', "att'l, tehickneee's (fekinn, 'thick"), ), 'ogennees's ["oper', "closed'), "hiight Pfall", "s "laaglhi's (llang', "shost"], taalinasat to p'rall', "ampty"], tage"s ("young's "old"],
falze"; (Mlaage', "srall"], 'patsern's Pehecterea", 'striped', 'dress', "dotted"), "ahapets fo [rouund", "cmntangular' "cziangulaz', 'squacul),
sattivey'' ('waiting', "staming', "drinking', 'playing', 'mating', "cooking', 'reeting', "xleeping', 'poxinn', "talking', "looking down', "lonking up", "driving', 'zeading", "beushing Lewth", 'fiying', 'ausfing', takiing', "hanging" "pose": (fwalking' 'standing', "lyingg, feitting', 'rorning', 'jmmping' 'erouching', ng'nding', 'miiling', 'grazing'),
Inaterial's
('wood, 'plastie', "mecal", "glass", 'leacher', Geather'', 'porcslaint, 'papent, "stone", "brick"), foular's [whitt', "ewd', "black', ', "green", "ailver', 'gald", "Khakit, "gray", "daak', "pink", "dark blue", "dark brown', "alue", 'yelloww, 'tan', 'brown', 'orarge', 'porgle', "belge', "blond', "brumelle", 'maroon', 'light bluw!, '1ight beown!]
The attributes and their ontology MetaShift, MetaShift covers over 100 attrihutes including activity (e.g., sitting, jumping), color (e.g., orange, white), material (e.g., wooden, metallic), shape (e.g., round, square) and so
Figure& 8:
A.2 COMPUTE REQUIREMENTS
All experiments were performed on an Amazon EC2 P3.8 instance with 4 NVIDIA V100 Tensor Core GPUs Each training trial in evaluating distribution shifts takes less than hour to finish.
A.3 DATASET LICENSES
Our MetaShift and the code would use the Creative Commons Attribution 4.0 International License. Visual Genome (Krishna et al., 2017) is licensed under a Creative Commons Attribution 4.0 Inter- national License. MS-COCO (Lin et al. 2014) is licensed under CC-BY 4.0. The Visual Genome dataset uses 108, 077 images from the intersection of the YFCC100M (Thomee et al. 2016) and MS-COCO. We use the pre- pre-processed and cleaned version of Visual Genome by GQA Hudson & Manning 2019).
14

---

Published as a conference paper at ICLR 2022
We use the implementation and hyper-ppaaee syper--araametes' from Domainbed (Guraaaai & Lopez- Paz 2020) for empirical risk minimization (ERM) and all domain generalization baselines. Domainbed is licensed under MIT License. It is worth noting that the implementation of empirical risk minimization (ERM) in Domainbed implicitly up-samples minority domains during training. Nevertheless, we stick with their implementation for consistency.
B EXPERIMENT DETAILS
B. EXPERIMENT SETUP DETAILS
In the experiments of evaluating domain generalization and subpopulation shifts, we use the plementation and the default hyperparameters from DomainBed (Gulrajani & Lopez- Paz 2020) to ensure a fair comparison. More specifically, we evaluated the following methods for training models to be robust to data shifts:
ERM: the standard Empirical risk minimization baseline.
IRM: Invariant risk minimization (Arjovsky et al. 2019), which penalizes feature distribu- tions that have different optimal linear classifiers for each domain.
GroupDRO: Group Distributionally robust optimization (Sagawa al. 2020), which uses distributionally robust optimization to explicitly minimize the loss on the worst-case domain during training.
CORAL: Correlation Alignment (Sun & Saenko 2016), which penalizes differences the means and covariances of the feature distributions for each domain.
CDANN: Conditional Adversarial Domain Adaptation (Long al. 2018), which uses adversarial learning to penalize differences in feature representations from different domains.
Following DomainBed (Gulrajani & Lopez-Paz 2020), we use ResNet-18 18 with the Adam optimizer, cross-entropy loss, learning rate 0.00005, batch size 32. The data augmentation steps include the following: crops of random size and aspect ratio, resizing 224 x 224 pixels, random horizontal flips, random color jitter, grayscaling the image with 10% probability. and normalization using the ImageNet channel means and standard deviations. We start from ImageNet pre-trained weights, which consistent with the standard practice of prior work (Gulrajani Lopez-Paz 2020 Koh et al. 20200)
Sagawa et al. 2020 For additional implementation details, refer readers DomainBed (Gulrajani
& Lopez-Pat Paz
2020
B.2
MULTI-CLASS CLASSIFICATION EXPERIMENTS
To demonstrate the generalizability of our findings, we extend our binary classification experiments into multi-class classification setting. More specifically, we augment the "Cat vs. Dog" classification task into a 10-class animal classification task by incorporating the following classes: "bear", "bird", "cow", "elephant", "horse", "sheep", "giraffe", "zebra".
B.2.1 MULTI-CLASS CLASSIFICATION DOMAIN GENERALIZATION
0OD Acc.
Algorithm
d=0.44 d=1. d=1.43
ERM
0.762 0.637 0.448
0.265
IRM
0.699 0.660
0.422
0.284
GroupDRO
0.690 0.683
0.451
0.252
CORAL
0.641 0.624
0.373
0.261
CDANN
0.676 0.647 0.392
0.304
Table 3: Evaluating domain generalization on a 10-class animal classification task. Out-of-domain (OOD) test accuracy with different amounts of distribution shift d. Higher d indicates more challenging problem.
We extend Task I in the domain generalization experiment (Section 4.1) into multi-class classi- fication setting. Overall, these new results very consistent with our findings. The multi-class
15

---

Published as a conference paper at ICLR 2022
classification setting is constructed follows: extend the "Cat vs. Dog" task to 10-class animal classification task. Similar to Section 4.1, we only change the training data dog, while keeping the training data of all other classes unchanged. in order avoid introducing additional confounding
effects.
We explore the effect of using 4 different sets of dog training data. The test is performed dog(shelf), and thus we also report the distance between the dog training data and the test data dog(shelf).
1. dog(cabinet + bed) as dog training data. Its distance to dog(shelf) is d=0.44
2. dog(bag + box) as dog training data. Its distance to dog(shelf) is d=0.71
3. dog(bench bike) as dog training data. Its distance to dog(shelf) is d=1.12
4. dog(boat surfhoard) dog training data. Its distance dog(shelf) is d=1.43
Results As shown in Table 3 in the multi-class classification setting, the standard empirical risk minimization still performs the best when shifts are moderate(i.e.. when is small), which is aligned with previous research, as the domain gencralization methods typically impose strong algorithmic regularization in various forms (Koh et al., 20200) Santurkar et al. 2020). However, when the amount of distribution shift increases, the domain generalization algorithms typically outperform the ERM baseline, though no algorithm a consistent winner compared to other algorithms for large shifts. This finding suggests that domain generalization an important and challenging task and that there's still a lot of room for new methods.
B.2.2 MULTI-CLASS CLASSIFICATION SUBPOPULATION SHIFT
We extend the "Cat vs. Dog" subpopulation Shift to a 10-class animal classification task. Similarly, avoid introducing additional confounding effects, we only vary the training data of cat and dog in different experiments, but keep the training data of other classes unchanged across experiments. We evaluate only on the cat and dog classes. The rationale of evaluating these two classes (instead of the other classes) is that we vary the training data of these classes, and thus these classes would be the most directly impacted.
Average Acc.
Worst Group Acc.
Algorithm
1=12% p=6% p=1% pblpp=1220 12% p=6% p= 1%
ERM
0.789 0.769 0.703
0.646 0.638 0.339
IRM
0.780 0.764 0.728
0.614
0.543 0.443
GroupDRO CORAL
0.785 0.778 0.726 0.630 0.669 0.520
0.787 0.764 0.713
0.614
0,543 0.386
CDANN
0.783 0.773 0.709
0.685 0.591 0.567
Table 4: Evaluating subpopulation shift in 10-class animal classification task: p is the percentage of minority groups in the training data. Lower p indicates more challenging setting. Evaluated halanced test. set (p=50%)
Results Table 4soos shows the evaluation results on subpopulation shifts in the multi-class classification setting. In terms of the average accuracy, ERM performs the best when p is large (i.e., the minority group is less underreprcsented) as expected. However, in terms of the worst group accuracy, the algorithms typically outperform ERM, showcasing their effectiveness in subpopulation shifts. Fur- thermore, it is worth noting that no algorithm a consistent winner compared to other algorithms for large shifts. This finding is aligned with our finding in the previous section, suggesting that subpopulation shift is an important and challenging problem and there's still lot of room for new methods.
B.3 APPLICATION: ASSESSING TRAINING CONFLICTS
As data is the fuel powering artificial intelligence, it important to understand the training conflicts on heterogeneous training data. We next demonstrate the utility of MetaDataset on visualizing how different subsets of data provide conflicting training signals. The key idea is to analyze the change of
16

---

Published as a conference paper at ICLR 2022
Evaluation Data
Aoodeeeii boog [Sp
Bop
Grop
Dop
Roo
(ood) led
caticar)
catdwindow
cat(biinds)
075
mputer
Normalized Perfomance Contribution
cal(keyacard)
cat(computer mouse
Training Data
carldrawer
celicabinet)
ca:lsofa)
catifaucet
catisink.
call towel catiblanket
000
pillow
dogicar) dog (wi)dww))
dogimimer
eog(frisbed doglCarpet eet dogi Blanke dog curaain
9/73
doglrematé control)
In-Domain Evaluation
Out-of-Domain Evaluation
Figure 9: The full visualization of the training conflicts experiment. Each row represents training subset, and each column represents evaluation subset. A positive value (green) indicates that training on the training subset would improve the performance evaluation subset. A negative value (red) indicates that the training would hurt the performance on the evaluation subset. See text for discussions. Coefficients are normalized to [],
validation loss on each validation subset after each gradient step, and then attribute the change to each training subset.
As shown in the "in-domain evaluation" region of Figure 9 we randomly sample 26 cat & dog subsets for training and in-domain evaluation: we split the data in each subset into training and an in-domain evaluation set. As shown in the "out-of-domain evaluation" region of Figure 9 we additionally sample 22 subsets as out-of-domain evaluation sets. We downsample the training data to make sure that every training subset has an equal number of data points (i.e., 50).
17

---

Published as a conference paper at ICLR 2022
C META-GRAPH AND META-DATA
C.I VISUALIZATION OF META-GRAPH SPECTRAL EMBEDDINGS
To provide more intuitive understanding of the meta-graph embeddings, visualize the spectral embeddings of all nodes in the "Cat" meta-graph in Figure 10 Overall, the spectral embeddings work well as similar nodes are close in the t-SNE visualization. Nodes with similar color (i.e., assigned to the same cluster by the Louvain community detection algorithm) are also clustered in the t-SNE visualization. Finally, we also note that the geometry of t-SNE visualization (Figure 10) also closely mirrors the geometry of the meta-graph visualization layout in Figure 2
CRL (bllppoot)
(oilEOw))
CBL Cremo TRpe!!
CBL
(couch)) (co niforto)))
ABL OWTRRDRY
Ip attur)
CBL (INIT))
[conaainee ver)
(bRques)
CRIKK
(mTNEFOT)
CSR (drawerr
(bmoks)
(OODA)
(rsupett
(cabunett
(Wallt))
TboSks
ITabie
(pspers)
Iconspiter)
(bRIttF)
AST
(moniiori
(water?
IPIDOFT
Itelesation)
(sciaen)
CBR
(computer mouse)
Jkey Aaperdt
(paper)
twomen)
(CORUP)))
(grawidd)
(hae))
(carpet EaR
ESU
t-SME-one
Figure 10: t-SNE Visualization of Meta-graph Spectral Embeddings for "Cat". Each node represents subset of the images. Node colors indicate the communities automatically detected graph-based algorithms. Overall. the spectral embeddings work as similar nodes are close Nodes with similar color (i.c., assigned the same cluster by the Louvain community detection algorithm) are also clustered in the t-SNE visualization. Finally, we also note that the geometry of the t-SNE visualization (Figure T0 also closely mirrrr the geometry of the meta-grap meta-graph visualization layout in Figure
t-SNE visualization.
C.2 LOUVAIN COMMUNITY DETECTION ALGORITHM
Community detection algorithms detect groups of nodes in graph that are more densely connected internally than with the rest the graph. We apply the Louvain community detection algorithm to recover the community structure of meta-graphs. One measure of how well network is partitioned into communities Modularity, which is the difference between the actual number of edges in community and the expected number of such edges. We apply the Louvain community detection algorithm [Blondel et al. 2008), which greedily maximizes modularity since optimizing modularity is NP-hard (Brandes et al. 2007). This is bottom-up algorithm: initially, every vertex belongs to separate community, and vertices are moved between communities iteratively in a way that maximizes the vertices' local contribution the overall modularity score. When consensus reached (i.e. single move would increase the modularity score), every community in the original graph is shrank to single vertex (while keeping the total weight of the adjacent edges) and the process continues
18

---

Published as a conference paper at ICLR 2022
the next level. The algorithm stops when it is not possible to increase the modularity anymore after shrinking the communities to vertices.
C.3 DISCUSSIONS ON IMPERFECT OR INCOMPLETE META-DATA
Even if there is some noise introduced by the imperfect incomplete metadata, the noise can make the classification tasks more challenging and more realistic, as in subpopulation shifts. For example, it is rare that someone would collect dog dataset of 100% indoor images, but it quite possible that the dataset contains 80% indoor images and 20% outdoor images. Therefore, noise from metadata would not invalidate our approach, but make the curated dataset more realistic and challenging.
D CONSTRUCTING METASHIFT FROM COCO DATASET
MetaShift from COCO demonstrate the generalizability of the MetaShift construction method- ology, apply our methodology on MS-COCO dataset (Lin et al., 2014). We construct 1321 subsets, with an average size of 389 and median size of 124, along with 80 meta-graphs that help to quantify the amount of distribution shift among MS-COCO subsets.
Subpopulation Shift Setup Based on the MetaShift from COCO, we constructed a subpopulation shift experiment similar Section We construct a "Cat vs. Dog" task, where the "indoor/outdoor" contexts are spuriously correlated with the class labels. The "indoor" context is constructed by merging the following super-categories: 'indoor', 'kitchen', 'electronic', 'appliance', 'furniture'. Similarly, the "outdoor" context is constructed by merging the following super-categories: 'outdoor', 'vehicle', 'sports'. In addition. in the training data, cat(ourdoor) and dog(indoor) subsets are the minority groups, while cat(indoor) and (og(outdoor) are majority groups. We keep the total size of training data as 3000 images unchanged and only vary the portion of minority groups. We use balanced test set with 524 images to report both average accuracy and worst group accuracy.
Average Acc.
Worst Group Ace.
Algorithm
1=13% p=7% p=1% p=13% p=7% y= %
ERM
0.826 0.821 0.739 0.722 0.698 0.452
IRM
0.805 0.794 0.791 0.765
0.672 0.458
GroupDRO CORAL
0.821 0,811 0.742 0.730
0.667 0.500
0.824 0.801 0.750 0.809
0.695 0.565
CDANN
0.834 0.805 0.748 0.794 0.690 0.527
Table Evaluating subpopulation shift on MctaShift from COCO. p is percentage of minority groups in the training data. Lower p indicates a more challenging setting. Evaluated on a balanced test set (p=50%).
Subpopulation Shift Result As shown in Ta overall, we found that the new results are consis- tent with our previous findings. When p is large (i.e., the minority group is less underrepresented) ERM outperforms most of the other algorithms in terms average accuracy. It is also worth noting that algorithm is consistent winner compared to other algorithms for large shifts. These new experiments also show that MetaShift can flexibly accommodate other source datasets including COCO.
19


Reviews:
[{"review_id": "afBFyLK5tvl", "reviewer": "Reviewer_ameR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**[Strengths]** Good idea to study the impact of distribution shift.\n+ This paper is well written and easy to understand. Section 3 gives a good introduction to the step-by-step construction of MetaShift.\n+ The main advantage of MetaShift is it contains systematic annotation about the differences between different shifts. This further helps to study the effects of distribution shifts (e.g., subpopulation shifts)\n+ The idea of generating a large number of real-world distribution shifts that are well-annotated and controlled is attractive. The proposed MetaShift is well illustrated and the figures are helpful to know the information of MetaShift. \n+ Section 4.3 is interesting and the results give some insights. \n\n**[Weaknesses]** The major concern is about the experimental evaluation where the constructed tasks are only binary classification. \n- In Section 4.1, this work constructs four binary classification tasks to study the impact of the shift under the generalization setting. One question is, how about constructing more challenging tasks which involve more classes? \n- When evaluating subpopulation shifts, the tasks are also binary which contain spurious correlation. The same question is multi-class classification tasks might be needed. \n- MetaShift is a collection of 12,868 sets of natural images from 410 classes. Why do the experiments only focus on binary classification (e.g., cat vs. dog, bus vs. truck)? In another word, it seems the same settings in Sec. 4 can be constructed based on other classes. It would be helpful if this work could discuss this. Again, more challenging multi-classification setting would be very useful.", "summary_of_the_paper": "This work proposes a collection called MetaShift to study the impact of dataset distribution. The major advantage of MetaShift is that it provides annotation/information to measure the amount of distribution shift between any two of its data sets. In the experiment, this work constructs two applications, 1) evaluating distribution shifts, assessing training conflicts. ", "summary_of_the_review": "This work introduces a good way to study the effects of distribution shifts. Specifically, this work proposes a framework called Metashift, which contains systematic annotation about the differences between different shifts. However, the major concern is about the constructed tasks in the experiment. More explanation/discussion can be included to eliminate the question.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RpA2QqPBi81", "reviewer": "Reviewer_T6b1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Pros\n+ The paper is well written and easy to follow.\n+ The proposed approach to leverage metadata (form previously published large-scale dataset) to create datasets of domain shifts is simple and well motivated. Splitting a large dataset (with multiple labels) in a meaningful ways to study dataset shift is not trivial. However the authors came up with an intuitive (and relatively simple) approach.\n+ The problem of studying shifts in dataset distribution is very relevant and important to machine learning. This dataset can benefit the community by allowing a more system evaluation of dataset shifts.\n\n### Cons\n- It would be nice to have more descriptions of the methods used to benchmark the dataset (ERM, IRM, DRO, CORAL and CDANN), architectures used (which model? pretrained or from scratch? what is the model capacity?) and training details (which loss, which optimization, learning rate, batch size, etc). If not enough space on the main text, these informations could be added on appendix.\n- It would be nice if the authors would give more detail on how the embeddings of meta-graphs are computed. For example what is the matrix A and how are the embeddings computed? Why using spectral embeddings specifically rather than other approaches (eg, using the word embedding (pretrained on large language corpus) of each context)?\n- It could be nice to show some more quantitative or qualitative (eg t-SNE) for the meta-graph embeddings (used to compute the shift between datasets).\n- The paper states it generates >12K datasets (across 410 classes), however experiments are done only on a very tiny number of datasets (cat/dog, bus/truck. elephant/horse and bowl/cup). The proposed dataset would be much more useful to the community if the authors would provide a much larger subset of “pre-made” datasets for easy experimentation.\n", "summary_of_the_paper": "In this paper, the authors introduce a new dataset (actully, a collection of datasets) called MetaShift. MetaShift is built on top of Visual Genome and leverages its metadata to cluster images, thus providing a context for each image (labels are of the form class+context, eg,  ‘cat in grass’, ‘dog in bathroom’). This context is then used to generate dataset shifts. Besides been much larger than similar (openly available) datasets, MetaShift explicitly provides the context, which can be used to compute a “distance score” pf distribution shift between any two datasets.", "summary_of_the_review": "I am inclined to accept this paper because of (i) the simplicity of the approach to generate the datasets and (ii) the usefulness to the community. However, I would be more confident with acceptance if the authors would address the weaknesses of the paper (se above). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8lu_9ajQYiG", "reviewer": "Reviewer_HKis", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nPaper has a strong motivation for building the dataset. The authors also present a detailed understanding of major applications of their dataset across ML models. It provides a good quantization for the population shift and show with experiments how it impacts domain generalizability. The generalizability of the dataset creation for any dataset with multilabel is another strong point of the paper. \n\nWeakness:\nThe paper only talks about all advantages for MetaShift that’s derived from Visual Genome data but don’t have any similar comparison or quality analysis for other datasets generated using their 4-step dataset creation, for example, analysis could have been provided on COCO dataset too. The paper also doesn’t address the dependency of dataset performance on metadata, what if there are inconsistencies in the metadata of dataset but images are perfect, how will the dataset creation and performance be impacted for a dataset with such metadata. It would also have been interesting to see the analysis of performance of certain model around underrepresented subsets vs over represented subset, set/characteristics of datum leading to training conflict in general, if there is any pattern in images. Other obvious underlying data bias issue has already been acknowledged in the paper too and I hope the authors will research more into solving it. \n", "summary_of_the_paper": "The paper provides a benchmark dataset that can be used for training & evaluation of Machine Learning models. Their contribution is that they have a large collection of 12,868 sets of natural images across 410 classes obtained from visual genome data and its metadata for annotations.  This helps in accounting for large natural shifts in the data. They also provide a way to measure distance between two subsets to quantize the distribution shift between any two of its data sets. \nThe paper provides a good justification for the need of such a dataset for computer vision tasks and motivate the idea well. It also talks in detail about the steps taken to generate MetaShift from Visual Genome, it also provides a generalization of their 4-step process of dataset creation on any dataset with multilabel, results presented for COCO dataset. The paper further discusses the use of this dataset for two major cases- Evaluating distribution shifts & Assessing training conflicts. They provide the impact of shift distance on the domain generalization by keeping test set same and varying training subsets randomly. Further, it talks about subpopulation shifts where the train and test distribution of same domain with different mixture weights. They show that no algorithm consistently performs better than other algorithms for larger shifts. It provides a detailed understanding of training conflict by analyzing the contribution of each training subset to the change of the validation loss of each validation set during the training process. \nOverall, it’s a well written paper about the motivation, use cases, applicability, and generalizability of their proposed data set. \n", "summary_of_the_review": "I feel the authors have done a good job highlighting the motivation of such a dataset, steps of creation of the dataset from Visual Genome, paying attention to the generalizability of the approach, discussing the major applications of the dataset in detail. The github link also provides holistic understanding of the work. This dataset will help the research in the field of CV. Once they overcome and address the current weaknesses of the dataset, it will become even better dataset asset. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work studies the impact of distribution shift via a collection of datasets-MetaShift. Reviewers all agreed that this work is simple, effective, and well-motivated, and has key implications, and will be quite useful to the community. There were some concerns about the lack of analysis of MetaShift, and the binary classification setting, which was addressed by the authors’ responses. Thus, I recommend an acceptance.
Decision: Accept (Poster)