MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts

1 INTRODUCTION . A major challenge in machine learning ( ML ) is that a model can have very different performances and behaviors when it ’ s applied to different types of natural data ( Koh et al. , 2020 ; Izzo et al. , 2021 ; 2022 ) . For example , if the user data have different contexts compared to the model ’ s training data ( e.g . users have outdoor dog photos and the model ’ s training was mostly on indoor images ) , then the model ’ s accuracy can greatly suffer ( Yao et al. , 2022 ) . A model can have disparate performances even within different subsets within its training and evaluation data ( Daneshjou et al. , 2021 ; Eyuboglu et al. , 2022 ) . In order to assess the reliability and fairness of a model , we therefore need to evaluate its performance and training behavior across heterogeneous types of data . However , the lack of well-structured datasets representing diverse data distributions makes systematic evaluation difficult . In this paper , we present MetaShift to tackle this challenge . MetaShift is a collection of 12,868 sets of natural images from 410 classes . Each set corresponds to images in a similar context and represents a coherent real-world data distribution , as shown in Figure 1 . The construction of MetaShift is different from and complementary to other efforts to curate benchmarks for data shifts by pulling together data across different experiments or sources . MetaShift leverages heterogeneity within the large sets of images from the Visual Genome project ( Krishna et al. , 2017 ) by clustering the images using metadata that describes the context of each image . The advantage of this approach is that MetaShift contains many more coherent sets of data compared to other benchmarks . Importantly , we have explicit annotations of what makes each subset unique ( e.g . cats with cars or dogs next to a bench ) as 1Dataset and code available at : https : //metashift.readthedocs.io/ well as a score that measures the distance between any two subsets , which is not available in previous benchmarks of natural data . We demonstrate the utility of MetaShift in two applications . First , MetaShift supports evaluation on both domain generalization and subpopulation shifts settings . Using the score between subsets provided by MetaShift , we study ML models ’ behavior under different carefully modulated amounts of distribution shift . Second , MetaShift can also shed light on the training dynamics of ML models . Since we have the subset membership information for each training datum , we could attribute the contribution of each gradient step back to the training subsets , and then analyze how different data subsets provide conflicting training signals . Our contributions : We present MetaShift as an important resource for studying the behavior of ML algorithms and training dynamics across data with heterogeneous contexts . Our methodology for constructing MetaShift can also be applied to other domains where metadata is available . We empirically evaluate the performance of different robust learning algorithms , showing that ERM performs well for modest shifts while no method is the clear winner for larger shifts . This finding suggests that domain generalization is an important and challenging task and that there ’ s still a lot of room for new methods . 2 RELATED WORK . Existing Benchmarks for Distribution Shift Distribution shifts have been a longstanding challenge in machine learning . Early benchmarks focus on distribution shifts induced by synthetic pixel transformations . Examples include rotated and translated versions of MNIST and CIFAR ( Worrall et al. , 2017 ) ; surface variations such as texture , color , and corruptions like blur in Colored MNIST ( Gulrajani & Lopez-Paz , 2020 ) , ImageNet-C ( Hendrycks & Dietterich , 2019 ) . Although the synthetic pixel transformations are well-defined , they generally do not represent realistic shifts in real-world images that we capture in MetaShift . Other benchmarks do not rely on transformations but instead pull together data across different experiments or sources . Office-31 ( Saenko et al. , 2010 ) and Office-home ( Venkateswara et al. , 2017 ) contain images collected from different domains like Amazon , clipart . These benchmarks typically have only a handful of data distributions . The benchmarks collected in WILDS ( Koh et al. , 2020 ) combine data from different sources ( e.g. , medical images from different hospitals , animal images from different camera traps ) . Similarly , some meta-learning benchmarks ( Triantafillou et al. , 2019 ; Guo et al. , 2020 ) focuses on dataset-level shift by combining different existing datasets like ImageNet , Omniglot . While valuable , they lack systematic annotation about what is different across different shifts . Santurkar et al . ( 2020 ) ; Ren et al . ( 2018 ) utilize the hierarchical structure of ImageNet to construct training and test sets with disjoint subclasses . For example , the “ tableware ” class uses “ beer glass ” and “ plate ” for training and testing respectively . Different from their work , we study the shifts where the core object remains the same while the context changes . NICO ( He et al. , 2020 ) query different manually-curated phrases on search engines to collect images of objects in different contexts . A key difference is the scale of MetaShift : NICO contains 190 sets of images across 19 classes while MetaShift has 12,868 sets of natural images across 410 classes . To sum up , the advantages of our MetaShift are : • Existing benchmark datasets for distribution shifts typically have only a handful of data distributions . In contrast , our MetaShift has over 12,868 data distributions , thus enabling a much more comprehensive assessment of distribution shifts . • Distribution shifts in existing benchmarks are not annotated ( i.e . we don ’ t know what drives the shift ) and are not well-controlled ( i.e . we can ’ t easily adjust the magnitude of the shift ) . The MetaShift provides explicit annotations of the differences between any two sub-datasets , and it quantifies the distance of the shift . Evaluating conflicts on training data Recent work has shown that different training data points play a heterogeneous role in training . To quantify this , Data Shapley ( Ghorbani & Zou , 2019 ; Kwon & Zou , 2022 ) provides a mathematical framework for quantifying the contribution of each training datum . Data Cartography ( Swayamdipta et al. , 2020 ) leverages a model ’ s training confidence to discover hard-to-learn training data points . Such understanding has provided actionable insights that benefit the ML workflow . For example , removing noisy low-contribution training data points improves the model ’ s final performance ( Liang et al. , 2020a ; 2021 ) . Furthermore , active learning identifies the most informative data points for humans to annotate ( Liang et al. , 2020b ) . Complementary to their work , our analysis sheds light on not only which but also why a certain portion of the training data are hard-to-learn—because different subsets are providing conflicting training signals . 3 THE METASHIFT CONSTRUCTION METHODOLOGY . What is MetaShift ? The MetaShift is a collection of subsets of data together with an annotation graph that explains the similarity/distance between two subsets ( edge weight ) as well as what is unique about each subset ( node metadata ) . For each class , say “ cat ” , we have many subsets of cats , and we can think of each subset as a node in the graph , as shown in Figure 2 . Each subset corresponds to “ cat ” in a different context : e.g . “ cat with sink ” or “ cat with fence ” . The context of each subset is the node metadata . The “ cat with sink ” subset is more similar to “ cat with faucet ” subset because there are many images that contain both sink and faucet . This similarity is the weight of the edge ; a higher weight means the contexts of the two nodes tend to co-occur in the same data . How can we use MetaShift ? It is a flexible framework to generate a large number of real-world distribution shifts that are well-annotated and controlled . For each class of interest , say “ cats ” , we can use the meta-graph of cats to identify a collection of cats nodes for training ( e.g . cats with bathroom-related contexts ) and a collection of cats nodes for out-of-domain evaluation ( e.g . cats in outdoor contexts ) . Our meta-graph tells us exactly what is different between the train and test domains ( e.g . bathroom vs. outdoor contexts ) , and it also specifies the similarity between the two contexts via graph distance . That makes it easy to carefully modulate the amount of distribution shift . For example , if we use cats-in-living-room as the test set , then this is a smaller distribution shift . Base Dataset : Visual Genome We leverage the natural heterogeneity of Visual Genome and its annotations to construct MetaShift . Visual Genome contains over 100k images across 1,702 object classes . For each image , Visual Genome annotates the class labels of all objects that occur in the image . Formally , for each image x ( i ) , we have a list of meta-data tags m ( i ) = { t ( i ) 1 , t ( i ) 2 , . . . , t ( i ) nm } , each indicating the presence of an object in the context . We denote the vocabulary of the meta-data tags as M = { m0 , . . . , m|M| } . MetaShift is constructed on a class-by-class basis : For each class , say “ cat ” , we pull out all cat images and proceed with the following steps . Step 1 : Generate Candidate Subsets We first generate candidate subsets by enumerating all possible meta-data tags . We construct |M| candidate subsets where the ith subset contains all images of the class of interest ( i.e. , “ cat ” ) that has a meta-tag mi . We then remove subsets whose sizes are less than a threshold ( e.g. , 25 ) . Step 2 : Construct Meta-graphs Since the meta-data are not necessarily disentangled , the candidate subsets might contain significant overlaps ( e.g. , “ cat with sink ” and “ cat with faucet ” ) . To capture this phenomenon , we construct a meta-graph to model the relationships among all subsets of each class . Specifically , for each class j ∈ Y , we construct meta-graph , a weighted undirected graph G = ( V , E ) where each node v ∈ V denotes a candidate subset , and the weight of each edge is the overlap coefficient between two subsets : overlap ( X , Y ) = |X ∩ Y | min ( |X| , |Y | ) , ( 1 ) We remove the edges whose weights are less than a threshold ( e.g. , 0.1 ) to sparsify the graph . As shown in Figure 2 , the meta-graph G captures meaningful semantics of the multi-modal data distribution of the class of interest . Step 3 : Quantify Distances of Distribution Shifts The geometry of meta-graphs provides a natural and systematic way to quantify the distances of shifts across different data distributions : Intuitively , if two subsets are far away from each other in the MetaGraph , then the shift between them tend to be large . Following this intuition , we leverage spectral embeddings ( Belkin & Niyogi , 2003 ; Chung & Graham , 1997 ) to assign an embedding for each node based on the graph geometry . Spectral embedding minimizes the expected square distance between nodes that are connected : min X : XT 1=0 , XTX=IK ∑ i , j∈V Aij ∥Xi −Xj∥2 ( 2 ) where Xi is the embedding for node i ∈ V and K is the dimension of the embedding , and A is the adjacency matrix . We denote by X the matrix of dimension n×K whose i-th row Xi corresponds to the embedding of node i . The constraint XT 1 = 0 forces the embedding to be centered and XTX = IK ensures that we do not get trivial solution like all node embeddings located at the origin ( i.e. , X = 0 ) . Denoting by L = D −A the Laplacian matrix of the graph , we have : tr ( XTLX ) = 1 2 ∑ i , j∈V Aij ∥Xi −Xj∥2 ( 3 ) min X : XT 1=0 , XTX=IK tr ( XTLX ) = K+1∑ k=2 λk ( 4 ) The minimum is reached for X equal to the matrix of eigenvectors of the Laplacian matrix associated with the eigenvalues λ2 , ... , λK+1 . After calculating the spectral embeddings , we use the euclidean distance between the embeddings of two nodes as their distance . Other off-the-shelf node embedding methods like Node2Vec ( Grover & Leskovec , 2016 ) can be readily plugged into MetaShift . We delay exploring them as future work . Although the subset overlap ( i.e. , edge weight ) can also be used as a similarity metric , it does not incorporate the structural information from neighboring nodes . Our node embedding-based distance captures not only such overlap , but also broader similarities . Step 4 : Simulating Distribution Shifts MetaShifts allows users to benchmark both ( 1 ) domain generalization and ( 2 ) subpopulation shifts in a well-annotated ( explicit annotation of what drives the shift ) and well-controlled ( easy control of the amount of distribution shift ) fashion . • In domain generalization , the train and test distributions comprise data from related but distinct domains . This arises in many real-world scenarios since it is often infeasible to construct a comprehensive training set that spans all domains . To simulate this setting , we can sample two distinct collections of subsets as the train domains and the test domains respectively ( e.g . bathroom vs. outdoor contexts ) . To adjust the magnitude of the shift , we can fix the test domains and change the train domains with different distances . For example , if we use cats-in-living-room as the test set , then this is a smaller distribution shift . • In subpopulation shifts , the train and test distributions are mixtures of the same domains , but the mixture weights change between train and test . It is also an important setting since ML models are often reported to perform poorly on under-represented groups . To simulate this setting , we can sample the training set and test set from the same subsets but with different mixture weights . To adjust the magnitude of the shift , we can use different mixture weights for the training set while keeping the test set unchanged . We demonstrate both settings in the experiment section . It is also worth noting that different subsets may share common images—e.g . a dog image can have both grass and frisbee would occur in both dog with grass and dog with frisbee . Therefore , a post-processing step is needed to remove the training images that also occur in the test set to ensure no data leakage . MetaShift Statistics Figure 3 shows the statistics of MetaShift across all tasks . We start from the pre-processed and cleaned version of Visual Genome ( Hudson & Manning , 2019 ) , which contains 113,018 distinct images across 1,702 object classes . After the dataset construction , we have 12,868 sets of natural images from 410 classes . Concretely , each class has 31.4 subsets , and each subset has 200.4 images on average . The subsets are characterized by a diverse vocabulary of 1,853 distinct contexts . Beyond 1,702 contexts defined by object presence , MetaShift also leverages the 37 distinct general contexts and 114 object attributes from Visual Genome . The general contexts typically describe locations ( i.e. , indoor , outdoor ) , weather ( e.g. , rainy , cloudless ) , and places ( e.g. , bathroom , ocean ) . The object attributes include color ( e.g. , white , red ) , material ( e.g. , wood , plastic ) , shape ( e.g. , round , square ) , and other object-specific properties ( e.g. , empty/full for plates ) . See Appendix A for examples and more information . Generalizability : Case Study on COCO The MetaShift construction methodology is quite simple , and can be extended to any dataset with metadata tags ( i.e. , multilabel ) . To demonstrate this , we apply the construction methodology on MS-COCO dataset ( Lin et al. , 2014 ) , which provides object detection labels for each image . Applying our construction methodology , we are able to construct 1321 subsets , with an average size of 389 and a median size of 124 , along with 80 meta-graphs that help to quantify the amount of distribution shift among MS-COCO subsets . Experiments on MetaShift from COCO can be found in Appendix D. Even though the data that we used here ( e.g . Visual Genome , COCO ) is not new , the idea of turning one dataset into a structured collection of datasets for assessing distribution shift is less explored . COCO ( Lin et al. , 2014 ) and many other datasets provide meta-data that could be naturally used for generating candidate subsets . The main contribution of our methodology is that after generating the candidate subsets , our method provides a meta-graph that could be used to determine the train and test domains , and also specifies the similarity between the two subsets via graph distance . MetaShift opens up a new window for systematically evaluating domain shifts .

Summary: This work proposes a collection called MetaShift to study the impact of dataset distribution. The major advantage of MetaShift is that it provides annotation/information to measure the amount of distribution shift between any two of its data sets. In the experiment, this work constructs two applications, 1) evaluating distribution shifts, assessing training conflicts.