MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts

1 INTRODUCTION . A major challenge in machine learning ( ML ) is that a model can have very different performances and behaviors when it ’ s applied to different types of natural data ( Koh et al. , 2020 ; Izzo et al. , 2021 ; 2022 ) . For example , if the user data have different contexts compared to the model ’ s training data ( e.g . users have outdoor dog photos and the model ’ s training was mostly on indoor images ) , then the model ’ s accuracy can greatly suffer ( Yao et al. , 2022 ) . A model can have disparate performances even within different subsets within its training and evaluation data ( Daneshjou et al. , 2021 ; Eyuboglu et al. , 2022 ) . In order to assess the reliability and fairness of a model , we therefore need to evaluate its performance and training behavior across heterogeneous types of data . However , the lack of well-structured datasets representing diverse data distributions makes systematic evaluation difficult . In this paper , we present MetaShift to tackle this challenge . MetaShift is a collection of 12,868 sets of natural images from 410 classes . Each set corresponds to images in a similar context and represents a coherent real-world data distribution , as shown in Figure 1 . The construction of MetaShift is different from and complementary to other efforts to curate benchmarks for data shifts by pulling together data across different experiments or sources . MetaShift leverages heterogeneity within the large sets of images from the Visual Genome project ( Krishna et al. , 2017 ) by clustering the images using metadata that describes the context of each image . The advantage of this approach is that MetaShift contains many more coherent sets of data compared to other benchmarks . Importantly , we have explicit annotations of what makes each subset unique ( e.g . cats with cars or dogs next to a bench ) as 1Dataset and code available at : https : //metashift.readthedocs.io/ well as a score that measures the distance between any two subsets , which is not available in previous benchmarks of natural data . We demonstrate the utility of MetaShift in two applications . First , MetaShift supports evaluation on both domain generalization and subpopulation shifts settings . Using the score between subsets provided by MetaShift , we study ML models ’ behavior under different carefully modulated amounts of distribution shift . Second , MetaShift can also shed light on the training dynamics of ML models . Since we have the subset membership information for each training datum , we could attribute the contribution of each gradient step back to the training subsets , and then analyze how different data subsets provide conflicting training signals . Our contributions : We present MetaShift as an important resource for studying the behavior of ML algorithms and training dynamics across data with heterogeneous contexts . Our methodology for constructing MetaShift can also be applied to other domains where metadata is available . We empirically evaluate the performance of different robust learning algorithms , showing that ERM performs well for modest shifts while no method is the clear winner for larger shifts . This finding suggests that domain generalization is an important and challenging task and that there ’ s still a lot of room for new methods . 2 RELATED WORK . Existing Benchmarks for Distribution Shift Distribution shifts have been a longstanding challenge in machine learning . Early benchmarks focus on distribution shifts induced by synthetic pixel transformations . Examples include rotated and translated versions of MNIST and CIFAR ( Worrall et al. , 2017 ) ; surface variations such as texture , color , and corruptions like blur in Colored MNIST ( Gulrajani & Lopez-Paz , 2020 ) , ImageNet-C ( Hendrycks & Dietterich , 2019 ) . Although the synthetic pixel transformations are well-defined , they generally do not represent realistic shifts in real-world images that we capture in MetaShift . Other benchmarks do not rely on transformations but instead pull together data across different experiments or sources . Office-31 ( Saenko et al. , 2010 ) and Office-home ( Venkateswara et al. , 2017 ) contain images collected from different domains like Amazon , clipart . These benchmarks typically have only a handful of data distributions . The benchmarks collected in WILDS ( Koh et al. , 2020 ) combine data from different sources ( e.g. , medical images from different hospitals , animal images from different camera traps ) . Similarly , some meta-learning benchmarks ( Triantafillou et al. , 2019 ; Guo et al. , 2020 ) focuses on dataset-level shift by combining different existing datasets like ImageNet , Omniglot . While valuable , they lack systematic annotation about what is different across different shifts . Santurkar et al . ( 2020 ) ; Ren et al . ( 2018 ) utilize the hierarchical structure of ImageNet to construct training and test sets with disjoint subclasses . For example , the “ tableware ” class uses “ beer glass ” and “ plate ” for training and testing respectively . Different from their work , we study the shifts where the core object remains the same while the context changes . NICO ( He et al. , 2020 ) query different manually-curated phrases on search engines to collect images of objects in different contexts . A key difference is the scale of MetaShift : NICO contains 190 sets of images across 19 classes while MetaShift has 12,868 sets of natural images across 410 classes . To sum up , the advantages of our MetaShift are : • Existing benchmark datasets for distribution shifts typically have only a handful of data distributions . In contrast , our MetaShift has over 12,868 data distributions , thus enabling a much more comprehensive assessment of distribution shifts . • Distribution shifts in existing benchmarks are not annotated ( i.e . we don ’ t know what drives the shift ) and are not well-controlled ( i.e . we can ’ t easily adjust the magnitude of the shift ) . The MetaShift provides explicit annotations of the differences between any two sub-datasets , and it quantifies the distance of the shift . Evaluating conflicts on training data Recent work has shown that different training data points play a heterogeneous role in training . To quantify this , Data Shapley ( Ghorbani & Zou , 2019 ; Kwon & Zou , 2022 ) provides a mathematical framework for quantifying the contribution of each training datum . Data Cartography ( Swayamdipta et al. , 2020 ) leverages a model ’ s training confidence to discover hard-to-learn training data points . Such understanding has provided actionable insights that benefit the ML workflow . For example , removing noisy low-contribution training data points improves the model ’ s final performance ( Liang et al. , 2020a ; 2021 ) . Furthermore , active learning identifies the most informative data points for humans to annotate ( Liang et al. , 2020b ) . Complementary to their work , our analysis sheds light on not only which but also why a certain portion of the training data are hard-to-learn—because different subsets are providing conflicting training signals . 3 THE METASHIFT CONSTRUCTION METHODOLOGY . What is MetaShift ? The MetaShift is a collection of subsets of data together with an annotation graph that explains the similarity/distance between two subsets ( edge weight ) as well as what is unique about each subset ( node metadata ) . For each class , say “ cat ” , we have many subsets of cats , and we can think of each subset as a node in the graph , as shown in Figure 2 . Each subset corresponds to “ cat ” in a different context : e.g . “ cat with sink ” or “ cat with fence ” . The context of each subset is the node metadata . The “ cat with sink ” subset is more similar to “ cat with faucet ” subset because there are many images that contain both sink and faucet . This similarity is the weight of the edge ; a higher weight means the contexts of the two nodes tend to co-occur in the same data . How can we use MetaShift ? It is a flexible framework to generate a large number of real-world distribution shifts that are well-annotated and controlled . For each class of interest , say “ cats ” , we can use the meta-graph of cats to identify a collection of cats nodes for training ( e.g . cats with bathroom-related contexts ) and a collection of cats nodes for out-of-domain evaluation ( e.g . cats in outdoor contexts ) . Our meta-graph tells us exactly what is different between the train and test domains ( e.g . bathroom vs. outdoor contexts ) , and it also specifies the similarity between the two contexts via graph distance . That makes it easy to carefully modulate the amount of distribution shift . For example , if we use cats-in-living-room as the test set , then this is a smaller distribution shift . Base Dataset : Visual Genome We leverage the natural heterogeneity of Visual Genome and its annotations to construct MetaShift . Visual Genome contains over 100k images across 1,702 object classes . For each image , Visual Genome annotates the class labels of all objects that occur in the image . Formally , for each image x ( i ) , we have a list of meta-data tags m ( i ) = { t ( i ) 1 , t ( i ) 2 , . . . , t ( i ) nm } , each indicating the presence of an object in the context . We denote the vocabulary of the meta-data tags as M = { m0 , . . . , m|M| } . MetaShift is constructed on a class-by-class basis : For each class , say “ cat ” , we pull out all cat images and proceed with the following steps . Step 1 : Generate Candidate Subsets We first generate candidate subsets by enumerating all possible meta-data tags . We construct |M| candidate subsets where the ith subset contains all images of the class of interest ( i.e. , “ cat ” ) that has a meta-tag mi . We then remove subsets whose sizes are less than a threshold ( e.g. , 25 ) . Step 2 : Construct Meta-graphs Since the meta-data are not necessarily disentangled , the candidate subsets might contain significant overlaps ( e.g. , “ cat with sink ” and “ cat with faucet ” ) . To capture this phenomenon , we construct a meta-graph to model the relationships among all subsets of each class . Specifically , for each class j ∈ Y , we construct meta-graph , a weighted undirected graph G = ( V , E ) where each node v ∈ V denotes a candidate subset , and the weight of each edge is the overlap coefficient between two subsets : overlap ( X , Y ) = |X ∩ Y | min ( |X| , |Y | ) , ( 1 ) We remove the edges whose weights are less than a threshold ( e.g. , 0.1 ) to sparsify the graph . As shown in Figure 2 , the meta-graph G captures meaningful semantics of the multi-modal data distribution of the class of interest . Step 3 : Quantify Distances of Distribution Shifts The geometry of meta-graphs provides a natural and systematic way to quantify the distances of shifts across different data distributions : Intuitively , if two subsets are far away from each other in the MetaGraph , then the shift between them tend to be large . Following this intuition , we leverage spectral embeddings ( Belkin & Niyogi , 2003 ; Chung & Graham , 1997 ) to assign an embedding for each node based on the graph geometry . Spectral embedding minimizes the expected square distance between nodes that are connected : min X : XT 1=0 , XTX=IK ∑ i , j∈V Aij ∥Xi −Xj∥2 ( 2 ) where Xi is the embedding for node i ∈ V and K is the dimension of the embedding , and A is the adjacency matrix . We denote by X the matrix of dimension n×K whose i-th row Xi corresponds to the embedding of node i . The constraint XT 1 = 0 forces the embedding to be centered and XTX = IK ensures that we do not get trivial solution like all node embeddings located at the origin ( i.e. , X = 0 ) . Denoting by L = D −A the Laplacian matrix of the graph , we have : tr ( XTLX ) = 1 2 ∑ i , j∈V Aij ∥Xi −Xj∥2 ( 3 ) min X : XT 1=0 , XTX=IK tr ( XTLX ) = K+1∑ k=2 λk ( 4 ) The minimum is reached for X equal to the matrix of eigenvectors of the Laplacian matrix associated with the eigenvalues λ2 , ... , λK+1 . After calculating the spectral embeddings , we use the euclidean distance between the embeddings of two nodes as their distance . Other off-the-shelf node embedding methods like Node2Vec ( Grover & Leskovec , 2016 ) can be readily plugged into MetaShift . We delay exploring them as future work . Although the subset overlap ( i.e. , edge weight ) can also be used as a similarity metric , it does not incorporate the structural information from neighboring nodes . Our node embedding-based distance captures not only such overlap , but also broader similarities . Step 4 : Simulating Distribution Shifts MetaShifts allows users to benchmark both ( 1 ) domain generalization and ( 2 ) subpopulation shifts in a well-annotated ( explicit annotation of what drives the shift ) and well-controlled ( easy control of the amount of distribution shift ) fashion . • In domain generalization , the train and test distributions comprise data from related but distinct domains . This arises in many real-world scenarios since it is often infeasible to construct a comprehensive training set that spans all domains . To simulate this setting , we can sample two distinct collections of subsets as the train domains and the test domains respectively ( e.g . bathroom vs. outdoor contexts ) . To adjust the magnitude of the shift , we can fix the test domains and change the train domains with different distances . For example , if we use cats-in-living-room as the test set , then this is a smaller distribution shift . • In subpopulation shifts , the train and test distributions are mixtures of the same domains , but the mixture weights change between train and test . It is also an important setting since ML models are often reported to perform poorly on under-represented groups . To simulate this setting , we can sample the training set and test set from the same subsets but with different mixture weights . To adjust the magnitude of the shift , we can use different mixture weights for the training set while keeping the test set unchanged . We demonstrate both settings in the experiment section . It is also worth noting that different subsets may share common images—e.g . a dog image can have both grass and frisbee would occur in both dog with grass and dog with frisbee . Therefore , a post-processing step is needed to remove the training images that also occur in the test set to ensure no data leakage . MetaShift Statistics Figure 3 shows the statistics of MetaShift across all tasks . We start from the pre-processed and cleaned version of Visual Genome ( Hudson & Manning , 2019 ) , which contains 113,018 distinct images across 1,702 object classes . After the dataset construction , we have 12,868 sets of natural images from 410 classes . Concretely , each class has 31.4 subsets , and each subset has 200.4 images on average . The subsets are characterized by a diverse vocabulary of 1,853 distinct contexts . Beyond 1,702 contexts defined by object presence , MetaShift also leverages the 37 distinct general contexts and 114 object attributes from Visual Genome . The general contexts typically describe locations ( i.e. , indoor , outdoor ) , weather ( e.g. , rainy , cloudless ) , and places ( e.g. , bathroom , ocean ) . The object attributes include color ( e.g. , white , red ) , material ( e.g. , wood , plastic ) , shape ( e.g. , round , square ) , and other object-specific properties ( e.g. , empty/full for plates ) . See Appendix A for examples and more information . Generalizability : Case Study on COCO The MetaShift construction methodology is quite simple , and can be extended to any dataset with metadata tags ( i.e. , multilabel ) . To demonstrate this , we apply the construction methodology on MS-COCO dataset ( Lin et al. , 2014 ) , which provides object detection labels for each image . Applying our construction methodology , we are able to construct 1321 subsets , with an average size of 389 and a median size of 124 , along with 80 meta-graphs that help to quantify the amount of distribution shift among MS-COCO subsets . Experiments on MetaShift from COCO can be found in Appendix D. Even though the data that we used here ( e.g . Visual Genome , COCO ) is not new , the idea of turning one dataset into a structured collection of datasets for assessing distribution shift is less explored . COCO ( Lin et al. , 2014 ) and many other datasets provide meta-data that could be naturally used for generating candidate subsets . The main contribution of our methodology is that after generating the candidate subsets , our method provides a meta-graph that could be used to determine the train and test domains , and also specifies the similarity between the two subsets via graph distance . MetaShift opens up a new window for systematically evaluating domain shifts .

Summary: The paper provides a benchmark dataset that can be used for training & evaluation of Machine Learning models. Their contribution is that they have a large collection of 12,868 sets of natural images across 410 classes obtained from visual genome data and its metadata for annotations.  This helps in accounting for large natural shifts in the data. They also provide a way to measure distance between two subsets to quantize the distribution shift between any two of its data sets.  The paper provides a good justification for the need of such a dataset for computer vision tasks and motivate the idea well. It also talks in detail about the steps taken to generate MetaShift from Visual Genome, it also provides a generalization of their 4-step process of dataset creation on any dataset with multilabel, results presented for COCO dataset. The paper further discusses the use of this dataset for two major cases- Evaluating distribution shifts & Assessing training conflicts. They provide the impact of shift distance on the domain generalization by keeping test set same and varying training subsets randomly. Further, it talks about subpopulation shifts where the train and test distribution of same domain with different mixture weights. They show that no algorithm consistently performs better than other algorithms for larger shifts. It provides a detailed understanding of training conflict by analyzing the contribution of each training subset to the change of the validation loss of each validation set during the training process.  Overall, it’s a well written paper about the motivation, use cases, applicability, and generalizability of their proposed data set.