Extending the WILDS Benchmark for Unsupervised Adaptation

1 INTRODUCTION . Distribution shifts—when models are trained on a source distribution but deployed on a different target distribution—are frequent problems for machine learning systems deployed in the wild ( Geirhos et al. , 2020 ; Koh et al. , 2021 ) . In this paper , we focus on the use of unlabeled data to mitigate these shifts . Unlabeled data is a powerful point of leverage as it is frequently much more available than labeled data . For example , in the crop detection task in Figure 1 , we might only have labeled training examples from one set of source domains ( farms ) available to learn a model that can extrapolate to a different set of target domains ( David et al. , 2020 ) . However , we might have many more unlabeled examples from the source domains , from extra domains , or even directly from the target domains . Many methods for leveraging unlabeled data have been highly successful on some types of distribution shifts ( Berthelot et al. , 2021 ; Zhang et al. , 2021 ) . However , the datasets typically used for evaluating these methods do not reflect many of the realistic shifts that might occur in the wild . These evaluations tend instead to focus on shifts between photos and stylized versions like sketches ( Li et al. , 2017 ; Venkateswara et al. , 2017 ; Peng et al. , 2019 ) or synthetic renderings ( Peng et al. , 2018 ) , or between variants of digits datasets like MNIST ( LeCun et al. , 1998 ) and SVHN ( Netzer et al. , 2011 ) . Unfortunately , prior work has shown that methods that work well on one type of shift need not generalize to others ( Taori et al. , 2020 ; Djolonga et al. , 2020 ; Xie et al. , 2021a ; Miller et al. , 2021 ) , which raises the question of how well they would work on a wider array of realistic shifts . In this paper , we make two contributions . First , we introduce the U-WILDS benchmark ( Figure 2 ) , which builds on the recent WILDS benchmark of in-the-wild distribution shifts ( Koh et al. , 2021 ) . WILDS datasets span a wide range of tasks and modalities , and each dataset reflects a domain generalization or subpopulation shift setting with a substantial gap between in-distribution and out-ofdistribution performance . However , WILDS only contains labeled data , and the leverage points for learning robust models are therefore limited . In U-WILDS , we augment 8 of the 10 WILDS datasets1 1We omitted PY150-WILDS , as code completion data is always labeled by nature of the task , and RXRX1WILDS , as unlabeled data for that genetic perturbation task is not typically available . with curated unlabeled data acquired from the same source and target domains in WILDS , as well as from extra domains of the same type : e.g. , in the GLOBALWHEAT-WILDS dataset pictured in Figure 1 , we acquired unlabeled photos of wheat fields from the source and target farms as well as extra farms that were not in the original labeled dataset . In total , U-WILDS adds 14.5 million unlabeled examples , expanding the number of examples by 3–13× and allowing us to combine the real-world relevance of WILDS with the leverage of unlabeled data . Second , to measure the effectiveness of contemporary methods that leverage unlabeled data , we developed a standardized and consistent protocol for benchmarking using U-WILDS . We evaluated representatives from three popular categories : methods for learning domain-invariant representations ( Sun & Saenko , 2016 ; Ganin et al. , 2016 ) , self-training methods ( Lee , 2013 ; Sohn et al. , 2020 ; Xie et al. , 2020 ) , and pre-training methods that rely on self-supervision ( Devlin et al. , 2019 ; Caron et al. , 2020 ) . These methods have been successful on some types of shifts , such as going from photos to sketches , or from handwritten digits to street signs ( Berthelot et al. , 2021 ; Zhang et al. , 2021 ) . Our results on U-WILDS are mixed : many methods performed similarly to or worse than purelysupervised training , and the only clear successes were on the two image classification datasets with unlabeled target data ( CAMELYON17-WILDS and FMOW-WILDS ) . On those , successful methods relied heavily on data augmentation ( Xie et al. , 2020 ; Caron et al. , 2020 ) , which limited their applicability to other modalities where augmentation techniques are not as well developed , such as text and molecular graphs . The same methods were unsuccessful on the image regression and detection tasks , which have been relatively understudied : e.g. , the notion of hard pseudolabels , which is common to many of the benchmarked methods , does not apply to regression . For the text datasets , continued language model pre-training did not help ; this could be because the online comments in CIVILCOMMENTS-WILDS and product reviews in AMAZON-WILDS are already more similar to the BERT pre-training corpus ( Devlin et al. , 2019 ; Hendrycks et al. , 2020 ) compared to other types of text studied in prior work , such as biomedical papers ( Gururangan et al. , 2020 ) . These results suggest potentially fruitful avenues for future work , such as developing data augmentation techniques for non-image modalities and more realistic hyperparameter tuning protocols . They also underscore the importance of developing and evaluating methods for unlabeled data on a wider variety of real-world shifts than is typically studied . To this end , we developed an open-source Python package containing data loaders , compatible implementations of the methods we benchmarked , and scripts to replicate all experiments in this paper ( Appendix G ) . In addition , we will release a public leaderboard . By allowing developers to easily test algorithms across the variety of datasets in U-WILDS , we hope to accelerate the development of methods that can leverage unlabeled data to improve robustness to real-world distribution shifts . 2 COMPARISON WITH EXISTING UNSUPERVISED ADAPTATION BENCHMARKS . The advantage of U-WILDS is that it builds on the diverse range of applications and modalities covered in WILDS ( Koh et al. , 2021 ) , while providing an extensive amount of unlabeled data that can be used as leverage for training robust models . In this section , we briefly compare with other existing ML benchmarks for unsupervised adaptation . Images . Evaluations of unsupervised adaptation methods for image classification have focused on generalizing from natural photos to a range of stylized images , such as sketches and cartoons ( PACS ( Li et al. , 2017 ) , Office-Home ( Venkateswara et al. , 2017 ) , and DomainNet ( Peng et al. , 2019 ) ) , product images ( Office-31 ( Saenko et al. , 2010 ) ) , and synthetic renderings ( VisDA ( Peng et al. , 2018 ) ) , though location-based shifts have also been recently explored ( Dubey et al. , 2021 ) . It is also popular to evaluate on shifts between digits datasets , such as MNIST ( LeCun et al. , 1998 ) , SVHN ( Netzer et al. , 2011 ) , and USPS ( Hull , 1994 ) . In contrast , U-WILDS contains satellite , microscope , agriculture , and camera trap images , and it includes image regression and detection tasks ; these modalities and settings are comparatively understudied . Existing adaptation benchmarks for image segmentation , which is closely related to detection , focus on generalizing from natural to synthetic scenes ( Ros et al. , 2016 ; Richter et al. , 2016 ; Cordts et al. , 2016 ; Hoffman et al. , 2018 ) . Text . Methods for unsupervised adaptation in NLP are typically evaluated on domain shifts between different textual sources , such as news articles , different categories of product reviews , Wikipedia , or social media platforms ( Blitzer et al. , 2007 ; Mansour et al. , 2009 ; Oren et al. , 2019 ; Miller et al. , 2020 ; Kamath et al. , 2020 ; Hendrycks et al. , 2020 ) , or even more specialized sources such as legal documents ( Chalkidis et al. , 2020 ) or biomedical papers ( Lee et al. , 2020b ; Gu et al. , 2020 ) . Multilingual tasks can also be a setting for unsupervised adaptation ( Conneau et al. , 2018 ; Conneau & Lample , 2019 ; Hu et al. , 2020a ; Clark et al. , 2020 ) , especially when generalizing to low-resource languages ( Nekoto et al. , 2020 ) . The U-WILDS text datasets differ in that they focus on subpopulation performance , either to particular demographics in CIVILCOMMENTS-WILDS or to tail populations in AMAZON-WILDS , rather than on adapting to a completely distinct domain . Molecules . While unlabeled molecules have been used for pre-training ( Hu et al. , 2020c ; Rong et al. , 2020 ) , no standardized unsupervised adaptation benchmarks have been developed . In U-WILDS , we extend OGB-MOLPCBA to include unlabeled data from source , validation , and target domains . 3 PROBLEM SETTING . Following WILDS , we study the domain shift setting where the data is drawn from domains d ∈ D. Each domain d corresponds to a data distribution Pd over ( x , y , d ) , where x is the input , y is the prediction , and all points from Pd have domain d. The domains come in four types : We consider domain adaptation , where all four types of domains are disjoint ; subpopulation shift , where the target domains are a subset of the source domains ; and hybrids of these two problem settings . Models are trained on labeled data from the source domains , as well as on unlabeled data from one or more types of domains , depending on what is realistic for the application . 4 DATASETS . The U-WILDS datasets augment the WILDS datasets with curated unlabeled data . For consistency , the labeled datasets are exactly the same as in WILDS , which allows direct evaluations of the utility of additional unlabeled training data . The labeled and unlabeled data are disjoint ; for instance , the unlabeled data from the target domains is different from the labeled target data used for evaluation . Here , we briefly describe each dataset , why unlabeled data is realistically obtainable for the corresponding task , and how it might help . We provide details for each dataset in Appendix A. IWILDCAM2020-WILDS : Species classification across different camera traps . The task is to classify the animal species captured in a camera trap image ( Beery et al. , 2020 ) . We aim to generalize to new camera trap locations despite variations in illumination , background , and label frequencies ( Beery et al. , 2018 ) . While hundreds of thousands of camera traps are active worldwide , only a small subset of these traps have had images labeled . On the other hand , unlabeled data from additional sensors operating across diverse conditions is plentiful . In this work , we add unlabeled images from 3,215 extra camera traps also in the WCS Camera Traps dataset ( Beery et al. , 2020 ) . This expands the number of camera traps by a factor of 11 and the number of examples by a factor of 5 . CAMELYON17-WILDS : Tumor identification across different hospitals . The task is to classify image patches from lymph node sections as tumor or normal tissue . We seek to generalize to new hospitals , which can differ in their patient demographics and data acquisition protocols ( Veta et al. , 2016 ; AlBadawy et al. , 2018 ; Komura & Ishikawa , 2018 ; Tellez et al. , 2019 ) . While obtaining labeled data for histopathology applications requires expensive and pain-staking annotations from expert pathologists , hospitals accumulate plentiful unlabeled slide images during normal operation . Unlabeled images could be used to adapt to differences in the test hospitals ( e.g. , different staining protocols might lead to different color distributions ) . We provide unlabeled patches from train and test hospitals , which expands the total number of patches by a factor of 7.5 . Both the labeled and unlabeled data are adapted from the Camelyon17 dataset ( Bandi et al. , 2018 ) . FMOW-WILDS : Land use classification across different regions and years . The task is to classify the type of building or land usage in a satellite image . Given training data from before 2013 , we aim to generalize to satellite imagery taken after 2013 , while maintaining high accuracy across all geographic regions . While labeling land use requires combining map data and expert annotations , unlabeled data is available in all locations in the world through constant streams of global satellite imagery . Prior work has shown that unlabeled satellite data can improve OOD accuracy in landcover and cropland prediction ( Xie et al. , 2021a ) as well as aerial object and scene classification ( Reed et al. , 2021 ) . We provide unlabeled satellite imagery from source and target timeframes across all regions , expanding the dataset by a factor of 3.5 . Both the labeled and unlabeled data are adapted from the FMoW dataset ( Christie et al. , 2018 ) . POVERTYMAP-WILDS : Poverty mapping across different countries . The task is to predict a real-valued asset wealth index of the area in a satellite image . We consider generalizing across different countries . Like FMOW-WILDS , unlabeled satellite imagery is available globally , while labeled data is expensive to collect as it requires conducting nationally representative surveys in the field . Prior work on poverty prediction has used unlabeled data for entropy minimization ( Jean et al. , 2018 ) and pre-training on auxiliary tasks such as nighttime light prediction ( Xie et al. , 2016 ; Jean et al. , 2016 ) , but these studies do not study generalization to new countries . We provide unlabeled satellite imagery from both train and test countries , expanding the dataset by a factor of 14 . Both the labeled and unlabeled data are adapted from Yeh et al . ( 2020 ) . GLOBALWHEAT-WILDS : Wheat head detection across different regions . The task is to localize wheat heads in overhead field images . We seek to generalize across image acquisition sessions , which each represent a particular location , time , and sensor . Wheat field images contain many densely packed and overlapping instances , making labeling wheat heads in images costly , tedious and sensitive to the individual annotator . However , hundreds of agricultural research institutes around the world collect terabytes of unlabeled field images which could be used for training . We add unlabeled field images from train , test , and extra acquisition sessions , expanding the dataset by a factor of 10 . The labeled and unlabeled data are adapted from the Global Wheat Head Detection dataset and its underlying sources ( David et al. , 2020 ; 2021 ) . OGB-MOLPCBA : Molecular property prediction across different scaffolds . The task is to predict the biological activity of small molecules represented as molecular graphs ( Wu et al. , 2018 ; Hu et al. , 2020b ) . We seek to generalize to molecules with new scaffold structures . Labels on biological activity are only available for a small portion of molecules , as they require expensive lab experiments to obtain . However , unlabeled molecule structures are readily available in large-scale chemical databases such as PubChem ( Bolton et al. , 2008 ) , and have been previously used for pretraining ( Hu et al. , 2020c ) and semi-supervised learning ( Sun et al. , 2020 ) . We provide 5 million unlabeled molecules from source and target scaffolds , which expands the number of molecules by a factor of 12.5 . The original labeled data was curated by MoleculeNet ( Wu et al. , 2018 ) from PubChem , and we similarly extracted the unlabeled data from PubChem ( Bolton et al. , 2008 ) . CIVILCOMMENTS-WILDS : Toxicity classification across demographic identities . The task is to classify whether a text comment is toxic or not . We consider the subpopulation shift setting , where the model must classify accurately across groups of comments mentioning different demographic identities . While labels require large-scale crowdsourcing annotations on both comment toxicity , unlabeled article comments are widely available on the internet . We provide unannotated comments as unlabeled data , which expands the size of the dataset by a factor of 4.5 . Both the labeled and unlabeled data are adapted from Borkan et al . ( 2019 ) . AMAZON-WILDS : Sentiment classification across different users . The task is to classify the star ratings of Amazon reviews . We seek to perform consistently well across new reviewers . While the labels ( star ratings ) are always available for Amazon reviews in practice , unlabeled data is a common source of leverage for sentiment classification more generally , with prior work in domain adaptation ( Blitzer & Pereira , 2007 ; Glorot et al. , 2011 ) and semi-supervised learning ( Dasgupta & Ng , 2009 ; Li et al. , 2011 ) . We provide unlabeled reviews from test and extra reviewers , which expands the total number of reviews by a factor of 7.5 . Both the labeled and unlabeled data are adapted from the Amazon review dataset by Ni et al . ( 2019 ) .

Summary: The authors present U-WILDS, an extension of the multi-task, large-scale domain-shift dataset WILDS. They provide a large quantity of unlabelled data complementing 8 of the existing multidomain labeled datasets in WILDS. They propose an extensive array of experiments evaluating the ability of a wide variety of algorithms to leverage the unlabelled data to address domain-shift. They present reasoned conclusions, and open-source datasets and implementations.