Extending the WILDS Benchmark for Unsupervised Adaptation

1 INTRODUCTION . Distribution shifts—when models are trained on a source distribution but deployed on a different target distribution—are frequent problems for machine learning systems deployed in the wild ( Geirhos et al. , 2020 ; Koh et al. , 2021 ) . In this paper , we focus on the use of unlabeled data to mitigate these shifts . Unlabeled data is a powerful point of leverage as it is frequently much more available than labeled data . For example , in the crop detection task in Figure 1 , we might only have labeled training examples from one set of source domains ( farms ) available to learn a model that can extrapolate to a different set of target domains ( David et al. , 2020 ) . However , we might have many more unlabeled examples from the source domains , from extra domains , or even directly from the target domains . Many methods for leveraging unlabeled data have been highly successful on some types of distribution shifts ( Berthelot et al. , 2021 ; Zhang et al. , 2021 ) . However , the datasets typically used for evaluating these methods do not reflect many of the realistic shifts that might occur in the wild . These evaluations tend instead to focus on shifts between photos and stylized versions like sketches ( Li et al. , 2017 ; Venkateswara et al. , 2017 ; Peng et al. , 2019 ) or synthetic renderings ( Peng et al. , 2018 ) , or between variants of digits datasets like MNIST ( LeCun et al. , 1998 ) and SVHN ( Netzer et al. , 2011 ) . Unfortunately , prior work has shown that methods that work well on one type of shift need not generalize to others ( Taori et al. , 2020 ; Djolonga et al. , 2020 ; Xie et al. , 2021a ; Miller et al. , 2021 ) , which raises the question of how well they would work on a wider array of realistic shifts . In this paper , we make two contributions . First , we introduce the U-WILDS benchmark ( Figure 2 ) , which builds on the recent WILDS benchmark of in-the-wild distribution shifts ( Koh et al. , 2021 ) . WILDS datasets span a wide range of tasks and modalities , and each dataset reflects a domain generalization or subpopulation shift setting with a substantial gap between in-distribution and out-ofdistribution performance . However , WILDS only contains labeled data , and the leverage points for learning robust models are therefore limited . In U-WILDS , we augment 8 of the 10 WILDS datasets1 1We omitted PY150-WILDS , as code completion data is always labeled by nature of the task , and RXRX1WILDS , as unlabeled data for that genetic perturbation task is not typically available . with curated unlabeled data acquired from the same source and target domains in WILDS , as well as from extra domains of the same type : e.g. , in the GLOBALWHEAT-WILDS dataset pictured in Figure 1 , we acquired unlabeled photos of wheat fields from the source and target farms as well as extra farms that were not in the original labeled dataset . In total , U-WILDS adds 14.5 million unlabeled examples , expanding the number of examples by 3–13× and allowing us to combine the real-world relevance of WILDS with the leverage of unlabeled data . Second , to measure the effectiveness of contemporary methods that leverage unlabeled data , we developed a standardized and consistent protocol for benchmarking using U-WILDS . We evaluated representatives from three popular categories : methods for learning domain-invariant representations ( Sun & Saenko , 2016 ; Ganin et al. , 2016 ) , self-training methods ( Lee , 2013 ; Sohn et al. , 2020 ; Xie et al. , 2020 ) , and pre-training methods that rely on self-supervision ( Devlin et al. , 2019 ; Caron et al. , 2020 ) . These methods have been successful on some types of shifts , such as going from photos to sketches , or from handwritten digits to street signs ( Berthelot et al. , 2021 ; Zhang et al. , 2021 ) . Our results on U-WILDS are mixed : many methods performed similarly to or worse than purelysupervised training , and the only clear successes were on the two image classification datasets with unlabeled target data ( CAMELYON17-WILDS and FMOW-WILDS ) . On those , successful methods relied heavily on data augmentation ( Xie et al. , 2020 ; Caron et al. , 2020 ) , which limited their applicability to other modalities where augmentation techniques are not as well developed , such as text and molecular graphs . The same methods were unsuccessful on the image regression and detection tasks , which have been relatively understudied : e.g. , the notion of hard pseudolabels , which is common to many of the benchmarked methods , does not apply to regression . For the text datasets , continued language model pre-training did not help ; this could be because the online comments in CIVILCOMMENTS-WILDS and product reviews in AMAZON-WILDS are already more similar to the BERT pre-training corpus ( Devlin et al. , 2019 ; Hendrycks et al. , 2020 ) compared to other types of text studied in prior work , such as biomedical papers ( Gururangan et al. , 2020 ) . These results suggest potentially fruitful avenues for future work , such as developing data augmentation techniques for non-image modalities and more realistic hyperparameter tuning protocols . They also underscore the importance of developing and evaluating methods for unlabeled data on a wider variety of real-world shifts than is typically studied . To this end , we developed an open-source Python package containing data loaders , compatible implementations of the methods we benchmarked , and scripts to replicate all experiments in this paper ( Appendix G ) . In addition , we will release a public leaderboard . By allowing developers to easily test algorithms across the variety of datasets in U-WILDS , we hope to accelerate the development of methods that can leverage unlabeled data to improve robustness to real-world distribution shifts . 2 COMPARISON WITH EXISTING UNSUPERVISED ADAPTATION BENCHMARKS . The advantage of U-WILDS is that it builds on the diverse range of applications and modalities covered in WILDS ( Koh et al. , 2021 ) , while providing an extensive amount of unlabeled data that can be used as leverage for training robust models . In this section , we briefly compare with other existing ML benchmarks for unsupervised adaptation . Images . Evaluations of unsupervised adaptation methods for image classification have focused on generalizing from natural photos to a range of stylized images , such as sketches and cartoons ( PACS ( Li et al. , 2017 ) , Office-Home ( Venkateswara et al. , 2017 ) , and DomainNet ( Peng et al. , 2019 ) ) , product images ( Office-31 ( Saenko et al. , 2010 ) ) , and synthetic renderings ( VisDA ( Peng et al. , 2018 ) ) , though location-based shifts have also been recently explored ( Dubey et al. , 2021 ) . It is also popular to evaluate on shifts between digits datasets , such as MNIST ( LeCun et al. , 1998 ) , SVHN ( Netzer et al. , 2011 ) , and USPS ( Hull , 1994 ) . In contrast , U-WILDS contains satellite , microscope , agriculture , and camera trap images , and it includes image regression and detection tasks ; these modalities and settings are comparatively understudied . Existing adaptation benchmarks for image segmentation , which is closely related to detection , focus on generalizing from natural to synthetic scenes ( Ros et al. , 2016 ; Richter et al. , 2016 ; Cordts et al. , 2016 ; Hoffman et al. , 2018 ) . Text . Methods for unsupervised adaptation in NLP are typically evaluated on domain shifts between different textual sources , such as news articles , different categories of product reviews , Wikipedia , or social media platforms ( Blitzer et al. , 2007 ; Mansour et al. , 2009 ; Oren et al. , 2019 ; Miller et al. , 2020 ; Kamath et al. , 2020 ; Hendrycks et al. , 2020 ) , or even more specialized sources such as legal documents ( Chalkidis et al. , 2020 ) or biomedical papers ( Lee et al. , 2020b ; Gu et al. , 2020 ) . Multilingual tasks can also be a setting for unsupervised adaptation ( Conneau et al. , 2018 ; Conneau & Lample , 2019 ; Hu et al. , 2020a ; Clark et al. , 2020 ) , especially when generalizing to low-resource languages ( Nekoto et al. , 2020 ) . The U-WILDS text datasets differ in that they focus on subpopulation performance , either to particular demographics in CIVILCOMMENTS-WILDS or to tail populations in AMAZON-WILDS , rather than on adapting to a completely distinct domain . Molecules . While unlabeled molecules have been used for pre-training ( Hu et al. , 2020c ; Rong et al. , 2020 ) , no standardized unsupervised adaptation benchmarks have been developed . In U-WILDS , we extend OGB-MOLPCBA to include unlabeled data from source , validation , and target domains . 3 PROBLEM SETTING . Following WILDS , we study the domain shift setting where the data is drawn from domains d ∈ D. Each domain d corresponds to a data distribution Pd over ( x , y , d ) , where x is the input , y is the prediction , and all points from Pd have domain d. The domains come in four types : We consider domain adaptation , where all four types of domains are disjoint ; subpopulation shift , where the target domains are a subset of the source domains ; and hybrids of these two problem settings . Models are trained on labeled data from the source domains , as well as on unlabeled data from one or more types of domains , depending on what is realistic for the application . 4 DATASETS . The U-WILDS datasets augment the WILDS datasets with curated unlabeled data . For consistency , the labeled datasets are exactly the same as in WILDS , which allows direct evaluations of the utility of additional unlabeled training data . The labeled and unlabeled data are disjoint ; for instance , the unlabeled data from the target domains is different from the labeled target data used for evaluation . Here , we briefly describe each dataset , why unlabeled data is realistically obtainable for the corresponding task , and how it might help . We provide details for each dataset in Appendix A. IWILDCAM2020-WILDS : Species classification across different camera traps . The task is to classify the animal species captured in a camera trap image ( Beery et al. , 2020 ) . We aim to generalize to new camera trap locations despite variations in illumination , background , and label frequencies ( Beery et al. , 2018 ) . While hundreds of thousands of camera traps are active worldwide , only a small subset of these traps have had images labeled . On the other hand , unlabeled data from additional sensors operating across diverse conditions is plentiful . In this work , we add unlabeled images from 3,215 extra camera traps also in the WCS Camera Traps dataset ( Beery et al. , 2020 ) . This expands the number of camera traps by a factor of 11 and the number of examples by a factor of 5 . CAMELYON17-WILDS : Tumor identification across different hospitals . The task is to classify image patches from lymph node sections as tumor or normal tissue . We seek to generalize to new hospitals , which can differ in their patient demographics and data acquisition protocols ( Veta et al. , 2016 ; AlBadawy et al. , 2018 ; Komura & Ishikawa , 2018 ; Tellez et al. , 2019 ) . While obtaining labeled data for histopathology applications requires expensive and pain-staking annotations from expert pathologists , hospitals accumulate plentiful unlabeled slide images during normal operation . Unlabeled images could be used to adapt to differences in the test hospitals ( e.g. , different staining protocols might lead to different color distributions ) . We provide unlabeled patches from train and test hospitals , which expands the total number of patches by a factor of 7.5 . Both the labeled and unlabeled data are adapted from the Camelyon17 dataset ( Bandi et al. , 2018 ) . FMOW-WILDS : Land use classification across different regions and years . The task is to classify the type of building or land usage in a satellite image . Given training data from before 2013 , we aim to generalize to satellite imagery taken after 2013 , while maintaining high accuracy across all geographic regions . While labeling land use requires combining map data and expert annotations , unlabeled data is available in all locations in the world through constant streams of global satellite imagery . Prior work has shown that unlabeled satellite data can improve OOD accuracy in landcover and cropland prediction ( Xie et al. , 2021a ) as well as aerial object and scene classification ( Reed et al. , 2021 ) . We provide unlabeled satellite imagery from source and target timeframes across all regions , expanding the dataset by a factor of 3.5 . Both the labeled and unlabeled data are adapted from the FMoW dataset ( Christie et al. , 2018 ) . POVERTYMAP-WILDS : Poverty mapping across different countries . The task is to predict a real-valued asset wealth index of the area in a satellite image . We consider generalizing across different countries . Like FMOW-WILDS , unlabeled satellite imagery is available globally , while labeled data is expensive to collect as it requires conducting nationally representative surveys in the field . Prior work on poverty prediction has used unlabeled data for entropy minimization ( Jean et al. , 2018 ) and pre-training on auxiliary tasks such as nighttime light prediction ( Xie et al. , 2016 ; Jean et al. , 2016 ) , but these studies do not study generalization to new countries . We provide unlabeled satellite imagery from both train and test countries , expanding the dataset by a factor of 14 . Both the labeled and unlabeled data are adapted from Yeh et al . ( 2020 ) . GLOBALWHEAT-WILDS : Wheat head detection across different regions . The task is to localize wheat heads in overhead field images . We seek to generalize across image acquisition sessions , which each represent a particular location , time , and sensor . Wheat field images contain many densely packed and overlapping instances , making labeling wheat heads in images costly , tedious and sensitive to the individual annotator . However , hundreds of agricultural research institutes around the world collect terabytes of unlabeled field images which could be used for training . We add unlabeled field images from train , test , and extra acquisition sessions , expanding the dataset by a factor of 10 . The labeled and unlabeled data are adapted from the Global Wheat Head Detection dataset and its underlying sources ( David et al. , 2020 ; 2021 ) . OGB-MOLPCBA : Molecular property prediction across different scaffolds . The task is to predict the biological activity of small molecules represented as molecular graphs ( Wu et al. , 2018 ; Hu et al. , 2020b ) . We seek to generalize to molecules with new scaffold structures . Labels on biological activity are only available for a small portion of molecules , as they require expensive lab experiments to obtain . However , unlabeled molecule structures are readily available in large-scale chemical databases such as PubChem ( Bolton et al. , 2008 ) , and have been previously used for pretraining ( Hu et al. , 2020c ) and semi-supervised learning ( Sun et al. , 2020 ) . We provide 5 million unlabeled molecules from source and target scaffolds , which expands the number of molecules by a factor of 12.5 . The original labeled data was curated by MoleculeNet ( Wu et al. , 2018 ) from PubChem , and we similarly extracted the unlabeled data from PubChem ( Bolton et al. , 2008 ) . CIVILCOMMENTS-WILDS : Toxicity classification across demographic identities . The task is to classify whether a text comment is toxic or not . We consider the subpopulation shift setting , where the model must classify accurately across groups of comments mentioning different demographic identities . While labels require large-scale crowdsourcing annotations on both comment toxicity , unlabeled article comments are widely available on the internet . We provide unannotated comments as unlabeled data , which expands the size of the dataset by a factor of 4.5 . Both the labeled and unlabeled data are adapted from Borkan et al . ( 2019 ) . AMAZON-WILDS : Sentiment classification across different users . The task is to classify the star ratings of Amazon reviews . We seek to perform consistently well across new reviewers . While the labels ( star ratings ) are always available for Amazon reviews in practice , unlabeled data is a common source of leverage for sentiment classification more generally , with prior work in domain adaptation ( Blitzer & Pereira , 2007 ; Glorot et al. , 2011 ) and semi-supervised learning ( Dasgupta & Ng , 2009 ; Li et al. , 2011 ) . We provide unlabeled reviews from test and extra reviewers , which expands the total number of reviews by a factor of 7.5 . Both the labeled and unlabeled data are adapted from the Amazon review dataset by Ni et al . ( 2019 ) .

Summary: The authors propose U-WILDS, which extends the WILDS benchmark (typically used for domain generalization or subpopulation shift) to the unsupervised domain adaptation scenario. They select eight datasets from WILDS, spanning a variety of data modalities, and add in additional unlabelled examples from a variety of data sources. They benchmark a comprehensive set of algorithms which make use of unlabelled data, and find that most methods do not significantly outperform ERM, except for some limited cases which the authors characterize in detail.