Extending the WILDS Benchmark for Unsupervised Adaptation
Authors: Shiori Sagawa
Pang Wei Koh
Tony Lee
Irena Gao
Sang Michael Xie
Kendrick Shen
Ananya Kumar
Weihua Hu
Michihiro Yasunaga
Henrik Marklund
Sara Beery
Etienne David
Ian Stavness
Wei Guo
Jure Leskovec
Kate Saenko
Tatsunori Hashimoto
Sergey Levine
Chelsea Finn
Percy Liang
Venue: ICLR 2022 Oral (2022)
Keywords: distribution shifts
adaptation
unlabeled data

Abstract: Machine learning systems deployed in the wild are often trained on a source distribution but deployed on a different target distribution. Unlabeled data can be a powerful point of leverage for mitigating these distribution shifts, as it is frequently much more available than labeled data and can often be obtained from distributions beyond the source distribution as well. However, existing distribution shift benchmarks with unlabeled data do not reflect the breadth of scenarios that arise in real-world applications. In this work, we present the WILDS 2.0 update, which extends 8 of the 10 datasets in the WILDS benchmark of distribution shifts to include curated unlabeled data that would be realistically obtainable in deployment. These datasets span a wide range of applications (from histology to wildlife conservation), tasks (classification, regression, and detection), and modalities (photos, satellite images, microscope slides, text, molecular graphs). The update maintains consistency with the original WILDS benchmark by using identical labeled training, validation, and test sets, as well as identical evaluation metrics. We systematically benchmark state-of-the-art methods that use unlabeled data, including domain-invariant, self-training, and self-supervised methods, and show that their success on WILDS is limited. To facilitate method development, we provide an open-source package that automates data loading and contains the model architectures and methods used in this paper. Code and leaderboards are available at https://wilds.stanford.edu.

Published as a conference paper at ICLR 2022
EXTENDING THE WILDS BENCHMARK FOR UNSUPERVISED ADAPTATION
Shiori Sagawa',, Pang Wei Koh Tony Lee't, Irena Gao
Sang Michael Xie'. Kendrick Shen', Ananya Kumar', Weihua Hu', Michihiro Yasunaga', Henrik Marklund', Sara Beery?, Etienne David Ian Stavness Wei Guo Jure Leskovec', Kate Saenko Tatsunori Hashimoto' Sergey Levine", Chelsea Finn', Percy Liang'
"Equal contribution Sttnfoor University "Caltech ÍNRAE UUniversity of Saskatchewan University Tokyo Boston University 'University of California, Berkeley
ABSTRACT
Machine learning systems deployed in the wild are often trained on a source dis- tribution but deployed on different target distribution. Unlabeled data can be a powerful point of leverage for mitigating these distribution shifts, as it is fre- quently much more available than labeled data and can often be obtained from distributions beyond the source distribution as well. However, existing distribu- tion shift benchmarks with unlabeled data do not reflect the breadth of scenarios that arise in weal-world applications. In this work, we present the WILDS 2.0 update, which extends 8 of the 10 datasets the WILDS benchmark of distribu- tion shifts to include curated unlabeled data that would be realistically obtainable in deployment. These datasets span a wide range of applications (from histol- ogy to wildlife conservation), tasks (classification, regression, and detection), and modalities (photos, satellite images, microscope slides, text, molecular graphs). The update maintains consistency with the original WILDS benchmark by using identical labeled training, validation, and test sets, as well as identical evaluation metries. We systematically benchmark state-of-the-art methods that unlabeled data, including domain-invane invariant, self-training, and self-supervised methods, and show that their success on WILDS is limited. To facilitate method development, we provide an open-source package that automates data loading and contains the model architectures and methods used in this paper. Code and leaderboards are available at https: /tww///w/////////e./..../ stanford edu.
INTRODUCTION
Distribution shifts shifts-when models trained source distribution but deployed different target distribution- are frequent problems for machine learning systems in the wild (Quiñonero- Candela et al., 2009; Geirhos et al., 2020; Koh et al., 2021). In this paper, focus on the of unlabeled data to mitigate these shifts. Unlabeled data is a powerful point of leverage as it is more readily available than labeled data and can often be obtained from distributions beyond the source distribution. For example, in the crop detection task in Figure 1, we wish to learn a model that can extrapolate to a of target domains (farms) (David et al., 2020), and while we only have labeled training examples from some source domains, we have many more unlabeled examples from the source domains, from extra domains, and even directly from the target domains.
Many methods for leveraging unlabeled data have been highly successful on some types of distri- bution shifts (Berthelot et al., 2021; Zhang et al., 2021). However, the datasets typically used for evaluating these methods do not reflect many of the realistic shifts that might occur in the wild. These evaluations tend instead to focus on shifts between photos and stylized versions like sketches (Li et al., 2017; Venkateswara et al., 2017; Peng et al., 2019) or synthetic renderings (Peng et al., 2018), 1)1 or between variants of digits datasets like MNIST (LeCun et al., 1998) and SVHN (Netzer et al., 2011). Unfortunately, prior work has shown that methods that work well need not generalize to others (Taori et al., 2020; Djolonga et al., 2020; Xie et al., 2021a: Miller et al., 2021), 1) which raises the question of how well they would work on wider array of realistic shifts.
type of shift

---

Published as a conference paper at ICLR 2022
Source
Extra
Validation
Target
Labeled
UK
Japan
Unlabeled
Belgium
France
Norway
Canada
Mexico
China.
United States
Figure 1: Each WILDS dataset (Koh et al., 2021) contains labeled data from the source domains (for training), validation domains (for hyperparameter selection), and target domains (for held-out evaluation). In the WILDS 2.0 update, we extend these datasets with unlabeled data from a com- bination of source, validation, or target domains, as well as extra domains from which there is no labeled data. The labeled data is exactly the same in WILDS 1.0. In this figure, we illustrate the setting with the GLOBALW HEAT-WETWDD WILDS dataset, where domains correspond to images acquired from different locations and different times.
In this paper, we make two contributions. First, we present WILDS 2.0 (Figure 2), an updated ver- sion of the recent WILDS benchmark of in-the-wild distribution shifts (Koh et al., 2021). WILDS datasets span wide range of tasks and modalities, and each dataset reflects domain generalization or subpopulation shift setting with substantial gap between in in-distribution and dfitt--ditti performance. However, WILDS 1.0 only contained labeled data, which limits the leverage for learn- ing robust models. In WILDS 2.0, we extend of the WILDS datasets! with curated unlabeled data acquired from the same source and target domains the labeled data, as well as from extra domains of the same type: c.g., in the GLOBAL WHALWHHAT-WWEE WILDS dataset pictured in Figure 1, we acquired unlabeled photos of wheat fields from the source and target farms as well extra farms that were not in the original labeled dataset. In total, WILDS 2.0 adds 14.5 million unlabeled ex- amples, expanding the number of examples for each dataset by 3-13x and allowing us to combine the real-world relevance of WILDS with the leverage of unlabeled data.
Second, we developed standardized and consistent protocol for evaluating methods that lever- age the unlabeled data in WILDS 2.0. We assessed representatives from three popular categories: representations (Sun & Saenko, 2016: Ganin et al., 2016), self-training methods (Lee, 2013; Sohn et al., 2020; Xie et al., 2020), and pre-training methods that rely on self-supervision (Devlin et al., 2019; Caron et al., 2020). These methods have been success- ful on some types of shifts, such as going from photos to sketches, or from handwritten digits to street signs (Berthelot et al., 2021: Zhang et al., 2021).
methods for learning domain-invariin
Our results on WILDS are mixed: many methods did not outperform standard supervised training despite using additional unlabeled data, and the only clear successes were on two im- age classification datasets (CAMElyON 1ONN17-WII 7T- and FMOW-WILDDS. Successful methods relied heavily on data augmentation (Xic et al., 2020; Caron et al., 2020). which limited their applicability to modalities where augmentations are not well developed, such as text and molecular graphs. The same methods were unsuccessful on image regression and detection tasks, which have been relatively understudied: e.g., beeudolabel-based methods do not straightforwardly apply regres- sion. For the text datasets, continued language model pre-training did not help, unlike in prior work (Gururangan et al., 2020). Our results suggest fruitful avenues for future work, such as developing data augmentations for non-image modalities and more effective hyperparameter tuning protocols.
Overall, our results underscore the importance of developing and evaluating methods for unlabeled data on a wider variety of real-world shifts than is typically studied. To this end, we have updated the open-source Python WILDS package include unlabeled data loaders, compatible implementa- tions of all the methods we benchmarked, and scripts to replicate all experiments in this paper (Ap-
We omitted PY PY150-wa 5O..D..S,, code completion data is always labeled by nature of the task, and RxRx1- WILDS,, as unlabeled data for that genetic perturhation task is not typically available.
2

---

Published as a conference paper at ICLR 2022
Ditared
IwleCamm
Cameboni 7
PXRoc
RMS
PoverbWllo
Skocalinedl
06B McPOBA ChrComment
Amazon
50
Impur 10
cemera trop shots Souu s00s
coll image
saloiito rrage
sixoilo mage
whea rigge
mocoaar g of comment pioduct rovlew
200e
Prediction b)
shimell specis
turor
persurted gene
land use
assel wisth
whest head D0ox
bicnssyy
toaicty
certiment
BURCCompion comples
Doma (8)
comera
pocpital
botcn
nogion
courtyy natrr
the
demogrtphic
WiLLE UCDT
Fnowrt ceeey
osample
Parniny
Croaais bakmeer yare batnavie
Towdd porteet cuu
Alpers
Target exarpple
Pup Fagan?
Origiaa paper
20,00
2016
2019
2015
2500
2321
2020
2019
2019
2016
domaine
323
51
16x5
23x
47
120,004
16
0,800
8.421
Esteled
cuamp as
800,099
455 964
125,500
41. 10.86
18,009
6,515
437 829
643/000
106 700
150,000
domait
13x5
13x2
18
44,000
exompp PM
1,795,247
6.577
0.22.077
domairo
3,P15
domains
42.445
1,561,216
2,807,841
Vill deric
dom
3x.5
5x2
11
31,361
1,554
Accampler
600,030
'55.315
24.175
2,000
430,325
205,208
Target
domaina
2x55
0x2
10
43,701
1,734
eecampeee Be
600,030
73,206
06.275
8.877
0,7,043
268,761
Figure 2: The WILDS 2.0 update adds unlabeled data to 8 WILDS datasets. For each dataset, we kept the labeled data from WILDS and expanded the datasets by 3-13x with unlabeled data from the same underlying dataset. The type of unlabeled data (i.e., whether it comes from source, extra, validation, or target domains) depends on what realistic and available for the application. Beyond these 8 datasets, WILDS also contains 2 datasets without unlabeled data: the PY150-WILDS code completion dataset and the RxRx1- - genetic perturbation dataset. For all datasets, the laheled data and evaluation metrics exactly the same as in WILDS 1 1.0. Figure adapted with permission from Koh et al. (2021).
pendix G). Code and public leaderboards are available at https:/ /wi lds. stanford. By allowing developers to easily test algorithms across the variety of datasets in WILDS 2.0, we hope to accelerate the development of methods that can leverage unlabeled data to improve robustness to real-world distribution shifts.
Finally, we note that WILDS 2.0 not separate benchmark from WILDS 1.0: the labeled data and evaluation metrics are exactly the same in WILDS 1.0 and WILDS 2.0, and future results should be reported on the overall WILDS benchmark, with a note describing what kind of unlabeled data (if any) was used. In this paper, we discuss the addition of unlabeled data and analyze the performance of methods that use the unlabeled data, For a more detailed description of the datasets, evaluation metrics, and models used, please refer to the original WILDS paper (Koh et al., 2021).
2 COMPARISON WITH EXISTING UNSUPERVISHD ADAPTATION BENCHMARKS
WILDS 2.0 offers a diverse range of applications and modalities while also providing an extensive amount of unlabeled data that be used leverage for training robust models. In this section, briefly compare with other existing ML benchmarks for unsupervised adaptation.
Images. Evaluations of unsupervised adaptation methods for image classification have focused on generalizing from natural photos to a range of stylized images, such as sketches and cartoons (PACS (Li et al., 2017), Office-Home (Venkateswara et al., 2017), and DomainNet (Peng al., 2019)), product images (Office-31 (Saenko et al., 2010)), and synthetic renderings (VisDA (Peng et al., 2018)), though location- based shifts have also been recently explored (Dubey al., 2021). It is also popular to evaluate on shifts between digits datasets, such as MNIST (LeCun et al., 1998), SVHN (Netzer et al., 201 1), and USPS (Hull, 1994). image detection and segmentation, existing adaptation benchmarks tend to focus
generalizing from synthetic to natural scenes (Ros et al., 2016; Richter et al., 2016; Cordts et al., 2016; Hoffman et al., 2018), which can be an important tool for realistic problems but is not the focus of this work. In contrast, WILDS considers distribution shifts, and it spans diverse modalities (satellite, microscope, agriculture, and camera trap images) and tasks (classification, regression, detection).
Text. Methods for unsupervised adaptation in NLP are typically evaluated on domain shifts between different textual sources, such news articles, different categories of product reviews, Wikipedia,
3

---

Published as a conference paper at ICLR 2022
or social media platforms (Blitzer et al., 2007; Mansour et al., 2009; Oren al., 2019; Miller al., 2020; Kamath et al., 2020; Hendrycks et al., 2020), or even more specialized sources such as legal documents (Chalkidis et al., 2020) or biomedical papers (Lee al., 2020b; Gu et al., 2020). Multi- lingual tasks can also be setting for unsupervised adaptation (Conneau et al., 2018; Conneau & Lample, 2019; Hu al., 2020a; Clark et al., 2020), especially when generalizing languages (Nekoto et al., 2020). The WILDS text datasets differ in that they focus on subpopulation performance, either to particular demographies in CIVIL COMMENTS- or to tail populations in AMAZON-WILDS, rather than on adapting to a completely distinct domain.
low-resource
Molecules. While unlabeled molecules have been used for pre- pre-training (Hu et al., 2020c; Rong et al., 2020), no standardized unsupervised adaptation benchmarks have been developed.
3 PROBLEM SETTING
As in WILDS 1.0, we study the domain shift setting where the data is drawn from domains d E 'D. Each domain corresponds prediction, and all points from Pa have domain d. See Koh et al. (2021) for more details. The domains come in four types:
data distribution Pa over (I, y, d), where æ is the input, y is the
Type of domain
Labeled data
Unlabeled data
Source domains
Used for training None
Extra domains
Can be used for training, if available
Validation domains
Used for hyperparameter tuning
Target domains
Used for held-out evaluation
All datasets have labeled source, validation, and target data, as well as unlabeled data from one or more types of domains, depending on what is realistic for the application.
Table 1:
We consider several variants of the domain shift setting. In some applications, all four types of domains are disjoint (c.g., if we are training on labeled data from some hospitals but seeking to generalize to new hospitals); in others, the target domains a subset of the source domains (c.g.. if we are training on a heterogeneous dataset but seeking to measure model performance on particular demographic subpopulations). Models are trained on labeled data from the source domains, well as unlabeled data of one or more types of domains, depending on what is realistic for the application.
4 DATASETS
WILDS 2.0 augments 8 WILDS datasets with curated unlabeled data. For consistency, the labeled datasets and evaluation metrics exactly the same in WILDS 1.0, which allows direct evalua- tions of the utility of unlabeled training data. The labeled and unlabeled data are disjoint, e.g., the unlabeled target data different from the labeled target data used for evaluation. Here, we briefly describe each dataset, why unlabeled data can be realistically obtained for the corresponding task, and how it might help. In Appendix A, we provide more information on each dataset, including data provenance and details on data processing. In general, all of the unlabeled datasets in WILDS 2.0 were processed in a similar way as their corresponding labeled datasets from WILDS 1.0.
Species classification across different camera traps. The task is to classify the animal species in camera trap image (Beery et al., 2020). We aim to generalize to new camera trap locations despite variations in illumination, background, and label frequencies (Beery et al., 2018). While hundreds of thousands of camera traps are active worldwide, only small subset of these traps have had images labeled, and the unlabeled data from the other camera traps capture diverse operating conditions that can be used to learn robust models. In this work, we add unlabeled images from 3,215 extra camera traps also in the WCS Camera Traps dataset (Beery et al., 2020). This expands the number of camera traps by x and the number of examples by 5x.
WWL.CCAM2220-WI..
CAMELYON WWWWIDS: Tumor identification across different hospitals. The task is to clas- sify image patches from lymph node sections as tumor or normal tissue. We seek to generalize to new hospitals, which can differ in their patient demographics and data acquisition protocols (Veta
4

---

Published as a conference paper at ICLR 2022
et al., 2016; AlBadawy et al., 2018; Komura & Ishikawa, 2018; Tellez et al., 2019). While ob- taining labeled data for histopathology applications requires pain-staking annotations from expert pathologists, hospitals typically accumulate unlabeled slide images during normal operation. These unlabeled images could be used to adapt to differences between hospitals (e.g., different staining protocols might lead to different color distributions). We provide unlabeled patches from train and test hospitals, which expands the total number of patches by 7.5x. Both the labeled and unlabeled data are adapted from the Camelyon17 dataset (Bandi et al., 2018).
-MOW-WILDE Land use classification across different regions and years. The task is to clas- sify the type of building or land usage in satellite image. Given training data from before 2013, we aim to generalize to satellite imagery taken after 2013, while maintaining high accuracy across all geographic regions. While labeling land use requires combining map data and expert annotations, unlabeled data is available in all locations in the world through constant streams of global satellite imagery. Prior work has shown that unlabeled satellite data can improve OOD accuracy in land- cover and cropland prediction (Xie et al., 202 well as aerial object and scene classification (Reed et al., 2021). We provide unlabeled satellite imagery across all regions from the train and test timeframes defined in WILDS, expanding the dataset by 3.5x. Both the labeled and unlabeled data are adapted from the FMoW dataset (Christie et al., 2018).
POVERTY MAP-WIDSS Poverty mapping across different countries. The task is to predict real-valued asset wealth index of the area in a satellite image. We consider generalizing across different countries. Like IMOW-WILDDS unlabeled satellite imagery is available globally, while labeled data is expensive to collect as it requires conducting nationally representative surveys in the field. Prior work on poverty prediction has used unlabeled data for entropy minimization (Jean al., 2018) and pre-training on auxiliary tasks such as nighttime light prediction (Xie et al., 2016; Jean al., 2016), but these studies do not study generalization to new countries. We provide unlabeled satellite imagery from both train and test countries. expanding the dataset by 14x. Both the labeled and unlabeled data are adapted from Yeh et al. (2020).
B.LOBA.WHAA--WLDD. Wheat head detection across different regions. The task is to localize wheat heads in overhead field images. We seek to generalize across image acquisition sessions, each of which represents a particular location, time, and sensor; these can differ in wheat genotype, wheat head appearance, growing conditions, background appearance, illumination, and acquisition protocols. Wheat field images contain many densely packed and overlapping instances, making labeling wheat heads in images costly, tedious and sensitive to the individual annotator. However, hundreds of agricultural research institutes around the world collect terabytes of unlabeled field images which could be used for training. We add unlabeled field images from train, test, and extra acquisition sessions, expanding the dataset by 10x. The labeled and unlabeled data are adapted from the Global Wheat Head Detection dataset and its underlying sources (David et al., 2020; 2021).
Molecular property prediction across different scaffolds. The task is to predict the biological activity of small molecules represented as molecular graphs (Wu et al., 2018; Hu et al., 2020b). We seek to generalize to molecules with new scaffold structures. Labels on biological activity are only available for small portion of molecules, as they require expensive lab experiments to obtain. However, unlabeled molecule structures are readily available in large-scale chemical databases such as PubChem (Bolton et al., 2008), and have been previously used for pre- training (Hu et al., 2020c) and semi-supervised learning (Sun et al., 2020). We provide 5 million unlabeled molecules from source and target scaffolds, which expands the number of molecules by 12.5x. The original labeled data was curated by MoleculeNet (Wu et al., 2018) from PubChem, and similarly extracted the unlabeled data from PubChem (Bolton et al., 2008)
ANVICOMMENTSS WILDS: Toxicity classification across demographic identities. The task is to classify whether a text comment is toxic or not. We consider the subpopulation shift setting, where the model must classify accurately across groups of comments mentioning different demographic identities. While labels require large-scale crowdsourcing annotations both comment toxicity, unlabeled article comments are widely available on the internet. We provide unannotated comments as unlabeled data, which expands the size of the dataset by 4.5x. Both the labeled and unlabeled data are adapted from Borkan et al. (2019).
A AMAZON-WILDS: Sentiment classification across different users. The task is to classify the star ratings of Amazon reviews. We seek to perform consistently well across new reviewers. While the
5

---

Published as a conference paper at ICLR 2022
labels (star ratings) always available for Amazon reviews in practice, unlabeled data is common source of leverage for sentiment classification more generally, with prior work in domain adaptation (Blitzer & Pereira, 2007; Glorot et al., 2011) and semi-supervised learning (Dasgupta & Ng, 2009; Li et al., 2011) I). We provide unlabeled reviews from test and extra reviewers, which expands the total number of reviews by 7.5x. Both the labeled and unlabeled data adapted from the Amazon review dataset by Ni et al. (2019).
5 ALGORIHTMM
For our evaluation, selected representative methods from the three categories described below. These methods exemplify current approaches to using unlabeled data to improve robustness, and they have been successful popular domain adaptation benchmarks like DomainNet (Peng al., 2019) and semi-supervised settings like improving ImageNet accuracy by leveraging unlabeled im- ages from the internet (Xie et al., 2020; Caron et al., 2020). For more details, see Appendix B.
Domain-invanee invariant methods. Domain invariian methods learn feature representations that are in- variant across different domains by penalizing differences between learned source and target rep- resentations (Long et al., 2015; Ganin et al., 2016; Sun & Saenko, 2016; Long et al., 2017; 2018; Saito et al., 2018; Zhang al., 2018; Xu al., 2019; Zhang et al., 2019b). We discuss these methods further in Appendix B.2. For experiments, evaluate two classical methods:
Neural Nenworks (DANN) (Ganin et al., 2016) penalize representations which an auxiliary classifier can easily discriminate between source and target examples.
Correlation Alignment (CORAL) (Sun et al., 2016; Sun & Saenko, 2016) penalizes differences between the means and covariances of the source and target feature distributions.
Self-training. Self-training methods "pseudo-label" unlabeled examples with the model's own pre- dictions and then train on them as if they were labeled examples. These methods often also use con- sistency regularization, which encourages the model to make consistent predictions views of unlabeled examples (Sohn et al., 2020; Xie et al., 2020; Berthelot et al., 2021). Self- training methods have recently been successfully applied unsupervised adaptation (Saito al., 2017; Berthelot et al., 2021; Zhang et al., 2021). We include three representative algorithms:
augmented
Pseudo-Label (Lee, 2013) dynamically generates pseudolabels and updates the model each batch.
FixMatch (Sohn et al., 2020) adds consistency regularization on top of the Pseudo-Labe algo- rithm. Specifically, it generates pseudolabels a weakly augmented view of the unlabeled data, and then minimizes the loss of the model's prediction on a strongly augmented view.
Noisy Student (Xie al., 2020) leverages weak and strong augmentations like FixMatch, but instead of dynamically generating pseudolabels for each batch, alternates between a few teacher phases, where generates pseudolabels, and student phases, where it trains to convergence on the (aseudo)labeled data.
Self-supervision. Self-supervised methods learn useful representations by training on unlabeled data via auxiliary proxy tasks. Common approaches include reconstruction tasks (Vincent et al., 2008; Erhan et al., 2010; Devlin et al., 2019; Gidaris et al., 2018; Lewis et al., 2020), and contrastive learning (He et al., 2020; Chen et al., 2020b; Caron et al., 2020; Radford et al., 2021b), and recent work has shown that self-supervised methods reduce dependence improve performance on domain adaptation tasks (Wang et al., 2021: Tsai et al., 2021; Mishra al., 2021). We these self-supervision methods for unsupervised adaptation by first pre-training models on the unlabeled data, and then finetuning them on the labeled source data (Shen et al., 2021). We evaluate popular self-supervised methods for vision and language:
spurious correlations and
SwAV (Caron et al., 2020) is a contrastive learning algorithm that maps representations to a set of clusters and then enforces similarity between cluster assignments.
Masked language modeling (MLM) (Devlin et al., 2019) randomly masks some the tokens from input text and trains the model to predict the missing tokens.

---

Published as a conference paper at ICLR 2022
6 EXPERIMENTS
To evaluate how well existing methods can leverage unlabeled data to be robust to in-the-will dis- tribution shifts, we benchmarked the methods above on all applicable WILDS 2.0 datasets.
6.1 SETUP
We used the default models, labeled training and test sets, and evaluation metrics from WILDS.
Unlabeled data. WILDS 2.0 contains multiple types of unlabeled data (from source, extra, valida- tion, and/or target domains). For simplicity, we ran experiments on single type of unlabeled data for each dataset. Where possible, used unlabeled target data to allow methods to directly adapt to the target distribution; for 1WILDCAM2020 WILDSWLLD and CIVILCOMMENTS -WINEN,SSWILDS, which do not have unlabeled target data, used the extra domains instead. All methods exactly the same sets of labeled and unlabeled training data (except ERM, which does not use unlabeled data).
Hyperparameters. We tuned each method on each dataset separately using random hyperparameter search. Following WILDS 1.0, we used the labeled out-of- -istriibbbtttni (OOD) validation set to select hyperparameters and for early stopping (Koh et al., 2021). This validation set is drawn from a different distribution than both the training and the OOD test set, so tuning on it does not leak information on the test distribution. We did not use the in-distribution (ID) validation set. For image classification and regression, we used both RandAAgment (Cubuk et al., 2020) and Cutout (DeVries & Taylor, 2017) as data augmentation for all methods. We did not use data augmentation for the remaining datasets. For some datasets, we also had ground truth labels for the "unlabeled" data, which we used to run fully-labeled ERM experiments. Overall, we ran 600+ experiments for 7,000 GPU hours on NVIDIA V100s. See Appendix B for discussion of which methods were applicable to which datasets; Appendix for augmentation details; Appendix for the fully-labeled experiments; Appendix D for further experimental details.
6.2 RESULTS
Table 2 shows mixed results WILDS: most methods do not improve standard empirical risk minimization (ERM) despite access to unlabeled data and careful hyperparameter tuning. In con- trast, these methods have been shown to perform well on prior unsupervised adaptation benchmarks; in Appendix E, we verify our implementations by showing that these methods (with the exception of CORAL) outperform ERM on the real + sketch shift DomainNet, standard unsupervised adaptation benchmark for object classification (Peng et al., 2019).
Image classification (IWILDCAM20200W.II))
CAMELYON17
and FMOW-WILDSS
Data augmentation improved OOD performance on all three image classification datasets. The gain was the most substantial on CAMELYON 1T-W.DS,, where vanilla ERM achieved 70.8% accuracy, while ERM with data augmentation achieved 82.0% accuracy.
On CAMELYON! 7- WILDS and FMOW-WILDS, where we had access to unlabeled target data, Noisy Student and SwAV pre-training consistently improved OOD performance and reduced variability across replicates. However, the other CORAL, DANN, Pseudo-Label, and FixMatch- underperformed ERM. This was especially surprising for FixMatch, which performed very well on DomainNet (Appendix E). Both FixMatch and Noisy Student use pseudo-labeling and consistency regulatization, but FixMatch dynamically computes pscudo-labels in each batch from the start of training, whereas Noisy Student first trains a teacher model to convergence on the labeled data and updates pseudolabels at much slower rate. As in Xie et al. (2020), this suggests that dynamically updating pseudo-labels might hurt generalization.
where we had access to 4x as many unlabeled images from extra domains (distinct camera traps) but not to any images from the target domains, none of the bench- marked methods improved OOD performance compared to ERM. This was surprising, as many of these methods were originally shown to work in semi semi-super settings. One difference could be that the labeled and unlabeled examples in WWLLDCAM2000 WILDS differ more significantly (as
On WILDCAMM020-WW
Thhe data augmentation involves color jitter, which simulates the difference in staining protocols hetween the source and target distributions CAMELYON 1I-WIL - (Koh al., 2021; Robey et al., 2021).
7

---

Published as a conference paper at ICLR 2022
WWLLDCAM2020-WILLL
FMOW-WILDS
(Unlabeled extra, macro F1)
(Unlabeled target, worst-region acc)
In-distribution Out-of-distribttion
In-distribution
Out-of-distribution
ERM (-data aug)
46.7 (0.6)
30.6 (1.1)
59.3 (0.7)
33.7 (1.5)
ERM
47.0 (1.4)
32.2 (1.2)
60.6 (0.6)
34.8 (1.5)
CORAL
40.5 (1.4)
27.9 (0.4)
58.9 (0.3)
34.1 (0.6)
DANN
48.5 (2.8)
31.9 (1.4)
57.9 (0.8)
34.6 (1.7)
Pscudo-Label
47.3 (0.4)
30.3 (0.4)
60.9 (0.5)
33.7 (0.2)
FixMatch
46.3 (0.5)
31.3
58.6 (2.4)
32.1 (2.0)
Noisy Student
47.5 (0.9)
32.1 (0.7)
61.3 (0.4)
37.8 (0.6)
SwAV
47.3 (1.4)
29.0 (2.0)
61.8 (1.0)
36.3 (1.0)
ERM (fully-labeled)
54.6 (1.5)
44.0 (2.3)
65.4 (0.4)
58.7 (1.4)
CAMELYONI7-WILDD
POVERTYMAP-WILDS
(Unlabeled target, avg acc)
(Unlabeled target, worst U/R corr)
In-distribution
Out-of-distribution
In-distribution
Out-of-distribttion
ERM (-data aug)
85.8 (1.9)
70.8 (7.2)
0.65 (0.03)
0.50 (0.07)
ERM
90.6 (1.2)
82.0 (7.4)
0.66 (0.04)
0.49 (0.06)
CORAL
90.4 (0.9)
77.9 (6.6)
0.54 (0.10)
0.36 (0.08)
DANN
86.9 (2.2)
68.4 (9.2)
0.50 (0.07)
0.33 (0.10)
Pscudo-Label
91.3 (1.3)
67.7 (8.2)
FixMatch
91.3 (1.1)
71.0 (4.9)
0.54 (0.11)
0.30 (0.11)
Noisy Suudent
93.2 (0.5)
86.7 (1.7)
0.61 (0.07)
0.42 10)
SWAV
92.3 (0.4)
91.4 (2.0)
0.60 (0.13)
0.45 (0.05)
GLOBBALWWEAT-WILDS
OGB-MoLPCBA (Unlabeled target, avg AP)
(Unlabeled target, avg domain acc) In-distribution Out-of-distribution
In-distribution
Out-of-distrriution
ERM
77.8 (0.1)
50.5 (1.7)
28.3 (0.1)
CORAL
26.6 (0.2)
DANN
20.4 (0.8)
Pscudo-Label
75.2 (1.2)
42.7 (4.8)
19.7 (0.1)
Noisy Student
78.8 (0.5)
49.3 (3.7)
27.5 (0.1)
COMILCCSMENTS WILDS (Unlabeled extra, worst-group acc) In-distribution Out-of-distribttion
AMAZON-WIIE WWIDDS
(Unlabeled target, 10th percentile acc)
In-distribution
Out-of-distrrbution
ERM
89.8 (0.8)
66.6 (1.6)
72.0 (0.1)
54.2 (0.8)
CORAL DANN
71.7 (0.1)
53.3 (0.0)
71.7 (0.1)
53.3 (0.0)
Pscudo-Label
90.3 (0.5)
66.9 (2.6)
71.6 (0.1)
52.3 (1.1)
Masked LM
89.4 (1.2)
65.7 (2.3)
71.9 (0.4)
53.9 (0.7)
ERM (fully-labeled)
89.9 (0.1)
69.4 (0.6)
73.6 (0.1)
56.4 (0.8)
Table 2: The in-distribution (ID) and out-of-distribution (OOD) performance of each method on each applicable dataset. Following WILDS 1.0, we ran 3-10 replicates (random seeds) for each cell, depending on the dataset. We report the standard deviation across replicates in parentheses; the standard error (of the mean) is lower by the square root of the number of replicates. Fuble-labeled experiments use ground truth labels on the "unlabeled" data. We bold the highest non-fully-labes labeled OOD performance numbers as well others where the standard error is within range. Below each dataset name, we report the type of unlabeled data and metric used.
they originate from different camera traps) than in the original FixMatch paper (Sohn et al., 2020), which used i.i.d. labeled and unlabeled data, or the Noisy Student paper (Xie et al., 2020), which used ImageNet labeled data (Russakovsky et al., 2015) and JFT unlabeled data (Hinton et al., 2015).
Fully-labeled ERM models that used ground truth labels for the "unlabeled" data were available for FMODSWILDS and CILLLCAM2020-WW WILDS. They significantly outperformed other methods, suggesting room for improvement in how we leverage the unlabeled data.
Image regression (POVERTYMAP-WIL)) SHEERTYMAP-WILLD. which differs from the above image datasets that a regression task and involves multi sueti-spectral satellite images (with 7 channels); both of these aspects relatively unstudied compared to standard RGB image classification. All applicable methods underperformed standard FRM, despite having access unlabeled data from the target domains (countries).
Data augmentation had no effect performance on
8

---

Published as a conference paper at ICLR 2022
Image detection (GLOBALWHEAT-WILDDS We did not apply data augmentation here, standard augmentation changes the labels (e.g., cropping the image might remove bounding boxes) and would violate the assumption that labels invariant under augmentations, which contrastive and consis- tency regularization methods like SWAV, Noisy Student, and Fix.Match rely on. Accordingly, we did not evaluate Fixc and SWAV, and we modified Noisy Student remove data augmentation noise. All applicable methods underperformed ERM.
Molecule classification (OGB-MOLPCBA)) We did not apply data augmentation techniques to OGB-MOL MCBA as they are not well-developed for molecular graphs. All methods underper- formed ERM. We did not report ID results as this dataset has no separate ID test set.
AMAZON-WILDS). Likewise, we did not apply data augmentation to the text datasets. On both datasets, other methods performed similarly to ERM (with daas-nna for CrVI.COMMENTS-VEE WILDS). Continued masked LM pre-training on the unlabeled data did not improve target performance, unlike in prior work (Gururangan et al., 2020); this might be because the BERT pre-training corpus (Devlin et al., 2019; Hendrycks et al., 2020) is more similar to the online comments in CWVI.COMMENTS-WLLL in AMAZON-WILDS than to the biomedical/CS text studied in Gururangan et al. (2020). Also, CIVILCOMMENTS WILDS and AMAZON-
Text classification (CIviL (CIViL COMMENTS-
and product reviews
measure subpopulation performance (on minority demographies and on the tail subpopulation, respectively), whereas prior work adapted models to new areas of the input space (e.g., from news to biomedical articles). Fully-labeled ERM models showed modest gains compared to SMOW-WILDD and WWI.DCAM2020-W1I
As our evalua- tions on these text datasets focus on subpopulations performance, these results are consistent with prior observations that ERM models can have poor subpopulation performance even with large la- beled training sets (Sagawa et al., 2020), necessitating other approaches to subpopulation shifts.
DISCUSSION
We conclude by discussing several takeaways and promising directions for future work.
The role of data augmentation. Many unsupervised adaptation methods rely strongly on data aug- mentation for consistency regularization or contrastive learning. This reliance on data augmenta- tion techniques which are largely image-specific restricts their generality, as they do not readily generalize other modalities (or even other types of images besides photos). Developing data aug- mentation techniques that can work well in other applications and modalities could be crucial for expanding the applicability of these methods (Verma et al., 2021).
Hyperparameter tuning. Unsupervised adaptation methods have even more hyperparameters than standard supervised methods, and consistent with prior work, we found that these hyperparameters can significantly affect OOD performance (Saito et al., 2021). Moreover, unlike in standard i.i.d. set- tings, we do not have labeled target data that we can use for hyperparameter selection. Improved methods for hyperparameter tuning could significantly improve OOD performance. Such methods might make use of the unlabeled target data, or even the combination of labeled and unlabeled OOD validation data, which is provided for most datasets in WILDS 2.0.
Pre-training on broader unlabeled data. Pre-training on huge amounts of unlabeled data improves robustness to distribution shifts some settings (Bommasani et al., 2021). The unlabeled data need not be related to the task: e.g., CLIP was pre-trained on text-image pairs from the internet but tested on tasks including histopathology and satellite image classification (Radford al., 2021a). Existing techniques for this type of broad pre-training appear insufficient for WILDS: many of our models were initialized with
weights or derivatives of BERT, but do not generalize well OOD, While we focused on providing curated unlabeled data that is closely tailored to the task, could be fruitful to use both broad and curated unlabeled data.
Leveraging domain annotations and task-specific structure. OOD robustness is ill-posed in gen- eral, as models cannot be robust to arbitrary distribution shifts. Beyond unlabeled data, WILDS also has domain annotations and other structured metadata for both labeled and unlabeled data (e.g.. in (WILDCAM2020- WILDS, we know which images were taken from which cameras). Exploit- ing this type of fine- fine-grained domain structure for unsupervised adaptation- e.g., through multi- source/multi-tarre target domain adaptation methods (Zhao et al., 2018; Peng et al., 2019) could be a promising avenue for learning models that are more robust to the domain shifts in WILDS.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
All WILDS datasets are curated and adapted from public data sources, with licenses that allow for public release. The datasets are all anonymized.
The distribution shifts in several of the WILDS datasets deal with issues of discrimination and bias that arise in real-world applications. For example, CIVILCOMMENTS WILDS studies disparate model performance across online comments that mention different demographic groups, while FMOW WILDS and POVERTY AP-AAP-WII AP-WILDS study countries and regions where labeled satellite data is less readily available. As our results suggest, standard models trained on these datasets will not perform well on those subpopulations, and their learned representations might also be biased in undesirable ways (Bolukbasi et al., 2016; Caliskan al., 2017; Garg et al., 2018; Tan & Celis, 2019; Steed & Caliskan, 2021). We also encourage caution in interpreting positive results on these datasets,
evaluation metries might not encompass all relevant facets of discrimination and bias: e.g., the "ground truth" toxicity annotations in CIVILCOMMENTS-VVE WILLSS can themselves be biased, and the particular choice of regions in FMOW-WILDS might obscure lower model perfor- mance in sub-regions.
For FMoW- -WILDS and POVERTYMAP- considered. In FMOW-WILDS, the image resolution is lower than that of other public satellite data (e.g., from Google Maps), and in POVERTYMAP-WIE privacy. For a deeper discussion of the ethics of remote sensing in the context of humanitarian aid and development, we refer readers to the UNICEF report by Berman al. (2018).
surveillance and privacy issues also need to be
the location metadata is noised protect
REPRODUCIBILITY STATEMENT
All WILDS datasets are publicly available at https://wi stanford.edu. together with code and scripts replicate all of the experiments in this paper. We also provide all trained model checkpoints and results, together with the exact hyperparameters used.
In our appendices, we provide
details the datasets and experiments:
In Appendix A, we describe each of the updated datasets in WILDS 2.0 and their sources unlabeled data as well as what data processing steps were taken.
Appendix B, we describe the implementations of each our benchmarked methods in detail. In particular, either for consistency with other methods or with prior implementations of these methods.
discuss any changes made to their original implementations,
In Appendix C, we describe details of the data augmentations (if any) that we used across each dataset.
In Appendix D, describe experimental protocol, including the hyperparameter lection procedure and hyperparameter grids for all of the methods and datasets.
In Appendix E, we describe the details of our experiments on DomainNet.
In Appendix F, we describe the details of our fully-labeled ERM experiments.
Finally, in Appendix G, we include an illustrative code snippet of how to use the data loaders in the WILDS library.
AUTHOR CONTRIBUTIONS
The project was initiated by Shiori Sagawa, Pang Wei Koh, and Percy Liang. Shiori Sagawa and Pang Wei Koh led the project and coordinated the activities below. Tony Lee developed the exper- imental infrastructure and ran the experiments. Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, and Michihiro Yasunaga designed the evaluation framework and implemented the algorithms. The unlabeled data loaders and corresponding dataset writeups were added by:
AMADON-WI.DS: Tony Lee
2AMELYONN1--WIDDS Tony Lee
10

---

Published as a conference paper at ICLR 2022
CIVILCOMMENTS- WILDSS Irena Gao
FMOW-WILDS: Sang Michael Xie
WW1.DCAW2020-WII.S. Henrik Marklund and Sara Beery
OGB-MOL,PCBA: Weihua Hu
POVERTYMAP-WII.D: Sang Michael Xie
GLOBALW Etienne David, Ian Stavness, and Wei Guo.
Tony Lee and Henrik Marklund set up the website and leaderboards. Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn and Percy Liang provided advice on the overall project direction and experimental design and analysis throughout. Shiori Sagawa, Pang Wei Koh, and Irena Gao drafted the paper; all authors contributed towards writing the final paper.
ACKNOWLEDGEMENTS
We would like to thank Ashwin Ramaswami, Berton Earnshaw, Bowen Liu, Hongseok Namkoong, Junguang Jiang, Ludwig Schmidt, Robbie Jones, Robin Jia, Ruijia Xu, and Yabin Zhang for their helpful advice.
The design of the WILDS benchmark was inspired by the Open Graph Benchmark (Hu et al., 2020b), and we are grateful to the Open Graph Benchmark team for their advice and help setting up our benchmark.
This project was funded by an Open Philanthropy Project Award and NSF Award Grant No. 1805310. Shiori Sagawa was supported by the Herbert Kunzel Stanford Graduate Fellowship and the Apple Scholars in AI/ML PhD fellowship. Sang Michael Xie was supported by a NDSEG Graduate Fellowship. Ananya Kumar was supported by the Rambus Corporation Stanford Graduate Fellowship. Weihua Hu was supported by the Funai Overseas Scholarship and the Masason Foun- dation Fellowship. Michihiro Yasunaga was supported by the Microsoft Research PhD Fellowship. Henrik Marklund was supported by the Dr. Tech. Marcus Wallenberg Foundation for Education in International Industrial Entrepreneurship. CIFAR, and Google. Sara Beery supported by NSF Graduate Research Fellowship and is a PIMCO Fellow in Data Science. Jure Leskovec is Chan Zuckerberg Biohub investigator. Chelsea Finn is CIFAR Fellow the Learning in Machines and Brains Program.
We also gratefully acknowledge the support of DARPA under Nos. N66001 (MCS); ARO under Nos. 191-1-16-110044 (MURI), W91INF-16111 16-1-0171 (DURIP); NSF under Nos. OAC- 1835598 (CINES), 1934579 (HDR), CCF-191 (Expeditions), IIS ISS-2030477 (RAPID)); Stanford Data Science Initiative, Wu Tssi Neurosciences Institute, Chan Zuckerberg Biohub, Ama- zon, JPMorgan Chase, Docomo, Hitachi, JD.com, KDDI, NVIDIA, Dell, Toshiba, and UnitedHealth Group.
REFERENCES
Abubakar Abid, Maheen Farooqi, and James Zou. Persistent anti-muslim bias in large language models. arXiv preprint arXiv:2101.05783, 2021.
Jorge A Ahumada, Eric Fegraus, Tanya Birch, Nicole Flores, Roland Kays, Timothy G OBrien, Jonathan Palmer, Stephanie Schuttler, Jennifer Zhao, Walter Jetz, et al. Wildlife insights: A platform to maximize the potential of camera trap and other passive sensor wildlife data for the planet. Environmental Conservation, 47(1):1-6, 2020.
Saad Ullah Akram, Talha Qaiser, Simon Graham, Juho Kannala, Janne Heikkilä, and Nasir Rajpoot. Leveraging unlabeled whole-slide-imager for mitosis detection. Computational Pathology and Ophthalmic Medical Image Analysis, 1:69-77, 2018.
EA AlBadawy, A Saha, and MA Mazurowski. Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing. Med Phys., 45, 2018.
11

---

Published as a conference paper at ICLR 2022
Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, et al. Big self-supervised models advance medical image classification. arXiv preprint arXiv:2101.05224, 2021.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the CAMELYON17 challenge. IEEE Transactions on Medical Imaging, 38(2)5550-560, 2018.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In European Con- ference on Computer Vision (ECCV). pp. 456 473, 2018
Sara Beery, Dan Morris, and Siyu Yang. Efficient pipeline for camera trap image review. arXiv preprint arXiv:1907.06772 2019.
Sara Beery, Elijah Cole, and Arvi Gjoka. The iwildcam 2020 competition dataset. arXiv preprint arXiv:2004.10344 10340, 2020.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine Learning, 1911)1551-175 2010.
Gabrielle Berman, Sara de la Rosa, and Tanya Accone. Ethical considerations when using geospatial technologies for evidence generation. Innocenti Discussion Paper, UNICEF Office of Research, 2018.
David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. Adamatch: A unified approach to semi-supervised learning and domain adaptation. arXiv preprint vrXiv:2106.04732 2021.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://wwk.
wandb. com/ Software available from wandb.com.
John Blitzer and Fernando Pereira. Domain adaptation of natural language processing systems. University of Pennsylvania, 2007.
John Blitzer, Mark Dredze, and Fernando Pereira. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics, pp. 440-447, 2007.
Evan E Bolton, Yanli Wang, Paul A Thiessen, and Stephen Bryant. Pubchem: integrated plat-
form of small molecules and biological activities. In Annual reports in computational chemistry, volume 4, pp. 217-241. Elsevier, 2008
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4359, 2016.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji. Annie Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Ste- fano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto. Pe- ter Henderson, John Hewitt, Daniel E. Ho. Jenny Hong. Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovee, Isabelle Levent, Xiang Lisa Li, Xuechen Li. Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadim- itriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher R, Dorsa
12

---

Published as a conference paper at ICLR 2022
Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Ro- han Taori, Armin W. Thomas, Florian Tramr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang. Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. arXiv preprint arXiv:2/08.0720 2021.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In World Wide Web (WWW), pp. 491-500, 2019.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Maxwell Burnette, Rob Kooper, J. D. Maloney, Gareth S. Rohde, Jeffrey A. Terstriep. Craig Willis, Noah Fahlgren, Todd Mockler. Maria Newcomb, Vasit Sagan, Pedro Andrade-Sanchel Nadia Shakoor, Paheding Sidike, Rick Ward, and David LeBauer. Terra-ref data processing infrastruc- ture. In Proceedings of the Practice and Experience on Advanced Research Computing, PEARC 18, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 978 145036446..
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):1833-8888 2017.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Un- supervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 9912-9924, 2020.
llias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androut- sopoulos. LEGAL-BERT: preparing the muppets for court". In Empirical Methods in Natural Language Processing (EMNIP), pp. 2898-2904, 2020.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML., pp. 16997-607, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020b.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Computer Vision and Pattern Recognition (CVPR), 2018.
Ozan Ciga, Anne L Martel, and Tony Xu. Self supervised contrastive learning for digital histopathol- ogy. arXiv preprint arXiv:201 13971, 2020.
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski. Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. arXiv preprint arXiv:2003.05002 2020.
Elijah Cole, Xuan Yang. Kimberly Wilber, Oisin Mac Aodha, and Serge Belongie. When does
contrastive visual representation learning work? arXiv preprint arXiv:2105.05833 2021.
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems (NeurIPS), pp. 7059-7069, 2019.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Empirical Methods in Natural Language Processing (EMNLP), pp. 2475-2485, 2018.
13

---

Published as a conference paper at ICLR 2022
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Computer Vision and Pattern Recognition (CVPR), pp. 702-703, 2020.
Sajib Dasgupta and Vincent Ng. Mine the easy, classify the hard: semi-supervised approach to automatic sentiment classification. Conference on Natural Language Processing (KONVENS), pp. 701-709, 2009.
Etienne David, Simon Madec, Pouria Sadeghi-Tchran, Helge Aasen, Bangyou Zheng, Shouyang Liu, Norbert Kirchgessner, Goro Ishikawa, Koichi Nagasawa, Minhajul Badhon, Curtis Poz- niak, Benoit de Solan, Andreas Hund, Scott C. Chapman, Frederic Baret, Ian Stavness, and Wei Guo. Global wheat head detection (gwhd) dataset: a large and diverse dataset high-resolution rgb-labelled images to develop and benchmark wheat head detection methods. Plant Phenomics, 2020, 2020.
Etienne David, Mario Serouart, Daniel Smith, Simon Madec, Kaaviya Velumani, Shouyang Liu, Xu Wang, Francisco Pinto, Shahameh Shafiee, Izzat S. A. Tahir, Hisashi Tsujimoto, Shuhei Nasuda, Bangyou Zheng, Norbert Kirchgessner, Helge Aasen, Andreas Hund, Pouria Sadhegi- Tehran, Koichi Nagasawa, Goro Ishikawa, Sébastien Dandrifosse, Alexis Carlier, Benjamin Du- mont, Benoit Mercatoris, Byron Evers, Ken Kuroki, Haozhou Wang, Masanori Ishii, Minhajul A. Badhon, Curtis Pozniak, David Shaner LeBauer, Morten Lillemo, Jesse Poland, Scott Chapman, Benoit de Solan, Frédéric Baret, Ian Stavness, and Wei Guo. Global wheat head detection 2021: An improved dataset for benchmarking wheat head detection methods. Plant Phenomics, 2021, 2021.
Olivier Dehaene, Axel Camara, Olivier Moindrot, Axel de Lavergne, and Pierre Courtiol, Self- supervision closes the gap between weak and strong supervision in histology. arXiv preprint TrXiv:2012.035883 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Association for Computational Lingnis- tics (ACL), pp. 4171-4186, 2019.
Terrance Dries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: 1708.04552 2017.
Josip Djolonga, Jessica Yung. Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander A'mour, Dan Moldovan, et On robustness and transferability of convolutional neural networks. arXiv preprint arXiv:2007.08558, 2020.
Abhimanyu Dubey, Vignesh Ramanathan, Alex Pentland, and Dhruv Mahajan. Adaptive methods for real-world domain generalization. In Computer Vision and Pattern Recognition (CVPR), 2021.
Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre- training help deep learning? In Artificial Intelligence and Statistics (AISTATS), pp. 201-208, 2010.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario March, and Victor Lempitsky. Domain-adversart training of neural networks. Journal of Machine Learning Research (JMLR), 17, 2016.
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. Word embeddings quantify 100 years of gender and ethnic stereotypes. Science, 115, 2018.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009. 1462, 2020.
14

---

Published as a conference paper at ICLR 2022
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations (ICLR), 2018.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Artificial Intelligence and Statistics (AISTATS), pp. 315-323, 2011.
Yves Grandvalet and Yoshua Bengio. Entropy regularization. In Supi--Suervised Learning, 2005.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. arXiv preprint arXiv:2007. 15779, 2020.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't 't stop pretraining: adapt language models to domains and tasks. arXiv preprint arXiv:2004. 10964, 2020.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wey Wei Chang. Realm: Retrieval- augmented language model pre-training. In icml, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Computer Vision and Pattern Recognition (CVPR), 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for un- supervised visual representation learning. In Computer Vision and Pattern Recognition (CVPR), 2020.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of-distribution robustness. arXiv preprint arXiv:2004.06100 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, and Trevor Darrell. Cycada: Cycle consistent adversarial domain adaptation. In International Conference on Machine Learning (ICML), 2018.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: massively multilingual multi-task benchmark for evaluating cross-lingual generaliza- tion. arXiv preprint arXiv:2003. 1080, 2020a.
Weihua Hu, Mauthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open Graph Benchmark: Datasets for machine learning graphs. In Ad- vances Neural Information Processing Systems (NeurIPS), 2020b.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. International Conference on Learning Representations (ICLR), 2020c.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
Jonathan J. Hull. A database for handwritten text recognition research. JEEE Transactions on pastern analysis and machine intelligence, 16(5):550-554 1994.
15

---

Published as a conference paper at ICLR 2022
Neal Jean, Marshall Burke, Michael Xie, W. Matthew Davis, David B. Lobell, and Stefano Ermon. Combining satellite imagery and machine learning to predict poverty, Science, 353, 2016.
Neal Jean, Sang Michael Xie, and Stefano Ermon. Semi-supervised deep kernel learning: Regres- sion with unlabeled data by minimizing predictive variance. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Junguang Jiang, Baixu Chen, Bo Fu, and Mingsheng Long. Transfer learning library. https: 2020.
//github. com/
Amita Kamath, Robin Jia, and Percy Liang. Selective question answering under domain shift. In Association for Computational Linguistics (ACL), 2020.
Benjamin Kellenberger. Diego Marcos, Sylvain Lobry, and Devis Tuia. Half percent of labels is enough: Efficient animal detection in uav imagery using deep cnns and active learning. IEEE Transactions Geoscience and Remote Sensing, 57(12))5524998 2019.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu. Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML), 2021.
Daisuke Komura and Shumpei Ishikawa. Machine learning methods for histopathological image analysis. Computational and structural biotechnology journal, 16:34-42, 2018.
Navid Alemi Koohbanani, Balagopal Unnikrishnan, Syed Ali Khurram, Pavitra Krishnaswamy, and Nasir Rajpoot. Self-path: Self-supervision for classification of pathology images with limited annotations. IEEE Transactions on Medical Imaging, 1, 2021.
Greg Landrum et al. Rdkit: Open-source cheminformatics, 2006.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( 26221222278223 1998.
Dong-Hyun Lec. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop Challenges in Representation Learning, 2013.
Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable self-supervised learning. arXiv preprint arXiv:2008. 01064 2020a.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jae- woo Kang. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 1634 2020b.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and comprehension. In Association for Computa- tional Linguistics (ACL), 2020.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Shoushan Li, Zhongqing Wang, Guodong Zhou, and Sophia Yat Mei Lec. Semi-supervised learning for imbalanced sentiment classification. In International Joint Conference on Artificial Intelli- gence (IJCAI), 2011.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International conference machine learning, pp. 2208-22011111 2017.
16

---

Published as a conference paper at ICLR 2022
Mingsheng Long, Zhangjie Cao. Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Ming Lu, Richard J Chen, Jingwen Wang. Debora Dillon, and Faisal Mahmood. Semi-supervised histology classification using deep multiple instance learning and contrastive predictive coding. arXiv preprint arXiv:1900. 1000005,, 2019.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadch. Domain adaptation with multiple sources. In Advances in Neural Information Processing Systems (NeurlPS), pp. 1041-1048, 2009.
John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribution shift on question answering models. arXiv preprint arXiv:2004.14444, 2020.
John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy the line: the strong correlation between outtof-distributtin and in-distribution generalization. In Interaational Conference on Machine Learning (ICMI.), 2021.
Samarth Mishra, Kate Saenko, and Venkatesh Saligrama. Surprisingly simple supi-super do- main adaptation with pretraining and consistency. arXiv preprint drXiv:2101.1227,, 2021.
Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456, 2020.
Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Tajudeen Kola- wole, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddee Hassan Muhammad, Sa- lomon Kabongo, Salomey Osei, Sackey Freshia, Rubungo Andre Niyongaho, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Meressa, Mofe Adeyemi, Masabata Mokgesi-Selinga, Lawrence Okegbemi, Laura Jane Martinus, Kolawole Tajudeen, Kevin Degila, Kelechi Ogueji, Kathleen Siminyu, Julia Kreutzer, Jason Webster, Jamiil Toure Ali, Jade Abbott, Iroro Orife, Ignatius Ezeani, Idris Abdulkabir Dangana, Herman Kamper, Hady Elsahar, Goodness Duru, Ghollah Kioko, Espoir Murhabazi, Elan van Biljon, Daniel Whitenack, Christopher Onyefuluchi, Chris Emezue, Bonaventure Dossou, Blessing Sibanda, Blessing Itoro Bassey, Ayodele Olabiyi, Arshath Ramkilowan, Alp ktem, Adewale Akinfaderin, and Abdallah Bashir. Participatory re- search for low-resoureed machine translation: A case study in African languages. Findings of Empirical Methods in Natural Language Processing (Findings of EMNLP). 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Jianmo Ni, Jiacheng Li, and Julian McAuley, Justifying recommendations using distantly-labelee re- views and fine- greinea aspects. In Empirical Methods in Natural Language Processing (EMNLP), pp. 188 197, 2019.
Mohammad Sadegh Norouzzadeh, Dan Morris, Sara Beery, Neel Joshi, Nebojsa Jojic, and Jeff Clune. A deep active learning system for species identification and counting in camera trap images. Methods in Ecology and Evolution, 12(1):150-161,
2021. 2021.
Jill Nugent. inaturalist. Science Scope, 41(7): 2018.
Yonatan Oren, Shiori Sagawa, Tatsunori Hashimoto, and Percy Liang. Distributionally robust lan- guage modeling. In Empirical Methods in Natural Language Processing (EMNLP). 2019.
Omiros Pantazis, Gabriel Brostow, Kate Jones, and Oisin Mac Aodha. Focus on the positives: Self- supervised learning for biodiversity monitoring. arXiv preprint arXiv:2108. 064335 2021.
Mohammad Peikari, Sherine Salama, Sharon Nofech-Mozes and Anne L Martel. A cluster-then- label semi superviie learning approach for pathology image classification. Scientific reports, (1):1-13, 2018.
Xingchao Peng, Ben Usman, Neela Kaushik, Dequan Wang. Judy Hoffman, and Kate Saenko. Visda: A synthetic-to-real benchmark for visual domain adaptation. In Computer Vision and Pallern Recognition (CVPR), pp. 2021-2026, 2018
17

---

Published as a conference paper at ICLR 2022
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In International Conference on Computer Vision (ICCV), 2019.
Joaquin Seliionero-Candeer Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence. Dataset shift in machine learning. The MIT Press, 2009.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning (ICML), volume 139, Pp. 8748-8763. 2021a.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103. 0..000000 2021b.
Colorado J. Reed, Xiangyu Yue, Ani Nrusimha, Sayna Ebrahimi, Vivek Vijaykumar, Richard Mao, Bo Li, Shanghang Zhang, Devin Guillory, Sean Metzger, Kurt Keutzer, and Trevor Darrell. Self- supervised pretraining improves self-supervised pretraining, arXiv, 2021.
Jian Ren, Ilker Hacihaliloglu, Eric A Singer, David Foran, and Xin Qi. Adversarial domain adapta- tion for classification of prostate histopathology whole-slide images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 201-209, 2018.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Patlern Analysis and Machine Intelligence (PAMI), 39:1 137--1194, 2015.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European conference on computer vision, pp. 102-118, 2016.
Tim Robertson, Markus Döring, Robert Guralnick, David Bloom, John Wieczorek, Kyle Braak, Javier Otegui, Laura Russell, and Peter Desmet. The gbif integrated publishing toolkit: facilitating the efficient publishing of biodiversity data on the internet. PloS one, (8) 102623, 2014.
Alexander Robey, George Pappas, and Hamed Hassani. Model-based domain generalization. arXiv preprint 1xXXi:22022.143366 2021.
David Rogers and Mathew Hahn. Extended-coomeccivtty fingerprints. Journal of Chemical Infor- mation and Modeling, 50(5):742-754, 2010. doi: 10.1021/c11000500.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer large-scale molecular data. arXiv preprint arXiv:2007.02835. 2020.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3234- 3243, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-2 1-22, 2015.
Kate Saenko, Brian Kulis, Mario Fritz,, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pp. 213-226, 2010.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. In International Conference on Learning Representations (ICLR), 2020.
Kuniaki Saito. Yoshitaka Ushiku, and Tatsuya Harada. Asymmetric tri tri-training for unsupervised domain adaptation. International Conference on Machine Learning (ICML), pp. 2988-2997, 2017.
18

---

Published as a conference paper at ICLR 2022
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier dis- crepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 3723-3733332 2018.
Kuniaki Saito, Donghyun Kim, Piotr Teterwak, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. arXiv preprint arXiv:2 10860, 2021.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprini arXiv:1910001108, 2019.
Shayne Shaw, Maciej Pajak, Aneta Lisowska, Sotirios A Tsaftaris, and Alison Q O'Nei.. Teacher- student chain for efficient semi-supervised histology image classification. arXiv preprint arXiv:2003.08797, 2020.
Kendrick Shen, Robbie Matthew Jones, Ananya Kumar, Sang Michael Xie, and Percy Liang. How does contrastive pre-training connect disparate domains? In NeurlPS Workshop on Distribution Shifts, 2021.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv, 2020.
Ryan Steed and Aylin Caliskan. Image representations learned with unsupervised pre-training contain human-like biases. In ACM Conference on Fairness, Accountability, and Transparency (FAceT), pp. 701-713, 2021.
Jong-Chyi Su and Subhransu Maji. The semi-supervised inaturalist-aves challenge at fgvc7 work- shop. arXiv preprint arXiv:2103.cm 06937, 2021.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European Conference on Computer Vision (ECCV), 2016.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Association for the Advancement of Artificial Intelligence (AAAI), 2016.
Yun-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. In Interna- tional Conference on Learning Representations (ICLR), 2020.
Yi Chern Tan and Elisa Celis. Assessing social and intersectional biases in contextualized word representations. arXiv preprint arXiv:1911.014580 2019.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. arXiv preprint arXiv:2007. 00644.. 2020.
David Tellez, Geert Litjens, Péter Bándi, Wouter Bulten, John-Melle Bokhorst, Francesco Ciompi, and Jeroen van der Laak. Quantifying the effects of data augmentation and stain color normaliza- tion in convolutional neural networks for computational pathology. Medical Image Analysis, 58, 2019.
Yao-Hung Hubert Tsai, Martin Q Ma, Han Zhao, Kun Zhang, Louis Philipp Morency, and Rus- lan Salakhutdinov. Conditional contrastive learning: Removing undesirable information in self- supervised representations. arXiv preprint Stt.::20060286666 2021.
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computa- tional Linguistics (TACL), 8:621 2020.
Grant Van Horn Elijah Cole. Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 12884 12884-12893, 2021.
19

---

Published as a conference paper at ICLR 2022
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Computer Vision and Pattern Recogni- fion (CVPR), pp. 5018 --218, 2017.
Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoc Le. Towards domain aggoostic contrastive learning. In International Conference Machine Learning (ICML), 2021.
Mitko Veta, Paul J Van Diest, Mehdi Jiwa, Shaimaa Al-Janabi, and Josien PW Pluim. Mitosis counting in breast cancer: Object-level interobserver agreement and comparison automatic method. PloS one, 11(8), 2016.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning (ICML), 2008
Rui Wang, Zuxuan Wu, Zejia Weng, Jingjing Chen, Guo-Jun Qi, and Yu-Gang Jiang. Cross-domain contrastive learning for unsupervised domain adaptation. arXiv, 2021.
Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in down- stream tasks? an analysis of head and prompt tuning. arXiv preprint arXiv:2106.09226, 2021.
Ben G Weinstein, Sergio Marconi, Stephanie Bohlman, Alina Zare, and Ethan White. Individual tree-crown detection in rgb imagery using semi-supervised deep learning neural networks. Remote Sensing, 11(11))309, 2019.
Ben Weinstein, Lindsey Gardner, Vienna Saccomanno, Ashley Steinkraus, Andrew Ortega, Kris- ten Brush, Glenda Yenni, Ann E McKellar, Rowan Converse, Christopher Lippitt, et al. A general deep learning model for bird detection in high resolution airborne imagery. bioRxiv, 2021.
John N Weinstein, Eric A Collisson, Gordon B Mills, Kenna R Mills Shaw, Brad A Ozenberger, Kyle Ellrott, Ilya Shmulevich, Chris Sander, Joshua M Stuart, Cancer Genome Atlas Research Network, et al. The cancer genome atlas pan-cancer analysis project. Nature genetics, 45(10), 2013.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learn- ing. Chemical science, 9(2):513-530, 2018.
Michael Xie, Neal Jean, Marshall Burke, David Lobell, and Stefano Ermon. Transfer learning from deep features for remote sensing and poverty mapping. In Association for the Advancement of Artificial Intelligence (AAAI), 2016
Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. arXiv, 2020.
Sang Michael Xie, Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma, and Percy Liang. In-N-out: Pre-training and self-training using auxiliary information for out-of-distribution robust- ness. In International Conference Learning Representations (ICLR), 2021a.
Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: A unified review. arXiv preprint arXiv:2102.10757, 2021b.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference Learning Representations (ICLR), 2018.
Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adap- tive feature norm approach for unsupervised domain adaptation. In International Conference Computer Vision (ICCV), pp. 1426-1435, 2019.
Christopher Yeh, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Ste- fano Ermon, and Marshall Burke. Using publicly available satellite imagery and deep learning to understand economic well-being in africa. Nature Communications, 11, 2020.
20

---

Published as a conference paper at ICLR 2022
Weichen Zhang, Wanli Ouyang, Wen Li, and Dong Xu. Collaborative and adversarial network for unsupervised domain adaptation. In Computer Vision and Pattern Recognition (CVPR), pp- 3801-3809, 2018.
Yabin Zhang, Haojian Zhang, Bin Deng, Shuai Li, Kui Jia, and Lei Zhang. Semi-supervised models are strong unsupervised domain adaptation learners. arXiv preprinl arXiv:2106.00417, 2021.
Yifan Zhang, Hanbo Chen, Ying Wei, Peilin Zhao, Jiezhang Cao, Xinjuan Fan, Xiaoying Lou, Hail- ing Liu, Jinlong Hou, Xiao Han, From whole slide imaging to microscopy: Deep microscopy adaptation network for histopathology cancer image classification. In International Conference on Medical Image Computing and Computer-Asisted Intervention, pp. 360-368, 2019a.
Yuchen Zhang. Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning (ICML), pp. 1404-7413, 2019b.
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gor- don. Adversarial multiple source domain adaptation. Advances neural information processing systems, 31:8559-8555 -8500, 2018.
21

---

Published as a conference paper at ICLR 2022
APPENDICES
Additional dataset details
23
A.I
IW1.DCAM2020-WII.D WIT.A
23
A.2
CAMELYON17-WILDS
24
A.3
FMOW-WILDS
25
A.4 POVERTYMAP-WILDS
26
A.5
HEATWALWWEAT-WILDS
27
A.6
OGB-MOLPCBA
A.7
WIVILCOMMENTS-WILLSS
32
A.8
AMAZON-WILDS
33
Algorithm details
35
B.1
Empirical risk minimization (ERM)
35
B.2
Domain-invarian methods
35
B.3 Self-training methods
37
B.4 Self-supervision methods
41
C Data augmentation
43
Experimental details
D.I In-distribution vs. out-of-distribution performance
44
D.2 Model architectures
44
D.3 Batch sizes and batch normalization
44
D.4 Hyperparameter tuning
45
D.5 Algorithm-specific hyperparameters
46
D.6 Compute infrastructure
46
E Experiments on DomainNet
47
Fully-labeled ERM experimental details
49
Using the WILDS library with unlabeled data
51
22

---

Published as a conference paper at ICLR 2022
A ADDITIONAL DATASET DETAILS
In this appendix, we provide additional details on the unlabeled data in WILDS 2.0. For more context on the motivation behind each dataset, the choice of evaluation metric, and the labeled data, please refer to the original WILDS paper (Koh et al., 2021).
A.I IWILDCAM2020-WILDS
The WWLDCCMM2020WILDD dataset was adapted from the iWildCam 2020 competition dataset made up of data provided by the Wildlife Conservation Society (WCS) (Beery et al., 2020) 3. Cam- era trap images are captured by motion-triggeene static cameras placed in the wild to study wildlife non-invasive manner. Images are captured high volumes single camera trap can capture 10K images in a month and annotating these images requires species identification expertise and time-intensive. However, there are tens of thousands of camera traps worldwide capturing images of wildlife that could be used as unlabeled training data. For example, Wildlife Insights (Ahumada et al., 2020) now contains almost 20M camera trap images collected across the globe, but a large proportion of that data is still unlabeled. Ideally could capture value from those images despite the lack of available labels. We extend WWILDCAM0200 -ILLDD with unlabeled data from a set of WCS camera traps entirely disjoint with the labeled dataset, representative of unlabeled data from newly-deployed sensor network.
Problem setting. The task is to classify the species of animals in camera trap images. The input an image from a camera trap, and the domain d corresponds to the camera trap that captured the image. The target y, provided only for labeled training images, is one of 182 classes animals. We seek to learn models that generalize well to new camera trap deployments, the test data comes from domains unseen during training. Additionally, we evaluate the in in-distribution performance on held-out images from camera traps the train set.
The data comes from multiple camera traps around the world, all provided by the Wildlife Conservation Society (WCS). The labeled data the same as in Koh al. (2021) and the unlabeled data comprise $19,120 images from 3215 15 WCS camera traps not included in iWildCam 2020:
Data.
1. Source: 243 camera traps.
2. Validation (OOD): 32 camera traps.
3. Target (OOD): 48 camera traps.
4. Extra: 3215 camera traps.
The four sets camera traps are disjoint. The distributions of the labeled and unlabeled camera traps are very similar, except that the labeled data does not contain cameras with photos taken before LandSat data was available.
Split
# Domains (camera traps)
# Labeled examples
# Unlabeled examples
Source
129,809
243
Validation (ID)
7,314
Target (ID)
8,154
Validation (OOD)
32
14,961
Target (OOD)
48
42,791
Extra (OOD)
3215
819, 120
Total
3538
203,029
819, 110
Table 3: Data for WWI.DCAM2020-WI..D Each domain corresponds different camera trap.
TThe WCS
Camera Traps Dataset can be found http://111a http:///.aaaaaaaaaiaeeeeeeea www.encceeattaseeeeeeeeeeeeeeeee
wcscameratraps
23

---

Published as a conference paper at ICLR 2022
Broader context. There are large volumes of unlabeled natural world data that have been collected in growing repositories such as iNaturalist (Nugent, 2018), Wildlife Insights (Ahumada et al., 2020), and GBIF (Robertson et al., 2014). This data includes images video collected by remote sensors or community scientists, GPS track data from an-animal devices, aerial data from drones or satellites, underwater sonar, bioacoustics, and eDNAA Methods that can harness the wealth of information in unlabeled ecological data are well-posed to make significant breakthroughs in how we think about ecological and conservation-focuss
research. Natural-world and ecological benchmarks that pro- vide unlabeled data include NEWT (Van Horn et al., 2021), investigating efficient task learning, and Semi-Supervie Superviser iNat (Su & Maji, 2021), which provides labeled data for only subset of the taxo- nomic tree. Recent work has begun adapt weakly- supervssuee and self-supervised approaches for these natural world settings, including probing the generality and efficacy of superviserii (Cole et al., 2021), incorporating domain-relevant context into self-supervision (Pantazis et al., 2021). or leveraging weak supervision from alternative data modalities (Weinstein et al., 2019) or pre-trained, generic models (Weinstein et al., 2021; Beery al., 2019). Active learning also plays a role here in seeking to adapt models efficiently to unlabeled data from novel regions with only a few targeted labels (Kellenberger al., 2019; Norouzzadeh al., 2021).
A.2 CAMELYON17-WILD
The CAMELYON17-WIS dataset (Koh et al., 2021) was adapted from the Camelyon17 7 dataset (Bandi et al., 2018), which a collection of whole-slide images (WSIs) of breast cancer metastases in lymph node sections from 5 hospitals in the Netherlands. The labels were obtained by asking expert pathologists to perform pixel-level annotations of each WSI, which is an expensive and pain- staking process. In practice, unlabeled WSIs (i.e., WSIs without pixel-level annotations) easier to obtain. For example, only a fraction of the WSIs the original Camelyon17 dataset (Bandi al., 2018) were labeled; the other WSIs, which taken from the same hospitals, were provided without labels. In this work, we augment the CAMELYON 17 WIWLLDS dataset with unlabeled data from these WSIs.
much
Problem setting. The task is to classify whether histological image patch contains any tumor tissue. We consider generalizing from a set of training hospitals to new hospitals at test time. The input æ corresponds to 96x96 96 image patch extracted from WSI of lymph node section, the label is a binary indicator of whether the central 32x32 patch of the input contains any pixel that was annotated from. Each patch also includes metadata on which WSI was extracted from, though we do not use this metadata for training or evaluation. Models are evaluated by their average accuracy on class-balanced test dataset.
tumor in the WSI, and the domain d identifies which hospital the patch came
Data. All of the labeled and unlabeled data are taken from the Camelyon17 dataset (Bandi et al., 2018), which consists of WSIs from 5 hospitals (domains) in the Netherlands. We provide unlabeled data from same domains as the labeled CAMELYON17-WILL dataset (no extra domains). The domains are split follows:
1. Source: Hospitals 1, 2, and 3.
2. Validation (OOD): Hospital 4.
3. Target (OOD): Hospital 5.
CAMELYON17-WILDS also includes a Validation (ID) set which contains data from the training hospitals.
The CAMELYON|77WII. dataset has a total of 455,954 labeled patches across these splits, derived from the 10 WSIs per hospital that have full pixel-level annotations. We augment the dataset with total of 2,999,307 unlabeled patches, extracted from an additional 90 unlabeled WSIs per hospital. There is no overlap between the WSIs used for the labeled versus unlabeled data. To extract and process each patch, we followed the same data processing steps that were carried out for the labeled data in Koh et al. (2021).
Unlike the labeled patches, which were sampled in a class-balance balanced manner (i.c., half of the patches have positive labels), we sampled the unlabeled patches uniformly random from the unlabeled
24

---

Published as a conference paper at ICLR 2022
Split
Domains (hospitals)
# Labeled examples
Unlabeled examples
Source
302,436
1,799,247
Validation (ID)
33,560
0
Validation (OOD)
34,904
600,030
Target (OOD)
85,054
600,030
Total
5
455,954
2,999,307
Table 4: Data for CAMELYON 17-WIDS.. Each domain corresponds to different hospital.
WSIs. We sampled 6,667 patches per unlabeled WSI, with the single exception of one WSI which had only 5,824 valid patches, resulting a total of 3,000, unlabeled patches (Table 4). While the labeled patches were sampled in plass-balanced manner, the underlying label distribution skews heavily negative (approximately 95% of the patches a WSI are negative), so we expect the unla- beled patches to be similarly skewed in their label distribution.
Broader context. We focused providing unlabeled data from the same hospitals (domains) as in the original labeled CAMELYON 17-WLDS dataset. This unlabeled data from the training and test hospitals be used develop and evaluate methods for semi-supervised learning (Peikari et al., 2018; Akram et al., 2018; Lu et al., 2019; Shaw et al., 2020) and domain adaptation (Ren al., 2018; Zhang et al., 201 9a; Koohbanani al., 2021), respectively. In practice, there is also large amount of unlabeled data from different domains that is publicly available: for example, The Cancer Genome Atlas (TCGA) hosts tens of thousands of publicly-available slide images across a variety of cancer types and from many different hospitals (Weinstein et al., 2013). These large and diverse datasets need not even directly relevant to the task at hand, e.g., one could pre- train a model on images for different types of cancer even if the goal were to develop a model for breast cancer. Recent work has started to explore the use of these large and diverse datasets for computational pathology applications (Ciga et al., 2020; Dehaene et al., 2020) and in other medical imaging applications (Azizi et al., 2021).
A.3 FMOW-WILDD
The WWILD dataset (Koh et al., 2021) was adapted from the FMoW dataset (Christie et al., 2018), which consists of global satellite images from 2002-2018, labeled with the functional pur- pose of the buildings or land in the image. The labels are collected by process which combines map data with crowdsourced annotations (from trusted crowd). contrast, unlabeled satellite imagery is readily available across the globe. In this work, we augment the TMOW-WILDD dataset with unused satellite images that were part of the original FMoW dataset but not in the FMoW- WWLLS dataset.
Problem setting. The task is to classify the building or land-use type of satellite image. We consider generalizing from images before 2013 to after 2013, as well as considering the performance on the worst-case geographic region (Africa, the Americas, Oceania, Asia, Europe). The input is an RGB satellite image (224 x 224 pixels). The label is one of 62 building or land use categories. The domain d represents both the year and the geographical region of the image. Each image also includes metadata on the location and time of the image, although we do not use these except for splitting the domains. Models are evaluated by their average and worst-region accuracies in the OOD timeframe.
The labeled and unlabeled data are taken from the FMoW dataset (Christie et al., 2018). We provide unlabeled data from same domains as the labeled FMOW- dataset (no extra domains). The domains follows:
Data.
1. Source: Images from 2002-2013.
2. Validation (OOD): Images from 2013-2016.
25

---

Published as a conference paper at ICLR 2022
Split
Domains (years x region)
# Labeled examples
Unlabeled examples
Source Validation (ID)
76,863
11,948
11
11,483
Target (ID)
11,327
Validation (OOD)
3 x
19,915
155,313
Target (OOD)
2 x 5
22,108
173,208
Total
16 x
141,696
340,469
Table 5: Data for FMoW-WILDD Each domain corresponds to a different year and geographical region.
3. Target (OOD); Images from 2016-2018.
All of these domains have disjoint locations. FMoW- also includes Validation (II) and Target (ID) sets which contain data from the training domains of 2002-2013.
The FMoW-w WILDS dataset has 141,696 labeled images across these splits. We augment the dataset with 340,469 unlabeled images. These images come from two sources:
1. We use a sequestered split of the dataset, which consists of new locations that are not in the original labeled MMOW-WILDS dataset; these unlabeled data are drawn from the same distribution as the labeled data.
2. For the unlabeled target and validation splits, also add unlabeled data in their respective timeframes from the training set locations. While the unlabeled data from the Validation (OOD) and Target (OOD) domains can come from the same locations as the labeled training data, we note that none of the locations in the labeled Validation (OOD) or Target (OOD) data, which is used for evaluation, is shared with any the unlabeled or labeled data used for training.
Broader context. We focus on providing unlabeled data from the years (domains) that were in the original FMOW-WILDS dataset. Prior works have used unlabeled satellite imagery for pre- training (Xie et al., 2016; Jean et al., 2016; Xie et al., 2021a; Reed et al., 2021), self-training (Xie al., 2021a), and semi-supervised learning (Reed al., 2021). Leveraging unlabeled satellite imagery is powerful since it is widely available and can reduce the frequency at which we need to re-collect labeled data.
A.4 POVERTYMAP-WILDD
The PAVERTYMAP-WIIN WILDS dataset (Koh et al., 2021) was adapted from Yeh et al. (2020). The dataset consists satellite images from 23 African countries, labeled with a village-level real- valued asset wealth index (measure of wealth). The labels are collected by conducting a nationally representative survey, which requires sending workers into the field to ask each household a number of questions and can be very expensive. In contrast, unlabeled satellite imagery is readily available across the globe. In this work, we augment the MAPERTYMMP-IIII dataset with satellite images from the same LandSat satellite.
Problem setting. The task is to predict real-valued asset wealth index from a satellite image. We consider generalizing across country borders (the dataset contains 5 different cross validation folds, each splitting the countries differently). The input æ is a multispectral LandSat satellite image with 8 channels (resized to 224 224 pixels). The output y is real-valued asset wealth index. The domain d represents the country the image was taken in, as well as whether the image was taken at urban rural area. Each image also includes metadata the location and time, although do not make use these except for defining the domains. Models are evaluated by the average Pearson correlation (r) across folds, well the lower of the Pearson correlations on the urban rural
26

---

Published as a conference paper at ICLR 2022
Split
Domains (countries rural-urban)
Labeled ex.
# Unlabeled ex.
Source Validation (ID)
9,797
181,948
13 2
1,000
0
Target (ID)
1,000
Validation (OOD)
x 2
3,909
24,173
Target (OOD)
5 x 2 3,963
55,275
Total
23 x 2
19,669
261,396
Table Data for POVERTYMAP-WIL (Fold A). Each domain corresponds a different country and whether the image was from a rural or urban area.
subpopulations to test generalization to these subpopulations. In particular, generalization to rural subpopulations is important as poverty is more common in rural areas.
Data. We provide unlabeled data from same domains as the labeled POVERTY WILLD dataset (no extra domains). The domains are split as follows:
Source: Images from training countries in the fold.
2. Validation (OOD): Images from validation countries in the fold.
3. Target (OOD): Images from test countries in the fold.
All the countries in these splits are disjoint. Folds also contain Validation (ID) and Target (ID) set with data from the training countries.
The POVERTY MAITH WILDS dataset has 19,669 labeled images across these splits. We augment the dataset with 261, ,,36 unlabeled images from the same 23 countries. These images are collected using the same process Yeh al. (2020) from the same LandSat satellite. The image locations are chosen to be roughly near survey locations from the Demographic and Health Surveys (DHS).
Broader context. We focus providing unlabeled data from the countries (domains) that were in the original POVERTYMAP-WO training unlabeled data (to predict auxiliary task such as nighttime light prediction) (Xie et al., 2016; Jean et al., 2016) and for semi-supervised learning via entropy minimization (Jean et al., 2018). However, these works focus generalization new locations in the countries in the training set. Poverty prediction is different from usual tasks in that the output is real-valued. Most methods for unlabeled data made for classification tasks, and hope that dataset will encourage more work on methods for using unlabeled data for improving OOD performance in regression tasks.
dataset. Prior works on poverty prediction have used pre-
A.5 GLOBBLLHHAT-WIIN
The GLOBALWIIETTWILLD dataset was extended from the Global Wheat Head Dataset developed by David al. (2020; 2021). The goal of the dataset is to localize wheat heads from field images to assist plant scientists to assess the density, size, and health of wheat heads in a particular wheat field. This imagery acquired during different periods cover the development of the vegetation, from the emergence to organ appearance. Examples in box annotations of each wheat head in the image. Wheat heads are densely packed and overlap- ping, making object annotation highly tedious. Thus, the Global Wheat Head Dataset (GWHD) is relatively small, while in reality more field images are available. We supplement GLOBALWIT WHETT WILDS with unlabeled examples from the same set of field vehicles and sensors but taken in different acquisition sessions, i.e., at different locations or the same location in different year. The inclusion of this unlabeled data allows: 1) much higher spatial coverage of field location when the data comes from an acquisition session which is already included, 2) much higher temporal resolution when the data comes from location which is already included, so we have larger range of wheat
are labeled by bounding
27

---

Published as a conference paper at ICLR 2022
Split
Domains (acquisition session)
Labeled examples
# Unlabeled examples
Source Validation (ID)
2,943
5,997
18
357
0
Target (ID)
357
Validation (OOD)
8
1,424
2,000
Target (OOD)
21
1,434
8,997
Extra
53
42,445
Total
100
6,515
59,439
Table Data for GLOBALWHHETTWILLDS
growth stages, and 3) slightly more diversity when the session comes from different location, but with the same image acquisition protocol (i.e., the same field vehicle and image sensor).
Problem setting. The task is to localize wheat heads in high resolution overhead field images taken from above the crop canopy. We consider generalizing across acquisition sessions represent- ing a particular location, time and sensor with which the images were captured. Variation across ses- sions includes changes wheat genotype, wheat head appearance, growing conditions, background appearance, illumination and acquisition protocol. The input is an overhead outdoor image of wheat canopy, and the label is a set of box coordinates bounding the wheat heads (the spike at the top of the wheat plant holding grain), omitting any hair-like awns that may extend from the head. The domain d designates an acquisition session, which corresponds to a certain location, time, and imaging sensor.
Data. We provide unlabeled data from same domains as the labeled GLOBALW SERAALWHEAT--IIHE WILDS dataset. Additionally, provide unlabeled data from extra acquisition sessions not in the labeled GL.OBAL. dataset (extra domains). The domains are split as follows:
1. Source: 18 acquisition sessions in Europe (France x x13, Norway x2, Switzerland, United Kingdom, Belgium).
2. Validation (OOD): 8 acquisition sessions: 7 in Asia (Japan China x 3) and 1 in Africa (Sudan).
3. Target (OOD): 21 acquisition sessions: 11 in Australia and 10 in North America (USA 6, Mexico x 3, Canada).
4. Extra (OOD): 53 acquisition sessions distributed across the world.
The source, validation, and target sessions are split by continent, while the extra sessions are taken from across the world. For acquisition sessions with both labeled and unlabeled data, we randomly selected new patches of 1024x 1024 pixels from the original underlying data. The images were preprocessed in the same way as described David al. (2021).
Broader context. Utilizing unlabeled data is relatively new in the context of plant phenotyping, due to the lack of large, unlabeled database of plant images. However, larger plant image datasets are starting become available, such as from the Terraphenotying Reference Platform TTERRA-Red Burnette et al. (2018)). Increasing the sample size and variation within plant datasets is an important goal, because plants from the same species are fairly self-similar within the same field and therefore increasing the number of locations, times and image types included a dataset can be beneficial for making fine- grained visual classifications for plants. Further, for plant phenotyping to be used in farming applications, such as for precisely spraying weeds in a field with herbicide, models must be highly robust variations between different fields.
28

---

Published as a conference paper at ICLR 2022
Split
Name
Country
Site
Date
Sensor
Stage
#Labelod
#Heads
#Unlabeled
Source
Arvalis_
France
Gréoux
6/2/2018
Handheld
PF
66
2935
0
Source
Arvalis
France
Gréoux
6/16/2018
Handheld
401
21003
Source
Arvalis_
France
Gréoux
7/1/2018
Handheld
F-R
588
21893
Source
Arvalis.4
France
Gréoux
5/27/2019
Handheld
204
Source
Arvalis_5
France
VLB*
6/6/2019
Handheld
F
448
8180
Source
6/26/2019
Handheld
Source
Arvalis 7
France
VLB*
6/1/2019
Handheld
F-R
24
1247
Source
6/1/2019
Handheld
Source
Arvalis 9
France
VLB
6/1/2020
Handheld R
32
1894
Source
Arvalis_
6/10/2020
Handheld
Source
Arvalis II
France
VIB
6/18/2020
Handheld
F
60
2818
Source
6/15/2020
Hancheld
Source
ETHZ
Switzerland
Eschikon
6/6/2018
Spidercam
747
49603
0
Source
INRAE_1
Toulouse
5/28/2019
Hancheid
Source
NMBU
Norway
NMBU
7/24/2020
Cart
82
7345
999
Source
NMBU.2
Norway
8/7/2020
Source
Ries
UK
Rothamsted
7/13/2015
Gantry
F-R
432
19210
Source
ULiège_
Belgium
Gembloux
7/28/2020
Cart
R
1847
1000
Validation
ARC 1
Sudan
WadMedani
3/1/2021
Handheld
30
1169
Validation
NAU_I
China
Baima
Handheld
PF
Validation
NAU
China
Baima
5/2/2020
Cart
PF
100
4918
1000
Validation
NAU_3
China
Baima
5///0200
4596
1000
Validation
Ukyoto_1
Japan
Kyoto
4/30/2020
Handheld
PF
60
2670
Validation
Utokyo_1
Japan
Tsukuba
5/22/2018
538
Validation
Utokyn_2
Japan
Tsukaha
5/22/2018
Cart
456
1000
Validation
Utokyo_2 3
Japan
Hokkaido
6/16/2021
Handheld
multiple
120
3085
Target
CIMMYT_I
Mexico
CindadObregon
3/24/2020
Cart
PF
69
2843
1000
Target
2
Mexico
CiudadObregoo
3/19/2020
PF
2771
1000
Target
CIMMYT
Mexico
CindaaObreggn
3/23/2020
Cart
PF
1561
1000
Target
KSU_I
US
Manhartan, KS
5/19/2016
Tractor
PF
6435
Target
KSU_2
US
Manhattan,K9 KS
5/12/2017
Tractor
PF
100
5302
1000
Target
KSU_3
US
Manhattan,K5
5/25/2017
Tractor
95
5217
1000
Target
KSU 4
US
Manhattan,KS
5/25/2017
Tractor
60
3285
1000
Target
Terrate[1
US
Maricopa
4/2/2020
144
3360
997
Target
Terraref
Maricopa
3/20/2020
Gantry
106
1274
1000
Target
UQ_1
Australia
Gatton
8/12/2015
Tractor
PF
22
640
Target
UQ
Australia
Gatton
9/8/2015
Tractor
PF
Target
UQ_3
Australia
Gatton
9/15/2015
Tractor
14
297
Target
UQ
Australia
Galton
10/1/2015
Tractor
1039
Target
UQQ.5
Australia
Gatton
10/9/2015
Tractor
F-R
30
3680
Target
UQ
Australia
Galton
10/14/2015
Tractor
F-R
1147
Target
UIQ.7
Australia
Galton
10/6/2020
Handheld
R
17
1335
Target
UQ
Australia
McAllister
10/9/2020
Handheld
R
4835
Target
UQ.9
Australia
Brookstead
10/16/2020
Handheld
F-R
33
2886
Target
UQ_10
Australia
Gatton
9/22/2020
Handheld
F-R
106
8629
Targer
UQ.11
Australia
Gatton
8/31/2020
Handheld
PF
84
4345
Target
Usask_1
Canada
Saskatoon
6/6/2018
Tractor
F-R
200
5985
Table 8: Source, validation, and test domains for GLOBALW HEA.WWEEAATWLAAA WILDS.
29

---

Published as a conference paper at ICLR 2022
Split
Country
Sensor
Stage
.Laabledd
Extra
Analis,
France
Mons
6/15/2018
Handheld
F-R
0
995
Extra
Arvalis_14
France
Gréoux
5/25/2020
Handheld
Extra
Aralis.11
France
VLB
6/2/2020
Handheld
1000
Extra
16
France
Gréoux
6/22/2020
Handheld
F-R
1000
Extra
Arvalis_17
France
Bignon
5/18/2021
Handheld
1000
Extra
Arvalis 18
France
VI_B*
5/28/2021
Handheld
PF
1000
19
France
Encrambade
6/2/2021
Handheld
Extra
Arvalis 20
France
OLM*
6/2/2021
Handheld
1000
Extra
Arvalis
France
Encrambade
6/11/2021
Handheld
PF
1000
Extra
Arvalis_22
France
VLB"
6/14/2021
Handheld
1000
Extra
Arvlls 23
France
OLM*
6/17/2021
Handheld F-R
1000
CIMMYT_4
CiudadOhregoo
3/11/2020
Extra
CIMMYT.5
Mexico
CiudadObregon
3/12/2020
Cat
1000
Extra
CIMMYT.6
Mexicu
CiudadObregon
3/13/2020
Cart
1000
Extra
CIMMYT_7
Mexico
CiudadOhregon
3/13/2020
Cart
1000
Extra
CIMMYT_8
Mexicu
CiudadOOregon
3/13/2000
Cart
1000
CindadOhregon
3/19/2020
Extra
CIMMYT_10
Mexico
CindadObregon
4/15/2020
Cart
E
1000
Extra
CIMMYT
Mexico
CiudadObregon
4/22/2020
Cart
Extra
CIMMYT_12
Mexico
CiudadObregon
4/22/2020
Cart
1000
Extra
CIMMYT 13
Mexico
CiudadObregon
4/22/2020
Cart
1000
CIMMYT
Mexico
CiudadObregon
4/22/2020
Cart
PF
1000
Extra
CIMMYT_15
Mexico
CiudadObregon
4/28/2020
Cart
PF
1000
Extra
CIMMYT.16
Mexico
CiudadObregon
$/3/2020
Cart
F-R
Extra
ETHZ_2
Switzerland
Eschikon
6/6/2018
Spidercam
750
Extra
INRAF_2
France
Clermont-Ferme
3/29/2019
Handheld
1000
KSU.S
Manhattan,
Tractor
Extra
KSUL6
US
Manhattan,
4/23/2017
Tractor
P.F
1000
Rothamsted
7/7/2015
Gantry
Extra
Rres
UK
Rothamsted
7/10/2015
Gantry
1000
Extra
Rres
UK
Rothamsted
7/13/2015
Gantry
F-R
1000
Rres
UK
Rothamsted
7/20/2015
Gantry
1000
Extra
ULiège
Belgium
Gembloux
6/11/2020
Cart
PF
1000
Extra
ULiège_3
Belgium
Gembloux
6/15/2020
Cart
Extra
ULiège
Belgium
Gembloux
6/16/2020
Car
1000
Extra
ULiège.5
Belgium
Gembloux
6/18/2020
Cart
1000
Extra
Uliège__
Belgium
Gembloux
6/23/2020
Cart
1000
Extra
ULiège_7
Belgium
Gembloux
6/26/2020
Cart
1000
iégege_
Belgium
Gembloux
Extra
ULiège_9
Belgium
Gembloux
7/13/2020
Cart
F-R
1000
Extra
Usask
Canada
Saskatchewan
8/6/2019
Tractor
800
Extra
Usask 3
Canada
Saskatchewan
8/12/2019
Tractor
F.R
800
Extra
Utokyo.4
Japan
Hokkaido
6/7/2021
Handheld
PF
100
Utokyo
Japan
Hokkaido
6/9/2021
Handheld
100
Extra
Utokyo.6
Japan
Hokkaido
6/16/2021
Handheld
PF
100
Extra
Utokyo
Japan
Hokkaido
6/23/2021
Handheld
100
Extra
8
Japan
Hokkaido
7/3/2021
Handheld
100
Extra
Utokyo_9
Japan
Hokkaido
7/10/2021
Handheld
100
Ulokyo_10
Jagan
Hokkaido
7/10/2021
Handheld
Exira
Utokyo_11
Japan
Hokkaido
7/11/2021
Handheld F-R
100
12
Japan
Hokkaido
7/20/2021
100
Extra
13
Japan
Hokkaido
7/20/2021
Handheld
100
Extra
14
Japan
Hokkaido
7/28/2021
Handheld
100
Table 9: Extra domains for GLOBAL SERALWWHEAT-WEE WHEA--ILDS.
30

---

Published as a conference paper at ICLR 2022
Split
Domains (scaffolds)
# Labeled examples
Unlabeled examples
Source
44,930
350,343
4,052,627
Validation (OOD)
31,361
43,793
430,325
Target (OOD)
43,793
43,793
517,048
Total
120,084
437,929
5,000,000
Table 10: Data for OGB-MoLPCBA Each domain corresponds different molecule scaffold structure.
A.6 OGB-MOLPCBA
The OGB-MOLPCBA dataset was adapted from the Open Graph Benchmark (Hu et al., 2020b) and originally curated by the MoleculeNet (Wu et al., 2018) from the PubChem database (Bolton al., 2008). The dataset is a collection molecules annotated with 128 kinds of binary labels indicating the outcome of different biological assays. Performing biological assays is expensive, and result, the assay labels are only sparsely available over tiny portion of the molecules curated in the large-scale PubChem database (Bolton al., 2008). On the other hand, unlabeled molecule data is abundant and readily available from the database. Prior work in graph machine learning has leveraged unlabeled molecules to perform pre-training (Hu et al., 2020c) and cemi-supervised learning (Sun et al., 2020). In this work, we augment the DGB-MoLPCHI dataset with unlabeled molecules subsampled from the PubChem database.
Problem setting. The task is multi-task molecule classification, and consider generalizing to new molecule scaffold structures at test time. The input I corresponds to a molecular graph (where nodes are atoms and edges are chemical bounds), the label y is a 128-dimensional binary vector, representing the binary outcomes of the biological assay results. y/ could contain NaN values, in- dicating that the corresponding biological assays were performed on the given molecule. The domain d indicates the scaffold group molecule belongs to. As the binary labels are highly-skewed, the model's classification performance is evaluated using the Average Precision.
Data. All of the labeled and unlabeled data are taken from the PubChem database (Bolton al., 2008). We provide unlabeled data from same domains as the labeled OGB- MMM-CBAL dataset (no extra domains). We curate the unlabeled data by randomly sampling 5 million molecules from the PubChem database. We then assign these unlabeled molecules to the existing labeled scaffold groups that contain the most similar molecules. Specifically, we first compute the 1024-dimensional Morgan fingerprints for all the molecules (Rogers & Hahn, 2010; Landrum et al., 2006). Then, for each unlabeled molecule, we compute its Jaccard similarity against the labeled molecules in OGB-Mor.Pe MOL..CAA and obtain labeled molecule with the highest Jaccard similarity. Finally, we assign the unlabeled molecule the scaffold group that the most similar labeled molecule belongs to. This way, the molecules within the same scaffold groups structurally similar to each other.
The domains in the OGB-MOLPCBA dataset are as follows:
1. Source: 44,930 scaffold groups
2. Validation (OOD): 31,361 scaffold groups.
3. Target (OOD): 43,793 scaffold groups.
The largest scaffolds are in the source split and the smallest scaffolds the target split. We assign all of the unlabeled molecules to the existing domains, there extra domains added.
While the unlabeled data are similar to the labeled data in that they were all derived from PubChem (Bolton et al., 2008), it is quite possible that there was some selection bias in which molecules in PubChem were chosen to be labeled, which would lead to an undocumented distribution shift between the unlabeled and labeled datasets.
31

---

Published as a conference paper at ICLR 2022
Broader context. We focused on providing unlabeled data for both training and OOD test do- mains. Unlabeled molecules can be used to develop and evaluate methods for domain adaptation, self-training, well as pre-training (Hu et al., 2020c) and semi-supervised learning (Sun et al., 2020). In terms of future directions, we think it is fruitful to explore both graph-agnostic meth- ods (e.g., pseudo-label training) and more graph-specific methods (e.g., self-supervised learning of graph neural networks (Xie et al., 2021b)).
A.7 CIVI.COMMENTS-WILLD
The WWW.CCOMMNTS-WIIT dataset (Koh et al., 2021) was adapted from the CivilComments dataset (Borkan et al., 2019), which is a collection of text comments made on online articles. The data in CIVILCOMMENTS- -OOOMEESSSWWLDS underwent a significant labeling and annotation process: each ex- ample was labeled toxic or non-toxic and annotated for whether they mentioned certain demographic identities by at least 10 crowdworkers. Such substantial labeling and identity annotation process is expensive and come-consuming. On the other hand, unlabeled, unannotated text comments are read- ily available. For example, CIVI.COMMENTS. SENNENS--WILDS only contains subset of all data available in the original CivilComments dataset (Borkan et al., 2019), most of which Koh et al. (2021) excluded because these examples were not annotated for mentioning identities. In this work, we augment the WWILCOMMENTS-WILEE dataset with these unlabeled, unannotated comments.
Problem setting. The task is to classify whether a text comment is toxic or not. The input æ is a text comment (at least one sentence long) originally made on an online article, the label y is a binary indicator of whether the comment is rated toxic not, and the domain d is an 8-dimensional binary vector, where each dimension corresponds whether the comment mentions each of 8 demographic identities: male, female, LGBTQ, Christian, Muslim, other religions, Black, or White, respectively. Each comment also includes metadata on which article the comment was made on, although we do not use this metadata for training or evaluation.
We consider the subpopulation shift setting, where the model must perform well across all subpop- ulations, which are defined based d. Koh et al. (2021) define 16 subpopulations (groups) based on d. Models are then evaluated by their worst-group accuracy, i.e., the lowest accuracy over the 16 groups considered. In work, use the same evaluation setup.
Data. All of the labeled and unlabeled data are taken from the CivilComments dataset (Borkan et al., 2019). After preprocessing, Koh et al. (2021) created the CIVILCOMMENTS- WILDS dataset using the 448,000 examples that were fully annotated for both toxicity y and the mention of de- COVI.COMMENTS-EO
mographic identities d. In this work, we augment 1,551,515 examples collected by Borkan et al. (2019). We use these examples as unlabeled data. We follow the same preprocessing steps as was done with the labeled data in Koh et al. (2021). The resulting unlabeled examples have no identity annotations d and no toxicity label y. We note that Borkan et al. (2019) actually do provide toxicity labels for these examples in the original CivilCom- ments dataset, but we ignore these labels and use them neither for training nor evaluation.
with an additional
Because our unlabeled examples have no identity annotations, we cannot group these examples as Koh et al. (2021) group the labeled examples; thus we refer to this data as unlabeled data coming from extra domains (Table 11). In practice, these comments may actually mention any number of identities.
A substantial amount (1,427,848 or 92%) of the unlabeled comments are drawn from the same articles as the labeled comments. In particular, 140,082 unlabeled comments are from the same articles as labeled comments in the test split.
CIVILCOMMENTS- WILDS exhibits class imbalance. We account for this when benchmarking meth- ods by sampling
batches of labeled data when applicable (see Appendix B).
with extra unannotated data from the original CivilComments dataset (Borkan et al., 2019). In practice, unannotated text comments are widely available data, as we do in this work, can help with bias is still an open question. Previous work suggests that training on large amounts of data alone is not sufficient avoid unwanted biases, since many
Broader context. In this work, we focused on supplementing CIVILCOMMENTS-VII
the internet. Whether using such unlabeled
32

---

Published as a conference paper at ICLR 2022
Split
Domains (label identity groups)
# Labeled examples
Unlabeled examples
Source
16
269,038
Validation
16
45,180
Target
16
133,782
Extra
0
1,551,515
Total
16
448,000
1,551,515
Table 11: Data for CIVI.COMMENTS-WE WILDDS All of the splits are identically distributed.
papers have pointed out biases in large language models (Abid et al., 2021; Nadeem et al., 2020; Gehman al., 2020). However, recent work has also suggested that pre-trained models can be trained to be more robust against some types of spurious correlations (Hendrycks et al., 2020; Tu al., 2020) and that additional domain- and task-specific pre- pre-training (Gururangan et al., 2020) can also improve performance. We hope our contributions to the CiVILCOMMENTE SWILDDS dataset can encourage future study on whether unlabeled data can leveraged to improve generalization across subpopulation shifts.
A.8 AMAZON-WILDS
The AMAZON-WILDS dataset (Koh et al., 2021) was adapted from the Amazon reviews dataset (Ni et al., 2019), which is a collection of product reviews written by reviewers. While Amazon reviews are always labeled by the star ratings in practice, unlabeled data is a common source of leverage more generally for sentiment classification, with prior work in domain adaptation (Blitzer & Pereira, 2007; Glorot et al., 201 and semi-supervise learning (Dasgupta & Ng, 2009; Li et al., 2011). 1). In this work, augment the AMAZON-WILDD dataset with unlabeled reviews, whose star ratings have been removed.
Problem setting. The task is sentiment classification, and we consider generalizing from set of reviewers to new reviewers test time. The input æ corresponds review text, the label y the star rating from to 5, and the domain d identifies which user wrote the review. For each review, additional metadata (product ID, product category, review time, and summary) are also available. Because the goal is to train model that performs well across wide range of reviewers, models are evaluated by their tail performance, concretely, their accuracy on the user at the 10th percentile.
Data. All of the labeled and unlabeled data are taken from the Amazon reviews dataset (Ni et al., 2019). We provide unlabeled data from same domains as the labeled AMAZON- WILDS dataset. Additionally, we provide unlabeled data from extra reviewers not in the labeled AMAZON-WILDS dataset (extra domains). The domains are split as follows:
1. Source: 1,252 reviewers.
2. Validation (OOD): 1,334 reviewers.
3. Target (OOD): 1,334 reviewers.
4. Extra (OOD): 21,694 reviewers.
The reviewers each split are distinct, and all reviewers have at least reviews. The distributions of reviewers in each split are identical. AMAZON-WILDS also includes Validation (ID) and Target (ID) sets which contain data from the source reviewers.
The AMAZON-WILDS dataset has a total of 539,502 labeled reviews across these splits, and augment the dataset with a total of 3,462,668 unlabeled reviews. For each split of the unlabeled data, include all available reviews that are written by the reviewer. For the Extra (OOD) split, we include all reviewers with at least 75 reviews that are not Source, Validation (OOD), or Target (OOD) splits.
33

---

Published as a conference paper at ICLR 2022
Split
# Domains (reviewers)
Labeled examples
# Unlabeled examples
Source Validation (ID)
245,502
1,252
46,950
0
Target (ID)
46,950
Validation (OOD)
1,334
100,050
266,066
Target (OOD)
1,334
100,050
268,761
Extra (OOD)
21,694
0
2,927,841
Total
25,614
539,502
3,462,668
Table 12: Data for AMAZON-WILDS. Each domain corresponds to different reviewer.
To filter and process reviews, we followed the same data processing steps as for the labeled data in AMAZON-WILDS (Koh et al., 2021).
Broader context. We focused on providing unlabeled data from OOD domains, including both test and extra domains. Unlabeled data from the test reviewers can be used to develop and evaluate methods for domain adaptation (Ren et al., 2018; Zhang et al., 2019a; Koohbanani et al., 2021), which has been well-studied in the context of sentiment classification (Blitzer & Pereira, 2007; Glorot et al., 201 While there is limited prior work on leveraging unlabeled data from extra domains, some domain adaptation techniques can be readily adapted to leverage such unlabeled data (Ganin et al., 2016). Finally, we focus on unlabeled data specific to the task in this work, varying only the domains, and this contrasts with the type of unlabeled data used for pre-training in NLP, which is much larger and more diverse (Devlin et al., 2019; Brown et al., 2020).
34

---

Published as a conference paper at ICLR 2022
B ALGORITHM DETAILS
B. EMPIRICAL RISK MINIMIZATION (ERM)
As a baseline, we consider Empirical Risk Minimization (ERM). ERM ignores unlabeled data and minimizes the average labeled loss. We additionally evaluate ERM with strong data augmentation on applicable datasets, i.c., on WVILDCAM020-WILDD CAMELYON 77-WLDS, MAP- WILDS, and FMOW-wILLD (see Appendix C). ERM with strong data augmentation learns model that minimizes the labeled training loss
L (h(
Σ mL
1
Th
ehoo Asting (x (1 ).n) (
where Astong is stochastic data augmentation operation, and l measures the prediction loss. We use L. throughout this appendix to refer to the above labeled loss with strong augmentations (on applicable datasets).
For all dataset except LWVI.COMMENTS-IILDD we sample labeled batches uniformly at random. In our experiments, we account for class imbalance in WWW..CCMMENTS-WITN by explicitly sam- pling class-balanced batches of labeled data when computing (h).
B.2 DOMAIN-INVARIANT METHODS
Domain-invariant methods seek to learn feature representations that are invariant across domains. These methods motivated by earlier theoretical results showing that the gap between in- and out- of-distribution performance depends on some measure of divergence between the source and target distributions (Ben-David et al., 2010). To minimize this divergence, the methods described below penalize divergence between feature representations across domains, i.e., they encourage the model produce feature representations that are similar across domains.
Consider a model / = f. where the featurizer / X' + F maps the inputs to some feature space, and the head g : F y maps feature representations prediction targets. methods seek to constrain f to output similar representations for labeled and unlabeled data.
In this work, adapt all of our conain-invariant methods use data augmentations on applicable datasets (see Appendix C), and thus the output of f on the labeled batch is
Br = {f a Astrong (x(()) : E (L,".",, ,nL)
(2)
Similarly, the output of f on an unlabeled batch is
Bu (f Astrong (2 : (1, (I,)),,)u))
(3)
Domain-invariant methods seek to minimize some divergence & F x J + R between the labeled data B, and the unlabeled data Bu, where the choice of divergence depends on the specific method. The divergence is expressed as penalty term:
Epeeatty (f) = E Br, E Bu)
(4)
The final objective is combination of the labeled loss and penalty loss. The balance between the two losses controlled by hyperparameter X, the penalty weight.
L(h) = (h) + XL (f)
(5)
In our experiments, we study two classical domain-invariant methods, Correlation Alignment (CORAL) (Sun et al., 2016; Sun & Saenko, 2016) and Domain-Ade Adversaal Neural Networks (DANN) (Ganin et al., 2016). These methods are well-known and established, but their perfor- mance can be lower than that of newer domain-invarian methods that employ different penalties to encourage the source and target representations to be similar (Jiang al., 2020; Zhang al., 2021). Examples of these newer methods are Joint Adaptation Networks (JAN) (Long et al., 2017), Con- ditional Domain Adversarial Networks (CDAN) (Long al., 2018), Collaborative and Adversarial
35

---

Published as a conference paper at ICLR 2022
Networks (CAN) (Zhang et al., 2018), and models with Adaptive Feature Norm (AFN) (Xu al., 2019), as well as methods that minimize the Maximum Classifier Discrepancy (MCD) (Saito et al,, 2018) and the Margin Disparity Diserepancy (MDD) (Zhang et al., 2019b).
All of the above methods were developed for the single-source single-targe target setting, where the source single distribution, and likewise for the target domain. As each WILDS 2.0 dataset comprises multiple source domains and multiple target domains, it is likely that methods that can leverage this additional structure could perform better. Examples of these methods include Multi-source Domain Adversarial Networks (MDAN) (Zhao et al., 2018) 8) and Moment Matching for Multi-Source Domain Adaptation (M3SDA) (Peng et al., 2019). The DomainBed (Gulrajani & Lopez- Paz-,a 2020) and WILDS (Koh et al., 2021) benchmarks also extended single-source algorithms like CORAL and DANN to take advantage of multiple source domains in the domain generalization setting, and similar extensions in the domain adaptation setting could be promising.
domain is treated
Correlation Alignment (CORAL). Algorithm 1 describes CORAL, proposed by Sun et al. (2016); Sun & Saenko (2016). CORAL measures the divergence & between batches of feature rep- resentations in terms of the deviation between their first and second order statistics. Given labeled batch and unlabeled batch of features By E gT X Bu E 1R define the feature means as
14. 1 Br.
(6)
TLL
BB Bu
1
IE nu
and covariance matrices
1
1 (1 BL) (L)BL (17
CL
B BL
(8)
TL -1 1
nL
C
BT Bu
1 ((TBb) ((TBB))((111))) (ITBb)
(9)
Yuu
nu
We then compute the CORAL penalty as
Br, Bu) || mell + 1|0, Cull..
(10)
We adapted our implementation from DomainBed (Gulrajani Lopez-Paz, 2020), as done in WILDS 1.0. We note that these implementations compute the penalty as a sum of deviations in means and covariances, whereas Sun et al. (2016); Sun & Saenko (2016) penalize deviations in covariances only. (Sun et al. (2016) considers features that are normalized to zero mean.) On applicable datasets, we also strongly augmented all labeled and unlabeled examples using Aurong whereas Sun et al. (2016); Sun & Saenko (2016) do not explicitly require data augmentations. We add augmentations to allow for a fairer comparison to other methods which use augmentations.
Note that CORAL has also been adapted by Guirajani & Lopez-Paz (2020); Koh et al. (2021) for domain generalization. In particular, where the original CORAL paper defines Lienalty as the diver- gence between just two kinds of batches (labeled and unlabeled), these works define Lpaaalty the divergence between many kinds of batches, where batches are grouped based on domain annotation d(+), For simplicity, we followed the original CORAL formulation and differentiate only between labeled and unlabeled batches. We leave leveraging the domain adaptations d to future work.
APPLICABLE DATASETS. We run CORAL on all datasets except GLOBALWIHEAT- CIVILCOMMENTS WILDS. We do not evaluate domain invariant methods on CIVILCOMMENTS- WILDS, since the labeled and unlabeled data drawn from the same distribution. We do not evaluate CORAL on GLOBALWHEAT- WHEATHEWWTTEEE WILDS because CORAL does not port straightforwardly to detection settings.
and
DANN. Algorithm 2 describes DANN, proposed by Ganin et al. (2016). DANN measures the divergence & between batches of feature representations using the performance of discriminator network ha that aims discriminate between domains. Given batch of features (either B,, or Bt),
36

---

Published as a conference paper at ICLR 2022
Algorithm 1: CORAL
Input: Labeled batch {(r). Mr.
(4) (10) (1, m.), unlabeled batch
{at (1) d()) :iE (1, mu)). strong augmentation function Aatrong penalty weight À E IR, dimension of feature representations m
Compute feature representations for labeled and unlabeled batches
Br (f Astonn (o ( (1, .n.))
Bu = (f o Astrong (r) (I,..,,uu)
Compute feature mean and covariances for labeled and unlabeled batches
L B
TT.
1
1 BU
YIU
C2 74.
B Bt.
(1" (1= (1m
n1.
1 (1 (1" T (1TBu)
1
Cp nu
B BT
nu
Update model h = o f on loss
l(h o Astong (x (.()).)())) + A (IIAL - well + +[[CL - Culli)
AL
this deep network ha must classify whether examples are from the labeled data or unlabeled data. ha is optimized using a binary classification loss
L(ha)
E(ha o Aatrong ( (z( 1))
l(ha o f Aatrong ( (z()),
(11)
mL
The loss ha is exactly related & as
{(BL, Bu) L(ha)
(12)
In other words, at the same time that ha is optimized to minimize its loss L(ha). the featurizer is incentivized minimize Lpenalty {Br, Bu) -(aa). or maximize L(ha). See Algorithm for details.
We adapted our implementation from the Transfer Learning Library (Jiang al., 2020) and matched all details to the formulation given by Ganin et al. (2016), except for two changes. On applicable datasets, have strongly all labeled and unlabeled examples using Aurnng whereas Ganin et al. (2016) do not explicitly require data augmentations. We add augmentations to allow for fairer comparison to other methods which use augmentations. Second, where Ganin et al. (2016) opti- mize f. g. and ha using the same learning rate т,, use three different learning rates nJ Nig Tha following the implementation of the Transfer Learning Library (Jiang et al., 2020).
APPLICABLE DATASETS. We run DANN on all datasets except GLOBALWHHEAT CIVILCOMMENTS WILDS. We do not evaluate domain invariant methods on COMILCOMMENTS- WILDS, since the labeled and unlabeled data drawn from the same distribution. We do not evalu- ate DANN on BOOBALAEEAT-WWE settings.
and
because DANN does not port straightforwardly to detection
B.3 SELF-TRAINING METHODS
Self-training methods leverage unlabeled data by "pseudo-labeling" unlabeled examples with the model's predictions and training them if they were labeled examples. In certain for-
37

---

Published as a conference paper at ICLR 2022
Algorithm 2: DANN
Input: Labeled batch {(r Mr.
(1) ( E (1, m.) unlabeled batch
(at (3) d()) : E (1, nu)). strong augmentation function Astrong, penalty weight À E IR, learning rates 7g. ng. The
Compute loss for domain discriminator ha
TAg e(ba fo A Astron ( ),
L(ha)
Σ
Elba fo Aarong(=10) 0)
THL
nu
Compute loss for model h g
1 n (( o Astrong (2 v.) AL(ha)
nL
Update f. g. ha using appropriate learning rates T] Ty. Tha
mulations, this is equivalent to minimizing the model's conditional entropy on the unlabeled data (Grandvalet & Bengio, 2005). Contemporary self-training methods also often make use of consis- tency regularization, i.c., encouraging the model to make similar predictions noisy/augmented versions of unlabeled examples. Self-training methods have recently been shown to be empirically successful unsupervised domain adaptation (Saito al., 2017; Berthelot et al., 2021; Zhang al., 2021).
The self- self-training methods we study follow this general structure: given unlabeled example ru, these algorithms generate pseudolabel ÿu = v((v)) where the pseudolabet-generating function wh: X' + ) differs between algorithms. For classification problems, we study algorithms that produce hard pseudolabels, which are one-hot class predictions, rather than soft pseudolabels, which are continuous distributions over the classes. Next, algorithms define an unlabeled loss (u) for model h by minimizing the loss between pseudolabels ju and the model's predictions. The algorithms we consider below augment zu during training; i.e., rather than minimizing the loss between ýu and the model's 's prediction on zu. the algorithms below minimize the loss of predictions on A(zv), where is a stochastic, abbe-ppreeeriing augmentation. Assuming model that outputs real-valued logits, the complete unlabeled loss is
(h(h)=
= TU Σ e(h A(x
(13)
This unlabeled loss jointly optimized with the standard ERM labeled loss. The balance between the two losses is controlled by hyperparameter A(t), which is function of the current step t.
L(h)= = (h) + (h)
(14)
Pseudo-Label Algorithm describes Pseudo-Label, proposed by Lee (2013). In this algorithm, the model dynamically generates pscudolabels and updates each batch. Formally, the pseudolabel- generating function given by
Tu ((r arg max h Aarong (x1))
(15)
y
where Astrong is the strong augmentation function described in Appendix C. Pseudo-Label then com- putes the loss between a strongly augmented example and its associated pseudolabel.
In order to more fairly compare Pseudo-Labed Label to FixMatch, we add on confidence thresholding to the Pscudo-Label algorithm, a feature also added in the implementation of Pscudo-Label by Sohn et al. (2020). When confidence thresholding, examples which the model has low confidence have zero loss, i.e., for some threshold hyperparameter т, the loss an example zu contributes is
of Softmax max o Astrong (zu) (m)) N 3 EE(h o Astrong (σV), vu)
(16)
38

---

Published as a conference paper at ICLR 2022
Finally, Pseudo-Label increases the balance X(t) between labeled and unlabeled losses over time, initially placing 0 weight on (h) and then linearly stepping the unlabeled loss weight until reaches the full value of hyperparameter À at some threshold step. We fix the step at which X(t) reaches its maximum value (A) to be 40% of the total number of training steps, matching the im- plementation of Sohn al. (2020). This scheduling allows the algorithm initially prioritize the labeled loss, as generated pseudolabels are mostly incorrect while the model has low accuracy, For- mally, step and given total number of steps T'
x() = min
0.47 13.1
(17)
We add augmentations to Pseudo-Label in order to allow for a fairer comparison to other meth- ods that use augmentations. On applicable datasets, we have strongly augmented all labeled and unlabeled examples using Astrong whereas Lee (2013) do not use any data augmentations, i.e., all instances of Astong replaced with the identity function.
Algorithm 3: Pseudo-Label
Input: Labeled batch ((x{ VL. (() E (1, m.)). unlabeled batch
(( d()) i E (1,..., nn)), strong augmentation function Aatong unlabeled loss weight for current step A(t) E R. confidence threshold E j0, 1]
Generate pseudolabels You arg max, h Astrong(Zu) [y] for the unlabeled data
Update model h on loss
1
nL
((h o Astong (2) 1()
TLL
A(1)
nu
1 Softmax max h o Astongg (zu)lv)) Annnng(Du)ll)
2) E(h Astron( (Fu); Tt)
nu
APPLICABLE DATASETS. We evaluate Pseudo-Label on all datasets except POVERTYMAP- WILDS, as DOVERTYMAP-WIT is a regression dataset, and hard pseudolabeling does not port straightforwardly to regression tasks.
FixMatch. Algorithm 4 describes FixMatch, proposed by Sohn et al. (2020). Like Pseudo-Label this algorithm dynamically generates pseudolabels and updates each batch. FixMatch additionally employs consistency regularization on the unlabeled data. While pseudolabels are generated on weakly augmented view of the unlabeled examples, the loss is computed with respect to predictions on strongly augmented view. This encourages models to make predictions strongly augmented example consistent with its prediction on the same example when weakly augmented. For details about the strong versus weak augmentations we use, Appendix C.
Formally, the pseudolabel-gennraain function &b is given by
ĪA = (av) = arg max o Aweak (xv)
(18)
Like Pseudo-Label. FixMatch uses confidence thresholding, and unlabeled examples on which the model has low confidence have zero loss. Following Sohn et al. (2020), we keep the balance between labeled and unlabeled losses constant X(t) at fixed magnitude (as opposed to slowly increasing X(t) as in Pseudo-Label) by noting that most predictions made by FixMatch are initially low contidence, so for sufficiently high confidence threshold т. most unlabeled examples have loss zero, keeping the magnitude of (u(h) initially small. This magnitude grows over time, providing a natural curriculum (Sohn et al., 2020).
x(t)
FixMatch's 's original authors justify keeping
We endeavored to match implementation of FixMatch to the formulation of Sohn et al. (2020), except in the use of augmentations for labeled data. Where we have strongly augmented all labeled examples using A Auroy in Algorithm Sohn al. (2020) explicitly choose weak instead of
39

---

Published as a conference paper at ICLR 2022
strong augmentations the labeled examples. However, results DomainNet in Appendix E suggest that using strong instead of weak augmentations for the labeled examples improves per- formance, so strong augmentations the labeled examples in order allow for fairer comparison to other methods.
Algorithm 4: FixMatch
Input: Labeled batch ((t, Vi. d( i (1,... nn)), unlabeled batch
((x)) d(") : i (1,... " "...,,n)),, nu)), weak augmentation function Aweak strong augmentation function A strong unlabeled loss weight À E IR confidence threshold [0, 1]
Generate pseudolabed ÿu = arg maxy h o Aweak (xt)(v) Awea for the unlabeled data
Update model h on loss
(h o Asrong ). 1)
70
Softmax maxh o Asuong (zu)(ui) 27} E(h o Astrnng (zv), (0)
TUU
i=1
APPLICABLE DATASETS. We evaluate FixMatch on image classification datasets (WILDCAM2020-WIHH WILDS, CAMELYONI7-WILDS, POVERTYMAP-WILE -WIDD, and FMOW-WILDS. We do not evaluate FixMatch on other datasets because FixMatch relies on enforcing consistency across data augmentations, which we only define for image datasets (see Appendix C).
Noisy Student. Algorithm 5 describes Noisy Student, proposed by Xie et al. (2020). Unlike Pseudo-Label and FixMatch, which update the model and re-generate new pseudolabels each batch, Noisy Student generates pseudolabels, fixes them, and then trains the model until convergence be- fore generating new pseudolabels. First, an initial teacher model is trained on the labeled data; next, the teacher model pseudolabels the unlabeled data, and student model is trained and pscudolabeled data; finally, the student model becomes the new teacher, and the cycle repeats (see Algorithm 5). Each (teacher, student) pair is termed iteration; we study the results of two iterations.
the labeled
We train Noisy Student using hard pseudolabels, which the teacher generates weakly aug- mented inputs:
iU = (au) = arg max fwacher o Aweak (x))
(19)
y
While the teacher generates pseudolabels on weakly augmented data, the student must make both labeled and unlabeled predictions on noisy (i.e., strongly augmented) data. Following Xie et al. (2020), add dropout layer (p 0.5) before the student's last layer, randomly corrupting final feature maps. Students thus are trained to be consistent across both data-based and model-based noise. We denote the model with inserted dropout as Dropout f. Xie et al. (2020) add even more model- based noise using stochastic depth; for simplicity. we do not use stochastic depth in our implementation.
We follow the original paper and fix the balance between labeled and unlabeled losses as A(t) = 1. Noisy Student does not confidence thresholding.
Note that Xie et al. (2020) use both dropout and strong data augmentations when training the initial teacher labeled data. We reuse models from ERM Data Augmentation experiments as initial teacher models; thus we differ from Xie et al. (2020) in that our initial teachers were trained with strong augmentations, but not dropout (see Algorithm 5).
APPLICABLE DATASETS. We evaluate Noisy Student on all datasets except text datasets CIVILCOMMENTS- WILDS and AMAZON-WIL WILDS. For GLOBALWHEAT-WILDE and OGB- MoL. PCBA Noisy Student without noise from data augmentations.
40

---

Published as a conference paper at ICLR 2022
Algorithm 5: Noisy Student
Input: Labeled dataset {TL ML, d divided into batches of size zt,, unlabeled dataset { du)) divided into batches of size Yu, total number of iterations S, weak augmentation function
Awesk. strong augmentation function Astong
Train an initial teacher model f(o) to convergence on labeled examples using the following batch-wise objective
71L. E(ho Astrong (.
for iteration (1,..... ,S) do
Generate fixed pseudolabels ÿu arg max, fls- Aweak (zu) for the unlabeled data
Train the next student model f|ا] to convergence on unlabeled and labeled examples using the following batch-wise objective
(Dropout h Asminn (1 (1()).
Drrooout o h Aurong (.
B.4 SELF-SUPERVISION METHODS
Self-supervised methods learn useful representations by training on unlabeled data via auxiliary "proxy" tasks. Common approaches include reconstruction tasks (Vincent et al., 2008; Erhan al., 2010; Devlin et al., 2019; Gidaris et al., 2018; Lewis et al., 2020), which remove or corrupt a small part of each training example and use it prediction goal, and contrastive learning (He et al., 2020; Chen et al., 2020b; Caron et al., 2020; Radford et al., 2021b), which aims to learn representation space such that similar example pairs stay close each other while dissimilar ones far apart. The underlying assumption is that feature encoders that solve the proxy tasks will also perform well on the downstream supervised task (Lee et al., 2020a; Wei al., 2021).
In our work, we consider two sulf-super methods: SWAV Caron et al. (2020) for images and masked language modeling (Devlin al., 2019) for text. We use these methods to pre-train models on the un unlabeled data. In all cases, we start with the same model initialization used for all of the other algorithms on that dataset; do additional pre-training via supervvipnvi on the unlabeled data; and then initialize a new classifier head and finetune the model via ERM with data augmentation. This follows the procedure Shen et al. (2021). As concrete example, for FMoW -WILD,, we use the following procedure to run our ERM experiments:
1. Initialize DenseNet- 121 model (Huang al., 2017) using Imageeet--retrrained weights.
2. Finetune the model on labeled data from the source domain.
3. Evaluate on held-out data from the target domain.
For SwAV, we use the exact same procedure but with the addition of a second step:
1. Initialize a DenseNet-122 model (Huang et al., 2017) using Imageeet--retrrinee weights.
2. Continue pre-training the model with SwAV unlabeled data from the target domain.
3. Finetune the model on labeled data from the source domain.
4. Evaluate on held-out data from the target domain.
Similarly, for text datasets, we initialized pre-trained BERT models and then continued pre-training them using masked language modeling on the unlabeled data in WILDS 2.0.
We tuned hyperparameters for finetuning, following the exact same procedure and hyperparameters as for ERM, but not for pre-training.
41

---

Published as a conference paper at ICLR 2022
SwAV. We directly the public SwAV repository available at https://git com/ facebookresearch. cswaav....... swav. We keep almost all of the hyperparameters used by the original paper for 400 epoch training with batch size 256. However, we make the following changes based on issues and tips from the original authors in the SwAV repository:
1. To stabilize training, we opt not to use a queue; this follows the suggestion in issue #69.
2. For each dataset, we the number of prototypes approximately 10x the number of classes; this follows the suggestion in issue #37. For POVERTYMAPE MAL-WLLSS,, which is regression problem, we use 1000 prototypes, which displayed more stable training than 10 or 100 prototypes.
3. We set c = 0.03 to avoid representation collapse; this follows the suggestion in the Com- mon Issues section of the repository's readme.
4. We set the base learning rate via the suggested "linear scaling" rule (issue #37). In other words, for total batch size (over GPUs) 2 512, the learning rate is scaled linearly. For smaller batch sizes (< 512), learning rate by x000 to obtain the final learning rate, since each of the base/final pairs that the paper reports differ by that factor.
the base learning rate at 0.6. We multiply the base
We set the maximum number of epochs to 400 but stop pre-training early when the loss does not decrease by more than 0.3% for 5 consecutive epochs.
APPLICABLE DATASETS. We evaluate SWAV IWILDAAM2200-WIT WILLDD CAMELYON17- WILDS, POVERTYMAP-WILDS and FMOW-WILDS We do not evaluate SWAV on other datasets because AV relies training with data augmentations, which we only define for image datasets (see Appendix C).
Masked language modeling (MLM). MLM popular supervisdd objective for text data and is commonly used to pre-train model representations (Devlin et al., 2019). Given unlabeled text corpus X' = {X] } (e.g., a set of comments for CivilComments; set of reviews for Amazon), training example (æc, y/ can be generated by randomly masking tokens in each text piece X (e.g., r = "The [MASK] is the currency [MASK] the UK"; = ("pound", "of")). The model is trained
its representation of the masked input predict the original tokens that should go in each mask. The MIL objective encourages the model to learn syntactic and semantic knowledge (e.g., to predict "of"") well as world knowledge (e.g., predict "pound") present in the text corpus (Guu et al., 2020).
For our implementation, with the MLM objective on the unlabeled data of each task (CivilComments, Amazon). Following the original BERT implementation (Devlin al., 2019), we randomly mask 15% of the tokens in each input text piece, of which 80% are replaced with [MASK], 10% are replaced with random token (according to the unigram distribution), and 10% are kept unchanged.
DistilBERT (Sanh et al., 2019) our initial model and pre-train
APPLICABLE DATASETS. We evaluate masked language modeling the text datasets CIVILCOMMENTS. WILDS and AMAZON-WILDS.
42

---

Published as a conference paper at ICLR 2022
C DATA AUGMENTATION
In this work, several methods we study leverage data augmentations to encourage generalization across domains. Below, we provide details on our implementations of these augmentations.
CAMELYONI7-WILDD, and FMOW-WILDS) We use consistent set of data augmentations across image datasets WII.DCAM2200-WIH WII.DS,, CAMELYON17-WILDD and (MOW-WILDS For methods other than SwAV, we define two strengths of data augmentations: weak function Aweak and a strong function Astrong and we specify both according to Sohn et al. (2020). The weak augmentation function Aweak flip. The strong augmentation function Aatong is a composition of (i) random horizontal flip. (ii) RandAugment (Cubuk et al., 2020), and (iii) Cutout (Devsies & Taylor, 2017). For the exact im- plementation of RandAugment, we directly use the implementation of Zhang et al. (2021), which is based the implementation used by Sohn et al. (2020). This implementation specifies pool of operations and sample magnitudes for each operation uniformly across a pre-specified range. The pool of operations includes: autocontrast, brightness, color jitter, contrast, equalize, posterize, rota- tion, sharpness, horizontal and vertical shearing, solarize, and horizontal and vertical translations. We apply N random operations for all experiments (see Appendix D.4).
Image classification (WWILDCAM2020
random horizontal
The labeled loss for all methods, including finetuning models pre-traine pre-trained with SwAV, uses this strong augmentation function.
For SwAV pre-training, we use the data augmentation pipeline used in the original paper (Caron et al., 2020), which is almost identical to the strong data augmentation introduced in SimCLR (Chen et al., 2020a) but with different random crop scales to accommodate the several additional lower- resolution crops. For each image, the pipeline is the following sequence random transformations: resized crop, horizontal flip, color jitter, grayscale, and Gaussian blur.
POVERTY WAP-W.DS.. As POVERTY WIP-II a dataset of multispectral images, we de- fine separate of data augmentations. For methods other than SwAV, we define two strengths of data augmentations: weak function Aweak and a strong function Astoog- The weak augmentation function Aweak random horizontal flip. The strong augmentation function Astrong is compo- sition of (i) random horizontal flip. (ii) random affine transformation, (iii) color jitter on the RGB channels, and (iv) Cutout on all channels (De Vries & Taylor, 2017).
We use the same augmentations for SwAV pre-training as above for WWLDDCMM200--WLDD JAMELYON17-WILN and FMOW- but note that the color jitter module is applied only to the RGB channels.
Other datasets. We do not define data augmentations for other datasets, i.c., GLOBALWHEAT- WILDS, DGB-MOLPCBA, COVILCOMMENTS-VLLL WHORAL.HEATTWWTEE is an image dataset and can be transformed using augmentations defined above, omit data augmentations for simplicity, because such augmentations would generally quire changing as well as x (e.g., random translations on the input image also require translating the bounding box labels). For OGB-MOLPCBA, we omit augmentations because data augmenta- tions on graphs are not well developed. CIVILCOMMENTS-WEE WILDS and AMAZON-WILL WIIDS are text datasets; although data augmentations have been proposed for text datasets, we do not use these augmentations because training with augmentations is not as standard on text datasets as on image datasets. For these datasets, methods are benchmarked without augmentations, we substitute all occurrences of Aweak, Astong with the identity.
and AMAZON-WILDS. Although
43

---

Published as a conference paper at ICLR 2022
D EXPERIMENTAL DETAILS
D.1 IN-DISTRIBUTION VS. OUT-OF-DISTRIBUTTE PERFORMANCE
We report both in-distribution and -itto--diitributionn performance metrics on all datasets, with the exception of NGB-MOLPCBA, which does not have a separate in-distribution test set. Using the terminology in WILDS (Koh al., 2021), we consider the train-to-train in-distribution comparison on IWILDCAM2020- WILDS, CAMELYON 7- 7-WILDS, FMoW-W and POVERTY MYP-RILMAP-WILLS. and the average comparison on CIVILCOMMENTS WILDS and AMAZON-WILDS.
D.2 MODEL ARCHITECTURES
For all experiments, we use the same models for each dataset as WILDS 1.0:
IWILDCAM2020-WILDS: ResNet-50 (He al., 2016).
CAMELYONI7-WILDD: DenseNet-121 (Huang al., 2017).
FMLDSWILDS: DenseNet- 121 (Huang et al., 2017).
POVERTYYMAP-WILDS: Multi-spectral ResNet- 18 (Yeh al., 2020).
GLOBALWIIETTWWLLDS:: Faster-RCNN (Ren et al., 2015).
OGB-MOLPCBA: Graph Isomorphism Network (Xu et al., 2018).
CIVI.COMMENTS- DistiBBERT (Sanh et al., 2019).
AMAZON-WILDS: DistiBBERT (Sanh et al., 2019).
The models for WILDCAM2020- WILDS, SMOW-WILDS, and GLOBBALWHAT-WILDD were ini- tialized with weights pre-trained ImageNet. Note that models for CAMELYONI7-WILDS not initialized with ImageNet weights. The DistilBERT models were also initialized with pre-trained weights from the Transformers library.
D.3 BATCH SIZES AND BATCH NORMALIZATION
For each dataset, set the total batch size (where batch contains both labeled and unlabeled data) to the maximum that can on 12GB GPU memory (Table 13). For all the methods that leverage unlabeled data, except the pre-training algorithms, resulting in a 4x larger effective batch size. For SwAV pre-training, we run with 4 GPUs in parallel, which achieves similar effect. For masked LM pre-training, we run with the default setting 256 steps of gradient accumulation. These larger batch sizes deviate from the defaults used in the WILDS paper (Koh et al., 2021). l). We use these larger batch sizes because methods that leverage unlabeled data tend to use larger batch sizes (Sohn et al., 2020; Xie et al., 2020; Caron et al., 2020).
with steps of gradient accumulation,
Dataset
WILDS 1.0 batch size
WILDS 2.0 batch size
CAMELYON/7-WIL.D
32
168
CIVIL WILDS FMOW-WILDS
16
48
32
72
HOVERTYMAP-WTINN
64
1 120
AMAZON-WIL.SS WWLDCAMM220-WLLL
8
24
16
24
32
4,096
GLOBALW HEAT- WILDS
4
8
Table 13: The batch sizes of each dataset from the original WILDS 1.0 paper and the batch sizes used in WILDS 2.0, which correspond to the maximum that fit into 12GB of GPU memory.
For models that batch normalization, the composition of each batch affects the way in which batch normalization is applied. For CORAL, DANN, and Pseudo-Label, we concatenate the labeled and unlabeled data together in each batch, so the labeled and unlabeled data are jointly normalized.
44

---

Published as a conference paper at ICLR 2022
Dataset
epochs
ERM
3:1 ratio
7:1 ratio
15:1 ratio
IWILDCAM2020- WILDS
12
6
3
2
CAMELYON 7-
10
5
3
2
FMOW -WILSWILDS
-WILDS
60
30
15
8
POVERTYMAP WIILD
150
75
38
GLOBALW
12
6
3
2
OGB-MOL -MOLPCAA
200
100
50
25
CIVILCOMMENTS- WILDDS
5
3
2
1
AMAZON-WIIE WILDS
Table 14: The number of epochs (complete passes over the labeled data) used for each dataset, specified for the ERM baseline as well as different ratios of unlabeled to labeled data within a batch.
For FixMatch, we jointly normalize the labeled data and the strongly augmented unlabeled data, but separately normalize the weakly augmented unlabeled data in separate forward pass; did two forward passes to keep the overall batch sizes consistent with the other algorithms, as in Table 13, while still fitting in GPU memory. For Noisy Student, MLM pre-training, and SwAV pre- training, the unlabeled data is processed separately from the labeled data, so each batch of laheled or unlabeled data separately normalized.
D.4 HYPERPARAMETER TUNING
We tune each algorithm separately for each dataset by randomly sampling 10 different hyperparame- ter configurations within the ranges defined below. We early stop and select the best hyperparameters based on the OOD validation performance, which is computed for each dataset; we do not use the labeled Validation (ID) data our experiments. We then run replicates using the best hyperparameters. For computational reasons, do not tune hyperparam- eters for the pre-training algorithms, though we tune the finetuning of their resulting pre-trained models usual.
the labeled Validation (OOD) data
Learning rates. For all the datasets, except for OGB-MOLPCBA, we multiply the learning rates used in WILDS by the ratio of the effective batch size to the original batch size used in WILDS 1.0. We center the learning rate grid around this modified learning rate r, and search over r (-,(-, where is the uniform distribution. For OGB-MOLPCBA, we pick r by multiplying the original learning rate by factor of 10 instead of 4096/32 = 128 (for ERM, which does not have grandient accumulation), because we found that the latter led to unstable optimization.
Iz-regularization. used in WILDS 1.0.
Across all datasets and methods, we used the same Lerregularization strengths
Ratio of unlabeled to labeled data in batch. For all the domain-invarian and self-training methods, we search over the ratio of unlabeled to labeled data in a batch, using the values {3 1, 1,15 1).
Number of epochs. We defined an epoch a complete pass over the labeled data. This means that the number of batches gradient steps taken per epoch varies with the ratio of unlabeled to labeled data We adjusted the number of epochs accordingly so that the total amount of compute was similar regardless of the ratio of unlabeled to labeled data batch. We allocated roughly twice as much compute (i.e., processing twice as many batches) to methods that used unlabeled data, compared to the supely-supervised ERM baseline. Overall, we set the number of epochs based on the WILDS 1.0 upwards adjustments (due to the different batch sizes and the of unlabeled
batch, a higher ratio means that each batch contains fewer labeled examples.
defaults, with data) we found that the best hyperparameter configuration had not converged on the validation set. Table 14 shows the total number of epochs used per dataset.
45

---

Published as a conference paper at ICLR 2022
D.5 ALGORITHM-SPECHEE HYPERPARAMETERS
We tuned the following algorithm-specife hyperparameters:
CORAL. We searched over penalty weights 10"(-,,))
We searched over penalty weights 10"(-1,1) and have separate learning rates for the fea- turizer, classifier and domain discriminator. We tuned the learning rate for the classifier and domain discriminator, then fixed the learning rate of the featurizer to be tenth of the learning rate of the classifier.
DANN.
Pseudo- Leeudo-Label, FixMatch, and Noisy Student. We fixed the penalty weight to be 1. For FixMatch and Pseudo-Label, we searched over confidence thresholds (0.7, 0.95). Noisy Student does not use confidence threshold.
SwAV. We did not tune SwAV hyperparameters. See Appendix B.4 for a description of the default hyperparameters used.
Masked language modeling. We did not tune masked LM hyperparameters, opting instead to use default hyperparameters. For both COMMLNC WILLD and AMAZON WILDS, we pre-trained DistilBERT for 1, 1,000 steps with a learning rate of 10 and a batch size of 8, 192 sequences using gradient accumulation. Following WILDS defaults, CivilComments, we set the max sequence length to be 300 and for Amazon, 512. We used FP16 training to speed pretraining.
D.6 COMPUTE INFRASTRUCTURE
We ran experiments on a mix of NVIDIA GPUs: V100, K80, GeForce RTX, Titan RTX, Titan Xp. and Titan V. SwAV pre-training took approximately 3 days 4 V100 GPUs for each dataset, while masked LM pre-training took approximately days for single GPU for each dataset. The other algorithms took less than day on V100 to run. The runtime estimates in Section for V100 GPUs. We used the Weights and Biases platform (Biewald, 2020) to monitor experiments.
estimated
46

---

Published as a conference paper at ICLR 2022
E EXPERIMENTS ON DOMAINNET
Prior work has shown that coomain-nvvriian self-training, and self-supervised methods can perform well on standard benchmarks for unsupervised domain adaptation. In this section, we describe our experiments DomainNet (Peng et al., 2019), standard unsupervised domain adaptation benchmark for object recognition. Our goal was to verify that our training/tuning protocol and our implementations of the methods benchmark in Section 6 6-which differ slightly from prior work in the ways described in Section B B-still result in models that can perform well on DomainNet. Consistent with prior work, the methods benchmark in Section 6, with the exception of CORAL,, all improve over standard ERM training in our DomainNet experiments.
E.1 SETUP
DomainNet is an object recognition dataset with approximately 600,000 images across six different domains: sketch, real, quickdraw. painting, infograph and clipart (Peng et al., 2019). Typically, one of these domains is selected as the source, and another domain as the target for evaluation. In our experiments, we use the real - sketch setting for two reasons: it is a common choice prior work on DomainNet, and as our models are pre-trained on ImageNet (following (Zhang et al., 2021)), we wanted to use the real domain as the source to be consistent with the realistic photographs used for ImageNet pretraining. While it is common to evaluate methods multiple pairs of source and target domains in DomainNet, in our experiments we only chose one pair, as our goal was only to verify consistency with prior results.
Data. The DomainNet dataset includes train and test splits for each of the domains, with 70%- 30% split between train and test examples. The real domain has 172,947 images total: 120,906 images in the train split and 52,041 images in the test split. The target domain, sketch, has total of 69,128 images: 48,212 in the train split and 20,916 images are in the test split. We used this data in the following way:
1. For training, we used the source training examples (with labels) and the target training examples (without labels).
2. For validation, we used the same set of target training examples, but with labels; this overestimates performance in a true domain adaptation setting (where one would not have labeled target data), but it is a common practice in the literature, and we followed it for consistency with Jiang et al. (2020) and Zhang et (2021).
3. For evaluation, we used the source test examples as the in-distribution test set, and the target test examples the out-of-distribution test set.
Hyperparameters and other details. Other experimental details followed our main experiments in Section 6. We used the strong and weak data augmentation described for image classification in Appendix C. We set the total batch size to 96, which is the maximum that can fit 12GB of GPU memory. We tuned hyperparameters with the protocol described in Section D.4. Specifically, 10-4 We then randomly sampled learning rates
for all methods, we fixed Læ-regularization from (0 to train the ERM with data augmentation model. For all other models, we took the best learning rate that we found for the ERM with data augmentation model and searched over one order of magnitude lower and higher from it. As in Zhang al. (2021), used a ResNet- 50 model initialized by pretraining on ImageNet. For SWAV pre-training, instead of following the carly-stopping procedure in Appendix B.4, we trained for the full 400 epochs used in Caron et al. (2020) since the experiment finished relatively quickly compared to the larger WILDS 2.0 datasets.
E.2 RESULTS
Table 15 shows the results of our experiments real + sketch. The use of (strong) data augmen- tation improved ERM performance from 34.9% to 35.9%. All unsupervised adaptation methods except CORAL improved over ERM. We also tested the use of strong vs. weak augmentation for la- beled examples for both Pseudo-Label and FixMatch, and we found that using strong augmentation for the labeled examples improves performance.
47

---

Published as a conference paper at ICLR 2022
In-distribution (real)
Out-of-distrilutiion (sketch)
ERM (-data aug)
82.6 (0.0)
34.9 (0.2)
ERM
82.5 (0.3)
35.9 (0.3)
CORAL
79.1 (0.4)
33.6 (0.6)
DANN
77.8 (0.2)
39.4 (0.8)
Pseudo-Label
79.9 (0.2)
36.1 (0.4)
Pscudo-Label (weak aug)
79.9 (0.6)
32.0 (0.8)
FixMatch
80.8 (0.2)
50.2 (0.4)
FixMatch (weak aug)
80.1 (0.1)
49.3 (0.2)
Noisy Student
82.0 (0.3)
39.7 (0.2)
SwAV
79.0 (0.3)
38.2 (0.4)
Table 15: The in-distribution vs. out-of- distiribbitttti test performance each method on DomainNet (real + sketch). We also included the results of applying weak instead of strong augmentation labeled examples for Pseudo-Label and FixMatch. Parentheses show standard deviation across replicates.
For DANN, Pseudo-Label, and FixMatch, compared our results against the results reported in Zhang et al. (2021). Performance was similar for DANN (ours, 39.4%; theirs, 40.0%). For Fix- Match, implementation performs better (ours, 50.2%; theirs, 45.3%), this is partially due to use of strong instead of weak augmentation for the labeled data, which increases performance by 0.9%. For Pseudo-Label, our implementation performs worse (ours, 36. %;; theirs, 40.6%), and believe is due to variation in hyperparameter tuning.
For Noisy Student, Berthelot al. (2021) reported significantly lower numbers (ours, 39.7%; theirs, 32.6%). However, this is expected as they trained their models from scratch, whereas we used ImageNet--retrained models.
We were unable to find comparable results in prior work for CORAL and SwAV pretraining on the real sketch split. Prior work has shown that these methods can improve performance on other unsupervised adaptation datasets (Sun & Saenko, 2016; Shen et al., 2021). ), On our DomainNet experiments, found that SwAV pre-training did improve performance over ERM, though CORAL did not (Table 15).
48

---

Published as a conference paper at ICLR 2022
F FULLY-LABELED ERM EXPERIMENTAL DETAILS
The self-training methods we evaluate in Section 5 generate pseudolabel it for each unlabeled example Ty and then train on (zu, Bu) as if the pseudolabels were true labels. However, these pseudolabels may not be accurate. In this section, we describe how we ran fully-labeled ERM experiments using ground truth labels the "unlabeled" data to establish informal upper bounds on how well we might expect a standard self-training approach perform with perfect pseudolabel accuracy.
For four of our datasets (AMAZON- -ILLD, CIVI.COMMENTS WILDS, WILDCCMM000-WIILD and FMOW-WILDS), we curated the "unlabeled" data by taking labeled examples and discarding the ground truth labels. For example, all 268,761 of the unlabeled target reviews in AMAZON- WILDS actually have associated star ratings; these available in data loaders, but in our main experiments we treat these reviews as unlabeled by not loading the star ratings. We evaluated models trained via empirical risk minimization (ERM) the combination of the standard labeled training set and the unlabeled data with these hidden labels revealed. For example, in SLAZON-WIIDDD we pool together the labeled source examples as well the unlabeled target examples with ground truth labels, and evaluate ERM models trained on all of that data. As with all of the other experiments in this paper, we evaluate test performance for all datasets on the labeled target splits, so no point are we training on our actual test examples.
F.1 HYPERPARAMETERS
Pooling labeled and unlabeled data. For all datasets, we pooled labeled source examples with examples from the same "unlabeled" split as in our main experiments (Table 2). We computed gra- dients for labeled minibatches and unlabeled minibatches separately, which means that for models using batch normalization, the labeled and unlabeled data were normalized separately. However, we fixed the labeled to "unlabeled" batch size ratios to match the overall labeled to unlabeled dataset size ratio, so other than the batch normalization effects, the training procedure can be viewed as running ERM the pooled labeled and "unlabeled" data.
Number of epochs. With the exception of WWI.DCAM2020-WI..D detailed below, we followed the procedure Appendix D.4 adjust the number of epochs based on the labeled unlabeled batch size ratios. This resulted in similar amount of computation allocated to these fully-labeled ERM experiments as the other experiments in Table 2.
Other details. Other experimental details were kept similar to the other experiments in the pa- per. Specifically, we tuned each experiment by randomly sampling different hyperparameters within the ranges defined in Appendix D.4; the only hyperparameter was the learning rate. We early stopped and selected the best hyperparameters based on the OOD validation performance, and then replicates using the best hyperparameters. We also used data augmentation for CWILDCAM2020-- WILDS and FMOW-WILDS but not for AMAZON- WWI.DS and CIVI.COMMENTS-WIII
tuned in these experiments
F.2 DATASET-SPECIRG DETAILS
A AMAZON-WILDS. We matched the experiments in Table by training on the unlabeled target data (268,761 examples). In addition, we ran a separate experiment where we trained on the unlabeled extra data instead of the unlabeled target data, the former has 10x the number of examples (2,927.88 841 examples). However, this did not improve performance. Using the unlabeled target data, we obtained an average accuracy of 73.6 (+= 0.1) and 10th percentile accuracy of 56.4 (#= 0.8), whereas using the unlabeled extra data, we obtained an average accuracy of 73.1 (#: 0.1) and a 10th percentile accuracy 54.7 (+t 0.0).
CIVIL COMMETEMNNTS-WIL We used the unlabeled extra split (1,551,515 examples). As in our other experiments on CIVILCOMMENTS WILDS, we accounted for label imbalance by sampling class-balanced labeled and "unlabeled" batches during training.
49

---

Published as a conference paper at ICLR 2022
We used the unlabeled extra split. Out of the 819, 120 unlabeled extra examples, 108,452 examples have ground truth labels (animal species) that are not present in the labeled training and test sets, we omitted those examples and trained the remaining 710,668 examples. We found that we required twice as many epochs compared to the other unlabeled meth- ods for the fully-laheled ERM training to converge, experiments.
WWLLDCAM2020-WILD
doubled the amount of compute allocated
to the fully-labeled WILDCAMM020-WLLL
FMOW-
We used the unlabeled target split (173,208 examples).
50

---

Published as a conference paper at ICLR 2022
G USING THE WILDS LIBRARY WITH UNLABELED DATA
We have extended the existing WILDS library (Koh et al., 2021) to add data loaders for each of the datasets with unlabeled data. These data loaders are compatible with the WILDS 1.0 APIs, allowing the unlabeled data to be accessed in a similar way to the labeled data:
>>> from wilds import get_dataset
from wilds. wonmon. loaders import get_train, Let__tainn__oo__ loader
>>> import torchvision. transsforrss as transforms
Load the labeled data
>>> dataset get get_gatett dataset(dataset=="fooo download=True)
labeled_subsed subset dataset. seraset.get_subeett gret_subset("traiin transsor ToTensor()
data_loader
pet_train__oader ("staddntta""dddrr"""" labeled labeled_subset, patch_size-1))
8 Load the unlabeled data
>>> dataset = get get__etaseee ddttaset(dattset=="fmo"" unlabeled-True, download=True)
>>> unlabeled_subsed
>>> unlabeled_dee data moader crrasst.get_subseet, sussssuuusettttttttets tesst__ullaeeedd transformstransfomm (ooterans
dataset.get, trriit_oaadee"""""
unlabeled subset, batch_ size=64)
Train loop
>>> for labeled_ batch, unlabeled batch batch in (oooddat_loader unlabeled data :ata_]o
x, y, metadata = labeled batch
unlabeled_x,
_etaddtaae unlabeled batch
Figure 3: Example of data loading for both labeled and unlabeled data.
As in the existing WILDS library, data downloading is automated. In addition, implemented CORAL, DANN, Pseudo-Label, FixMatch, and Noisy Student using the existing WILDS interfaces. This allows developers easily extend these algorithms and evaluate them in standardized way on all of the WILDS datasets with unlabeled data. The WILDS repository also contains scripts for masked language model pre-training and for SwAV pre-training, which uses modified version of the public SWAV repository that can interface with the WILDS data loaders.
51


Reviews:
[{"review_id": "iU0Yv_qVtGX", "reviewer": "Reviewer_3NYa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) The problem of unsupervised domain adaptation is well-grounded in a practical learning setting where labels are scarce, but observations are not.  This makes the extension of the WILDS data set to this setting a very useful contribution that can facilitate impactful future work.\n2) The empirical evaluation was fairly extensive in it's inclusion of a variety of unsupervised domain adaptation techniques.  I think it is important to validate the trend established in prior work that these techniques typically perform poorly in more realistic domain adaptation problems.  I also think the finding that data augmentation techniques work relatively well in image domains to be an interesting one that can motivate future work.\n3) Perhaps a necessary consequence of this kind of work is that the paper serves as a nice survey of modern unsupervised domain adaptation, both in terms of data sets and methods.  Someone interested in the topic could use this paper to begin a literature search.\n\nWeaknesses:\n1) Ultimately, the novelty of this work is low.  The main novel contributions are the extension to the existing WILDS data set (using already established data sets used by the original WILDS authors) and some new empirical results using previously published results.  However, I do not think for this kind of work that novelty is paramount.  \n2) The descriptions of the data sets are relatively shallow.  Besides raw increase in instances, not much else is reported about the data sets.  Is there anything interesting to say about the instances that were chosen to be labeled by the original WILDS authors versus the ones added by U-WILDS?  Namely, is there anything unique about the new instances?  Can you characterize and measure how different the new instances are from the original ones?\n3) Data preparation was not discussed at all.  The major practical concern I have about this endeavor is standardization across new and old instances.  If the original WILDS data set was released with some effort to process the data before repackaging and release, the U-WILDS data set would need to undertake the same process.  I would like to see some comments on whether effort was taken to ensure that the U-WILDS instances were preprocessed for consumption in the same way the WILDS data set were, or why that was not a necessary step.", "summary_of_the_paper": "The authors introduce an extension to the new, but popular, data shift benchmark WILDS data sets called U-WILDS.  U-WILDS increases the number of instances in these data sets significantly (by a factor 3.5-14x times, depending on the data set), but includes no additional labels.  Such an extension allows unsupervised domain adaptation techniques to now be evaluated on the various WILDS data sets.  The authors evaluate a sampling of the state of the art in unsupervised domain adaptation on these new data sets.  The main finding is that most of the techniques performed poorly, except data augmentation in the image problems.  The authors suggest that this could motivate the need for better data augmentation techniques for other modalities.", "summary_of_the_review": "While I recognize the low novelty of the work, I believe the contribution of the extended WILDS data set as well as the insights provided in the empirical evaluation are of significant enough value to the machine learning community to warrant acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PKvZJN4zxe7", "reviewer": "Reviewer_RpU9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths \n\n- The dataset is a useful addition to the WILDS dataset, and the provided unlabeled data would be very useful to design many unsupervised generalization algorithms.\n- The paper is very well written, easy to understand and well organized, although the experimental evaluation could be explained in greater detail in main paper.\n\n### Required Clarifications\n\n- The models chosen for benchmarking the methods are not suitable. While domain adaptation methods like DANN and CORAL work strictly with an assumption of single source to target domains, semi-supervised and self-supervised works are well-known to work only for within-domain samples. In this respect, the observations found with respect to the benchmarking, although useful, aren't surprising. \n- self-supervised, semi-supervised and DA methods make different assumptions about the availability of target data. self-supervised requires target labels for fine-tuning, while DA methods only require unlabeled target data during training and nothing more. So, comparing both of them together using a common yard-stick is, in my opinion, not appropriate.  \n- The main paper needs to have more experimental detail such as what are the exact labeled and unlabeled data from source and target for each of the experiments.\n- The DA methods considered are quite old and primitive. The authors are encouraged to consider more recent benchmarks like CDAN, CAN, AFN etc.\n- The authors are also encouraged to use multi-source/multi-domain adaptation benchmarks [1,2,3] which seem most appropriate for the dataset proposed. at least for the visual recognition datasets. While self-supervised and semi-supervised methods aren't particularly designed keeping in mind cross-domain transfer, UDA methods like DANN and CORAL aren't designed to handle multi-domains. \n- The authors are also encouraged to compare and contrast with [1] in terms of the datasets proposed in both papers (although these can be considered contemporary works).\n- The key takeaways from sec 7 are not that surprising, and things like lack of augmentation strategies for many modalities and model selection for DA  are already well known to the community, while ineffectiveness of pre-training seems interesting.\n\n1. Dubey, Abhimanyu, et al. \"Adaptive Methods for Real-World Domain Generalization.\" _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2021.\n2. Li, Ya, et al. \"Deep domain generalization via conditional invariant adversarial networks.\" _Proceedings of the European Conference on Computer Vision (ECCV)_. 2018.\n3. Li, Da, et al. \"Deeper, broader and artier domain generalization.\" _Proceedings of the IEEE international conference on computer vision_. 2017.", "summary_of_the_paper": "### New large scale dataset for unsupervised transfer learning\n\nThe paper proposed an extension to the popular WILDS benchmark dataset by augmenting the different domain data with additional unlabeled examples. The dataset consists of data of various modalities including images, graph and text from various domains. Additionally, many recent methods that leverage unlabeled images to achieve generalization are bench marked on the proposed datasets. ", "summary_of_the_review": "Although the dataset itself is very useful and important, the analysis and bench marking needs some improvement. Nevertheless, this paper definitely has merit, and if the authors could clarify the questions raised, I would be happy to update the score. I haven't gone through the supplementary material in detail. If any of the questions above have a direct answer in the suppl. material, the authors can directly point to that and I would be happy to update my comments. \n\nPost Rebuttal \n****************\n\nI thank the authors for answering all my queries patiently, which answered most of the questions I had regarding suitability of self-supervised and semi-supervised algorithms to the task. I would like to raise my score, and suggest the authors to include these discussions in the main paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BNOqXSBCqZc", "reviewer": "Reviewer_NvLs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The work behind the data collection and baseline computation appears sizeable and very valuable\n- The authors implemented many of the top-performing algorithms meant to address domain shift with unlabelled data. They provide their implementation, as well as a unified, didactic, and detailed explanation.\n- The conclusions are both measured and essential: we lack a definitive answer to domain shift in the wild, and unlabelled only provides a very partial solution.\n- The writing is remarkably clear and to the point.\n\nWeakness:\nI am struggling to fault the paper.  The extensive appendix answered most of my questions.\n- The paper does not have a methodological contribution per se, but the answers provided from the meta-analysis are new - at least to me, even if the fact that current methods would not hold too well for in-the-wild data was suspected\n- I have some questions about missing details, detailed below\n\nQuestions:\n- \"Models are trained on labeled data from the source domains, as well as unlabeled data from one or more of the other sources, depending on what is realistic for the application.\" I couldn't understand which problems were allowed to use OOD unlabelled data for training and which ones were not. From my understanding, unlabelled data from all domains (source, val, target, extra) are merged and used for training; is this correct? And what would be the use of unlabelled data if not for helping the training?\n- As a follow-up, are unlabelled data from all domains (ID/OOD) used identically by all methods? DANN, for example, may work better if only using target domain unlabelled data, if available.\n- The justification to not use augmentation on GlobalWheat are weak: translating and rotating bounding boxes seems easy enough?\n- It seems that you did not try approaches that explicitly use the domain as weak labels or adversarially, such as \"Adversarial Multiple Source Domain Adaptation, Zhao et al. NeurIPS2018. I am not suggesting that you do (the comparison is already more than sufficient), but you should mention these approaches. Note that these would only work when the domains are not too numerous and meaningful.\n\nDetails:\nThe domains of PovertyMap are missing in Fig2", "summary_of_the_paper": "The authors present U-WILDS, an extension of the multi-task, large-scale domain-shift dataset WILDS. They provide a large quantity of unlabelled data complementing 8 of the existing multidomain labeled datasets in WILDS. They propose an extensive array of experiments evaluating the ability of a wide variety of algorithms to leverage the unlabelled data to address domain-shift. They present reasoned conclusions, and open-source datasets and implementations.\n", "summary_of_the_review": "An impressive and exemplary dataset paper. The data collection and baseline implementation could be an important stepping stone for the ML community to address its biggest challenge yet: domain shift in the Wild. This paper highlights that we are not there yet, and that unlabelled data give an encouraging venue that has not reached maturity yet.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7uvHeLm_KF", "reviewer": "Reviewer_Dxvt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem is well-motivated, and the use of realistic problems to benchmark unsupervised adaptation methods presents a major gain over prior datasets which rely on different stylized images (e.g. PACS). The datasets used span a wild variety of modalities and domains, and are fairly robust with large sample sizes. The paper is easy to read and easy to follow, and the experimental evaluations are robust.\n\nI would suggest the following improvements:\n\n- In order to present an informal upper bound for the performance of algorithms on the OOD domain, the authors should consider adding in two additional \"oracle\" baselines which have access to labelled OOD-domain data: 1) training an ERM model only on the OOD domain, 2) training an ERM model on all available data. \n\n- The authors should justify why the OOD domain was the one selected in each dataset. If possible, the authors could consider allowing for the ability to swap around the various training/validation/test domains, though it might be impractical if some domains do not have sufficient data.\n\n- The authors should clarify the model selection procedure for each of the algorithms. For example, how does each method make use of the Validation (ID) labelled examples versus the Validation (OOD) unlabelled examples for model selection?", "summary_of_the_paper": "The authors propose U-WILDS, which extends the WILDS benchmark (typically used for domain generalization or subpopulation shift) to the unsupervised domain adaptation scenario. They select eight datasets from WILDS, spanning a variety of data modalities, and add in additional unlabelled examples from a variety of data sources. They benchmark a comprehensive set of algorithms which make use of unlabelled data, and find that most methods do not significantly outperform ERM, except for some limited cases which the authors characterize in detail.", "summary_of_the_review": "Though the paper does not propose any novel methodology, I believe that it is a solid step towards the use of real-world datasets for benchmarking unsupervised domain adaptation algorithms, and would be a valuable contribution to the conference. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents U-WILDS, an extension of the multi-task, large-scale domain-shift dataset WILDS. The authors propose an extensive array of experiments evaluating the ability of a wide variety of algorithms to leverage the unlabelled data to address domain-shift problems. The vision behind sounds quite ambitious and convincing to me, namely, the proposed U-WILDS benchmark would be a useful and well-motivated resource for the ML community, and their experiments were very comprehensive. Although they did not introduce any new methods in this paper, U-WILDS significantly expands the range of modalities, applications, and shifts available for studying and benchmarking real-world unsupervised adaptation.

The clarity, vision and significance are clearly above the bar of ICLR. While the reviewers had some concerns on the novelty, the authors did a particularly good job in their rebuttal. Thus, all of us have agreed to strongly accept this paper for publication! Please include the additional rebuttal discussion in the next version.
Decision: Accept (Oral)