RLSBench: A Large-Scale Empirical Study of Domain Adaptation Under Relaxed Label Shift
Authors: Saurabh Garg
Nick Erickson
James Sharpnack
Alex Smola
Sivaraman Balakrishnan
Zachary Chase Lipton
Venue: Submitted to ICLR 2023 (2023)
Keywords: domain adaptation
distribution shift
label shift
large scale study

Abstract: Despite the emergence of principled methods for domain adaptation under label shift (where only the class balance changes), the sensitivity of these methods to natural-seeming covariate shifts remains precariously underexplored. Meanwhile, popular deep domain adaptation heuristics, despite showing promise on benchmark datasets, tend to falter when faced with shifts in the class balance. Moreover, it's difficult to assess the state of the field owing to inconsistencies among relevant papers in evaluation criteria, datasets, and baselines. In this paper, we introduce \textsc{RLSbench}, a large-scale benchmark for such \emph{relaxed label shift} settings, consisting of 11 vision datasets spanning $>$200 distribution shift pairs with different class proportions. We evaluate 12 popular domain adaptation methods, demonstrating a more widespread susceptibility to failure under extreme shifts in the class proportions than was previously known. We develop an effective meta-algorithm, compatible with most deep domain adaptation heuristics, that consists of the following two steps: (i) \emph{pseudo-balance} the data at each epoch; and (ii) adjust the final classifier with (an estimate of) target label distribution. Furthermore, we discover that batch-norm adaption of a model trained on source with aforementioned corrections offers a strong baseline, largely missing from prior comparisons. We hope that these findings and the availability of \textsc{RLSbench} will encourage researchers to include rigorously evaluate proposed methods in relaxed label shift settings. Code is publicly available at https://github.com/ICLR2023Anon.



Under review as a conference paper at ICLR 2023
RLSBENCH: A LARGE-SCALE EMPIRICAL STUDY OF DOMAIN ADAPTATION UNDER RELAXED LABEL SHIFT
Anonymous authors
Paper under double-blind review
ABSTRACT
Despite the emergence of principled methods for domain adaptation under label shift (where only the class balance changes), the sensitivity of these methods to natural-seeming covariate shifts remains precariously underexplored. Meanwhile, popular deep domain adaptation heuristics, despite showing promise on bench- mark datasets, tend to falter when faced with shifts in the class balance. More- over, it's difficult to assess the state of the field owing to inconsistencies among relevant papers in evaluation criteria, datasets, and baselines. In this paper, we introduce RLSBENCH, a large-scale benchmark for such relaxed label shift set- tings, consisting of 14 datasets across vision, tabular, and language modalities spanning 500 distribution shift pairs with different class proportions. We eval- uate 13 popular domain adaptation methods, demonstrating a more widespread susceptibility to failure under extreme shifts in the class proportions than was pre- viously known. We develop an effective meta-algorithm, compatible with most deep domain adaptation heuristics, that consists of the following two steps: (i) pseudo-balane -alannce the data at each epoch; and (ii) adjust the final classifier with (an estimate of) target label distribution. In our benchmark, the meta-algorithm im- proves existing domain adaptation heuristics often by 2-10% accuracy points when label distribution shifts are extreme and has minimal (i.e., <0.5%) no effect on accuracy in cases with no shift in label distribution. We hope that these findings and the availability of RLSBENCHI will encourage researchers to rigorously evalu- ate proposed methods in relaxed label shift settings. Code is publicly available at wtt.c/////I///11.....2..
INTRODUCTION
Real-world deployments of machine learning models are typically characterized by distribution shift, where data encountered in production exhibits statistical differences from the available training data (Quinonero-Candee
et al.) 2008 Torralba & Efros 2011 Koh et al.) 2021). Because continually labeling data can be prohibitively expensive, researchers have focused on the unsupervised domain adaptation (DA) setting, where only labeled data sampled from the source distribution and unlabeled from the targes distribution are available for training.
Absent further assumptions, the DA problem is well known to be underspecified (Ben-David David et al.) 2010b) and thus no method is universally applicable. Researchers have responded to these challenges in several ways. One approach is to investigate additional assumptions that render the problem well- posed. Popular examples include covariate shift and label shift, for which identification strategies and principled methods exist whenever the source and target distributions have overlapping support (Shi) modaira 2000 Schölkopf al., 2012 Gretton et al. 2009). Under label shift in particular, recent research has produced effective methods that are applicable in deep learning regimes and yield both consistent estimates of the target label marginal and principled ways to update the resulting clas- sifier (Lipton et al. 2013 & Alexandari et al.) 202 I Azizzadenesheli et al. 2019; Garg et al. 2020). However, these assumptions are typically, to some degree, violated in practice. Even for archetypat cases like shift in disease prevalence (Lipton al. 2018), the label shift assumption can be violated. For example, over the course of the COVID-19 19 epidemic, changes in disease positivity have been coupled with shifts in the age distribution of the infected and subtle mutations of the virus itself.
complementary line of research focuses on constructing benchmark datasets for evaluating meth- ods, in the hopes of finding heuristics that, for the kinds of problems that arise in practice, tend

---

Under review as a conference paper at ICLR 2023
to incorporate the unlabeled target data profitably. Examples of such benchmarks include Office-
Home (Venkateswara et al., 2017), Domainnet (Peng et al., 2019)). WILDS (Sagawa et al.
2021).
However, most academic benchmarks exhibit little no shift in the label distribution ply).
Conse-
quently, benchmark-driven research has produced a variety of heuristic methods (Ganin et al.
2016
Sohn et al., 2020; Wang et al.)
2021 Li et al. 2016) that despite yielding gains in benchmark perfor- mance tend to break when p p(y) shifts. While this has previously been shown for ramain-adversarian
methods (Wu et al. 2019 Zhao et al. 2019), we show that this problem is more widespread than previously known. Several recent papers attempt to address shift in label distribution compounded by natural variations in p(zly) (Tan et al., 2020 Tachet des Combes et al. 2020 Prabhu al., 2021). However, the experimental evaluations are hard to compare across papers owing to discrepancies in how shifts in p(y) are simulated and the choice of evaluation metrics. Moreover, many methods violate the unsupervised contract by peeking at target validation performance during model selec- tion and hyperparameter tuning. In short, there is a paucity of comprehensive and fair comparisons between DA methods for settings with shifts in label distribution.
In this paper, we develop RLSBENCI,, a standarized test bed of relaxed label shift settings, where p(y) can shift arbitrarily and the class conditionals p(zly) can shift in seemingly natural ways (following the popular DA benchmarks). We evaluate a collection of popular DA methods based on domain-invariant representation learning, self-training, and test-time adaptation across 14 multi- domain datasets spanning vision, Natural Language Processing (NLP), and tabular modalities. The different domains in each dataset present a different shift in p( ly). Since these datasets exhibit minor to no shift in label marginal, we simulate shift in target label marginal via stratified sampling with varying severity. Overall, we obtain 560 different source and target distribution shift pairs and train > 30k models in our testbed.
Based on our experiments on RLSBENCH suite, we make several findings. First, we observe that while popular DA methods often improve over a source-only classifier absent shift in target label distribution, their performance tends to degrade, dropping below source-only classifiers under severe shifts in target label marginal. Next, we develop a meta-algorithm with two simple corrections: (i) sampling the data to balance the source and pscudo-balance the target; (ii) re-weighting the final classifier using an estimate of the target label marginal. We observe that in these relaxed label shift environments, the performance of existing DA methods (e.g. CDANN, FixMatch, and BN-adapt) source-only classifier. On the other
when paired with our meta-algorithm tends to improve hand, existing methods specifically proposed for relaxed label shift (e.g., IW-CDANN and SENTRY) often fail to improve over source-only classifier and significantly underperform when compared to existing DA methods paired with our meta-algorithm.
Our findings underscore the importance of fair comparison to avoid false sense of scientific progress in relaxed label shift scenarios. Moreover, we hope that the RLSBENCH testbed and our meta-algorithm (that can be paired with any DA method) provide a framework for rigorous and reproducible future research in relaxed label shift scenarios.
2 PRELIMINARIES AND PRIOR WORK
We first setup the notation and formally define the problem. Let X' be the input space and y (1,2,..., k} the output space. Let P., P. X' x y - [0. 1] be the source and tar- get distributions and let ps and pt denote the corresponding probability density (or mass) func- tions. Unlike the standard supervised setting, in unsupervised DA, we possess labeled source data {(=1, W1), (I2, :,)...., (I }n)} and unlabeled target data MEn+1,nnt2: In+m). With f: - A k-1 we denote a predictor function which predicts ÿ = arg max, (1) on an input x. For vector u, we use By to access the element at index y.
In the traditional label shift setting, one assumes that p(x|y) does not change but that p(3) can. Under label shift, two challenges arise: (i) estimate the target label marginal pe(y); and (ii) train classifier if to maximize the performance on target domain. This paper focuses on the relaxed label shift setting. In particular, we assume that the label distribution can shift from source to target arbitrarily but that p(zly) varies between source and target in some comparatively restrictive way (e.g., shifts arising naturally in the real-world like ImageNet (Russakovsky et al. 2013) to ImageNetV2 (Recht et al. 2019).. Mathematically, we assume a divergence-basee based restriction p( ((xl)). That is, for some small E <> 0 and distributional distance D, we have max, (alp()))) p. & and allow an arbitrary
shift in the label marginal p(y). We discuss several precise instantiations in App. F However, in
2

---

Under review as a conference paper at ICLR 2023
practice, it's hard to empirically verify these distribution distances for small enough with finite samples. Moreover, we lack a rigorous characterization of the sense in which those shifts arise in popular DA benchmarks, and since, the focus of our work is on the empirical evaluation with real- world datasets, we leave a formal investigation for future work. While prior work addressing relaxed label shift has primarily focused on classifier performance, we also separately evaluate methods for estimating the target label marginal. This can be beneficial for two reasons. First, it can shed more light into how improving the estimates of target class proportion improves target performance. Second, understanding how the class proportions are changing can be of independent interest.
2.1 PRIOR WORK
Unsupervised domain adaption In our work, we focus on unsupervised DA where the goal is
to adapt predictor from source distribution with labeled data to a target distribution from which
we only observe unlabeled examples. Two popular settings for which DA is well-posed include
(i) covariate shifi (Zhang al., 2013 Zadrozny 2004 Cortes al. 2010: Cortes & Mohri 2014
Gretton et al. 2009) where p [x] can change from source to target but p(y] x:) remains invariant; and
(11) label shift (Saerens et al. 2002 Lipton al., 2018 Azizzadenesheli
al. 2019 Alexandari
et al. 2021: Garg et al. 2020 Zhang et al. 202 where the label marginal p(y) can change but
p(Zly g) is shared across source and target. Principled methods with strong theoretical guarantees
exists for adaptation under these settings when target distribution's 's support is subset of the source
support. Ben-David et al. (2110 a): Mansour et al. (2009); /Zhao et al. (2019): Wu al. (2019)
present theoretical analysis when the assumptions of contained support is violated. More recently,
massive literature has emerged exploring a benchmark-
heuristic approach (Long et al. 2015
2017 Sun & Saenko 2016 Sun et al., 2017 Zhang et al. 2019 2018 Ganin et al. [016] /Shnn et al., 2020 However, rigorous evaluation of popular DA methods is typically restricted to these carefully curated benchmark datasets where their is minor to no shift in label marginal from source to target.
Relaxed Label Shift Exploring the problem of shift in label marginal from source to target with natural variations in p(=ly)), few papers highlighted theoretical and empirical failures of DA methods based on coominn-advessan
neural network training (Yan et al. 2017: Wu et al.] 2019 Zhao et al., 2019). Subsequently, several papers attempted to handle these problems in
training (Tachet et af.) 2020; Prabhu et al.] 2021 Liu et al., 2021]] Tan et al.] 2020 Manders et al. 2019). However, these methods often lack comparisons with other prominent DA methods and are evaluated under different datasets and model selection criteria. To this end, we perform a large scale rigorous comparison of prominent representative DA methods in a standardized evaluation framework.
Domain generalization In domain generalization, the model is given access to data from multiple different domains and the goal is to generalize to a previously unseen domain at test time (Blanchard et al.] 2011 Muandet M et al. 2013). For a survey of different algorithms for domain generalization, we refer the reader to Gulrajani & Lopez-Paz (2020). crucial distinction here is that unlike the domain generalization setting, in DA problems, we have access to unlabeled examples from the test domain.
Distinction from previous distribution shift benchmark studies Previous studies evaluating ro- bustness under distribution shift predominantly focuses on transfer learning and domain generaliza- tion settings Wenzel et al. (2022); Gulrajani & Looezz-Paz (2020); Djolonga et al. (202T); Wiles et al.) (
(202T); Koh et al. (2021 Taori et al. (2020); Hendrycks al. (2021) studies the impact of robustness interventions (c.g. data augmentation techniques, adversarial training) on target (out of distribution) performance. Notably, (Saawa al. (202T) focused evaluating DA methods on WIL.SS2.00 an extended WILDS benchmark for DA setting. Our work is complementary to these studies, as we present the first extensive study of DA methods under shift in p(y) and natural variations in p(xlly).
3 RLSBENCH: A BENCHMARK FOR RELAXED LABEL SHIFT
In this section, we introduce RLSBENCH, a suite of datasets and DA algorithms that are at the core of our study. Motivated by correction methods for the (stricter) label shift setting (Saerens et al. 2002 Lipton et al., 2018)) and learning under imbalanced datasets (Wei al., 2021 Cao al. (019a). we also present meta-algorithm with simple corrections compatible with almost any DA method.
3.1 DATASETS
RLSBENCH builds on fourteen open-source multi-domain datasets for classification. We include tasks spanning applications in object classification, satellite imagery, medicine, and toxicity detection:
3

---

Under review as a conference paper at ICLR 2023
(i) CIFAR-10 which includes the original CIFAR-10 (Krizhevsky & Hinton 2009). CIFAR-10-
C (Hendrycks & Dietterich 2019) and CIFAR-10V2 (Recht et al. 2018); (i) CIFAR- 100 including
the original dataset and CIFAR-I0O-C: (iii) all four BREEDs datasets (Santurkar et al. 202T). i...
Entity13, En Nonliving26, Living17. BREEDs leverages class hierarchy in ImageNet [Rus] sakovsky al. 2015) to repurpose original classes be the subpopulations and define classifica- tion task on superclasses. We consider distribution shift due to subpopulation shift which is induced by directly making the subpopulations present in the source and target distributions disjoint. We also consider natural shifts induced due to differences in the data collection process of ImageNet, i.e. Ima-
geNetv2 (Recht et al. 2019) and a combination of both. (iv) OfficeHome (Venkateswara et al. (017) which includes four domains: art, clipart, product, and real: (v) DomainNet (Peng et al. 2019) where
we consider four domains: clipart, painting, real, sketch; (vi) Visda (Peng et al. 2018)) which contains
three domains: train, val and test; (vii) FMoW (Koh et al.] 202T Christie et al.) 20188) from WILDS benchmark which includes three domains: train, OOD val, and OOD with satellite images taken in different geographical regions and at different times; (viii) Camelyon (Bandi et al. 2018 from WILDS benchmark which includes three domains: train, OOD val, and OOD test, for tumor identifi- cation with domains corresponding to different hospitals; (ix) Civilcomments (Borkan et al. 2019)
which includes three domains: train, OOD val, and OOD test, for toxicity detection with domains cor- responding to different demographic subpopulations: (x) Retiring Adults (Ding al., 202 where we consider the ACSIncome prediction task with various domains representing different states and
time-period; and (xi) Mimic Readmission (Johnson et al. 2020 PhysioBank| 2000) where the task is to predict readmission risk with various domains representing data from different time-perion
Throughout the paper, we represent each multi-domain dataset with the name highlighted in the boldface above. Across these datasets, we obtain total of 56 different source and target pairs. We relegate other details about datasets in App. D For vision datasets, we show example images in Fig. 7
Simulating a shift in target marginal The above datasets present minor to no shift in label marginal. Hence, we simulate such shift by altering the target label marginal and keeping the source target distribution fixed (to the original source label distribution). Note that, unlike some previous studies, we do not alter the source label marginal because, in practice, we may have an option to carefully curate the training distribution but might have little to no control over the test data.
For each target dataset, we have the true labels which allow us to vary the target label distribution. In particular, sample the target label marginal from Dirichlet distribution with parameter a E (0.5, 1, 3.0, 10) multiplier to the original target marginal. Specifically, Pe()) ~ Dir(B) where By (v) and pv.o(y) is the original target label marginal. The Dirichlet parameter a controls the severity of shift in target label marginal. Intuitively, as a decreases, the severity of the shift increases. For completeness,
also include the target dataset with the original target label marginal. For ease of exposition, we denote the shifts as NONE (no external shift) in the set of Dirichlet parameters, i.c. the limiting distribution as a co. After simulating the shift in the target label marginal (with two seeds for each a),, we obtain 560 pairs of different source and target datasets.
3.2 DOMAIN ADAPTATION METHODS
With the current version of RLSBENCH, we implement the following algorithms (a more detailed description of each method is included in App
Source only As a baseline, we include model trained with empirical risk minimization (aannik 1999) with cross-entropy loss on the source domain. We include source only models trained with and without augmentations. We also include adversarial robust models trained on source data with augmentations (Source (adv)). In particular, we use models adversarially trained against fz-perturbations.
Domain alignment methods These methods employ domain-adversarial training schemes aimed to learn invariant representations across different domains (Ganin et al., 20166) /Zhang et al.) 2019
Tan et al.) 2020). For our experiments, we include the following five methods: Domain Adversarial Neural Networks (DANN (Ganin et al. (0166), Conditional DANN (CDANN (Long et al. 2018).
Maximum Classifier Diserepancy (MCD (Saito et al., 2018)). importance-reweighhee DANN and
CDANN (i.e. IW-DANN & IW-CDANN Tachet des Combes et al. (2020)).
Self-training methods These methods "pseudo-label" unlabeled examples with the model's own predictions and then train on them as if they were labeled examples. For vision datasets, these methods often also use consistency regularization, which encourages the model to make consistent predictions
4

---

Under review as a conference paper at ICLR 2023
Algorithm I Meta algorithm to handle shift in class proportions
input Source training and validation data: (Xs, Ys) and (X's, Yg), unlabeled target training and validation data: Xr and Xy. classifier and DA algorithm A
1: Xs, Ys- < SampleClasBallnccddXX.
Balance source data
2: for t = 1 to T do
3: Y1 arg max,, (X.r)
4: XT - SampleCCassBBllaccdd..X7 Pr.)
>> Pseudo balliicbbaaceeeee target data 5: Run epoch of A to update f balanced source data (Xs, Ys) and target samples (Xr)
6: end for
7: Estimate target marginal (ey))= EstimaceLabel/Mrginnal X Ygg X7))
PL(y j) fj Σn k) for all to Re-eweight predictor with estimated label marginal
8:
output Target label marginal (y) and classifier f'
augmented views of unlabeled examples (Lee al. 2013 Xie al. 2020; Berthelot et al.
2021)
We include the following three algorithms: FixMatch (Sohn et al. (020). Noisy Student (Xie et al. 2020), Selective Entropy Optimization via Committee Consistency (SENTRY (Prabhu et 2021
For NLP and tabular dataset, where we do not have strong augmentations defined, we consider
PseudoLabel algorithm (Lee et al. 2013).
Test-time adaptation methods These methods take a source-trained model and adapt few parameters (e.g. batch norm parameters, batch norm statistics) on the unlabeled target data with an
aim to improve target performance. We include the following methods: CORAL (Sun et al. (206)) or Domain Adjusted Regression (DARE (Rosenfeld et al. 2022)) BatchNorm adaptation (BN-adapt (Li
al. 2016 Schneider et al. 2020)), Test entropy minimization (TENT [Wang et al. 2021
3.3
META ALGORITIIM TO HANDLE SHIFTS IN TARGET CLASS PROPORTIONS
Here we discuss two simple general- purpose corrections that we implement in our framework. First, note that, as the severity of shift in the target label marginal increases, the performance of DA methods can falter as the training is done over source and target datasets with different class proportions. Indeed, failure of domain adversarial training methods (one category of deep DA methods) has been theoretically and empirically shown in the literature (Wu et al.] 2019) Zhao et al.] 2019). In
our experiments, we show that a failure due shift in label distribution is not limited domain adversarial training methods, but is common with all the popular DA methods (Sec. 4
Re-sampling To handle label imbalance in standard supervised learning, re-sampling the data to balance the class marginal is a known successful strategy (Chawla et al. 2002 Buda et al. 2018 Cao al. 2019b). In relaxed label shift, we seek to handle the imbalance in the target data (with respect to the source label marginal), where we do not have access to true labels. We adopt an alternative strategy of leveraging pseudolabels for target data to perform pseudo class-balanced re- re-sampling! [I (Zou et al.) 2018| et al. (222). For relaxed label shift problems, (Prabhu et al.] (221) employed this technique with their committee consistency objective, SENTRY. However, they did not explore re-sampling based correction for existing DA techniques. Since this technique can be used in conjunction with any DA methods, we employ this re-sampling technique with existing DA methods and find that re-sampling benefits all DA methods, often improving over SENTRY in our testbed (Sec. 4
Re-weighting With re-sampling, we can hope to train the classifier f on mixture of balanced source and balanced target datasets in an ideal case. However, this still leaves open the problem of adapting the classifier f to the original target label distribution which is not available. If we can estimate the target label marginal, correction (Lipton et al. 2018 Alexandari et al., 2021). turn to techniques developed under the stricter label shift assumption (recall, the setting where p(zly) (y) remains domain invariant). These approaches leverage off the-shelf classifiers to estimate
post-hoc adapt the classifier ff with simple re-weighting I). To estimate the target label marginal, we
A different strategy here could he to re-sample target pseudolabel marginal to match source label marginal. For simplicity, in work, we choose to balance source label marginal and balance target pseudolahel marginal.
5

---

Under review as a conference paper at ICLR 2023
target marginal and provide O(1/vn) convergence rates under the label shift condition with mild assumptions on the classifier (Lipoon et al. 2018: Azizzadenesheli et al. 2019; Garg et al. 2020).
While the relaxed label shift scenario violates the conditions required for consistency of label shift cstimation techniques, we nonetheless employ these techniques and empirically evaluate efficacy of these methods in our testbed. In particular, to estimate the target label marginal, we experiment with: (i) RLLS Azizzadenesheli et al. 2019); (ii) MILS (Alexandari et al., 3 and (iii) baseline estimator that simply averages the prediction of a classifier f on unlabeled target data. We provide precise details about these methods in App. E Since these methods leverage off- off-the-shelf classifiers, classifiers obtained with any DA methods can be used conjunction with these estimation methods.
Summary Overall, in Algorithm[] we illustrate how to incorporate the re-sampling and reiieighting correction with existing DA techniques. Algorithm AA can be any DA method and in Step 7, we can use any of the three methods listed above to estimate the target label marginal. We instantiate Algorithm T with several algorithms from Sec. .3.2 App. Intuitively, in an ideal scenario when the re-sampling step in our meta-algorithm perfectly corrects for label imbalance between source and target, we expect DA methods to adapt classifier f to p(aly) shift, The re- re-weighting step in our meta-algorithm can then adapt the classifier to the target label marginal Po()). We emphasize that in our work, we do not claim to propose these corrections. But, to the best of our knowledge, our work is the first to combine these two corrections together in relaxed label shift scenarios and perform extensive experiments across diverse datasets.
3.4 OTHER CHOICES FOR REALISTIC EVALUATION
For fair evaluation and comparison across different datasets and domain adaptation algorithms, we re-implemented all the algorithms with consistent design choices whenever applicable. We also make several additional implementation choices, described below. We defer the additional details to App. L
Model selection criteria and hyperparameter choices Given that we lack validation i.i.d data from the target distribution, model selection in DA problems not follow the standard workflow used in supervised training. Prior works often omit details on how to choose hyperparameters leaving open a possibility of choosing hyperparameters using the test set which can provide false and unreliable sense of improvement. Moreover, inconsistent hyperparameter selection strategies can complicate fair evaluations mis-associating the improvements to the algorithm under study.
In our work, we use source hold-out performance to pick the best hyperparameters. First, for & regularization and learning rate, we perform sweep over random hyperparameters to maximize the performance of source only model on the hold-out source data. Then for each dataset, we keep these hyperparamettrs fixed across DA algorithms. For DA methods specific hyperparameters, we use the same hyperparameters across all the methods incorporating the suggestions made in corresponding papers. Within a run, we use hold out performance on the source to pick the early stopping point. In appendices, we report oracle performance by choosing the early stopping point with target accuracy.
Evaluation criteria To evaluate the target label marginal estimation, we report & error between the estimated label distribution and true target label distribution. To evaluate the classifier performance on target data, we report performance of the (adapted) classifier on a hold-out partition of target data.
Architectural and pretraining details We experiment with different architectures (e.g.. Den1eNet121, Resenet18, Resnet50, DistilBERT, MLP and Transformer) across different datasets. We experiment with randomly- -initall-ii models and Imagenet, and DistillBert pre-trained models. Given a dataset, we use the same architecture across different DA algorithms.
Data augmentation Data augmentation is standard ingredient to train vision models which can help approximate some of the variations between domains. Unless stated otherwise, we train all the vision datasets using the standard strong augmentation technique: random horizontal flips, random crops, augmentation with Cutout (DeVsies & Taylor, 2017). and RandAugment (Cubuk et al., 2020). To understand help with data augmentations alone, we also experiment with source-only models trained without any data augmentation. For tabular and NLP datasets, we do not use any augmentations.
4 MAIN RESULTS
We present aggregated results vision datasets in testbed in Fig. In App. we present aggregated results on NLP and tabular datasets. We include results on each dataset in App. !! Note that we do not include RS results with a source only model as it is trained only on source data and

---

Under review as a conference paper at ICLR 2023
10P
Tret-rime training methods
Domain aignment
SelStraning methods
101
Relabe Aaaraey
308
-10P
-10
-10P
Nore
10.0
3.0
1.0
0.5
None
10.0
3.0
1.0
0.5
Nune
10.0
3.0
1.0
0.5
Dinchlen Shift (сipтs)
Dirchler Shift (siphe)
Dirchlet Shift (alpha)
.... Source
UN-o
TENT
CORAL
DANN
IW-DANN
CDANN
IW-CDAN
FaMarch
Stteyttudert
SENTRY
(a) Performance of DA methods relative to source-only training with increasing target label marginal shift
None
RW
RS
R$ RW
DANN
BN- adaat
FirMatc
11P
Nome
33.0
3.0
1.0
0.5
Kom
12.0
32
1.
8.5
Node
1000
3.
1.0
85
Uirctie (Abba)
Unichit Skitt (epm))
Stitt (mpss))
CDANN
TENT
Student
30
-30
Noms
2.0
0.5
2.0 1.0
None
2.0
1.0
Diichirn Stift (aips)
Dricdes Stltt (aipaa)
Delcctin Skift (AbRa)
(b) Relative performance of DA methods when paired with our meta-algorithm (RS and RW corrections)
Figure I: Performance of different DA methods relative to a source-only model across all distribution
shift pairs in vision datasets grouped by shift severity in label marginal. For each distribution shift pair and DA method, we plot the relative accuracy of the model trained with that DA method by subtracting the accuracy of the source-only model. Hence, the black dotted line at 0 captures the performance of the source-only model. Smaller the Dirichlet shift parameter, the more severe is the shift in target class proportion. (a) Shifts with a (NONE, 10.0, 3.0) have little to no impact on different DA methods whereas the performance of all DA methods degrades when a E {1.0, 0.5) often falling below the performance of source-only classifier (except for Noisy Student). (b) RS and RW (in our meta-algorithm) together significantly improve aggregate performance over no correction for all DA methods. While RS consistently helps (over no correction) across different label marginal shift severities, RW hurts slightly for BN-adapt, TENT, and NoiseStudent when shift severity is small. However, for severe shifts (a E (3.0, 1.0, 0.5)) RW significantly improves performance for all the methods. Parallel results tabular and language datasets in App. B Detailed results with all methods on individual datasets in App. 0 A more detailed description of the plotting technique in App. A
observed no differences with just balancing the source data (as for most datasets source is already balanced) in our experiments. Unless specified otherwise, we use source validation performance as the early stopping criterion. Based on running our entire RLSBBENC suite, we distill our findings into the following takeaways.
Popular deep DA methods without any correction falter. While DA methods often improve over a source-only classifier for cases when the shift in target label marginal is absent or low, the performance of these methods (except Noisy Student) drops below the performance of a source-only classifier when the shift in target label marginal is severe (i.e., when 0.5 in Fig. D 4h. and ()). On the other hand, DA methods when paired with RS and RW correction, significantly improve over source-only model even when the shift in target label marginal is severe (Fig. Tb and 5p).
7

---

Under review as a conference paper at ICLR 2023
Vision Datazets
Tabular Datasets
Language Datasets
10
68
CORAT
CORAL (6S 9000
Sourne pei las (999
Sonee SEMTTR
Soure (e aug) (PW)
TEXT
Nanggoodee (RSS (0S--990)
(RS (es_oo0
COMM (RS (999
Fublact
Fiehlus (RS (99)
IW-DAM
COAMN
No-CDANN
Soute (w/o axg)
Pasuect abel
Paudi and (RS RWO
Sunte (wo wng) (RW)
CDeat (SS- (oo)
DAN
Source (mi (w/o a0d)
CDWW
boudd (eS (00)
DAM (PS RRV)
co4M (IS (w)
Sune (wi wag) (RO)
Dabii (eSS
Daway
Figure 2: Average accuracy of different DA methods aggregated across all distribution pairs in each
modality. Parallel results with all methods on individual datasets in App.
Viece Datautts
Viece Datamets
Tabulor Dacautts
Izegugge
1.4
Edrmaien Eret
Scurce (wr may,
FiaMacd (Mors]
Source aig)
COAM (Nees)
Eeimatee Erer
Sourcee
(Mane))
0.25
COMNN (Nios)
Source
FoMinch (FS)
CDANU (RS)
CIAAN (RS)
0:20
COANN (RS)
0.00
0.00
3.3
3.0 (ebrhs
0.5
Diriche:
Drrihlet
(absta)
Figure 3: Target label marginal estimation (l)) error with RLLS and classifiers obtained with different DA methods. Across all shift severities in vision datasets, RLLS with classifiers obtained with DA methods improves over RLLS with a source-only classifier. For tabular datasets, RLLS with classifiers obtained with DA methods improves over RLLS with a source-only classifier for severe target label marginal shifts. Conversely, for NLP datasets, RLLS with source-only classifiers performs better than RLLS with classifiers obtained with DA methods.
Re-sampling to pseudobalance target often helps all DA methods across all modalities. When the shift in target label marginal is absent or very small (i.c., a (NONE, 10.0) in Fig. [..4p.pppp............ and 5D)) we observe no (significant) differences in performance with re-sampling. However, as the shift severity in target label marginal increases (i.e., a E [3.0, 1.0, 0.5) in Fig. lb. and 5)), we observe that re-sampling typically improves all DA methods in our testbed.
Benefits of post-hoc re-weighting of the classifier depends on shift severity and the underlying DA algorithm. For domain alignment methods (i.e. DANN and CDANN) and self-training methods, in particular FixMatch and PseudoLabell, we observe that RW correction typically improves (over no correction) significantly when the target label marginal shift is severe (i.c., a E (3.0, 1.0, 0.5) in Fig. Ab, and (b) and has no (significant) effect when the shift in target label marginal is absent or very small (i.e.. a E {NONE, 10.0) in Fig. pp. 5p) For BN-adapt, TENT, and NoisyStudent RW correction can slightly hurt when target label marginal shift is absent low (i.e., E (NONE, 10.0) in Fig. [b) but continues to improve significantly when the target label marginal shift is severe (i.e., E (3.0, 1.0, 0.5) in Fig. [b). Additionally, we observe that in specific scenarios of the real-world shift in p(xly) (c.g.. subpopulation shift in BREEDs datasets, camelyon shifts, and replication study in CIFAR- 10 which are benign relative to other vision dataset shifts in our testbed), RW correction does no harm to performance for BN-adapt, TENT, and NoisyStuden even when the target label marginal shift is less severe or absent (refer to datasets in App.
DA methods paired with our meta-algorithm often improve over source-only classifier but no one method consistently performs the best. First, we observe that our source-only numbers are better than previously published results. Similar to previous studies (Gulrajani & Lopez-Paz 2020), this can be attributed to improved design choices (e.g. data augmentation, hyperparameters) which make consistent across all methods. While there is no consistent method that does the best across datasets, overall, FixMatch with RS and RW (our meta-algorithm) performs the best for vision datasets. For NLP datasets, source-only with RW (our meta-algorithm) performs the best overall. For tabular datasets, CDANN with RS and RW (our meta-algorithm) performs the best overall (Fig. 2
Existing DA methods when paired with meta-algorithm significantly outperform other DA methods specifically proposed for relaxed label shift. We observe that, with consistent experimental design across different methods, existing DA methods with RS and RW corrections often
8

---

Under review as a conference paper at ICLR 2023
improve over previously proposed methods specifically aimed to tackle relaxed label shift, i..., IW- CDANN, IW-DANN, and SENTRY (Fig. 6). For severe target label marginal shifts, the performance of FI-DANN, IW-CDANN and SENTRY often falls below that of the source-only model. Moreover, while the importance weighting (i.e., IW-CDANN and IW-DANN) improves over CDANN and DANN resp. (Fig.
T and 5b), RS and RW corrections significantly outweigh those improvements (Fig. 6
BN-adapt and TENT with our meta-algorithm are simple and strong baselines. For models with batch norm parameters, BN-adapt (and TENT) with RS and RW steps is computationally efficient and strong baseline. We observe that while the performance of B BN-adapt (and TENT) can drop substantially when the target label marginal shifts (i.e., a E {1.0, 0.5] in Fig. Ta), RS and RW correction improves the performance often improving Bd-atpp (and TENT) over all other DA methods when the shift in target label marginal is extreme (i.e., a 0.5 in Fig. [b).
Deep DA heuristics often improve target label marginal estimation on tabular and vision modalities. Recall that we experiment with target label marginal estimation methods that leverage off-the-shelf classifiers to obtain an estimate. We observe that estimators leveraging DA classifiers tend to perform better than using source-only classifiers for tabular and vision datasets (vision in Fig. and others in App. G.. For NLP, we observe improved estimation with source-only model over DA classifiers. Correspondingly, as one might expect. better estimation yields greater accuracy improvements when applying RW correction. In particular, RW correction with DA methods improves over the source-only classifier for vision and tabular datasets and vice-versa for NL.P datasets. (App- G Moreover, for all modalities, we observe trade-off between estimation error with the baseline method (i.e. binning target pseudolabels) and RLLS (or MLLS) method with severity in target marginal shift (Fig.
Early stopping criterion matters. We observe a consistent R 25% and 88% accuracy difference on vision and tabular datasets respectively with all methods (Fig. 12p On NLP datasets, while the early stopping criteria have 22% accuracy difference when RW and RS corrections are not employed, the difference becomes negligible when these corrections are employed (Fig.
12p These results highlight that subsequent works should describe the early stopping criteria used within their evaluations.
Data augmentation helps. Corroborating findings from previous empirical studies in other set- tings (Gulrajani& & Lopez-Paz 22200, Saggwa et al., (022]. we observe that data augmentation tech- niques can improve the performance of a source-only model on vision datasets in relaxed label shift scenarios (refer to result on each dataset in App.[]. Thus, whenever applicable, subsequent methods should use data augmentations.
5 CONCLUSION
Our work is the first large-scale study investigating methods under the relaxed label shift scenario. Relative to works operating strictly under the label shift assumption, RLSBENCH provides an opportunity for sensitivity analysis, allowing researchers to measure the robustness of their methods under various sorts of perturbations to the class-conditional distributions. Relative to the benchmark- driven deep domain adaptation literature, our work provides comprehensive and standardized suite for evaluating under shifts in label distributions, bringing these benchmarks one step closer to exhibit the sort of diversity that we should expect encounter when deploying models in the wild. On one hand, the consistent improvements observed from label shift adjustments are promising. At the same time, given the underspecified nature of the problem, practitioners must remain vigilant and take performance on any benchmark with a grain of salt, considering the various ways that it might (or might not) be representative of the sorts of situations that might arise in their application of interest.
In the future, we hope to extend RLSBENCH to datasets from real applications in consequential domains such as healthcare and self-driving, where both shifts in label prevalences and perturbations in class conditional distributions can be expected across locations and over time. We also hope to incorporate eup-suuee methods that learn representations by training on union of unlabeled data from source and target via proxy tasks like reconstruction (Gidaris et al. [0188] (He et al.) (2022) and contrastive learning (Caron et al., 2020 Chen et al. 2020). While re-weighting predictions using estimates of the target label distribution yields significant gains, the remaining gap between our results and oracle performance should motivate future work geared towards improved estimators. Also, observe that the success of target label marginal estimation techniques depends the nature of the shifts in p(zly). Mathematically characterizing the behavior of label shift estimation techniques when the label shift assumption is violated would be an important contribution.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Our code with all the results is released on htttp:://gtttttueeetes com http://wwwccccmcccc2222222201 We implement our RLSBENCH library in PyTorch (Paszke et al. 2017) and provide an infrastructure to run all the experiments to generate corresponding results. We have stored all models and logged all hyperparameters and seeds to facilitate reproducibility. In our appendices, we provide additional details on datasets and experiments. In App.[) D we describe dataset information and in App. L we
describe hyperparameter details.
REFERENCES
Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Adapting label shift with birs-coreected calibration. In International Conference on Machine Learning (ICML), 2021.
Jing An, Lexing Ying, and Yuhua Zhu. Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients. arXiv preprint arXiv:2009. 13447, 2020.
Kamyar Arizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. In International Conference Learning Representations (ICLR), 2019.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging, 2018
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1-2), 2010a.
Shai Ben- David. Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility Theorems for Domain Adaptation. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2010b.
David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. Adamatch: A unified approach to cunissuper learning and domain adaptation. arXiv preprint arXiv:2106.04732, 2021.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. Advances in neural information processing systems, 24, 2011.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of The 2019 World Wide Web Conference, 2019.
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 106:249-259, 2018
Jonathon Byrd and Zachary C Lipton. What is the effect of importance weighting in deep learning? In International Conference on Machine Learning (ICML), 2019.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with aabel-disributiin-aaaaa margin loss. In Advances in Neural Information Processing Systems, volume 32, 2019a.
Zhangjie Cao, Kaichao You, Mingsheng Long, Jianmin Wang, and Qiang Yang. Learning to transfer examples for partial domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2985-2994, 2019b.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
10

---

Under review as a conference paper at ICLR 2023
Ting Chen, Simon Komblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2018.
Corinna Cortes and Mehryar Mohri. Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science, 2014.
Corinna Cortes, Yishay Mansour, and Mehryar Mohri. Learning Bounds for Importance Weighting. In Advances in Neural Information Processing Systems (NIPS), 2010.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Terrance Deiei and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: /00iv./700..000 178 2017.
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 34:6478 6490,, 2021.
Josip Djolonga, Jessica Yung. Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' Amour, Dan Moldovan, et al. On robustness and transferability of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16458-16468, 2021.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Dovain-adversarial training of neural networks. The journal of machine learning research, 2016.
Jacob Gardner, Geoff Pleiss, Kilian Q Weinberger, David Bindel, and Andrew G Wilson. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 1833v:1803.07728, 2018.
Arthur Gretton, Alexander J Smola, Jiayuan Huang. Marcel Schmittfull, Karsten M Borgwardt, and Bernhard Schölkopf. Covariate Shift by Kernel Mean Matching. Journal of Machine Learning Research (JMLR), 2009.
Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Computer Vision and Pattern Recognition (CVPR), 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 16000-16009, 2022.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903. /2211, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of diitr---istiittiinn generalization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3340,8349, 2021.

---

Under review as a conference paper at ICLR 2023
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7700-4708, 2017.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst- porst-group-aceureee In Conference Causal Learning and Reasoning, pp. 336-351. PMIR, 2022.
Junguang Jiang, Yang Shu, Jianmin Wang, and Mingsheng Long. Transferability in deep learning: A survey, 2022.
Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horn,, Leo Anthony Celi, and Roger Mark. Mimic-iv. PhysioNet. Available online at: https:/pphysonnet wttp:/ttee//miiiiiiioo./oaaaaaae August 23, (021),2020
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, lan Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A bench- mark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML), 2021.
Alex Krizhevsky and Geoffrey Hinton. Learning Multiple Layers of Features from Tiny Images. Technical report, Citeseer, 2009.
Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. In International Conference Machine Learning, pp. 5468 5479. PMLR, 2020.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896, 2013.
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint 1603v:1603.0477 2016.
Zachary C Lipton, Yu-Xiang Wang, and Alex Smola. Detecting and Correcting for Label Shift with Black Box Predictors. In International Conference on Machine Learning (ICML), 2018.
Xiaofeng Liu, Zhenhua Guo, Site Li, Fangxu Xing, Jane You, C.-C. Jay Kuo, Georges El Fakhri, and Jonghye Woo. Adversarial Unsupervised Domain Adaptation with Conditional and Label Shift: Infer, Align and Iterate. In 202/ IEEE/CVF International Conference Computer Vision (ICCV), pp. 10347-10356, Montreal, QC, Canada, October 2021. IEEE. ISBN 778-1-66542-88 81255. doi: N0.11099/CCV499922222.000 0100200 URL 97162057
eee . org/ orgee.r///ommmeetttt document/
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105. PMLR, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International conference on machine learning. PMLR, 2017.
Mingsheng Long, Zhangjie Cao, Jianmin Wang. and Michael Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 2017.
Jeroen Manders, Twan van Laarhoven, and Elena Marchiori. Adversarial Alignment of Class Prediction Uncertainties for Domain Adaptation, January 2019. URLhttp: /ttp://axxvv.o////// abs/ 1804 04448 Number: arXiv: 00400444 arXiv:1804.044 04404448
[cs, stat].
12

---

Under review as a conference paper at ICLR 2023
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.34400, 2009.
John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International Conference Machine Learning. PMLR, 2021.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pp. 10-18. PMLR, 2013.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Xingchao Peng, Ben Usman, Kuniaki Saito, Neela Kaushik, Judy Hoffman, and Kate Saenko. Syn2real: new benchmark forsynthetic-to-rn to--ral visual domain adaptation. arXiv preprint mrXiv::800.09955, 2018.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Toolio PhysioBank. Physionet: components of a new research resource for complex physio- logic signals. Circulation, 101(23):e21--2220 2000.
Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. Sentry: Selective entropy opti- mization via committee consistency for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8558-8567, 2021.
Joaquin Quinonero-Candela. Masashi Sugiyama, Anton Schwaighofer, and Neil Lawrence. Dataset shift in machine learning. Mit Press, 2008
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers 2018.
generalize to cifar-10? arXiv preprint 1806.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400. PMIR, 2019.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint arXiv:2202.06856, 2022.
Olga Russakovsky. Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 1522, 2015.
Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the Outputs of Classifier to New a Priori Probabilities: A Simple Procedure. Neural Computation, 2002.
Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, and Percy Liang. Extending the wilds benchmark for unsupervised adaptation. In NeurIPS Workshop Distribution Shifis, 2021.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01 108, 2019.
13

---

Under review as a conference paper at ICLR 2023
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. In International Conference on Learning Representations (ICLR), 2021.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann. Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. arXiv preprint arXiv:2006. 169771,10..6,1, 2020.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On Causal and Anticausal Learning. In International Conference on Machine Learning (ICML), 2012.
Hidetoshi Shimodaira. Improving Predictive Inference Under Covariate Shift by Weighting the Log- Likelihood Function. Journal of Statistical Planning and Inference, 2000.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying cumi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33, 2020.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision. Springer, 2016.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2016.
Baochen Sun, Jiashi Feng, and Kate Saenko. Correlation alignment for unsupervised domain adaptation. In Domain Adaptation in Computer Vision Applications. Springer, 2017.
Remi Tachet, Han Zhao, Yu-Xiang Wang, and Geoff Gordon. Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift. arXiv:2003.00 04475 [cs, stat], December 2020. URL http:://axxii..orttt org/ abs 2003. 04475)
arXiv: 2003.04475.
Remi Tachet des Combes, Han Zhao, Yu-Xi Yu-Xiang Wang, and Geoffrey J Gordon. Domain adaptation with conditional distribution matching and generalized label shift. Advances in Neural Information Processing Systems, 33, 2020.
Shuhan Tan, Xingchao Peng, and Kate Saenko. Class-imbalanced domain adaptation: An empirical odyssey. In European Conference Computer Vision, pp. 585-602. Springer, 2020.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33: 33:18583-18599 2020.
Antonio Torralba and Alexei A Efros. Unbiased look dataset bias. In CVPR 2011, pp. 1521-1528. IEEE, 2011.
Antonio Torralba, Rob Fergus, and William T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. JEEE Transactions on Pattern Analysis and Machine Intelligence, (0(11):1958 1970, 2008
Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions on neural networks, 10(5):988-99) 1999.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5018-5027, 2017.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test- time adaptation by entropy minimization. In International Conference Learning Representations, 2021. URL https: http::/s::///ppeeviieeeee
net. http:/ooummmmm 33x_mmm22.__xx111322333333
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A claallneba self-training framework for imbalanced semi seni-supervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10857-10866, 2021.
14

---

Under review as a conference paper at ICLR 2023
Florian Wenzel, Andrea Dittadi, Peter Vincent Gehler, Carl-Johann Simon-Gabrie Max Horn, Dominik Zietlow, David Kernert Chris Russell, Thomas Brox, Bernt Schicle, et al. Assaying out-of-distribution generalization in transfer learning, arXiv preprint arXiv:2207.09229, 2022.
Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre Alvise-Rebufi Ira Ktena, Taylan Cemgil, et al. A fine-grained analysis on distribution shift. arXiv preprint arXiv:21 10. 1328, 2021.
Yifan Wu, Ezra Winston, Divyansh Kaushik, and Zachary Lipton. Domain adaptation with asymmetrically-relxxed distribution alignment. In International Conference on Machine Learning (ICML), 2019.
Xinpeng Xie, Jiawei Chen, Yuexiang Li, Linlin Shen, Kai Ma, and Yefeng Zheng. Self-supervised cyclegan for object-preserving image-to-image domain adaptation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pp. 498-513, Springer, 2020.
Da Xu, Yuting Ye, and Chuanwei Ruan. Understanding the role of importance weighting for deep learning. arXiv preprint vrXiv:2103.15209 2021.
Hongliang Yan, Yukang Ding, Peihua Li, Qilong Wang. Yong Xu, and Wangmeng Zuo. Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2272-2281, 2017.
Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei Koh, and Chelsea Finn. Wild- time: A benchmark of in-the-wild distribution shift over time. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https
//openreview net. Feeee//orrmmiidFFnne ottpa//eewe...tttoooooooooooooo
Bianca Zadrozny. Learning and Evaluating Classifiers Under Sample Selection Bias. In International
Conference on Machine Learning (ICML), 2004.
Jingzhao Zhang, Aditya Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. In International Conference Learning Representations (ICLR), 2021.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikum Wang. Domain Adaptation Under Target and Conditional Shift. International Conference on Machine Learning (ICML), 2013.
Richard Zhang. Making convolutional networks shift-invariant again. 2019.
2019.
Weichen Zhang, Wanli Ouyang, Wen Li, and Dong Xu. Collaborative and adversarial network for unsupervised domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, 2018.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning. PMLR, 2019.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pp. 7523-7532. PMLR, 2019.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV) pp. 289-305, 2018.
15

---

Under review as a conference paper at ICLR 2023
APPENDIX
A DESCRIPTION OF PLOTS
For each plot in Fig. obtain all the distribution shift pairs with a specific alpha (i.e., the value on the x-axis) Then for each distribution shift pair (with a specific alpha value), we obtain relative performance by subtracting the performance of source-only model trained on the source dataset of that distribution shift pair from the performance of the model trained on that distribution shift pair with the DA algorithm of interest. Thus for each alpha and each DA method, we obtain 112 relative performance values. We draw the box plot and the mean of these relative performance values.
For (similar-looking) plots, we use the same technique throughout the paper. The only thing that changes is the group of points over which aggregation is performed.
B TABULAR AND NLP RESULTS OMITTED FROM THE MAIN PAPER
B.1 TABULAR DATASETS
alggment
Self-training methods
10P
10P
Relaiiv Accurcyy
-10
-10
10
WP
-10P
Norw
10.0
Nons
10.0
0.5
Dirichlet Shitt (sipha)
Dirichie Shift (spha]
Source only
DANN
IW-DANN
COANN
NW-CDANN
PseudoLabel
(a) Performance of DA methods relative to source-only training with increasing target label marginal shift
None
RW
R5
RS
RW
DANN
CDANN
Panudalabi
10/
-10
100
Delchte: Skitt (ethn))
Drickes Stitt (algha)
Drickls Shitt (abbh)
(b) Relative performance of DA methods when paired with our meta- algorithm (RS and RW corrections)
Figure 4: Performance of different DA methods relative to a source-only model across all distribution shift pairs in tabular datasets grouped by shift severity in label marginal. For each distribution shift pair and DA method, we plot the relative accuracy of the model trained with that DA method by subtracting the accuracy of the source-only model. Hence, the black dotted line at 0 captures the performance of the source-only model. Smaller the Dirichlet shift parameter, the more severe is the shift in target class proportion. (a) Shifts with (NONE, 10.0, 3.0) have little to impact different DA methods whereas the performance of all DA methods degrades when a E (1.0, 0.5) often falling below the performance of source-only classifier. (b) RS and RW (in our meta-algorithm) together significantly improve aggregate performance over no correction for all DA methods. While RS consistently helps (over no correction) across different label marginal shift severities. RW hurts slightly when shift severity is small. However, for severe shifts (a E (3.0, 1.0, 0.5)) RW significantly improves performance for all the methods. Results with all methods on individual datasets in App.
16

---

Under review as a conference paper at ICLR 2023
B.2 NLP DATASETS
siggment
Sel--taining
124
32"
Reaarbe Accurey
Relatbe
-10
-00
-12
-30
Nons
10.0
3.0
1.0
0.5
Nors
10.0
3.0
1.0
0.5
Diricnter Shitt (alpha)
Uiricher Shirt (alpha)
Source only
DANN
IW-DANN
CDANN
IW-CDANN
Psoudolabel
(a) Performance of DA methods relative to source-only training with increasing target label marginal shift
None
RW
RS
RS RW
DANN
CDANN
Paeudolabel
107
1P
-1P
-10
-10
(b) Relative performance of DA methods when paired with our meta-algorithm (RS and RW corrections)
Figure 5: Performance of different DA methods relative to a source-only model across all distribution shift pairs in NLP datasets grouped by shift severity in label marginal. For each distribution shift pair and DA method, we plot the relative accuracy of the model trained with that DA method by subtracting the accuracy of the source-only model. Hence, the black dotted line at 0 captures the performance of the source-only model. Smaller the Dirichlet shift parameter, the more severe is the shift in target class proportion. (a) Performance of DANN and IW-DANN methods degrades with increasing severity of target label marginal shift often falling below the performance of source- only classifier (except for Noisy Student). Performance of PsuedoLabel, CDANN, and IW-CDANN show less susceptibility to increasing severity in target marginal shift. (b) RS and RW (in our meta-algorithm) together significantly improve aggregate performance over no correction for all DA methods. While RS consistently helps (over no correction) across different label marginal shift severities, RW hurts slightly for Bd-atppt, TENT, and NoisyStudent when shift severity is small. However, for severe shifts (a (3.0, 1.0, 0.5)) RW significantly improves performance for all the methods. Detailed results with all methods on individual datasets in App. !
C COMPARISON BETWEEN IW-CDANN, IW-DANN, AND SENTRY WITH EXISTING DA METHODS PAIRED WITH OUR META-ALGGRIIE
Fig. 6 shows the relevant comparison.
Note. On Officehome dataset, we observe a slight discrepancy between SENTRY results with our runs and numbers originally reported in the paper (Prabhu et al. 2021). We find that this is due to differences in batch size used in original work versus in our runs (which we kept the same for all the algorithms). In App. M we report SENTRY results with the updated batch size. With the new batch size, we reconcile SENTRY results but also observe a significant improvement in FixMatch results. We refer reader to App. M for more detailed discussion.
17

---

Under review as a conference paper at ICLR 2023
Vision Datasets
Vision Datasets
10P
10P
10
Helabve Acursay
10P
10
MUDAAN TACODAN
DANN (RS-RRI
Relatro
10P
100P
-10
SENTRY
FicMaack
Fanhlach (85 RNN
100
None
10.0
3.0
1.0
0.5
None
10.0
3.0
0.5
Dirchlet (aloha)
Dirchlet (alsha)
Tabular Datasets
Language Datasets
107
107
IN-DANN
DANN (RS. RM)
COANM (IS)+1I))
107
Rolaiive Acturacy
100
10P
100"
-10P
-10'
IWS DANN
DANN (FS (AW
IV COANN
CDANN (IS+RRW)
None
3.0
None 10.0
3.0
0.5
Dirichet (alpha)
Dinihhet Stift (Abbha)
Figure 6: Comparison of existing DA methods paired with our RS and RW correction and DA methods specifically proposed for relaxed label shift problems. Across vision and tabular datasets, we observe the susceptibility IW-DAN, IW-CDAN, and SENTRY with increasing severity of target label marginal shifts. In particular, for severe target label marginal shifts, the performance of IW-DAN, IW-CDAN, and SENTRY often falls below that of the source-only model. However, existing DA techniques when paired with RS RW correction significantly improve over the source-only model. For NLP, datasets we observe similar behavior but with relatively less intensity.
D DATASET DETAILS
In this section, we provide additional details about the datasets used in our benchmark study.
CIFAR1O We use the original CIFARIO dataset (Krizhevsky & Hinton 2009) as the source dataset. For target domains, we consider (i) synthetic shifts (CIFARIO-C) due common corrup- tions (Hendrycks & Dietterich 2019): and (ii) natural distribution shift, i.e., CIFAR 10v2 (Recht) jtt al. 2018 Torralla et al. 2008) due differences in data collection strategy. We randomly sam- ple 3 set of CIFAR-IO-C datasets. Overall, we obtain 5 datasets (i.e., CIFAR10v1, CIFARIOVE 10v2, CIFAR10C-Frost (severity 4), CIFARIOC-Pixelate (severity 5), CIFARIO-C Saturate (severity 5)).
CIFAR100 Similar to CIFAR10, we use the original CIFAR 100 set as the source dataset. For target domains consider synthetic shifts (CIFAR100-C) due common corruptions. We sample 4 CIFAR 100-C datasets, overall obtaining 5 domains (i.e., 100, CIFAR 100C-FFo (severity 4), CIFAR 100C-Motion Blur (severity 2), CIFAR 100C-Contratt (severity 4), CIFARR100C-ppater (severity 2) ).
FMoW In order to consider distribution shifts faced in the wild, we consider FMoW-WILDS Koh et al. 2021 Christie et al. 2018) from WILDS benchmark, which contains satellite images taken in different geographical regions and at different times. We use the original train as source and OOD val and OOD test splits as target domains as they are collected over different time-period. Overall, we obtain 3 different domains.
Camelyon17 Similar FMoW, consider tumor identification dataset from the wilds bench- mark (Bandi et al. 2018). We use the default train as source and OOD val and OOD test splits as tar- get domains they are collected across different hospitals. Overall, we obtain 3 different domains.
18

---

Under review as a conference paper at ICLR 2023
Dataset
Domains
Cifar10v1
Cifar10v2
Cifar10C-Frost
Cifar10C-Pixelate
Cifarl0C-Saturate
CIFAR10
Cifar100v1
Fifar100C-Fog
Cifar100C-M. blur
Cifr1100C-Contrast
Cifar100C-Spatter
CIFAR100
Hospital 1-3
Hospital
Hospital
Camelyon
Idisloint sub.)
v2
Idisjoin sub.)
Entity13
v1 [disjoint sub.)
v2 [disjoin sub.)
Entity30
Idisjoint sub.)
v2
[disjoin sub.
Living17
vi (disjoint sub.)
(disjoin sub.)
Nonliving26
Years 2002/113
Year 2013-116
Year 2016-11 618
FMoW
Product
RealWorld
ClipArt
Officehome
Real
ClipArt
Sketch
Painting
Domainnet
Rendering
Real
Real
Visda
Figure 7: Examples from all the domains in each vision dataset.
19

---

Under review as a conference paper at ICLR 2023
Dataset
Source
Tagett
CIFARIOvI, CIFARIOV2 CIFARIOC-Fres (severity 4).
CIFAR10
CIFART0y1
CHFARIOC- Pixelllteee (severity 5), CIFARIO-C Saturate (severity 5)
CIFARIOO, CIFAR10OC-Fog (severity 4),
CIFARI00
CIFARI00
CIFAR100C- Blur (severity 2), CIFARIOUC-CCnmraa (severity 4),
CIFAR spatter (severity
Camelyon
Camelyon
Camelyon (Hospital 1-3), Camelyon (Hospital Camelyon (Hospital 5)
(Hospital 1-3)
FMoW
FMoW (2002-113)
FMoW (2002-113), FMoW (2013-116) 16), FMoW (2016-118)
Entity (ImageNetv] sub-population 1),
Entity
Entity13 (ImageNetvl sub sab-population
Entit13
(ImageNety1 sub-population I)
Eotity13 (ImageNetv2 sub-population 1).
Entiyl3 (ImugeNetv2 sub-populution 2)
Entity30 (ImageNetvl sub-population 1).
En
Entity30
(ImageNetvl
Entit (ImageNetv) sub-population 2),
Entity30 (lmageNetv2 sub population 1),
sub uuh-population 1)
Entit300 (ImageNetv2 cub--oopulaton
Living17 (ImageNetv] population 1),
Living17
Living17
IImageNetv1 sub-population I)
Living17 (ImageNetvl sub-population 2).
Livingl7 1 (ImageNetv2 sub-population 1),
Living17 (ImageNetv2 population 2)
Nonliving26 (ImageNetvl sub-population 1),
Nonliving26 (ImageNerv1 sub-population I)
N6 (ImageNetvl sub aot-pooulation
Nonliving22
Nonliving26 (ImageNetv2 sub-population 1].
Nonliving26 (ImageNetv2 sub-population 2)
Officehome
Product
Product, Art, ClipArt, Real
DomainNet
Real
Real, Painiting, Sketch, ClipArt
Synthetic (originally referred train)
Synthetic. Real-1 (originally referred as val). Real-2 (originally referred to as test)
Visda
Civilcomments
Train
Train, Val and Test (all formed by disjoint partitions of online articles)
Mimic Readmissions (year: 2008), Mimic Readmissions (year: 2009),
Mimic Readmissions (year: 2008)
Mimic Readmissions
Mimic Readmissions (year: 2010), Mimic Readmissions (year: 2011)
Mimic Readmissions (yeur: 2012), Mimic Readmissions (yeur: 2013)
Retiring Adults (year: 2015;
("MD', 'NJ' "MAT).
Retiring Adults (year: 2014
Retiring Adults (year: 2016; states: ['MD', 'NJ", 'MAD.
Retiring Adults
Retiring Adults (year: 2017: states: I'MD', 'NI' 'MA'].
states: ('MD'.' 'NJ' 'MA'D
Retiring Adults (year: 2018; ('MD', 'NJ. 'MA')
Table 1: Details of the datasets considered in our RLSBENCH.
We also consider BREEDs benchmark (Santurkar et al. 2021) in our setup to assess robustness to subpopulation shifts. BREEDs leverage class hierarchy in ImageNet re-purpose original classes to be the subpopulations and defines a classification task on superclasses. We consider distribution shift due subpopulation shift which is induced by directly making the subpopulations present in the training and test distributions disjoint. BREEDs benchmark contains 4 datasets Entity-13, Entity-30. Living-17, and Non-living-26, each focusing on different subtrees and levels in the hierarchy. We also consider natural shifts due to differences in the data collection process of ImageNet (Russakovsky et al. 2015). e.g, ImageNetv2 (Recht al. 2019)) and combination of both. Overall, for each of the 4 BREEDD datasets (i.e., Entity- 13, Entity-30, Living- 17, and Non-living-26), we obtain four different domains. We refer them as follows: BREEDsv1 sub-population (sampled from ImageNetv1), BREEDsvl sub-population 2 (sampled from ImageNetv BREEDsv2 sub-population (sampled from ImageNetv2), BREEDsv2 sub- population 2 (sampled from ImageNetv2). For each BREEDs dataset, we use BREEDsv sub- population A as source and the other three as target domains.
BREEDs
OfficeHome We use four domains (art, clipart, product and real) from OfficeHome dataset (Venkateswara et al. 2017). ) We use the product domain as source and the other domains as target.
20

---

Under review as a conference paper at ICLR 2023
DomainNet We use four domains (clipart, painting, real, sketch) from the Domainnet dataset PPng et al. 2019). We use real domain as the source and the other domains as target.
Visda We use three domains (train, val and test) from the Visda dataset (Peng et al. 2018). While 'train" domain contains synthetic renditions of the objects, 'val" and 'test" domains contain real world images. To avoid confusing, the domain names with their roles as splits, we rename them as 'synthetic', 'Real-1' and 'Real-2'. We use the synthetic (original train set) as the source domain and use the other domains as target.
Civilcomments (Borkan et al. 2019) from the wilds benchmark which includes three domains: train, OOD val, and OOD test, for toxicity detection with domains corresponding to different demographic subpopulations. The dataset has subpopulation shift across different demographic groups as the dataset in each domain is collected from a different partition of online articles.
Retiring Adults (Ding et al. 2021 where we consider the ACSIncome prediction task with various domains representing different states and time-period; We randomly select three states and consider dataset due to shifting time across those states. Details about precise time-periods and states are in
Table 1
Mimic Readmission JJohnson et al. 2020 PhysioBank 2000 where the task is to predict readmis- sion risk with various domains representing data from different time-period. Details about precise time-periods are in Table
We provide scripts to setup these datasets with single command in our code. To investigate the performance of different methods under the stricter label shift setting, also include hold-out partition of source domain in the set of target domains. For these distribution shift pairs where source and target domains i.i.d. partitions, we obtain the stricter label shift problem. We summarize the information about source and target domains in Table
Train-test splits We partition each source and target dataset into 80% and 20% i..d. splits. We use 80% splits for training and 20% splits for evaluation (or validation). We throw away labels for the 80% target split and only use labels in the 20% target split for final evaluation. The rationale behind splitting the target data is to use completely unseen batch of data for evaluation. This avoids evaluating on examples where a model potentially could have overfit. over-fitting to unlabeled examples for evaluation. In practice, if the aim is to make predictions on all the target data (i.e., transduction), we can simply use the (full) target set for training and evaluation.
E METHODS TO ESTIMATE TARGET MARGINAL UNDER THE STRICTER LABEL SHIFT ASSUMPTION
In this section, we describe the methods proposed to estimate the target label marginal under the stricter label shift assumption. Recall that under the label shift assumption, p. (t) can differ from Pe(y) but the class conditional stays the same, i.e., pr (xly) = (x(lly). We focus our discussion on recent methods that leverage off-the-shelf classifier to yield consistent estimates under mild assumptions (Lipton et al.] 2018) (Azizzaennssheli et al. 2019: Alexandari et al. 2021 Garg et al.) 2020). For simplicity, we assume we possess labeled source data {(x1, 81), (I2, (22,82)... ... (Z Fr) and unlabeled target data [In+1. (aamm
U2)......
RLLS First, we discuss Regularized Learning under Label Shift (RLLS) Azizzadeneshelii et al.
2019) (a variant of Black Box Shift Estimation (BBSE, Lipton et al. (2018) )): noment-matching based estimators that leverage (possibly biased, uncalibrated, or inaccurate) predictions estimate the shift.
RLLS solves the following optimization problem to estimate the importance weights we wx(v) (.())
p(())
as:
RLIS 0
= arg min 1Č - 111 + ARLLL |W - 1112122-
(I)
WEW
where W {w E Rd E, wv))p.(v) = 1 and Vy E y w(y) > 0}. č2 is empirical confusion matrix of the classifier f on source data and J is the empirical average of predictions of the classifier on unlabeled target data. With labeled source data data, the empirical confusion matrix can be
21

---

Under review as a conference paper at ICLR 2023
computed as:
R n Σ f((x))
L
- ]]
To estimate target label marginal, we can multiple the estimated importance weights with the source label marginal (we can estimate source label marginal simply from labeled source data).
In our relaxed label shift problem, we use validation source data compute the confusion matrix and use hold portion of target unlabeled data to compute μ.s. Unless specified otherwise, we use RLLS to estimate the target label marginal throughout the paper. We choose ARLLS as suggested in the original paper (Azizzadenesheli et al. 2019).
MLLS Next, we discuss Maximum Likelihood Label Shift (MLLS) (Saerens et 2002 Alexandari
et al.] (02T): an Expectation Maximization (EM) algorithm that maximize the likelihood of observed unlabeled target data to estimate target label marginal assuming access to a classifier that outputs the source calibrated probabilities. In particular, MLLS uses the following objective:
1 arg min - Σ WEW m71 2-1
MLIL t
log(w" f(zs-n)),
(2)
where f' is the classifier trained on source and W the same constrained set defined above. We can again estimate the target label marginal by simply multiplying the estimated importance weights with the source label marginal.
Baseline estimator Given classifier f. we can estimate the target label marginal as simply the average the classifier output on unlabeled target data, i.c.
Shuseline
Σ f(ita))
(3)
Note that all of the methods discussed before leverage off-the-shelf classifier f. Hence, experiment with classifiers obtained with various deep domain adaptation heuristics to estimate the target label marginal.
Having obtained an estimate of target label marginal, we can simply re- re-weight the classifier with P. Pe j). fj as fj = Σ. (y k)
for all j E V. Note that, we train f on a non-uniform source class-balance (and without re-balancing as in Step 1 of Algorithm [ then we can re-weight the classifier with Ti = j) fj
for all j e y. aeportance-weight @u as f3 ΣS ( = k) fx
F THEORETICAL DEFINITION FOR RELAXED LABEL SHIFT
Domain adaptation problems are, in general, ill-posed David et al. 2010b Several attempts have been made to investigate additional assumptions that render the problem well-posed. One such example includes the label-shift setting, where p(xly) (xly) does not change but that p(y) can. Under label shift, two challenges arise: (i) estimate the target label marginal Pr(y); and (ii) train a classifier f to maximize the performance on the target domain. However, these assumptions are typically, some degree, violated in practice. This paper aims to relax this assumption and focuses on relaxed label shift setting. In particular, we assume that the label distribution can shift from source target arbitrarily but that p(arly) varies between source and target in some comparatively restrictive way (e.g., shifts arising naturally in the real world like ImageNet (Russakovsky et al. 2015) to ImageNetV2 (Recht al. 2.19))
Mathematically, we assume a divergence- restriction on p(xly), i.e., for some small > 0 and distributional distance D, we have max,, ((p:(rly); px (xly)) & but allowing an arbitrary shift in the label marginal p(y). Previous works have defined these constraints in different ways (Wu et al.
2019 Tachet des Combes al. 2020 Kumar et al. 2020).
In particular, we can use Wasserstein-infiitty distance to define our constraint. First, we define Wasserstein given probability measures p,q q on X::
Wa(pp.) inf 4
[f(x)) æ122 f Rd - IR. ..P = g).
22

---

Under review as a conference paper at ICLR 2023
where # denotes the push forward of measure, i.e. for every set S R p(S) p(f Intuitively, We moves points from the distribution p to by distance at most to match the distributions. Hence, our D = max, Wa (p.ll)) Pt distribution constraint KI. or TV distances. We can define our constraint in a representation space 2 obtained by projection inputs E X' with function h X - 2. Intuitively, want to define the distribution distance with some h that captures all the required information for predicting the label of interest but satisfies small distributional divergence in the projected space. However, in practice, it's hard to empirically verify these distribution distances for small enough with finite samples. Moreover, we lack a rigorous characterization of the sense in which those shifts arise in popular DA benchmarks, and since, the focus of our work is on the empirical evaluation with real-world datasets, we leave a formal investigation for future work.
(zlpi) & Similarly,
can define our
G
TARGET MARGINAL ESTIMATION AND ITS EFFECT ON ACCURACY
G.1
VISION DATASETS
Visio Oatzads
Datzmes
Datzatts
(Hee))
(45)
(S5)
Entration
Nose
10.0
3.5
1.5
0.5
Nose
10.0
1.5
0.5
Noxe
10.0
(alpha)
Dinicble Strit (alpha)
Dinichie Stift (alpha]
Vision Datzsets
Viston Datzoets
Vision Datasets
Seare (n eag)
Source be eag.
Soarcs (e eat.
DMaadap: (ham)
TENT (Nora)
Sate/tader (hone)
1.0
Bleadap: (1S)
TENT (85)
Edrmuton Sme
Nalaytader (RS)
08
Edrnuton
0.
0.6
0.2
0.2
0.0
Noee
12.0
3/2
1.2
0.5
None
12.0
5-4
3.2
01.0
Nonn
10.0
Dirchle SkIt (200o]
Divible Skit [26ba]
Diichle
Skift (2pbo)
(a) Target label marginal estimation (61) error with RLLS and classifiers obtained with different DA methods
201
Baxtss Azcra
No
38.0
Nore
30.1
05
1.D Drickle (apta)
Sekk (asta)
(epta)
100
Vairr Detasts
UPP
Vaier Detastss
or Detests
Redatis Acarzy
RedAbre Zeancy
-00
-107
Soanca (TWW)
1000
-10
Tours 340 (890
TENT (FS
100
-107
Fax (TTN)
Polodtaarr (TS
190
DRvadast (197)
T23T (14))
HelgStuddei (199)
009
109
None
25.0
3.0
1.
05
Noma
25.0
3.0
1.0
0.5
Nonc
23.0
3.0
100
0.5
Direttee Suth (sipho
Dinealee Sue (spho)
Devehie Stth ppp)
(b) Relative performance of DA methods when paired with RW corrections
Figure& 8: Target label marginal estimation (f,) error and relative performance with RLLS and classifiers obtained with different DA methods. Across all shift severities (except for a = 0.5) in vision datasets, RLLS with classifiers obtained with DA methods improves over RLLS with a source-only classifier. Correspondingly, we see significantly improved performance with post-hoe RW correction applied to classifiers trained with DA methods as compared to when applied to source-only models.
23

---

Under review as a conference paper at ICLR 2023
G.2
TABULAR DATASETS
1.0
Eatirat Eneer
0.8
10.5
12.0
3.2
Diribles [abpha)
Drrihle: Shitt (agpta)
Diriclee Stiit (elpha]
(a) Target label marginal estimation (C, error with RLLS and classifiers obtained with different DA methods
Temeia Dalsets
Taaela Datwet
Tanalo Dakset
LAP
UPP
Datry (905)
Soarce
Deccket 5ahn (ocks)
Stin (xichs)
Chicklet bHt
(b) Relative performance of DA methods when paired with RW corrections
Figure 9: Target label marginal estimation (E,) error and relative performance with RLLS and classifiers obtained with different DA methods. For tabular datasets, RLLS with classifiers obtained with DA methods improves over RLLSS with a source-only classifier for severe target label marginal shifts. Correspondingly for severe target label marginal shifts, see improved performance with post-hoc RW correction applied to classifiers trained with DA methods as compared to when applied to source-only models.
G.3
NLP DATASETS
0.30
Lenguage
0.30
Languag:
0.30
Lengzage Dauezo
(Noe))
[Noes)
0.30
01:30
Esseatto Exer
0.33
0.33
10.35
0.30
0.30
0.30
0.30
Diickir et Shitt (s(tto)
Dirck Stit
Detcher Sub
(a) Target label marginal estimation (e1) error with RLLS and classifiers obtained with different DA methods
guge
-10
I9a9 1900 (96)
-10
(FW)
D000 (909)
-10P
10:0
5.
Deckes (agka)
Stat (Auba)
(b) Relative performance of DA methods when paired with RW corrections
Figure 10: Target label marginal estimation (f1) error and relative performance with RLLS and classifiers obtained with different DA methods. For NLP datasets, RLLS with source-only classifiers performs better than RLLS with classifiers obtained with DA methods. Correspondingly, see improved performance with post-hoc RW correction applied to source-only models over classifiers trained with DA methods.
24

---

Under review as a conference paper at ICLR 2023
G.4
COMPARISON OF DIFFERENT TARGET LABEL MARGINAL ESTIMATION METHODS
1ong Flrowes
MiLS
MiLS5 Rookew
Bookre
Eat Eut 1:
11.30
0.35
$2.0
(2()00
(appho)
Figure 11: Comparison of different target label marginal estimation methods. We plot estimation errors with different methods with the source-only classifier. For all modalities, we observe a trade- off between estimation error with the baseline method and RLLS (or MLLS) method with severity in target marginal shift.
H RESULTS WITH ORACLE EARLY STOPPING CRITERION
In this section, we report results with oracle early stopping criterion. On vision and tabular datasets,
we observe differences in performance when using target performance versus source hold-out
performance for model selection. This highlights more nuanced behavior than the accuracy-on-the line phenomena (Miller et al. 2021 Recht et al. 2019). We hope to study this contrasting behavior
in more detail in future work.
Vision
target
Accurady
8
sug) (ROM)
Baing BaiiyStaaenn Stuaten (RS (RO))
Stioyttueent
DANN
BN adaps (RS (RO))
CDAMN
IW CDANN
DANN (RS. (RW))
CDANN (R5- RRW)
FuMarthh
FaMaucch (RS. (RW)
Sourcs
Tabular Datasets
Language Datasets
80.0
77.5
source
target
52
source
target
Nut Reanac
75.0
Avg. Aocuracy
72.5
20.0
67.5
65.0
62.5
60.0
DAWN
IW- DANN
COMN
to COMNN
Sounte (w/j eng)
Source angl (RRV)
Ponald (RA)
CDANN (RS (RV)
DANN
CLANN
IW DNN
IW-CDAM
COMNN RWW
(o ave) (B9))
Paoute
DORN (eS
Source
DANN
Poeadoo abel
Figure 12: Average accuracy of different DA methods aggregated across all distribution pairs in each modality. We compare the performance with early stopping point obtained with source validation performance and target validation performance.
25

---

Under review as a conference paper at ICLR 2023
1
RESULTS ON INDIVIDUAL DATASETS
cifar10
50
80
70
Source (adv) (RW)
Source (ade)
CORAL
CORAL (RS (RS+-WW) RW)
Source ( ang)
TENT
Source (w ang) (RW)
IWN-CU CW-N
CDANN
(RWW)
NoioyS tuders (RS (RS-+RW) RWW
NeicyStuder Student
IW-DANN
DANN
SENTRR
FilMaach
COWNN (RS: (RS--RV) FRW)
DR adooc (RS (RW)
TENT (RS (3---
Fachlaach (RS) (RW)
Bl
Source (wi
Source
None
RW
RS
RS RW
(w aug)
-15
Deckkie Skik
Desiae 9 Japa
Dosc Sull (449)
DANN
COANN
IN-CDANN
Hooes
30.8
0.5
Mare
300
33
03
Mene
X00
33
05
Dricle St Japa
Figure 13: CIFARIO. Relative performance and accuracy plots for different DA algorithms across various shift pairs CIFAR10.
26

---

Under review as a conference paper at ICLR 2023
cifar100
80
Accuracy
60
80
30
Source (adv) (RW)
Source (alk)
Sourca (wy aug (RW)
aug)
CORAL
CORAL (RS+RWW)
CDANN
IW: WDADAN
SENTRR
DANN
IW-DAMN
CDANN (RS (RW)
(RS-+RW)
Source (w aug)
(RW)
Saurce BR adaant
TENT
Noisy Studkon
SoloyStudass (AS) (RW)
RWW)
FarMarch
FoMMatc (RS RRW)
TENT
BN- SN-asdape
None
RW
RS
RS RW
(w aug)
Deick Skil C
Dricle St Japan
Diicles (4gia)
DANN
COANN
IW-CDANN
Daaee Actune
Tore
30:8
1.0
0.5
-17
Mare
300
33
03
-17
35e
X00
33
05
20ich
Dricle St Jagan
D Ditclen (agia
Figure 14: CIFAR100. Relative performance and accuracy plots for different DA algorithms across various shift pairs CIFAR100.
27

---

Under review as a conference paper at ICLR 2023
camelyon
91
Accuracy
90
80
75
70
Source (weo anj)
Sourca (wy lo aug) (RW)
Source (w aug) (RW)
CORAL (RS #RW)
Source (adv) (RW)
NoieyStudemt
NoloyStudent (RS-+ RRW)
Source (Mik)
IW CDANN
CDANN
30V-adappt
IVL IWLDANN
FiaMaach
DANN
Faraatc (RS) (RW)
DANN (RS-+RW) RRW)
TENT (RS (RS- (RW)
BN adape (RS FRW)
Source
CDANN
None
RW
RS
RS RW
(w axg)
Deick Sk.l (1ah))
Dricae 2 Japa
Dricle Skltt Japla)
DANN
COANN
IW-CDANN
Deaee Acount
Tore
30:8
E5
Mare
3000
33
1G
03
-17
3ene
X000
33
05
NoloSSudant
D0ich
Driclet St Jainn
D Ditcle 918 (agaa
Figure 15: Camelyon. Relative performance and accuracy plots for different DA algorithms across various shift pairs in Camelyon.
28

---

Under review as a conference paper at ICLR 2023
fmow
75
Accuracy
65
55
5
SENTRY
Source (ode) (RW)
DSKNN (RS. (RW)
Sourgs (adv)
CDANN (RS- RRW)
DANN
IW-DANN
CDMNN
IW- WDCCANN
Source (we auc)
CORAL
Source (wy aug (RW)
RN-adapt (RS RWW)
TENT (RS RW)
TENT
FirMatch R(W)
(AS RRW)
SoloyStuden: (RS RRW)
NoieyStudent
Source aug)
Source (w aug (RW)
None
RW
RS
RS RW
(w axg)
Deiik Sk.l (1
Driclee St Japa
Dilcle (4p9a
DANN
COANN
IW-CDANN
70mh
30:8
30
0.5
-17
3000
33
100
03
-17
35ne
30,0
33
05
D0ich
Driclet St Japan
D Ditcier (agit
Figure 16: FMoW. Relative performance and accuracy plots for different DA algorithms across various shift pairs FMoW.
29

---

Under review as a conference paper at ICLR 2023
entity13
Accuracy
80
75
20
65
60
CORAL
CORAL RW))
Soornn (adl))
SENTRY
Source (adv) (RW)
Source (w/ ang) (RW)
TENT
DANN
WNNDANN
IW CDANN
CDANN
Source [w aug)
FulMarch
NoieyStudden
BN sddaps (RS- RWW)
(HS RRW)
DANN (RS) (RW)
CDANN (RS (RS- RWW)
Noiey (RS (RW)
FaMMath (RS RRW)
BN
aug)
None
RW
RS
RS RW
(w aug)
Deick Skil G1
Dricae St papa
Dricke (4pla)
DANN
COANN
IW-CDANN
Daaee Aconto
70mh
30.8
0.5
Mare
3000
33
03
-17
1ene
X00
33
05
20/ch S8.1
Di Driclet St Jagan
D Dilcler (agia)
Figure 17: 13. Relative performance and accuracy plots for different DA algorithms across various shift pairs 13t
30

---

Under review as a conference paper at ICLR 2023
entity30
75
Accuracy
50
CORAL
SENTRY
CORAL (RS. RWW)
Sourgs (adv)
Source (adv) (RW)
Source (w/ aug) (RW)
DANN
Bkkataap
TENT
IN-DANN
COANN
NoieyStudden
IW-C IW-CDANN
DANN (RSI (RW)
Source (wr aus)
FortArrch
aadaaaae (MS (FW)
TENT (RS) (RW)
Student (RS RWW)
CDMNN (RS (RW)
FuMatch (RS RRW)
aug)
None
RW
RS
RS RW
(w aug)
Deick Skik G
Dricle S Japa
Dricke (4g1a)
DANN
COANN
IW-COANN
Deaee Actune
Tore
30.8
0.5
Mare
3000
33
03
-17
X00
33
05
20/ch
Driclet St Jagan
D Ditcle S00 (agia)
Figure 18: Entity30. Relative performance and accuracy plots for different DA algorithms across various shift pairs Entity30.
31

---

Under review as a conference paper at ICLR 2023
living17
80
Accuracy
Source (adv) (RW)
Source (ank)
SEN TYY
CORAL
CORAL (RS++RW)
Source (w/ aug (RW)
TENT
39-aatapt
Source [wr aud)
RWA)
IW DANN
aug) (RW)
BN--adape (KS (RW)
TENT (RS RW)
WW-CDANN
CDANN
FadMarch (HS- RWW)
DANN
NoisyStucdon
DANN (RS- RWW)
CDANN (RS (RS-URW) (RW)
Faabasch
Noiey
None
RW
RS
RS RW
(w aug)
Deick Skll G
Driclet St Japa
Diicke Sul. (4gia)
DANN
COANN
IW-COANN
30.8
055
Mare
3000
33
03
-17
35ne
X00
33
05
20/ch S8/
Driclet St Jaipna
Ditcle 9100 (agia
Figure 19: Living 17. Relative performance and accuracy plots for different DA algorithms across various shift pairs Living17.
32

---

Under review as a conference paper at ICLR 2023
nonliving26
20
Accuracy
60
SENTRY
CORAL
CORAL (RS. (RW)
Source (w) aug)
Source (w/s aug) (RW)
Source (adv)
DANN
WADDAN
CDANU
BN-adays
FixMaach
DANN (RS--RW)
TENT
NoieyStucie Stucien
TENT (RS+ (RW)
Source aug)
CDARN (RS: RRW)
IW-CDANN
Studlet (RS RWA)
FodMatc (RS RRW)
BN SN-adapp (RS-
Source (ade)
None
RW
RS
RS RW
(w axg)
Decck Skil G
Dricle St Japan
Dricke Suit (agia
DANN
COANN
IW-CDANN
30.8
0.5
Mare
3000
33
03
-17
35ne
X00
33
05
20/ch
Driclet 21 24i
Dilcle (agaa
Figure 20: Nonliving 26. Relative performance and accuracy plots for different DA algorithms across various shift pairs Nonliving26.
33

---

Under review as a conference paper at ICLR 2023
domainnet
65
Accuracy
50
65
40
Source (RW)
Source (ark))
CORAL C.RRALL
Source (w aug) (RW)
NoieyStuder (RS RRW)
addaa (RS. (RS+ RRW)
TENT (RS (RW)
Source (w aug)
TENT
Source (wy aug (RW)
Noies Studeet
DANN
IW NNNDAN
Source (we aug)
DSNN (RS+RWW)
WC-DAAN
CDANN (RS. (RW)
CDANN
SENTRY Furatth (RS (RS-URW) (WW)
FaaMasch
None
RW
RS
RS RW
(w axg)
Deick (A (190))
Dricae S Japa
Diicke Su0 (4g90
DANN
COANN
IW-CDANN
Torh
30:8
E5
Mare
3000
33
03
-17
X00
33
05
NoloSStdant
Deiciat
Dricle St Jagnn
D Diicie 908 (agit
Figure 21: NomainNet. Relative performance and accuracy plots for different DA algorithms across various shift pairs Net.inNet.
34

---

Under review as a conference paper at ICLR 2023
officehome
Accuracy
60
80
CORAL
CORAL (AS+RRW)
Source (adv) (RW))
Sourcs (2dv)
Source aug) (RW))
Source aug) (RW)
Source [w aug)
NoioyStuden:: (RS (RW)
Source (we aug)
Bi--adap (RS RFW)
TENT RWW)
SoudeStudent
Bl-eddapt
TENT
CDMNN (RS RW))
DANN (RW-
SENT TRY
FasMarch IW WILDANN
DANN
FieMatch (RS (RW+
(W-CDANN
CDANN
None
RW
RS
RS RW
(w axg)
-17
Deick Skik (rana)
Diicae Sut (4gor)
DANN
COANN
IW-CDANN
Netee Acoute
30.8
E5
Mare
3000
33
03
-17
35ne
30,0
33
05
201ch
Driclet 2t Jagan
D. Ditclec (agit
Figure 22: Officchome. Relative performance and accuracy plots for different DA algorithms across various shift pairs in Officehome.
35

---

Under review as a conference paper at ICLR 2023
visda
80
Accuracy
70
50
Source aug (RW)
CORAL (RS+
CORAL
(w aug)
StudeStudent
(RS--RW)
Saurce (adv) (RW)
Source (w/ aug) (FWW
Source (weo aug)
SNLadaps RW)
TENT (RS (RW)
TENT
DANN
IW WN-DANV
FixMArch
CDANN
(VA- WA-CDANN
DANN (RS (RS-RRW)
FurMatch (RS (RS- (RW)
CDMNN (RS FRW)
Source
None
RW
RS
RS + RW
(w axg)
Deck Sal (1an))
Dricae S Japa
Dicie Sun (4p90
DANN
COANN
IW-CDANN
70mh
30:8
E.5
Mare
30.0
33
03
35ne
30,0
33
05
SuloyStdantt
20/ch
Drichtc St Jagan
D. Diicien (agit1
Figure 23: Visda. Relative performance and accuracy plots for different DA algorithms across various shift pairs in Visda.
36

---

Under review as a conference paper at ICLR 2023
civilcomments
92
Aug Accoracy
88
86
92
DANN
CDANN
IW-DANN
IN- CDANN
Foudolabb Labal
Pacudll (RS (RW)
DANN (RS (RV)
Source (wio sug) (RW)
RW
Source (w/o
DVAN
CJANN
-30
-1P
-37
-30
-39
6.6
Mmm
10:3
IV DANN
MW-CMMNN
PacudoLabe
107
107
187
Teckict Sant
Deuker Sart (scke)
Figure 24: Civilcomments. Relative performance and accuracy plots for different DA algorithms across various shift pairs in Civilcomments.
37

---

Under review as a conference paper at ICLR 2023
retiring_adult
84
Accuracy
82
80
78
76
74
DANN
IW-DANN
Source (w/ aug)
CDANN
PseudeLako Lbbll
IW NN-C
DANN (RV)
Source (wio aug) (RW)
Paeudol abel (RS RRV
RW
Source (w/o
DANN
CJANN
dounce
-30
-10
-37
-30
-39
6.6
1mmm
12:3
2.50
IV DANN
MW-COMNN
PaoddLabd
1.7
107
187
Tertict Sant
Deuker Sart (mich))
Figure 25: Retiring Adults. Relative performance and accuracy plots for different DA algorithms across various shift pairs in Retiring Adults.
38

---

Under review as a conference paper at ICLR 2023
mimic_readmission
64
62
Aug. Acciracy
60
66
54
52
DANN
IW-DANN
CDANN
Poeudol abel
Source (w jo aug)
Pacudol (RS (RW))
Source (wio aug) (RW)
COAM (RS RR)
DAMI
RW
Source (w/o
DANN
CLANN
dounce
-33
-30
-10
-37
-30
-39
6.6
Mmm
12:3
IV DANN
MW-CMMNN
PaoddLLad
107
107
187
Sant
Decker Sart (scke)
Edghtr
Figure 26: Mimic Readmissions. Relative performance and accuracy plots for different DA algorithms across various shift pairs in Mimic Readmissions.
39

---

Under review as a conference paper at ICLR 2023
J
AGGREGATE ACCURACY WITH DIFFERENT DA METHODS ON EACH DATASET
Dataset
Source
DANN
IW-DANN CDANN IW-CDANN PscudoLabel
Civilcomments
86.85
86.62
86.95
86.91
87.16
87.4
Source
DANN
CDANN
PseudoL.abll
Dataset
RS+ RW
RS+ RW
RS+ RW
None RW
None
RS
None
RS
None RW
Civilcomments
86.8 89.1
86.6
88.8
87.1
88.8
86.9
89.0
86.9 88.9
87.4 89.3
86.9
88.6
Table 2: Results with different DA methods on NLP datasets aggregated across target label marginal shifts.
Dataset
Source
DANN
IW-DANN
CDANN IW-CDANN PseudoLabel
Retiring Adult
77.44
77.17
77.35
78.15
78.44
78.30
Mimic Readmission
57.57
56.36
56.48
56.67
56.71
57.35
Source
DANN
CDANN
PseudoLabel
Dataset
RS+
RS+ RW
RS+
None RW
None RW
RS
None RW
RS
None RW
RS
RW
RW
Retiring Adults
77.4 80.0
77.2
79.5
77.4
79.4
78.1
80.5
78.1
80.4
78.3
80.8
78.5
80.8
Mimic Readmissions
57.6
59.0
55.1
57.3
59.2
56.7
56.8
57.4
57.4
57.7
57.7
57.9
Table 3; Results with different DA methods on tabular datasets aggregated across target label marginal shifts.
Dataset
Source
(wo aug)
Source (w aug)
BN- adapt
TENT
DANN
IW- DAN
CDAN
IW- CDAN
Fix- Match
Noisy- Student
Sentry
CIFAR-10
89.69
89.14 14
89.21
89.20
90.86
90.78
90.00
89.93
91.87
90.72
91.83
CIFAR-100
65.99
76.69
77.57
77.58
74.80
74.81
74.57
74.66
79.03
77.60
74.74
FMoW
64.00
68.99
65.52
66.55
60.11
60.33
60.79
61.05
68.37
68.90
51.06
Camelyon
77.42
76.95
85.70
82.48
86.66
85,89
85.45
84.27
86.29
79.29
86.81
Domainnet
52,37
50.50
50.66
51,12
51.91
52.05
54.40
54.29
57.96
51.49
55.16
13
76,93
80.07
77.99
78.04
78.26
78.75
79.74
79.28
80.25
80.37
73.58
Entity30
62.61
69.83
68.09
68.09
67.90
68.36
68.51
69.34
69.95
69.10
58.51
Living17
64.13
69.30
68.84
68.82
72.12
69.87
70.72
70.65
72.86
72.16
53.44
Nonliving26
$4.75
63,95
62.60
63.02
61.69
61.99
62.53
64.51
62.98
63.60
44.82
Officehome
59.89
59.45
60.59
60.82
66.05
65.79
66.19
66.15
65.48
60.47
65.37
Visda
58.47
53.41
59.98
60.96
69.69
69.79
72.55
72.80
72.02
53.51
72.23
Avg
66.02
68.94
69.70
69.70
70.92
70.77
71.40
71.54
73.37
69.75
66.14
Table 4: Results with different DA methods on vision datasets aggregated across target label marginal shifts. While no single DA method performs consistently across different datasets, FixMatch seems to provide the highest aggregate improvement over source-only classifier in our testbed.
40

---

Under review as a conference paper at ICLR 2023
Source
BN-adapt
CDANN
FixMatch
Dataset
RS+ RW
RS+ RW
RS+
None RW
None
RW
RS
None
RW
RS
None
RW
RS
RW
CIFAR-10
89.1
89.4
89.2
91.4
92.1
92.9
90.0
91.3
91.4
92.5
91.9
93.1
93.6
94.1
CIFAR-100
76.7
77.5
77.6
78.8
77.9
79.0
74.6
75.8
74.1
75.3
79.0
79.6
79.1
79.8
PMoW
69.0
70.3
65.5
67.2
66.2
65.6
60.8
61.9
57.0
55.2
68.4
69.4
64.9
66.7
Camelyon
77.0
77.9
85.7
85,9
88.5
89.3
85.5
85.8
87.9
88.5
86.3
87.0
86.6
86.8
Domainnet
50.5
48.2
50.7
50.1
51.4
49.8
54.4
54.2
54.7
54.3
58.0
57.5
58.4
57.8
En
80.1
80.9
78.0
79.4
79.8
80.7
79.7
80.2
80.6
81.4
80.3
81.4
82.4
En
69.8
70.1
68.1
69.2
69.1
70.0
68.5
69.6
69.4
70.5
70.0
71.6
70.1
71.2
Living17
69.3
69.9
68.8
69.7
69.6
70.1
70.7
71.3
72.9
72.7
72.9
72.8
72.3
71.9
Nonliving26
63.9
64.5
62.6
63.0
63.7
63.9
62.5
62.9
63.8
64.0
63.0
64.7
63.9
64.8
Officehome
59.4
57.9
60.6
60.5
60.9
60.4
66.2
66.3
66.1
65.1
65.5
64.9
66.5
66.1
Visda
53.4
52.1
60.0
60.6
59.5
58.8
72.6
72.6
75.3
75.3
72.0
72.5
73.5
73.8
Avg
68.9
69.0
69.7
70.5
70.8 70.9
71.4
72.0
72.1
72.3
73.4
74.1
73.7
74.1
TENT
DANN
NoisyStudent
Dataset
RS+
RS+ RW
RS+
None
RW
None
None RW
RW
RW
CIFAR-10
89.2
91.4
92.1
92.9
90.9
92.3
91.5
92.6
90.7
90.8
90.6
90.7
CIFAR-100
77.6
78.8
78.0
79.0
74.8
75.9
74.8
76.1
77.6
78.0
77.9
78.0
FMoW
66.6
67.4
66.7
66.1
60.1
61.6
56.4
54.5
68.9
69.8
67.1
68.0
Camelyon
82.5
82.7
87.8
88.9
86.7
87.3
88.4
88.8
79.3
79.1
79.2
79.3
Domainnet
50.6
51.8
50.3
51.9
52.1
53.6
51.5
49.8
51.3
49.5
Entity13
78.0
79.5
79.8
80.8
78.3
79.4
79.7
80.8
80.4
81.5
80.6
81.7
Entity30
68.1
69.2
69.1
70.1
67.9
69.2
69.0
69.8
69.1
70.1
69.3
70.3
Living17
68.8
69.7
69.6
70.1
72.1
73.0
71.8
72.3
72.2
71.1
69.3
69.4
Nonliving26
63.0
63.4
63.3
61.7
62.4
63.1
63.6
64.3
63.2
Officehome
60.8
60.4
60.9
60.4
66.1
66.1
66.5
65.3
60.5
59.5
60.8
59.5
Visda
61.0
61.5
60.3
59.6
69.7
69.9
73.1
73.2
53.5
51.5
55.7
54.3
Avg
69.7
70.4
70.8
71.1
70.9
71.7
71.6
71.8
69.7
69.6
69.5
69.6
Table Results with DA methods paired with re-sampling (RS) and re-weighting (RW) correction (with RLLS estimate) aggregated across target label marginal shifts for vision datasets. RS and RW seem to help for all datasets and they both together significantly improve aggregate performance over correction for all DA methods.
41

---

Under review as a conference paper at ICLR 2023
K DESCRIPTION OF DEEP DOMAIN ADAPTATION METHODS
In this section, we summarize deep DA methods compared our RLSBENCH testbed. We also discuss how each method combines with our meta-algorithm to handle shift in class proportion.
K.1 SOURCE ONLY TRAINING
As a baseline, we consider empirical risk minimization on the labeled source data. Since this simply ignores the unlabeled target data, call this as source only training. As mentioned in the main paper, we perform source only training with and without data augmentations. Formally, we minimize the following ERM loss:
(U(((:)) n 2
I T only only(5)
where T is the stochastic data augmentation operation for vision datasets and is a loss function. For NLP and tabular datasets, T is the identity function. Throughout the paper, we cross-entropy loss minimization. Unless specified otherwise, we use strong augmentations as the data augmentation technique for vision datasets. For NLP and tabular datasets, we do not use any data augmentation.
As mentioned in the main paper, we do not include re-sampling results with source only model as trained only on source data and we observed no differences with just balancing the source data (as for most datasets source is already balanced) in our experiments. After obtaining a classifier f. we can first estimate the target label marginal and then adjust the classifier f with post-hoe re-weighting with importance ratios (y) = (v)pP
Adversarial training of a source only model Along with standard training of source only model with data augmentation, we experiment with adversarially robust models (Madry et al. 2017). To train adversarially robust models, we replace the standard ERM objective with robust risk minimization objective:
1 a
T only (adv)
Σ ((R(((()))))) n
where R(-) performs the adversarial augmentation. In our paper, we use targeted Projected Gradient Descent (PGD) attacks with & perturbation model.
K.2 DOMAIN-ADVERSARIA TRAINING METHODS
Domain-adversarial trianing methods seek to learn feature representations that are invariant across domains. These methods aimed at practical problems with non-overlapping support and motivated by theoretical results showing that the gap between in- and out-of outtof-distribution performance depends on some measure of divergence between the source and target distributions (Ben-David al.. 2010a Ganin et al., 2016). While simultaneously minimizing the source error, these methods align the representations between source and target distribution. To perform alignment, these methods penalize divergence between feature representations across domains, encouraging the model to produce feature representations that are similar across domain.
Before describing these methods, we first define some notation. Consider a model f = go h. where h: ddd the featurizer that maps the inputs some d dimensional feature space, and the head g: d - A k-1 maps the features to the prediction space. Following Sagawa et al. (2021) with all of our domain invariant methods, we use strong augmentations with source and target data for vision datasets. For NLP and tabular datasets, do not use any data augmentation.
DANN DANN was proposed in Ganin et al. (016)). DANN approximates the divergence between feature representations of source and target domain by leveraging domain discriminator classifier. Domain discriminator fa aims to discriminate between source and target domains. Given a batch of inputs from source and target, this deep network fa classifies whether the examples from the source data or target data. In particular, the following loss function is used:
Leuomiia (fd) n
Σ {(fa(b(((())))0))
1
Σ I)
(6)
m i-n+1
42

---

Under review as a conference paper at ICLR 2023
where {21, ..,............................. In. are source examples and (In+11,- .... Im+a) target examples. Over- all, the following loss function is used to optimize models with DANN:
LDANN (h, g, fa) Liource only (g o h) Aaomain disc. (fa)
(7)
LIDANN(h, 9: Sa] is maximized with respect the domain discriminator classifier and LDANN (h, g, Sa) minimized with respect to the underlying featurize and the source classifier. This is achieved by gradient reversal layer in practice. To train, three networks, we use three different learning rate ny Tly
and fe We discuss these hyperparameter details in App. L We adapted our DANN implementation
from Sagawa al. (202T) and Transfer learning library (Jang al. 2022
CDANN Conditional Domain adversarial neural network is variant of DANN Long et al. 2018). Here the domain discriminator is conditioned on the classifier g's prediction. In particular, instead of training the domain discriminator on the representation output of h, these methods operate the outer product between the feature presentation h(x) at an input æ and the classifier's probabilistic prediction f g h(x)) (i.e., h(r) @ f())). Thus instead of training the domain discriminator classifier fa on the d dimensional input space, they train it on d x k dimensional space. In particular, the following loss function is used:
(UALTMT(T))) 1 ntm TE Σ ((A))MM(()))) 1), (8)
I LCDAN domain disc.. (sa: 9; h)
where { T2,,.... 3n) are n source examples and (Ia+1 1, Im+n) are m target examples. The
overall loss is the same as DANN where Ldomain disc. (Ja) is replaced with LEDAN domain disc. (Sa, 9, ht.
We adapted our implementation for CDANN from Transfer learning library (Jiang et al. 2022).
To adapt DANN and CDANN to our meta algorithm, at each epoch we can perform re-balancing of source and target data as in Step and 4 of Algorithm [ After obtaining the classifier f. we can use this classifier to first obtain an estimate of the target label marginal and then perform re-weighting adjustment with the obtained estimate.
IW-DANN and IW-CDANN Tachet et al. (2020) proposed training with importance re-weighting correction with DANN and CDANN objectives to accommodate for the shift in the target label proportion. In particular, at every epoch of training they first estimate the importance ratio oe (with BBSE on training source and training target data) and then re- re-weight the domain discriminator objective and ERM objective. In particular, the domain discriminator loss for IW-DANN can be written as:
1 a Σ (w
n+m Σ b=++1
L 1 domain disc. (fa)
((fa(M((())))) 1),
(9)
where we multiply the source loss with importance weights. Similarly, we can re-write the source only training objective with importance re-weighting as follows:
1 n n
L Laurce only (f)
Σ (()))((((((a))..u)))
(10)
Overall, the following objective is used to optimize models with IW-DANN:
LIW.DAN(h, g, fa) _ only h) domain disc. (fa)
(11)
where the importance weights updated after every epoch with classifier obtained in previous step. Similarly, with using importance re-weights with the CDANN objective, we obtain IW-CDANN objective.
In population, IW-CDANN and IW-DANN correction matches the correction with our meta-algorithm for DANN and CDANN. However, the behavior this importance re- re-weighting correction can be different from our mlga-algo for over-paramen pver-parameterized models with finite data (Byrd & Lipton 2019). Recent empirical and theoretical findings have highlighted that importance re-weighting have
43

---

Under review as a conference paper at ICLR 2023
minor effect overparameterizze models when trained for several epochs (Byrd & Lipton 2019; Xu et al. 2021). On the other hand, with finite samples, re-sampling (when class labels are available) has shown different and promising empirical behavior (An et al. 2020 Idrissi et al. 2022). This may highlight the differences in the behavior of IW-CDANN (or IW-DANN) with our meta algorithm CDANN (or DANN).
We refer to the implementation provided by the authors Tachet et al. 2020).
K.3 SELF-TRAINING METHODS
Self-training methods leverage unlabeled data by 'pseudo-labeling' unlabeled examples with the classifier's 's own predictions and training on them as if they were labeled examples. Recent self- training methods also often make use of consistency regularization, for example, encouraging the model to make similar predictions augmented versions of unlabeled example. In our work, we experiment with the following methods:
PseudoLabel [eee et al. 2013) proposed PseudoLabel that leverages unlabeled examples with classifier's own prediction. This algorithm dynamically generates psuedolabels and overfits on them in each batch. In particular, while pseudolabels are generated on unlabeled examples, the loss is computed with respect to the same label. PseudoLabel only overfits to the assigned label if the confidence of the prediction is greater than some threshold т.
Refer to T as the aatteauugmntttion technique (i.e., identity for NLP and tabular datasets and strong augmentation for vision datasets). Then, PscudoLabel uses the following loss function:
1 Σ (U(T(()) A Ti
m
LPscddoLbbel (f)
Σ (f(T(x))))) (-++1
max (T(())))
where Dr arg max,, (z:))) PseudoLabel increases X between labeled and unlabeled losses over epochs, initially placing 0 weight on unlabeled loss and then linearly increasing the unlabeled loss weight until it reaches the full value of hyperparameter À some threshold step. We fix the step at which dG reaches its maximum value be 40% of the total number of training steps, matching the implementation (Sohn al.
2020 Sagawa et 2021).
FixMatch Sohn et al. 2020) proposed FixMatch as variant of the simpler Pseudo-label method (Lee et al. 2013). This algorithm dynamically generates psuedolabels and overfits on them in each batch. FixMatchh employs consistency regularization on the unlabeled data. In particular, while pseudolabels are generated on weakly augmented view of the unlabeled examples, the loss is com- puted with respect to predictions to encourage a model to make predictions on weakly augmented data consistent with the strongly augmented example. Moreover, FixMatch only overfits to the assigned labeled with weak augmenta- tion if the confidence of the prediction with strong augmentation is greater than some threshold т.
strongly augmented view. The intuition behind such update
Refer to Tweak the weak-augmentattoon and Tamong as the strong- -uggmetatiminoon function. Then, FixMatch uses the following loss function:
TrixMatch (J)
Σ ((((Tsinong
E(f(T (2TT) V:)) -1 max fy (Tarong (x:)) 3 T Σ
i++1
where G. arg max, fy (Tweak We adapted our implementation from Sagawa et al. 1021
which matches the implementation of SShnn et al. (2020) except for one detail. While Sohn al. (2020) augments labeled examples with weak augmentation, Saaawa et al. (021) proposed to strongly augment the labeled source examples.
NoisyStudent Xie et al. (2020) proposed different variant of Pseudo-labeling, Unlike FixMatch, Noisy Student generates pseudolabels, fixes them, and then trains the model until convergence before generating pseudolabels. The first of pseudolabels are obtained by training initial teacher
44

---

Under review as a conference paper at ICLR 2023
model only on the source labeled data. Then in each iteration, randomly initialized models fit the labeled source data and pseudolabeled target data with pseudolabels assigned by the converged model in the previous iteration. Noisy student objective can be summarized as:
NoisyStudeent (f
Elf (trrong(()) v1)) + 1 Σ 71
770 Σ
1
124-
(2:). (i)).
where 3. = arg max, f f (Twekk is computed with the classifier obtained at - step. Note that the randomly initialized model each iteration uses dropout of p 0.5 in the penultimate layer. We adopted our implementation of NoisyStudent to Sagawa et al. (202T). To initialize the initial teacher model, we use the source-only model trained with strong augmentations without dropout.
Prabhu et al. 2021 proposed different variant of pseudolabeling method. This method aimed tackle DA under relaxed label shift scenario. SENTRY incorporates a target instance based on its predictive consistency under a committee of strong image transformations. In particular, SENTRY makes strong augmentations of an unlabeled target example and makes prediction on those. If the majority of the committee matches the prediction on the sample example with weakk-ugmentttion then entropy is minimized on that example, otherwise the entropy is maximized. Moreover, the authors employ an 'information-entropy objective aimed match the prediction at every example with the estimated target label marginal. Overall the SENTRY objective is defined as follows:
SENTRY
1 n
mtn
ISENTRY(f)
Σ
(( strong (ttong((:))))))) (Ri), +
Σ
Σ
fklv log(P((y j))
n2
m+n
+ Aunsup
fuly = jla;) Sog(sa(v (1)))) (21(x) - 1),
i=
where 1(x) e {0, 1} is majority vote output of the committee consistency. For more details, we refer the reader to Prabbu al. (202T). Additionally, each training epoch, SENTRY balances the source data and pseudo-balances the target data. We adopted our implementation with the official implementation in Prabhu et al. (202T) with minor differences.
Since Fix-Match, Stunnt, and Sentry use strong aata-augmeetationn in their implementation, the applicability of these algorithms is restricted vision datasets. For NLP and tabular datasets, we only train models with PseudoLabel as it doesn't 't rely on any augmentation technique.
K.4 TEST-TIME TRAINING METHODS
These take an already trained source model and adapt few parameters (e-g. batch norm parameters, batch norm statistics) on the unlabeled target data with an aim to improve target performance. Hence, we restrict these methods to vision datasets with architectures that use batch norm. These methods computationally cheaper than other DA methods in the suite as they adapt classifier on-the-fly. We include the following methods our experimental suite:
BN-adapt Li et al. 2016) proposed batch norm adaptation. More recently, Schneider et al. (2020) showed gains with BN -adat on common corruptions benchmark. Batch norm adaptation is applicable for deep models with batch norm parameters. With this method we simply adapt the Batchnorm statistics, in particular, mean and std of each batch norm layer.
TENT
Wang et al. (021) proposed optimizing batch norm parameters minimize the entropy of the predictor on the unlabeled target data. In our implementation of TENT, we perform BN-adapt before learning batch norm parameters.
CORAL Sun et al. 2016) proposed CORAL to adapt model trained on the source target by whitening the feature representations. In particular, say Σ, is the empirical covariance of the target data representations and Σ, is the empirical covariance of the source data representations, CORAL adjusts a linear layer g on target by re-training the final layer on the outputs: Σ 1112
45

---

Under review as a conference paper at ICLR 2023
DARE (Rosenfeld al. 2022 simplified the procedure and showed that this is equivalent training
((x) a linear head h on Σ,-/2 (xx and whitening target data representations with Σ, E-1/"((>>> before input
to the classifier. We choose to implement the latter procedure it is cheap to train single classifier in multi-domain datasets.
With our meta-algorithm, before adapting the source-only classifier with test time adaptation methods, we use it to perform the re-sampling correction. After obtaining the adapted classifier, we estimate target label marginal and use it to adjust the classifier with re- weightiing.
L HYPERPARAMETER AND ARCHITECTURE DETAILS
L.1 ARCHITECTURE AND PRETRAINING DETAILS
For all datasets, we used the same architecture across different algorithms:
CIFAR-10: Resnet- 18 (He et al. 2016
pretrained on Imagenet
CIFAR-100: Resnet-18 (He et al.
2016)
pretrained on Imagenet
Camelyon: Densenet- 121
(Huang et al.
2017) not pretrained on Imagenet as per the
suggestion made in (Koh et al. 2021
FMoW: Densenet-12) 121 (Huang et al.
2017 pretrained on Imagenet
BREEDs (Entity13, 30, Living
Nonliving26): Resnet- 18 (He et al., 2016) nor
pretrained on Imagenet as per the suggestion in (Santurkar al. 2021). The main rationale
is to avoid pre-training on the superset dataset where we are simulating sub- cab-population shift.
Officehome: Resnet-50 (He et al.
2016)
pretrained on Imagenet
Domainnet: Resnet-50) (He et al.
2016) pretrained on Imagenet
Visda: Resnet-50 HHe et al. 2016) pretrained on Imagenet
Civilcomments: Pre-trained DistiBBERT-bas-uucaaee (Sanh et al. 2019
Retiring Adults: We
MI.P with 2 hidden layers and 100 hidden units in both of the
hidden layer
Mimic Readmissions: We use the transformer architecture described in Yao et al. 2022
Except for Resnets on CIFAR datasets, we used the standard pytorch implementation (Gardner al. 2018). For Resnet on cifar, we refer the implementation here: https::/gitthb com/ kuangliu wwwpilupppphchhhhoooooo cifar For all the architectures, whenever applicable, we add antialias-
ing (Zhang 2019). We use the official library released with the paper.
For imagennt-pretrraneed models with standard architectures, we use the publicly available models here: htttt:://ytttrreeee wwg/vviion...... sttalaaaaaeeeeeeeee wwwaaleeemodels..ommmaooooeeeale .htll For imagenet-pretraine models on the reduced input size images (e.g. 10), we train model on Imagenet on reduced input size from scratch. We include the model with our publicly available repository. For bert- based models, we use the publicly available models here: formers/
L.2 HYPERPARAMETERS
First, we tune learning rate and regularization parameter by fixing batch size for each dataset that correspond to maximum we can fit to 15GB GPU memory. We set the number of epochs for training as per the suggestions of the authors of respective benchmarks. Note that we define the number of epochs as full pass over the labeled training source data. We summarize learning rate, batch size, number epochs, and & regularization parameter used our study Table
For each algorithm, we use the hyperparameters reported in the initial papers. For donain-adversarial methods (DANN and CDANN), we refer to the suggestions made in Transfer Learning Library (Jiang al. 2022). We tabulate hyperparameters for each algorithm next:
46

---

Under review as a conference paper at ICLR 2023
Epuch
Batch
Eg regularization
Learning rull
CIFARIO
50
200
0.0001 (chosen from (0.0301,0.00..1ee) 1e-5,, Je.....5)))
0.01 (chosen from (0.001, (.01,0.0001))
CIFAR100
50
200
0.0001 (chosen from (0.0001, 0.001,1e-5, 1e-5,
0.01 (chosen from (0.001, 0.01, 0.0001)
Camelyon
96
(chosen from [0.01,0.07), 0.001, 0.0001, 0.0))
0.03 (chosen from (0.003, 0.3, 0.0003 0.03))
FMoW
30
64
0.0
(chosen from (0.0001, (0.001, 0.000111--5500000
0.0001 (chosen from (0.001, 0.01, (.0000)))
En
256
(5e-5. 1c-4, lo-5))
0.2 (chosen from (0.1, 0.5, 0.01,
En
40
256
Se-5 (chosen from (5e-5, Ie-4, Ie-5))
0.2 (chosen from {0.1, 0.5, 0.2, 0.010,00000)
Living!?
256
5e-55 Se.4, Ie-4, le-5))
0.2 (chosen from [0.1, 0.5, 0.2, 0.01,0.0))
Nonliving26
40
256
0 5e-5 (chosen from [5e-5, Se-4, Ie-4, le-5))
0.2 (chosen from [0.1, 0.5, 0.2, 0.00000,00000
Offiechome
96
0.0001
(chowen from (0.0001,0.00,1ee.
0.01 (chosen from (0.001, 0.01, 0.000.0011)
DomainNet
15
96
0.0001 (chosen from 0.0000,,.00111e4.
0.01 (chosen from (0.001, 0.01,000001)
Visda
96
0.0001 (chowen from 0.00000,.00011e-5500000
0.01 (chosen from (0.001, 0.001).)
Civilcomments
32
0.01
(chosen from (0.01, 0.001,,00001,,000000 0.0))
2e-5 (chosen from (2c 4, 2e 5))
Retiring Adults
200
0.0001 (chosen from (0.01, 0.001, 0.0001, 0.0))
0.01 (chosen from (0.001, 0.01, 0.0001))
Mimic Readmissioos
100
128
(chosen from
0.00000000000000000
So-4 (chosen from 10.005, 0.000000000))
Table 6: Details of the learning rate and batch size considered our RLSBENCII
DANN, CDANN, IW-CDANN and IW-DANN As per Transfer Learning Library sug- gestion, we use learning rate multiplier of 1 for the featurizer when initializing with pre-trained network and 1.0 otherwise. We default to penalty weight of 1.0 for all datasets with pre-trained initialization.
FixMatch We use the lambda is 1.0 and use threshold T as 0.9.
NoisyStudent We repeat the procedure for 2 iterations and use drop level of 0.5.
SENTRY We Asee 1.0, Aen 1.0, and Aussup = 0.1. We use a committee of size 3. PsuedoLabel We use the lambda is 1.0 and use threshold 5 as 0.9.
L.3 COMPUTE INFRASTRUCTURE
Our experiments were performed across a combination of Nvidia T4, A6000, P100 and V100 GPUs. Overall, to run the entire RLSBENCH suite a T4 GPU machine with 8 CPU cores would approximately need 70k GPU hours of compute.
L.4 DATA AUGMENTATION
In our experiments, we leverage data augmentation techniques that encourage robustness to some variations between domains for vision datasets.
For weak augmentation, we leverage random horizontal flips and random crops of pre-defined size. For strong augmentation, we apply the following transformations sequentially: random horizontal flips, random crops of pre-defined size, augmentation with Cutout (DeV & Taylor] (017)) and RandAugment (Cubuk et al. 2020). For the exact implementation of RandAugment, we directly use the implementation of Sohn et al. (2020). The pool of operations includes: autocontrast, brightness, color jitter. contrast, equalize, posterize, rotation, sharpness, horizontal and vertical shearing, solarize, and horizontal and vertical translations. We apply N = 2 random operations for all experiments.
M COMPARISON WITH SENTRY ON OFFICEHOME DATASET WITH DIFFERENT HYPERPARAMETERS
In this section, shed more light the discrepancy observed between SENTRY results reported in the original paper (Prabhu et al. 2021) and our implementation.
We note that for the main experiments on Officehome dataset, we used batch size of 96 for all methods including SENTRY. However, SENTRY reported results with a batch size 16 in their work. Hence, re-run the SENTRY algorithm with batch size of 16. To investigate the impact of
47

---

Under review as a conference paper at ICLR 2023
the decreased batch size, make comparison with Fix.aatch (the best algorithm Officehome in our runs) by re-running it with the decreased batch size.
In Table we report results individual shift pairs in officehome. We observe that SENTRY improves over FixMatch for the default minor shift in the label distribution the officehome dataset. However, as the shift severity increases observe that SENTRY performance degrades. Overall, we observe that RS-FixMatch performs similar or superior to SENTRY on out shift pairs in officehome.
Algorithm
Alpha None
Alpha 10.0
Alpha 3.0
Alpha 1.0
Alpha 0.5
Avg
FixMatch
92.5
95.2
98.0
100.0
100.0
97.1
RS-FixMatch
92.5
96.4
98.0
100.0
100.0
97.4
SENTRY
93.0
94.0
98.0
83.3
87.5
91.2
(a) Product to Product (in-distribution)
Algorithm
Alpha None
Alpha = 10.0
Alpha = 3.0
Alpha 1.0
Alpha 0,5
Avg
FixMatch
71.4
71.5
70.7
73.1
75.5
72.4
RS-FixMatch
74.7
74.0
72.1
73.1
72.9
SENTRY
78.1
78.0
75.1
71.7
65.3
73.6
(b) Product to Real
Algorithm
Alpha None
Alpha 10.0
Alpha 3.0
Alpha 1.0
Alpha 0.5
Avg
FixMatch
41.5
44.0
44.2
48.4
39.4
43.5
RS-FixMatch
45.5
44.8
43.6
50.0
37.4
44.2
SENTRY
45.8
46.5
41.4
40.3
27.3
40.3
(c) Product ClipArt
Algorithm
Alpha = None
Alpha = 10.0
Alpha = 3.0
Alpha = 1.0
Alpha = 0.5
Avg
FixMatch
54.4
51.3
54.7
57.3
55.9
54.7
RS-FixMatch
57.2
53.6
55.9
57.3
58.8
56.6
SENTRY
63.7
62.0
62.1
65.3
55.9
61.8
(d) Product Art
Table 7: Officehome results with batch size 16 instead of 96 used throughout our experiments.
More generally, across our runs, also observed model training with SENTRY to be unstable. Investigating further, we observe that the maximization objective to enforce consistency cause instabilities. This behavior is specifically prevalent for experiments where we don't 't use initiale the underlying model with pre-traine trained weights (for example, in BREEDs datasets).
48


Reviews:
[{"review_id": "qC12kRD4Tp0", "reviewer": "Reviewer_kXyy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a benchmark for evaluating domain adaptation methods under the relaxed label shift setting. The authors leverage the introduced benchmark on various domain adaptation methods showcasing the inability of prior domain adaptation methods in coping with the relaxed label shift setting. In addition, this paper also proposes a set of meta-algorithm that could bring general improvements to existing domain adaptation methods.", "summary_of_the_review": "Generally speaking, this paper introduces a comprehensive benchmark that is of good significance towards the development of domain adaptation methods that are robust for real-world applications. The technical novelty of the paper may not be outstanding, but the empirical results do shed light on how relaxed label shift can be tackled if the authors could further elaborate in their analysis. I recommend a \"marginally above the acceptance threshold\" score.", "strength_and_weaknesses": "It is true that comprehensive and fair benchmarking of domain adaptation methods, especially in an under-explored but more realistic setting such as relaxed label shift. Therefore, the motivation of this paper is solid which investigates the effectiveness of prior domain adaptation methods under relaxed label shift in a comprehensive and fair manner. The experiments conducted are comprehensive and the observations of which generally support the authors' claims. The details of the implementation of the experiments are provided in a meticulous manner, with the rather detailed results provided in an anonymous GitHub form. These details greatly help the reproduction of the results and solidify the presentation of experiments. Overall, this paper deals with a domain adaptation setting of great research interest with comprehensive empirical results.\n\nDespite the relatively solid motivation and comprehensive empirical results, there are a few weaknesses that the authors would consider addressing:\na) Even though the authors have proposed a set of meta-algorithm such that it could be leveraged with any domain adaptation (DA) methods and improve the capability of the DA methods in tackling label shift. However, the strategies used (i.e., re-sampling and re-weighting) are not novel techniques, as have also been acknowledged in the paper. Therefore, it would be interesting to analyze why such techniques, though useful, have not been considered in past literature. Additionally, while these strategies may not be leveraged previously on DA with relaxed label shift, they have been leveraged for other tasks with solid proof. The authors should therefore not only show that such methods work empirically, but also showcase why such methods work by providing theoretical proofs or theorems. This would enable a better understanding of why such methods should be employed for DA with relaxed label shifts.\nb) Another concern about the proposed algorithm lies in the authors' claims (on Page 5, the \"Summary\" paragraph): \"we expect DA methods to adapt classifier $f$ to $p(x|y)$ shift\". This seems to contradict what the authors have stated in their review of previous domain adaptation methods in Section 2.1, where the authors noted that: \"a few papers highlighted the theoretical and empirical failure of DA methods\" for relaxed label shift setting. If so, it seems that the algorithm is only effective due to perhaps the meta-algorithm is able to adapt $f$ to the shift in $p(y)$. Can the authors elaborate on this contradiction?\nc) While the authors describe in detail their observation through comprehensive sets of empirical results, the analysis behind these observations fall short of being satisfactory. For example, on Page 8 the authors observe that \"early stopping criterion\" and \"data augmentation\" would impose certain effects on the final result. However, other than a short description no further analysis of why these effects are imposed and to what degree subsequent methods should pay attention to these two aspects.\nd) It is noted that the authors seem to take a different source/target domain setting approach than how certain domain adaptation benchmarks are normally leveraged. One example is the Office-Home dataset which normally benchmarks DA methods by setting 12 pairs of source/target domains with each domain being set as both the source and target domain. In this paper, though only the \"Product\" domain is set as the source domain. Could the authors prove that such settings would not affect the conclusions obtained? Meanwhile, to my understanding, the authors provide the average scores across all source/target domain pairs. Are there any outlier source-target domain pairs whose results are notably different than the rest? If there are, they should be discussed separately.\ne) There are some typos and sentences that are not fluent enough, hampering the clarity of the presentation. The authors may check through their papers for these minor but sometimes obvious mistakes. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JaPOTfq4e5-", "reviewer": "Reviewer_VEqd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper gives the empirical study of domain adaptation algorithms under relaxed label shift scenarios. Many algorithms such as domain-invariant representation learning, self-training, and test-time adaptation methods across 11 multi-domain datasets are conducted. ", "summary_of_the_review": "Based on the above discussion, I tend to reject.", "strength_and_weaknesses": "Strength:\n1) This paper gives a large-scale evaluation of existing domain adaptation algorithms under relaxed label shift scenarios. This helps related researchers focus on this problem.\n2) Commonly adopted datasets and many related methods are conducted.\n\nWeakness:\n1) This paper evaluated domain adaptation methods on relaxed label shift scenarios. But there is no formal definition of the relaxed label shift.\n2) The authors only report the mean accuracy of different methods. How about the variance? \n3) The observation from the experimental results all seems to be well-known conclusions. Are there some inspirational or unusual conclusions?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IyRIltyTuj7", "reviewer": "Reviewer_PHqQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies popular domain adaptation methods under scenarios where both label distribution and conditionals may shift. Specifically, the authors introduce a large-scale benchmark for relaxed label shift settings which consists of 11 vision datasets spanning more than 200 distribution shift pairs. 12 popular domain adaptation methods are evaluated on this benchmark and several observations are provided. This paper shed light on domain adaptation methods in relaxed label shift settings.", "summary_of_the_review": "Strengths:\n+ This paper is well-written. It is enjoyable to read.\n+ This paper studies an important problem. While existing DA approaches work well under covariance shift and label shift. More general scenarios should be introduced to test the effectiveness of the proposed method.\n+ Experiments results are self-contained, and many insightful observations and discussions are provided.\n\nWeaknesses:\n+ It can be observed that some semi-supervised learning methods (e.g. Fixmatch) work well in relaxed label shift settings. Fixmatch is a consistency-based method. In experiments we find that adversarial learning based methods show worse performance than consistency-based methods. In DA, MCD[1] is an adversarial learning based method. And it is also a consistency-based method. What about this method in the benchmark?\n[1] Maximum classifier discrepancy for unsupervised domain adaptation\n+ More insightful analysis should be provided. For example, why Fix-match works better than other methods?\n+ It would be interesting if more benchmarks (more than vision datasets, for example, NLP?) can be introduced.\n\nOverall, the authors should give more insightful discussions to further improve the paper. And this paper can shed light on DA in relaxed label shift setups. ", "strength_and_weaknesses": "Strengths:\n+ This paper is well-written. It is enjoyable to read.\n+ This paper studies an important problem. While existing DA approaches work well under covariance shift and label shift. More general scenarios should be introduced to test the effectiveness of the proposed method.\n+ Experiments results are self-contained, and many insightful observations and discussions are provided.\n\nWeaknesses:\n+ It can be observed that some semi-supervised learning methods (e.g. Fixmatch) work well in relaxed label shift settings. Fixmatch is a consistency-based method. In experiments we find that adversarial learning based methods show worse performance than consistency-based methods. In DA, MCD[1] is an adversarial learning based method. And it is also a consistency-based method. What about this method in the benchmark?\n[1] Maximum classifier discrepancy for unsupervised domain adaptation\n+ More insightful analysis should be provided. For example, why Fix-match works better than other methods?\n+ It would be interesting if more benchmarks (more than vision datasets, for example, NLP?) can be introduced.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5QqC37TsAq", "reviewer": "Reviewer_mTKd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper works on benchmarking the domain adaptation methods on datasets with target label marginal shift and category data distribution p(x|y) shift. The benchmark datasets are based on the current popular DA datasets with resampled target label marginal distribution by resampling target domain data. The paper proposed resampling and reweighting techniques upon the current DA methods on this benchmark, and shows improvement over DA methods commonly, in the relaxed label shift setting.", "summary_of_the_review": "The paper builds a benchmark for DA with target label marginal shift and the per-category data distribution shift. The benchmark dataset is simply by sampling target domain data using the popular DA datasets. The proposed RS and RW are reasonable for remedying the target label distribution shift, however, may heavily rely on the quality of pseudo-labels, and with limited contribution in methodology for DA. ", "strength_and_weaknesses": "1. Strength: This paper benchmarks DA considering the target label marginal mismatch in relaxed label shift setting. It compares different DA methods with the proposed reweighting method on classifiers and resampling method for balancing the target data label distribution. The proposed two techniques can commonly improve DA methods in the relaxed label shift setting.\n\n2. Weakness\n\nI overall appreciate the benchmark in relaxed label shift setting, and the extensive comparisons, especially with the resampling and reweighting techniques. However, my major concerns on this work are the limited novelty. \n\n(1) The benchmark datasets are all based on the previous popular DA datasets. The work of this paper is to resample the target domain data based on Dirichlet distribution to simulate the target label marginal shift.\n\n(2) To handle the label marginal shift, the proposed RS (resampling based on pseudo-label) and RW (reweighting classifiers) are straightforward. These two techniques are combined with the other SoTA DA methods. This is reasonable, but the novel contribution in methodology for DA is limited.\n\n(3) The RS and RW depend on the estimation of target domain label distribution, which are based on the pseudo-labels. The analysis of the accuracy of pseudo-labels and their effects on the DA performance need to be deepened.  From Table 3, it seems that the RS do not improve the target marginal estimation significantly. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject