Source-Free Adaptation to Measurement Shift via Bottom-Up Feature Restoration
Authors: Cian Eastwood
Ian Mason
Chris Williams
Bernhard Schölkopf
Venue: ICLR 2022 Spotlight (2022)
Keywords: Transfer learning
dataset shift
unsupervised domain adaptation
source-free domain adaptation

Abstract: Source-free domain adaptation (SFDA) aims to adapt a model trained on labelled data in a source domain to unlabelled data in a target domain without access to the source-domain data during adaptation. Existing methods for SFDA leverage entropy-minimization techniques which: (i) apply only to classification; (ii) destroy model calibration; and (iii) rely on the source model achieving a good level of feature-space class-separation in the target domain. We address these issues for a particularly pervasive type of domain shift called measurement shift which can be resolved by restoring the source features rather than extracting new ones. In particular, we propose Feature Restoration (FR) wherein we: (i) store a lightweight and flexible approximation of the feature distribution under the source data; and (ii) adapt the feature-extractor such that the approximate feature distribution under the target data realigns with that saved on the source. We additionally propose a bottom-up training scheme which boosts performance, which we call Bottom-Up Feature Restoration (BUFR). On real and synthetic data, we demonstrate that BUFR outperforms existing SFDA methods in terms of accuracy, calibration, and data efficiency, while being less reliant on the performance of the source model in the target domain.


Published as a conference paper at ICLR 2022
SOURCE-FREE ADAPTATION TO MEASUREMENT SHIFT VIA BOTTOM-UP UP FEATURE RESTORATION
Cian Eastwood"1 18
Ian Mason +
Christopher K. I. Williams it
Bernhard Schölkopf 8
f School of Informatics, University of Edinburgh
: Alan Turing Institute, London
8 MPI for Intelligent Systems, Tübingen
ABSTRACT
Source-free domain adaptation (SFDA) aims to adapt model trained on labelled data in source domain to unlabelled data in target domain without access the source-domain data during adaptation. Existing methods for SFDA leverage entropy-minimizztion techniques which: (i) apply only to classification; (ii) destroy model calibration; and (iii) rely on the source model achieving good level of feature-space separation in the target domain. We address these issues for particularly pervasive type of domain shift called measurement shift which can be resolved by restoring the source features rather than extracting new ones. In particular, propose Feature Restoration (FR) wherein (i) store a lightweight and flexible approximation of the feature distribution under the source data; and (ii) adapt the ceature-extracto such that the approximate feature distribution under the target data realigns with that saved on the source. We additionally propose a bottom- up training scheme which boosts performance, which we call Bottom-Uj Up Feature Restoration (BUFR). On real and synthetic data, we demonstrate that BUFR outper- forms existing SFDA methods in terms of accuracy, calibration, and data efficiency, while being less reliant on the performance of the source model in the target domain.
INTRODUCTION
In the real world, the conditions under which a system is developed often differ from those in which it is deployed-a concept known as datasel shift (Quitonero-Candlla et al. 2009). In contrast, conventional machine learning methods work by ignoring such differences, assuming that the development and deployment domains match or that it makes no difference if they do not match (Storkey. 2009). As a result, machine learning systems often fail spectacular ways upon deployment in the test or targel domain (Torralba & Efros 2011 Hendrycks & Dietterich) 2019)
One strategy might be to re-collect and annotate enough examples in the target domain re-train or fine-tune the model (Yosinski et al.) 2014). However, manual annotation can be extremely expensive. Another strategy is that of wnsupervised domain adaptation (UDA), where unlabelled data in the target domain is incorporated into the development process. A common approach is to minimize the domain "gap' by aligning statistics of the source and target distributions in feature space (Long al. 2015) 2018, Ganin & Lempitsky 2015). However, these methods require simultaneous access to the source and target datasets an often impractical requirement due to privacy regulations or transmission con- straints, e-g. in deploying healthcare models (trained on private data) to hospitals with different scan- ners, or deploying image-processing models (trained on huge datasets) to mobile devices with different cameras. Thus, UDA without access the source data at deployment time has high practical value.
Recently, there has been increasing interest in methods to address this setting of source-free free domain adaptation (SFDA, Kundu et al. 2020 Liang al. 2020 Li et al. 20200; Morerio et al. (2220) where the source dataset is unavailable during adaptation the deployment phase. However, to adapt to the target domain, most of these methods employ entropy-minie cinrii-mmiiiizztiion techniques which: (i) apply only to classification (discrete labels); (ii) destroy model calibration minimizing prediction-entrope causes every sample to be classified (correctly or incorrectly) with extreme confidence: and (iii) assume that, in the target domain, the feature space of the unadapted source model contains reasonably well-separated data clusters, where samples within a cluster tend to share the same class label. As
"Equal contribution. Correspondence c. eastwwoddedd ac annxa sonagma com.

---

Published as a conference paper at ICLR 2022
demonstrated in Section 5 even the most innocuous of shifts can destroy this initial feature-space class-separution in the target domain, and with it, the performance these techniques.
We address these issues for a specific type of domain shift which we call measurement shift (MS). Measurement shift is characterized by change in measurement system and is particularly pervasive in real-world deployed machine learning systems. For example, medical imaging systems often fail when deployed to hospitals with different scanners (Zech al. 2018 AlBadawy et al. 2018 Beede ct al. 2020) or different staining techniques (Tellez et al. 2019). while self-driving cars often struggle under "shifted" deployment conditions like natural variations in lighting (Dai & Van Gool 2018 or weather conditions (Volk et al. (019)). Importantly, in contrast to many other types domain shift, measurement shifts can be resolved by simply restoring the source features in the target domain we do not need learn new features in the target domain discriminate well between the classes. Building on this observation, we propose Feature Restoration (FR)-a method which seeks to extract features with the same semantics from the target domain were previously extracted from the source domain, under the assumption that this is sufficient to restore model performance. At development time, train source model and then use boftly-binned histograms to save lightweight and flexible approximation of the feature distribution under the source data. At deployment time, we adapt the source model's 's feature-extractor such that the approximate feature distribution under the target data aligns with that saved on the source. We additionally propose Bottom-U Up Feature Restoration (BUFR)-4 a bottom-up training scheme for FR which significantly improves the degree to which fea- tures restored by preserving learnt structure in the later layers of network. While the assumption of measurement shift does reduce the generality of our methods-ther do not apply to all domain shifts, but rather subset thereof performance on this important real-world problem. To summarize our main contributions, we:
experiments demonstrate that, in exchange, we get improved
Identify subset of domain shifts, which we call measurement shifts, for which restoring the source features in the target domain is sufficient to restore performance (Sec. :::
Introduce lightweight and flexible iatributiionallgnmment method for the source- free setting in which softly-binned histograms approximate the marginal feature distributions (Sec. 3
Create & release EMNIST-DA, simple but challenging dataset for studying MS (Sec. 5.1
Demonstrate that BUFR generally outperforms existing SFDA methods in terms of accuracy, calibration, and data efficiency, while making less assumptions about the performance of the source model in the target domain (i.c. the initial feature-space separatio))) (Sec. (,3):
Highlight& & analyse issues with entropy-minimizzatio in existing SFDA methods (Sec. 5.5
2 SETTING: SOURCE-FREE ADAPTATION TO MEASUREMENT SHIFT
We now describe the two phases of source-free domain adaptation (SFDA), development and de- ployment, before exploring measurement shift. For concreteness, we work with discrete outputs (i.e. classification) but FR can easily be applied to continuous outputs (i.e. regression).
Source-free adaptation. At development time, source model is trained with the expectation that an unknown domain shift will occur upon deployment in the target domain. Thus, the primary objective is to equip the model for source-free adaptation at deployment time. For previous work, this meant storing per-class means in feature space (Chidlovski et al., 2016). generating artificial negative datasets (Kundu et al. 2020), introducing special training techniques (Liang et al. 22200). For us, this means storing lightweight approximate parameterizations of the marginal feature distributions, as detailed in the next section. More formally, a source model fa Xg + V., is trained on n, labelled (1) X, and ys V.. before examples from the source domain D. {(x!") .ys J with x
(3)
saving any lightweight statistics of the source data S.. At deployment time, we are given a pre- trained source model f., lightweight statistics of the source data S., and n. unlabelled examples from
the target domain D. = (x( n with x(") Xr. The goal is to learn a target model St X, + Yr
which accurately predicts the unseen target labels {v(" (1) ) with (i) De. Importantly, the source dataset D. is not accessible during adaptation in the deployment phase.
Domain shift. As depicted in Figure Ia domain shift (Storkey) 2009, Section 9) can be understood
by supposing some underlying. domain- invariant latent representation L of a sample (X.Y). This combines with the domain (or environment) variable E to produce the observed covariates X (T.), where mg is some domain-dependent mapping. For example, I. could describe the shape,
domamn-nnvriiant
2

---

Published as a conference paper at ICLR 2022
E
m
ml
X,
Xt
x
Y
9+
9t
Z
(a) Domain shift
(b) Feature restoration
(c) Measurement shifts (d) Non measurement shifts
Figure 1: Domain shift, feature restoration and measurement shift. (c.d): Top=source, bottom=target.
(c): CIFAR-1O-C 'frog' & CAMELYON17 'tumor'. (d): Office-31 'desk chair' & VisDA-C 'person'.
appearance and pose parameters of scene objects, with X obtained by "rendering" the scene L, taking into account parameters in E that prescribe e.g. lighting, camera properties, background etc.
Feature restoration. In the source domain we leam a feature space Z (X.) = (m) where our source model fa decomposes into feature-extractor g. and classifier h. with f. ho g. (left path of Figure Tbp. For our source model Js achieve good predictive accuracy, the features Z must capture the information in I. about Y and ignore the variables in E = & that act as "nuisance variables" for obtaining this information from X, (e.g. lighting camera properties). In the target domain (E- = t), we often cannot extract the same features& & due to a change in nuisance variables. This hurts predictive accuracy as it reduces the information about L in Z Is (Xe) (and thus about Y). We can restore the source features in the target domain by learning a target feature-extractov ge such that the target feature distribution aligns with that of the source (right path of Figure [b] i.e. ((gx(X,)) p(g.( (X.)). Ultimately, we desire that for any L we will have (m,m(L) = gr (m,((L) Le. that for source X, m, (L) and target X, = my m,(L) images generated from the same L, their corresponding Z's s will match. We can synthetic data, where we have source and target images generated from the same L, to quantify the degree to which the source features are restored in the target domain with 1g. (L)) ge (e(, (L)]l In Section 5.5) we use this to compare quantitatively the degree of restoration achieved by different methods.
Measurement shifts. For many real-world domain shifts, restoring the source features in the target domain is sufficient restore performance- we do not need learn new features in order to discriminate well between the classes in the target domain. We call these measurement shifts as they generally arise from change in measurement system (see Figure [c].. For such shifts, it is preferable to restore the same features rather than learn new ones via c.g. entropy minimization as the latter usually comes at the cost of model calibration as we demonstrate in Section 5
Common UDA benchmarks are not measurement shifts. For many other real-world domain shifts, restoring the source features in the target domain is not sufficient to restore performance-vt we need new features discriminate well between the classes in the target domain. This can be caused by concept shift (Moreno-Tone Torres et al.) 2012 Sec. 4.3), where the features that define a concept change across source and target domains, or by the source model exploiting spurious correlations or "shortcuts" " (Arjovsky et al., 2019: Geirhos et al.) 2020 in the source domain which are not discriminative- or do not even exist in the target domain. Common UDA benchmark datasets like Office-31 (Saenko al.) 2010) and VisDA-C (Peng al., 2018) fall into this category of domain shifts. In particular, 31 is an example concept shift- desk chair' has very different meanings (and thus features) in the source and target domains (left column of Fig. [d] -idlle
-while VisDA-C is an example of source models tending to exploit shortcuts. More specifically, in the synthetic-to-real task of VisDA-C (right column of Fig. ld). source models tend not to learn general geometric aspects of the synthetic classes. Instead, they exploit peculiarities of the c.g. person-class which contains only 2 synthetic "people" rendered from different viewpoints with different lighting. Similarly, if we consider the real-to-synthetic task, models tend to exploit textural cues in the real domain that do not exist in the synthetic domain (Geirhos et al. 2019). As a result, the standard approach is to first pretrain on ImageNet to gain more "general" Visual features and then carefully] fine-tune these features on (i) the source domain and then (ii) the target domain, effectively making the adaptation task ImageNet - synthetic -> real. In Appendix D we illustrate that existing methods actually fail without this ImageNet pretraining as successful discrimination in the target domain requires learning new combinations of the general base ImageNet features. In summary, common UDA benchmarks like Office and VisDA-C do not contain measurement shift and thus are not suitable for evaluating our methods. We nonetheless report and analyse results on VisDA-C in Appendix D
Many works lower the learning rate of carly layers in source and target domains, c.g. I Liang et al. 2020
3

---

Published as a conference paper at ICLR 2022
Pa
Pa.. Iz
o
Source data
Target data
Feat, Extractor
Classifter h
Feat, Extractor
Classifier
9a
9t
h
Development
Deployment
Figure 2: The Feature Restoration framework. Left: At development time, a source model is trained before saving approximations of the D marginal feature distributions under the source data {p=a H Right: At de- ployment time, the -eature-extractor is adapted such that the approximations of marginal feature distributions on the target data {gaa
1. align with those saved on the source.
3 FEATURE RESTORATION
Below we detail the Feature Restoration (FR) framework. During development we train model and then save lightweight approximation of the feature distribution under the source data. At deployment time, we adapt the model's 'ss feature-extractor such that the approximate feature distribution under the target data aligns with that saved on the source. Figure 2 gives an overview of the FR framework.
3.1 DEVELOPMENT
Setup. The source model fa is first trained using some loss, e.g. cross-entropy. Unlike most existing SFDA methods (Chidlovskii et al.] 2016, [Liang et al.] 2020 Kundu al., (020), we make no modification to the standard training process, allowing pretrained source models to be utilized. We decompose the source model fa into a feature-extractor 9a X, + RRD and a classifier h : RR P + Ve.
where is the dimensionality of the feature space. So Z g, (x(x)) ( denotes the features extracted h(g (x.")
denotes the model's output for source sample i. Under the assumption of measurement shift, the feature extractor should be adapted to unlabelled target data to give z;
for source sample i, and g. (x!"))
(x!*)
(8)), ), but the classifier h should remain unchanged, so that (+) ge (xxx))
(6) = [x()))) = (xx((x))))
Choosing an approximation of the feature distribution. For storing the full joint distribution can be prohibitively expensive Thus, we choose to store only the marginal feature distributions. To accurately capture these marginal distributions, we opt to use soft binning (Dougherty et al. 1995) for its (i) feexibliyy-bbnn//istttggg
feature spaces,
make few assumptions about distributional form, allowing us to accurately capture marginal feature distributions which we observe empirically to be heavily-skewer size does not scale with dataset size (Appendix A T5b permitting very large source datasets (for a fixed number of bins B and features D, soft binning requires constant O (D storage and simple to compute soft counts); and (iii) differentiiailitty- the use of soft (rather than "hard") binning, detailed in the next section, makes our approximation differentiable.
and bi-modal (see Appendix D) (ii) scalability-sstrrgee
Estimating the parameters of approximation binning function of Yang et al. (2018) Sec. 3.1) to approximately parameterize the D marginal feature distributions on the source data {p=z) ) where Pz, denotes the marginal distribution of the d-th feature zι- Specifically, we approximately parameterize p=, using B normalized bin counts
the source data. We use the soft
Zy [T 1 ...... Bl. where T
T represents the probability that a sample 2/ falls into bin b is calculated using
under the source data and Dm T w 1. 22
u(
u(g(x(()) ) ; 23
11.
T1
where 210 = g(x(1)) denotes the d-th dimension of the i-th sample in feature space, the vector-
II we assume features are jointly Normal, computational complexity is O(N per update, where N is the batch size. If we hin the feature space into histograms (B bins per dimension), memory complexity is 0 (BB).
4

---

Published as a conference paper at ICLR 2022
valued soft binning function (see Appendix =d =mman min 22 (1)
Fd and z is defined analogously to =m Repeating this for all D features, we get ア [ " 221- r In the left-hand "cloud" of Figure 2 the blue curve depicts one such approximate marginal feature distribution _. We find it useful to additionally store approximate parameterizations of the marginal logit distributions on the source data a where the logit (i.e. pre-softmax) activations a are linear combination of the feature activations z and T. is defined analogously to T... Note that we can parameterize a similar distribution for regression. Intuitively, aligning the marginal logit distributions further constrains the ways in which the marginal feature distributions can be aligned. We validate this intuition in the ablation study of Appendix J.2 Finally, we equip the model for source-free adaptation deployment time by saving the parameters/statistic of the source data S = { Ta z z zmaz, a am ,a" mar) }.
where z man [amtin .22 min
mmn) and z maz a min and mar are defined analogously. ......D
3.2 DEPLOYMENT
At deployment time, we adapt the feature-extractor such that the approximate marginal distributions on the target data (T/, 7 align with those saved on the source (Tg. mz)) More specifically, we learn the target feature-extractor It by minimizing the following loss on the target data,
D.
K
Legt T2z mas mh)
Σ DSKL (T ( +
Σ DskL (TT.))
where DskL (pll)) + (ol(()) is the symmetric KL divergence, and DEL ( (=t is the KI. divergence between the distributions parameterized by normalized bin counts Th, and x which is calculated using
B
(T (
Σ
log
(3)
6=1
zg.b
with Th.., representing the probability of sample from feature falling into bin b under the source data, and T under the target data. Practically, to update on a batch of target samples, we first approximate nे and T on that batch using Eq. [ and then compute the loss. Appendix B details the FR algorithm at development and deployment time, while Appendix L summarizes the notations.
3.3 BOTTOM-UP UP FEATURE RESTORATION
A simple gradient-based adaptation of dt would adapt the weights of all layers at the same time. Intuitively, however, we expect that many measurement shifts like brightness or blurring can be resolved by only updating the weights of early layers. If the early layers can learn to extract the same features from the target data as they did from the source (e.g. the same edges from brighter or blurrier images of digits), then the subsequent layers shouldn' 't need to update. Building on this intuition, we argue that adapting all layers simultaneously unnecessarily destroys learnt structure in the later layers of a network, and propose a bottom-up training strategy to alleviate the issue. Specifically, we adapt Ii in a bottom-up manner, training for several epochs on one "block" before "unfreezing" the next. Here, a block can represent a single layer or group of layers (c.g. a residual block, He et al. (016)), and "unfreezing" simply means that we allow the block's ' weights to be updated. We call this method Bottom-Up Up Feature Restoration (BUFR). In Section 5 we illustrate that BU training significantly improves accuracy, calibration, and data efficiency by preserving learnt structure in later layers of gt-
RELATED WORK
Fine-tuning. well-established paradigm in deep learning is to first pretrain model on large- "source" data (e.g. ImageNet) and then fine-tune the final layer(s) on "target" data of interest (Girshick) al. 2014 Zeiler & Fergus. 2014). This implicitly assumes that new high-level concepts should be learned by recombining old (i.e. fixed) low-level features. In contrast, under the assumption of measurement shift, we fix the final layer and fine-tune the rest. This assumes that the same high-level concepts should be restored by learning new low-level features. Royer & Lampert (2000) tun- each layer of network individually and select the one that yields the best performance. For many domain shifts, they find it best to fine-tune an early or intermediate layer rather than the final one. This supports the idea that which layer(s) should update depends on what should be transferred.
Unsupervised DA. Inspired by the theory of Ben- David et al. (007 2010), many UDA methods seek to align source and target domains by matching their distributions in feature space (Long et al. 2015 2018
Ganin & Lempitsky 2015 Ganin al. 2016 Tzeng al. 2017 Shu al. 2018
5

---

Published as a conference paper at ICLR 2022
However, most of these methods are nonparametric (i.e. make no assumptions about distributional form), they require the source data during adaptation to align the distributions. In addition, parametric methods like Deep CORAL (Sun & Saenko, (016) are not designed for the source-free setup-they prevent degenerate solutions during alignment with a classification loss on the source data and have storage requirements that are at least quadratic in the number of features, In contrast, our method works without the source data and its storage is linear in the number of features.
Source-free DA. Recently, Liang et al. (2000) achieved compelling results by re-purposing the semi-supervised information cotemaatioomaxmmizattooo loss (Krause et al. 2010) and combining it with pseudo- labelling loss (Lee et al. 2013). However, their entropy-minimizing losses are classification-speefff destroy model calibration, and rely on good initial source- source-model performance in the target domain (as demonstrated in the next section). Other works have trained expensive generative models so that the source data-distribution can be leveraged in the target domain (Li et al. 20200 Morerio et al. 2020
Kundu et al.) 2020; Kurmi et 2021 Yeh al. 2021 Stan & Rostami (021). However, these meth- ods are still classification muasificationnspecffe and rely on good initial Seature-space c class-separation for entropy minimization (Li et al. 2020; Kundu al. 2020), beeudo-labelling (Morerio et al. [020)] [Stan & Ros- tami 202I), and aligning the predictions of the source and target models (Kurmi et
et al., 2021). Another approach is to focus on the role of natal-normalization (BN). Li et al. (2017) propose Adaptive BN (AdaBN) where the source data BN-statistics are replaced with those of the target data. This simple parameter-free free method is often competitive with more complex techniques. Waag et al. (202T) also the target data -N-statistics but additionally train the -N-paameeters the target data via entropy minimization, while Ishii & Sugiyama (202T) retrain the feature-extractor to align BN- statistics. Our method also attempts to match statistics of the marginal feature distributions, but is not limited to matching only the first two moments -hence can better handle non-Ganne -au-Sausii distributions.
5 EXPERIMENTS
In this section we evaluate our methods multiple datasets (shown in Appendix F). compare to various baselines, and provide insights into why our method works through a detailed analysis.
5.1
SETUP
Datasets and implementation. Early experiments on MNIST-M (Ganin et al. 2016) and MNIST- C (Mu& & Gilmer, 2019) could be well-resolved by number of methods due to the small number of classes and relatively mild corruptions. Thus, to better facilitate model comparison, we additionally create and release EMNIST-DA a domain adaptation (DA) dataset based on the 47-class Extended MNIST (EMNIST) character-recognition dataset (Cohen al., (017). We also evaluate on object recognition with CIFAR-10-C and CIFAR- 1-0-R (Hendrycks & Dietterich 2019), and on real-world measurement shifts with CAMELYON17 (Bandi et al.) 2018). We use a simple l 5-layer convolutional neural network (CNN) for digit and character datasets and a ResNet- 18 (He et al. (016)) for the rest. Full dataset details are provided in Appendix and implementation details in Appendix G Code is available at /
com/ ci aneast bbufr.
Baselines and their relation. We show the performance of the source model on the source data as No corruption, and the performance of the source model on the target data (before adapting) as Source- only. We also implement the following baselines for comparison: AdaBN (Li et al. (0077) replaces the source BN-statisties with the target BN-statistics; PL is a basic pseudo- labelling approach (Lee et al. 2013). SHOT-IM is the information-maxxiizzaiio loss from Liang et al. (2020) which consists of a prediction-entropy term and prediction-diversin term; and targef-supervised is upper-bound that uses labelled target data (we use a 10-10-11 10 training-vallo validation-test split, reporting accuracy on the test set). For digit and character datasets we additionally implement SHOT (Liang et al. 2020). which uses the SHOT-IM loss along with special pre-training techniques (e.g. label smoothing) and self- supervised PL loss; and BNM-IM (Ishii & Sugiyamal 2021). which combines the SHOT-IM loss from Liang et al. with BN-matching (BNM) loss that aligns feature mean and variances on the target data with B--staiistics of the source. We additionally explore simple alternative parameterizations to match the source and target feature distributions: Marg. Gauss. is the BNM loss from Ishii & Sugiyama which is equivalent to aligning 1D Gaussian marginals; and Full Gauss. matches the mean and full covariance matrix. For object datasets we additionally implement TENT (Wang et al.) 2021). l), which updates only the -N-parmmeters to minimize cediitiion-entroge and also compare some UDA methods. For all methods we report the classification accuracy and Expected Calibration Error (ECE, Naeini et al. 2015) which measures the difference in expectation between confidence and accuracy.

---

Published as a conference paper at ICLR 2022
Table I: Digit and character results. Shown are the mean and standard deviation.
EMNIST-DA
EMNIST-DA-SEVERR
EMNIST-DA-MILD
Model
ACC 1
ECE 4
ACC
ECE 4
ACC
ECE
No corruption
89.4 0.1
2.3 1 0.1
89.4 0.1
2.3 0.1
0.1
2.3
- 0.1
Source-only
29.5 +: 0.5
30.8 + 1.6
42.6 0.4
78.5
4.8
+
AdaBN [Li et al. 2017
46.2 30.3
0.7
4.94
84.9
4.9 -
Marg. Gauss. [shu & Sugiyama 2021
51881:11 1.1
0.5
6.
0.3
4.3
Full Gauss.
0.7 1.7
2.8
8.4
0.2
4.9
PL Lee ct al. 2013
5..0 0.6
2.7
0..
0.1
16.4
BNM-IM (Ishn & Sugiyama
2021
6..7 2.2
1.3
90.2
0.1
13.0
SHOT-IM (Liang et al. 2020
3.7 3.76
2..
7..0
0.1
13.7
SHOT (Liang et al. 2020
8. 1..7
42.7
86.1
14.
FR (ours)
7..4
12.9+0.9
15.3
6.8
5..0+
8.14+0.1
4.6
BUFR (ours)
86.1 + 0.1
4.7
84.6 + 0.2
5.6 0.3
87.0 + 0.2
4.2 0.2
Target-supervised
8688 0.6 7.3 0.7
85.7 0.6 7.0 0.5
87.3 + 0.7 84
5.2
RECOGGTTIN RESULTS
Table Treports classification accuracies and ECEs for EMNIST-DA. with Appendix (aaaiixxKeeppoee repporting results for MNIST datasets (K.1) and full, per-shift results (K.4 and K.5). The severe and mild columns represent the most and least "severe" shifts respectively, where a shift more severe if it has lower AdaBN performance (see Appendix K.5). On EMNIST-DA, BUFR convincingly outperforms all other methods- particulaly on severe shifts where the initial feature-space class-separation likely poor. Note the large deviation in performance across random runs for SHOT-IM and SHOT, suggesting that initial feature-space clustering has a big impact how well these entropy- miniopy-mmiiizzaiion methods can separate the target data. This is particularly true for the severe shift, where only BUFR achieves high accuracy across random runs. For the mild shift, where all methods perform well, we still see that: (i) BUFR performs the best; and (ii) PI., BNM-IM, SHOT-IM and SHOT are poorly calibrated due to their entropy- munropy-minimizing (i.e. confidence- maximizing) objectives. In fact, these methods are only reason- ably calibrated if accuracy very high. In contrast, our methods, and other methods that lack entropy terms (AdaBN, Marg. Gauss., Full Gauss.), maintain reasonable calibration as they do not work by making predictions more confident. This point is elucidated in the reliability diagrams of Appendix H
sonfidence-maximizing
5.3 OBJECT-RECOGNITTON RESULTS
Table reports classification accuracies and ECEs for CIFAR-10-C and CIFAR-100-C. Here observe that FR is competitive with existing SFDA methods, while BUFR outperforms them on almost all fronts (except for ECE on CIFAR-100-C). We also observe the same three trends as EMNIST-DA: (i) while the entropy- miiniiiiinii methods (PL, SHOT-IM, TENT) do well in terms of objectives lead to higher OCC--articullrr on CIFAR- 100-C where their ECE is even higher than that of the unadapted source-only model; (ii) the addition of bottom-up training significantly boosts performance; (iii) BUFR gets the largest boost on the most severe shifts-for example, as shown in the full per-shift results of Appendix K.6 BUFR achieves 89% accuracy on the impulse-noise shift of 10-CC- with the next best SFDA method achieving just 75%. Surprisingly, BUFR even outperforms target-supervised fine-tuning both CIFAR-10-C and CIFAR- 100-C in terms of accuracy. We attribute this to the regularization effect of bottom-up training, which we explore further in the next section.
accuracy, their confidence-maximizinng
We also report results for the "online" setting of Wang et al. (202T. where we may only use a single pass through the target data, applying mini-batch updates along the way. As shown in [3 pendix K.2 FR outperforms existing SFDA methods CIFAR-10-C and is competitive
o3 of Ap- CIFAR-100- C. This includes TENT (Wang et al. 2021 method designed specifically for this online setting.
5.4 REAL-WORLD RESULTS
rreootts results on CAMBLYON17 dataset containing real-world (i.e. naturally occurring) measurement shift. Here we report the average classification accuracy over 4 target hospitals. Note that the accuracy on the source hospital (i.e. no corruption) was 99.3%. Also note that this particular dataset is an ideal candidate for entropy citroo--mmiiizzttionn techniques due to: (i) high AdaBN accuracy on the target data (most pseudo-labels are correct since updating only the BN-statisties gives ~84)); :); (ii) a low number of classes (random pseudo- have 50% chance of being correct); and (iii) large tar- get dataset. Despite this, our methods achieve competitive accuracy and show greater data efficiency- with 50 examples-per-clat or less, only our methods meaningfully improve upon the simple AdaBN baseline which uses the datget-data BN- These results illustrate that: (i) our method performs
7

---

Published as a conference paper at ICLR 2022
Table 2: Objoct-rocognition results. ": result adopted from
Wang et al.
2021
Table 3: EMNIST-DA degree of restoration.
Model
CIFAR-10-C
CIFAR-100-C
Model
D
ACC
ECE,
ACC1
ECF. 4
Source-only.
3.2 + 0.0
No corruption
95.3 + 0.2
2.100.
76.4 = 0.2
4.8
0.1
AdaBN
3.1 0.1
DANN Ganin al.
2016
81.7
61.1
Marg. Gauss.
2.0
UDA-SS. SS.
(Sun al.
2019
83.3
53
Full Gauss.
2.0
PL BNM-IM SHOT-IM
2.6
Source-only
57.8 + 0.7
28.2 0.4
36.4 0.5
19.4
2.5
AdaBN eL et al. 2018
80.4
11.2 0.1
56.6 0.3
1.1
2.9
PL (Lee ct al. 2015
x2.5
17.5 0.3
0.2
37.7 0.2
(ours)
1.0
SHOT-IM [Liang ct al. 2020
85.4
14.6 0.2
0.2
32.9
BUFR (ours)
1.0
TENT (Wang et al. 2021
86.6
12.8
0.4
25.7 75.7
FR (ours)
0.7
11.3 0.3
65.5 0.2
15.7
BUFR (ours)
89.4 + 0.2
10.0 + 0.2
0.2
14.5 +.3 0.3
Turget-supervised
88.4 + 0.9
6,4 0.6
68.1 = 1.2
0.7
Table 4: CAMELYON17 accuracies for varying numher of examples-per-elass in the target domain.
Model
5
10
50
500
All(> 15k)
Source-only
55.8 + 1.6
55.8
+ 1.6
55.8 + 1.6
55.8 + 1.6
55.8 + 1.6
AdaBN (Li al 2018
82.6 2.2
83.3
+ 2.3
83.7 + 1.0
83.9 - 0.8
84.0 = 0.5
PL (Lec et al. 2013
82.5 2.0
83.7 + 1.7
83.6
1 1.2
85.0 0.8
90.6 + 0.9
SHOT-IM (Liang et al. 2020
2.2
83.4 2.5
83.7 = 1.2
0.7
89.9 + 0.2
FR (ours)
84.6 0.6
86.0 0.7
86.0
0.6
89.5 + 0.4
BUPR (ours)
84.5 0.8
86.1 0.2
8
89.1 0.8
89.7 + 0.5
well in practice; (ii) measurement shift is an important real-world problem; and (iii) source-free methods are important to address such measurement shifts as, e.g., medical data is often kept private.
5.5 ANALYSIS
Feature-space class-separation. Measurement shifts cause the target data be poorly-separated in feature space. This point is illustrated in Figure 3 where we provide t-SNE visualizations of the feature-space class-separation
the EMNIST-DA crystals shift. Here, Figure Bal shows the initial plas-sepaaattin before adapting the source model. We see that the source data well separated in feature space (dark colours) but the target data is not (light colours). Figure3h shows the performance of mntropy-minimizatii
method when applied such a "degraded" feature space where initial class-separation is poor on the target data. While accuracy and class-separation improve, the target- data clusters are not yet (i) fully homogeneous and (ii) returned to their original location (that of the source data clusters). As shown in Figure dd.d, our methods of FR and BUFR better restore class-separation on the target data with more homogeneous clusters returned to their previous location.
Quantifying the degree of restoration. We quantify the degree to which the EMNIST source features are restored in each of the EMNIST-DA target domains by calculating the average pairwise distance:
D D= # E E I9. (x,((())) gi (m, (x() where 7' is the number of EMNIST-DA target domains, N is the number of EMNIST images, ( is clean or uncorrupted EMNIST image, m, is the identity transform, and m, is the shift of target domain (e.g. Gaussian blur). Table 3 shows that the purely alignment-based methods (Marg. Gauss., Joint Gauss., FR, BUFR) tend better restore the features than the entropy-basee based methods (PL, BNM-IM, SHOT-IM), with our alignment based methods doing it best. The only exception is Marg. Gauss. the weakest form of alignment. Finally, is worth noting the strong rank correlation (0.6) between the degree of restoration in Table and the ECE in Table [ This confirms that, for measurement shifts, it is preferable to restore the same features rather than learn new ones as the latter usually comes at the cost of model calibration.
Restoring the semantic meaning of features. The left column of Figure shows the activation distribution (bottom) and alttiaally-activaiinng image patches (top) for specific filter in the first layer of a CNN trained on the standard EMNIST dataset (white digit, black background). The centre column shows that, when presented with shifted target data (pink digit, green background), the filter detects similar patterns of light and dark colours but no longer carries the same semantic meaning of detecting a horizontal edge. Finally, the right column shows that, when our BUFR method aligns the marginal feature distributions on the target data (orange curve, bottom) with those saved on the source data (blue curve, bottom), this restores a sense of semantic meaning to the filters (image patches, top). Note that we explicitly align the first-layer feature/filter distributions in this illustrative experiment.
8

---

Published as a conference paper at ICLR 2022
(a) Source-only (19%)
(b) SHOT-IM (67%)
(c) FR (77%)
(d) BUFR (83%)
Figure 3: t-SNE Van der Maaten & Hinton 2008 visualization of features for classes of the EMNIST-DA
crystals shift. Dark colours show the source data, light the target. Model accuracies are shown parentheses.
An
Accuracy (%6)
86G
-44
82
mpdite poke
seanges
80
zirpe
Epochs
(a) Restoring feature semantics
(b) BUFR training curves
Figure 4: (a) Activation distributions (bottom) and maximally-activannl image patches (top) for specific filter the first layer CNN. Left: Source model, source data (white digit. black backgr.). Centre: Source model, target data (pink digit, green backgr.). Right: Target model (adapted with BUFR), target data, (b) BUFR training curves on selected EMNIST-DA DA corruptions. Dashed-grey beeeeddgeyy lines indicate when the next block unfrozen.
Efficacy of BU training. Figure 4b shows that, when training bottom-up manner, updating only the first two blocks is sufficient to resolve many measurement shifts. This confirms the previous intuition that updating only the early layers should be sufficient for many measurement shifts. BUFR exploits this by primarily updating early layers, thus preserving learnt structure in later layers (see Appendix 1.3H1.4) 4). To examine the regularization benefits of this structure preservation, we compare the accuracy of BUFR to other SFDA methods as the number of available target examples reduces. As shown in Table of Appendix []1]] the performance of all competing methods drops sharply reduce the number of target examples. In contrast, BUFR maintains strong performance. With only examples-per placcess, it surpasses the performance of many methods using all 400 caappee--pe--llaee
Ablation study. We also conduct an ablation study on the components of our loss from Equation 2 Table [0 of Appendix [.2] shows that, for casier tasks like CIFAR-10-C, aligning the logit distributions and using the symmetric KL divergence (over more commonly-used asymmetric one) make little difference to performance. However, for harder tasks like CIFAR-100-C, both improve performance.
DISCUSSIONS
Aligning the marginals may be insufficient. Our method seeks to restore the joint feature distri- bution by aligning (approximations of) the marginals. While found that this is often sufficient, it cannot be guaranteed unless the features are independent. One potential remedy is to encourage feature independence in the source domain using "disentanglement" (Bengio et al. 2013 Eastwood Williams 2018) methods, allowing the marginals to better capture the joint.
Model selection. Like most UDA & SFDA works, we use a torget-domain validation set (Gulrajani Lopez-Pars Paz, 2021) for model selection. However, such labelled target data is rarely available in real-world setups. Potential solutions include developing benchmarks (Gulrajani & Lopez-Paz 2021 and validation procedures (You al. 2019) that allow more realistic model selection and comparison.
Conclusion. We have proposed BUFR, a method for source- free adaptation to measurement shifts. BUFR works by aligning histogram-based approximations of the marginal feature distributions on the target data with those saved on the source. We showed that, by focusing on measurement shifts, BUFR can outperform existing methods in terms of accuracy, calibration and data efficiency, while making less assumptions about the behaviour of the source model on the target data. We also highlighted issues with the matropy-minimizztioon techniques on which existing SFDA-methods rely, namely their dlasiffcationnsspccffiii tendency be poorly calibrated, and vulnerability to simple but severe shifts.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We thank Tim Hospadales, Amos Storkey, Oisin Mac Aodha, Luigi Gresele and Julius von Kügelgen for helpful discussions and comments. CE acknowledges support from The National University of Ireland via his Travelling Studentship in the Sciences. IM is supported by the Engineering and Physical Sciences Research Council (EPSRC).
REFERENCES
Ehab A AlBadawy, Ashirbani Saha, and Maciej A Mazurowski. Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing. Medical Physics, (5(3):1150-115: 2018.
Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence, 33(5):898-911 May 2011.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprini arXiv: 1907.02893, 2019.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging, 38(2):550-560, 2018.
Emma Beede, Elizabeth Baylor, Fred Hersch, Anna Turchenko, Lauren Wilcox, Paisan Ruamviboon- suk, and Laura M. Vardoulakis. A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. In Proceedings of the 2020 CHI Conference Human Factors in Computing Systems, pp. 1-12. Association for Computing Machinery, 2020.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances in Neural Information Processing Systems, pp. 137-144, 2007.
Shai Ben-David David. John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 79()):11--77, 2010.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1588) 2013.
Boris Chidlovskii, Stéphane Clinchant, and Gabriela Csurka. Domain adaptation in the absence of source domain data. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 451-4600 2016.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: an extension of MNIST to handwritten letters. arXiv preprint arXiv:1702.05373. 2017.
Dengxin Dai and Lue Van Gool. Dark model adaptation: Semantic image segmentation from daytime to nighttime. In International Conference on Intelligent Transportation Systems, pp. 3819-3824. IEEE, 2018.
Shai Ben David, Tyler Lıı, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 129-136. 2010.
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 32(1-2):12-22, 1983.
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D'Amous Dan Moldovan, et al. On robustness and transferability convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16458-16468, 2021.
James Dougherty, Ron Kohavi, and Mehran Sahami. Supervised and unsupervised discretization of continuous features. In International Conference on Machine Learning, pp. 194-202, 1995.
10

---

Published as a conference paper at ICLR 2022
John Duchi. Derivations for linear algebra and optimization, 2007. staanorrd 2021.
URL hetps: /web Accessed: 5th October
_ow.inotess..... p
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry, Exploring the landscape of spatial robustness. In International Conference on Machine Learning, pp. 1802- 1811, 2019.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In
International Conference Machine Learning, pp. 1180-1189. PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, (7())2206-2033) 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
Robert Geirhos, Morn-Herrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2:665-675 673, 2020.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580-587, 2014.
Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In International Conference on Learning Representations, 2019.
Masato Ishii and Masashi Sugiyama. Source-free domain adaptation via distributional alignment by matching batch normalization statistics. arXiv preprint arXiw:2101.10842, 2021.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo. Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovee, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML), 2021.
Andreas Krause, Pictro Perona, and Ryan Gomes, Discriminative clustering by regularized infor- mation maximization. In Advances in Neural Information Processing Systems, pp. 775-783, 2010.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009,
Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et Universal source- ffeee domain adap- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4544 4553, 2020.
11

---

Published as a conference paper at ICLR 2022
Vinod K Kurmi, Venkatesh K Subramanian, and Vinay P Namboodiri. Domain impression: A source data free domain adaptation method. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 615-625, 2021.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1)):278-23224 1998.
Dong-Hyun Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML, volume 3, 2013.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1641-9650, 2020.
Yanghao Li, Naiyan Wang. Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normaliza- tion for practical domain adaptation. In International Conference on Learning Representations Workshop, 2017.
Yanghao Li, Naiyan Wang. Jianping Shi, Xiaodi Hou, and Jiaying Liu. Adaptive batch normalization for practical domain adaptation. Patlern Recognition, 80: 109-117, 2018.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning (ICML), pp. 6028-6039, July 13-18 2020.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning, pp. 97-105, 2015.
Mingsheng Long, Zhangjie Cao, Jianmin Wang. and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, 2018.
C. Michaelis, B. Mitzkus, R. Geirhos, E. Rusak, O. Bringmann, A. S. Ecker, M. Bethge, and W. Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. In Machine Learning for Autonomous Driving Workshop, NeurIPS 2019, 2019.
Jose G Moreno-Torres, Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. Pattern Recognition, 45:521-530, 2012.
Pietro Morerio, Riccardo Volpi, Ruggero Ragonesi, and Vittorio Murino. Generative pseudo-label re-
finement for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3130-3139, 2020.
Norman Mu and Justin Gilmer. MNIST-C: A robustness benchmark for computer vision. arXiv preprint arXiv:1906.0233: 02337, 2019.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht, Obtaining well calibrated proba- bilities using Bayesian binning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 29, 2015.
Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In International Conference on Machine Learning, pp. 625 632, 2005.
Xingchao Peng, Ben Usman, Necla Kaushik. Dequan Wang, Judy Hoffman, and Kate Saenko. VISDA: A synthetic-to-real benchmark for visual domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2021-2026, 2018.
Joaquin Quiñonero-Candela, Candela, Masashi Sugiyama, Neil D Lawrence, and Anton Schwaighofer. Dataset Shift in Machine Learning. MIT Press, 2009.
Prasun Roy, Subhankar Ghosh, Saumik Bhattacharya, and Umapada Pal. Effects of degradations on deep neural network architectures. arXiv preprint
2019.
Amélic Royer and Christoph Lampert. A flexible selection scheme for minimum-effort transfer learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2191-2200, 2020.
12

---

Published as a conference paper at ICLR 2022
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 1 115(3):11-252, 2015.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European Conference on Computer Vision, pp. 213 113-226. Springer, 2010.
Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. A DIRT- approach unsupervised domain adaptation. In International Conference on Learning Representations, 2018.
Patrice Simard, Bernard Victorri, Yann LeCun, and John S Denker. Tangent prop-a formalism for specifying selected invariances in an adaptive network. In Advances in Neural Information Processing Systems, pp. 895-903, 1991.
Serban Stan and Mohammad Rostami. Unsupervised model adaptation for continual semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2601,,601,220 2021.
Amos J Storkey. When training and test sets are different: characterising learning transfer. In Dataset Shift Machine Learning, pp. 3-28. MIT Press, 2009.
Baochen Sun and Kate Saenko. Deep CORAL: Correlation alignment for deep domain adaptation. In European Conference on Computer Vision, pp. 443-450. Springer, 2016.
Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A Efros. Unsupervised domain adaptation through self-supervision. arXiv preprint arXiv: /8825, 2019.
David Tellez, Geert Litjens, Péter Bándi, Wouter Bulten, John-Melle Bokhorst, Francesco Ciompi, and Jeroen van der Laak. Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology. Medical Image Analysis, 58:101544, 2019.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In Proceedings The IEEE Conference on Computer Vision and Pattern Recognition, pp. 1521-1528, 2011.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7167-7176, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE, Journal of Machine Learning Research, 9(11), 2008.
Georg Volk, Stefan Müller, Alexander von Bernuth, Dennis Hospach, and Oliver Bringmann. Towards robust CNN-based object detection through augmentation with synthetic rain variations. In IEEE Intelligent Transportation Systems Conference, pp. 285-292, 2019.
Dequan Wang. Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. TENT: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
Yongxin Yang, Irene Garcia Morillo, and Timothy M. Hospedales. Deep neural decision trees. In ICML Workshop on Human Interpretability in Machine Learning (WHI), 2018.
Hao- Wei Ych, Baoyao Yang, Pong C Yuen, and Tatsuya Harada. SoFA: Source-data-free feature alignment for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4744883, 2021.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems, pp. 3320-3328, 2014.
Kaichao You, Ximei Wang, Mingsheng Long, and Michael Jordan. Towards accurate model selection in deep unsupervised domain adaptation. In International Conference on Machine Learning, pp. 7124-7133, 2019.
13

---

Published as a conference paper at ICLR 2022
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. International Conference on Machine Learning, pp. 609-616, 2001.
John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS Medicine, 15(11)xe100268,, 2018.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In
European Conference on Computer Vision, pp. 818-833, 2014.
14

---

Published as a conference paper at ICLR 2022
Appendix
Table of Contents
Soft binning
16
B FR algorithm
17
When might FR work?
D Common UDA benchmarks not measurement shifts
18
E Further related work
19
F Datasets
G Further implementation details
22
Reliability diagrams and confidence histograms
23
Activation distributions
25
Further analysis
27
J.1
Efficacy of bottom-up training
27
J.2
Loss ablation study
27
J.3
Who affected
28
J.4
Who moves
28
K
Full Results
29
Digit and character summary results
2
Online results)
................................
30
K.3
CAMELYON17 results
30
MNIST-C full results
31
K.5
EMNIST-DA full results
32
K.6
CIFAR-10- full results
33
K
7
CIFAR-100-C full results
34
K.8 8
CIFAR-1O-C full online results
35
9
CIFAR-100-C full online results
36
Notations
37
15

---

Published as a conference paper at ICLR 2022
A SOFT BINNING
Function. Let 2 ~ Pz be continuous ID variable for which we have n samples {2(0) in The goal is approximately parameterize Ps using B normalized bin counts T= 1, ... B where :5 represents the probability that 2 falls into bin and ΣB h= Rz = 1. We achieve this The first step is to find the
/im1
using the soft binning function of Yang et al. (2018) Section 3.1). range of 2, i.e. the minimum and maximum denoted z = min; 200) and z
max; 200 respectively. This will allow us to normalize the range of our samples 2 to be [0, and thus ensure that binning "softness", i.e. the degree to which mass is distributed into nearby bins, is comparable across variables with different ranges. The second step is to define B uniformly- spaced and monotonically-incraas.
cut points (i.e. bin edges) over this normalized range [0, 1]. denoted c = [c1, c...... CB-1] = -ー [0, 1, 2, B -3,, 3, B --2], 2]. The third step is to compute the B- dimensional vector of soft counts for sample 2f)), denoted u( u(z(1)) using soft binning vector-valued function u.
(0) min
uf u ; min = o((w
wo)/T);
(4)
mer
2
cyl. T is temperature factor, is the softmax function, u(2(")) )o is the mass assigned to bin b, and Σ u(2())) ) 1. Note that: (i) both w and Wo are constant vectors for a pre-specified number bins B: (ii) as - u ()) tends one-hot vector; and (iii) the B 1 cut points e result in B bins, where values 2
where 21, BJ. Wo [0, -1, -1 C2,
B Σド=1
or 26) >> 1 are handled sensibly by the soft binning function in order to catch new samples that lie outside the range of our original n samples (as r + 0, they will appear in the leftmost or rightmost bin respectively). Finally, get the total counts per bin by summing
the per-sample soft counts u( u(z())) before normalizing by the total number of samples to get the normalized bin counts π.. un(a (8) 1a min 2 maa))
i.e.
1i=1
Memory cost.
When using 32-bit floating point numbers for each (soft) bin count, the memory cost of soft binning is 32 x B x D bits-depending only on the number bins B and the number of features D. and not on the dataset size. For concreteness, Table compares the cost of storing bin counts to that of: (i) storing the whole source dataset; and (ii) storing the (weights assume 8 bins per feature and the following network architectures:
the) source
model. As in our experiments, a variation of LeNet (LeCun et al.) (998) for MNIST; ResNet- 18 (He et al., 2016) for CIFAR- and ResNet-101 (He et al. 2016) for both VisDA-C (Peng et al. 2018) and ImageNer (Russakovsky et al. 2015).
Table 5: Storage size for different datasets and their corresponding source models.
Storage size (MB)
MNIST
CFR-100
VisDA-C
ImageNet
Source dataset
33
150
7885
138000
Source model
0.9
49
173
173
Source bin-counts
0.004
0.02
0.5
0.5
16

---

Published as a conference paper at ICLR 2022
B FR ALGORITHM
Algorithm| 1 gives the algorithm for FR at development time, where a source model is trained before saving approximations of the feature and logit distributions under the source data. Algorithm 2 gives the algorithm for FR at deployment time, where the feature-extractor is adapted such that the approximate feature and logit distributions under the target data realign with those saved the source.
Algorithm 1: FR at development time.
FR deployment time. Input: Source model fa. unlabelled target
Algorithm 2:
Input: Source model Ss. labelled source data
D. = (X.. Y.), number of bins B, number of training iterations I.
data Xx, source data statistics S,, number of adaptation iterations I.
/N Train src model ===he h g-
/ Init trgt model ft = h
x/
for i in range(I) do
fe x fa:
Li + Lfre D.):
Adapt trgt leat.-extracta 9t.
*/
fs + SGD(S Li):
for in range(L) do
x + CALC CALC_BC(fi, X1; K1iz"M min maz B) T t CALC_BC(Je, X min max B)
* calc. feat. &logit ranges
*/
2 zmin, maz t CALC_ CALC_RANGE(fs, X,): a min a mas t CALC (A.
L, + Legel (TT Thz Tá Ti):
/+ Calc. feat. &logit bin onts +/
9t + SGD(ge: Li)
22 + CALC_BC(fa,, Xo; z min maz B);
Output: 9t
T + CALC_BC(Ja, Xsj a minn mma B);
*+ Gather source stats S.
S, < {Tg na, min maa miin maz]
Output: S,
C WHEN MIGHT FR WORK?
Toy example where FR will work.
Let L take two values (-1, 1), and let
== L
(5)
= 0.5, 0
(6)
where U denotes uniform distribution and E domain Figure Lap. Then the optimal classifier f X + Y can be written as f(X) = sign(X - E). Imagine the source domain has F. = 0, and the target domain has F 2. Then all points will be initially the target domain, but will restore optimal performance by essentially rrrmanrall X to achieve an intermediate feature representation Z with the same distribution as before (in the source domain).
offset (this setup is depicted in
classified as positive
Toy example where FR will not work. Let L be a rotationally-syminn (e.g.a standard multivariate Gaussian), and let X be rotated version of L where the rotation depends on E. Now let Y = L,, the first component of L. Then any projection of X will have the correct marginal distribution, hence FR will not work here matching the marginal distributions of the intermediate feature representation Z will not be enough to yield the desired invariant representation.
multivariate distribution
How to know if FR is suitable. We believe it reasonable to assume that has knowledge of the type of shifts that are likely to occur upon deployment. For example, if deploying a medical imaging system to new hospital, one may know that the imaging and staining techniques may differ but the catchment populations are similar in e.g. cancer rate. such cases, we can deduce that measurement shift is likely and thus FR is suitable.
17

---

Published as a conference paper at ICLR 2022
D COMMON UDA BENCHMARKS ARE NOT MEASUREMENT SHIFTS
Overview. The standard approach for common UDA benchmarks like VisDA-C (Peng al. 2018 is to first pretrain on ImageNet to gain more "general" visual features and then carefully tne-tune these features on (i) the source domain, and then (ii) the target domain, effectively making the adaptation task ImageNet + synthetic + real. Here, we use VisDA-C to: (i) investigate the reliance of existing methods on ImageNet pretraining; (ii) evaluate our FR and BUFR methods on domain shifts that require learning new features (i.e. non measurement shifts); and (iii) investigate the effect of label shift on our methods (which violates the assumption of measurement shift and indeed even domain shift).
Reducing label shift. For (ii), we first note that VisDA-C contains significant label shift. For example, 8% of examples are labelled 'car' in the source domain, while 19% of examples are labelled 'car' in the target domain. To correct for this while retaining as many examples as possible, we randomly drop examples from some classes and oversample examples from others so that all classes have 11000 examples in the source domain and 3500 examples in the target domain -ths labelled as "No label shift" in T
Results. In Table 6 we that: (i) without ImageNet pre-training, all (tested) methods fail despite similar accuracy being achieved the source domain with or without ImageNet pre-training (compare XX vs. vx); (ii) with the standard VisDA-C setup (i.e. / XX) AdaBN < FR << SHOT, SHOT learns new discriminative features in the target domain; and (iii) correcting for label shift boosts the performance of FR and closes the gap with SHOT (compare /X vs. v/), but some gap remains VisDA-C is not a measurement shift but rather more general domain shift. Finally, we note that ImageNet pretraining makes the features in early layers quite robust, reducing the advantage of bottom-up training.
Implementation details. These results were achieved using a standard VisDA-C implentation/setup: we train ResNet- 101 (He et al. (016) (optionally pre-trained on IetgeNet) for 15 epochs using SGD, learning rate of 0.001, and batch size of 64. We additionally adopt the learning rate scheduling of (Ganin& & Lempitsky 20155; Long et al. 2018: Liang et al. 2020) in the source domain, and reduce the learning rate 0.0001 the target domain.
Table VisDA-C -C results (ResNet-101). No label shift: examples were dropped or oversampled to correct for label shift.
Model
ImageNet pretrain
No label shift
Avg. Acc.
No corruption
x
99.8
Source-only
X
10.4
AdaBN (Li et al. 2017)
X
X
15.9
SHOT [Lang et al. 2020)
X
X
17.1
FR
X
16.8
BUFR
X
16.2
No corruption
X
99.6
Source-only
47.0
AdaBN (Li et al. 2017)
65.2
SHOT [Lang et 2020)
82.9
FR
73.7
BUFR
72.9
No corruption
99.7
Source-only
44.6
AdaBN (Li et al. 2017)
68.7
SHOT Liang et al. 2020)
85.0
FR
82.8
BUFR
83.1
18

---

Published as a conference paper at ICLR 2022
E FURTHER RELATED WORK
Domain generalization. Domain generalization seeks to do well in the target domain without updating the source model. The goal is achieve this through suitable data augmentation, self- supervision, and inductive biases with respect to a perturbation of interest (Simard et al. 1991, Engstrom al. 2019; Michaelis et al. 2019 Roy al. 2019 Djolonga et al. 2021). One may view this as specifying the shifts that a model should be robust to a prior.. Practically, however, generally do not know what shift will occur upon deployment there will always be unseen shifts. Furthermore, the condition that our augmented development process be sufficiently diverse is untestable-with the worst-case error still being arbitrarily high (David et al.) 2010; Arjovsky et al. 2019). Permitting adaptation the target domain is one reasonable solution to these problems.
Common corruptions. Previous works (Hendrycks & Dietterich, 2019) have used common cor- ruptions to study the robustness of neural networks to simple transformations of the input, e-g. Gaussian noise (common in low-lighting conditions), defocus blur (camera is not properly focused or calibrated), brightmess (variations in daylight intensity), and impulse noise (colour analogue of salt-and- pepper noise, caused by bit errors). We see common corruptions as one particular type of measurement shift, with all the aforementioned corruptions arising from a change in measurement system. However, not all measurement shifts are common corruptions. For example, the right column of Figure Ic depicts tissue slides from different hospitals. Here, the shift has arisen from changes staenng procedures, patient populations and image acquisition (e.g. different sensing equipment). This measurement shift cannot be described in terms of simple input transformations like Gaussian noise blurring, and thus do not consider it a common corruption. In addition, EMNIST-DA shifts like bricks and grass use knowledge of the object type (i.e. a digit) to change the background and foreground separately (see Figure 7b. We do not consider these to be common corruptions as common corruptions rarely have knowledge the image content-e.g. blurring all pixels or adding noise randomly. In summary, we consider measurement shifts to be superset of common corruptions, thus warranting their own definition.
SFDA and related settings. Table 7 compares the setting of SFDA to the related settings of fine- tuning, unsupervised domain adaptation (UDA), and domain generalization (DG).
Table 7: Source-free domain adaptation and related settings. Adapted from Wang et al. 2021
Setting
Source data
Target data
Adapt. Loss
Fine-tuning UDA Domain gen. Source-free DA
L(x', y²) L(x", v²) + L(x", xx)) (x²,y²) L(xt)
x 8 wx y
F DATASETS
Figures
6 and below visualize the different datasets we use for evaluation and analysis.
MNIST-M (Ganin et al. 2016) constructed by combining digits from MNIST with random back- ground colour patches from BSDS500 (Arbelaez et al. 2011). The source domain is standard MNIST
and the target domain is the same digits coloured (see Figure 5). MNIST-C (Mu & Gilmer 2019)
contains 15 different corruptions of the MNIST digits. Again, the source domain is standard MNIST
and the corruptions of the same digits make up the 15 possible target domains (see Figure 6)).
As shown in Appendix K.I many methods achieve good performance on these MNIST datasets. For this reason we create and release the more challenging EMNIST-DA dataset. EMNIST-DA contains 13 different shifts chosen give diverse range of initial accuracies when using source model trained on standard EMNIST. In particular, a number of shifts result in very low initial performance but are conceptually simple to resolve (see Figure 7). Here, models trained on the training set of EMNIST (source) before being adapted to a shifted test set of EMNIST-DA (target, unseen examples).
We also use the CIFAR- 10-C and CIFAR- corruption datasets (Hendrycks & Dietterich 2019) to compare methods on object-reoognition tasks. These datasets contain 19 different corruptions of the C1FAR-10 and CIFAR-100 test sets (see Figure 8). Here, a model is trained on the training set of CIFAR-1O/CIFAR (source, Krizhevsky 2009) before being adapted to corrupted test set (target).
19

---

Published as a conference paper at ICLR 2022
Finally, show real-world measurement shift with CAMELYON17 (Bandi et al. 2018). medical dataset with histopathological images from different hospitals which use different staining and imaging techniques (Figure99). The goal is determine whether or not an image contains tumour tissue. We train on examples from a single source hospital (hospital 3) before adapting one of the 4 remaining target hospitals. We use the WILDS (Koh et al. 2021) implementation of CAMELYON17.
9
4
Figure 5: Top: samples from MNIST. Bottom: samples from MNIST-M.
Identity
Brightness Canny Edg Dotted Line
Fog
Glass Blr Impulse Ns Motion Bir
8
g
Rotate
Scale
Shear
Shot Noise
Spatter
Stripe
Translate
Zigzag
Figure 6: MNIST-C corruptions.
Identity
Bricks
Crystals
Dotted Line
Fog
Gauss Blur
Grass
N
N
MM
Impulse Ns
Inverse
Oranges
Shot Noise
Sky
Stripe
Zigzag
N
N
N
N
M
W
Figure EMNIST-DA shifts.
Identity
Brightness
Contrast Defocus
Elastic
Fog
Frost
Gauss
Gauss Ns
Glass
Impulse Jpeg Cmpr.
Motion Blr
Pixelate
Saturate
Shot Noise
Snow
Spatter
Speckle Ns Zoom Bir
Figure 8; CIFAR corruptions. The same corruptions are used for CIFAR-10-C and CIFAR-100-C.
20

---

Published as a conference paper at ICLR 2022
Figure 9: CAMELYON17. Columns show different hospitals. Top row: no tumour tissue. Bottom row: tumour tissue present.
21

---

Published as a conference paper at ICLR 2022
G FURTHER IMPLEMENTATION DETAILS
Architectures. The architecture of the simple 5-layer CNN (a variant of LeNet, LeCun et al. 1998) which we use for digit and character datasets, is provided in Table medical datasets, we use standard ResNet-18 (He et al. 2016).
g For the object-rocognition and
Training details. For all datasets and methods we train using SGD with momentum use a batch size of 256, and report results over 5 random seeds. In line with previous UDA & SFDA works (although often not made explicit), we use test-domain validation set for model selection (Gulrajani & Lopez-Paz 202I). In particular, we select the best-performing learning rate from {0.0001, 0.001, 0.01, 0.1, IJ. and for BUFR, train for 30 epochs per block and decay the learning rate as a function of the number of unfrozen blocks in order further maintain structure. For all other methods, including FR, we train for 150 epochs with constant learning rate. The temperature parameter 7 (see Appendix Eq. is set 0.01 in all experiments.
0.9,
Tracking feature and logit distributions. To track the marginal feature and logit distributions, we implement a simple StatsLayer class PyTorch that can be easily inserted into network just like any other layer. This seamlessly integrates distribution-trackinn In the source domain, we simply: (i) add StatsTa Layers our (pre)trained source model; (ii) pass the source data through the model; and (iii) save the model as normal in Py Torch (the tracked statistics, Le. bin counts, domain, the source model can be loaded as normal and the inserted StatsLayers will contain the source-data statistics. Code is available at wttttt:::/// //gp::///iii thub. com/ c anea fur
into standard training processes.
automatically saved as persistent buffers akin to BN- statisssss.... In the target
The Full Gauss, baseline. This baseline models the distribution of hidden features as joint multivariate Gaussian, with dimensionality equal to the number of hidden units. After training a model on the source data, the source data is passed through once more and the empirical mean vector and covariance matrix are calculated and saved. To adapt to the target data the empirical mean and covariances divergence Dr.. (Q) where Q is the Gaussian distribution estimated on the target data minibatch and P from the source data. This divergence has analytic form (uuchi 2007| Sec. 9) which use as the loss function. We use this direction for the KI. divergence as we only need to invert the covariance matrix once (for saved P) rather than the covariance matrix for Q on every batch.
calculated for each minibatch and the distributions
aligned using the KL
Online setup. In the online setting, where only a single epoch is permitted, we find that all methods are very sensitive to the learning rate (unsurprising, given that most methods will not have converged after a single epoch). For fair comparison, we thus search over learning rates in {0.1, 0.01, 0.001, 0.0001} } for all methods, choosing the best-performing one. Additionally, when learning speed is of critical importance, find it beneficial slightly increase т. We thus T = 0.05 for all online experiments, compared to 0.01 for all "offline" experiments.
Table 8: Architecture of the CNN used on digit and character datasets. For conv. layers, the weights-
shape is: num. input channels mum. output channels filter height filter width.
Block
Weights-Shape
Stride
Padding
Activation
Dropout Prob.
Conv + BN
3 x 64 x 5 x
2
2
ReLU
0.1
Conv + BN
64 x 128 3 3
2
2
ReLU
0.3
Conv + BN
128 256 x 3
2
ReLU
0.5
Linear BN
6400 x 128
N/A
N/A
ReLU
0.5
Linear
128 x Number of Classes
N/A
N/A
Softmax
0
22

---

Published as a conference paper at ICLR 2022
H RELIABILITY DIAGRAMS AND CONFIDENCE HISTOGRAMS
This section shows reliability diagrams (DeGroot & Fienberg 1983 Mculescu-Mizzl & Caruana 2005) and confidence histograms (adrrzzny Elkan 2001): over all EMNIST-DA shifts (see Figure 10 (ii) a severe EMNIST-DA shift (see Figure I and (ii) mild shift AMNIST-DA shift
(see Figure 12). Reliability diagrams given along with the corresponding Expected Calibration
Error (ECE, Naeini et al. 2015) and Maximum Calibration Error (MCE, Naeini et al. (015). ECE calculated by binning predictions into 10 evenly-spaced bins based confidence, and then taking weighted average of the absolute difference between average accuracy and average confidence of the samples in each bin. MCE is the maximum absolute difference between average accuracy and average confidence over the bins. Figures 10 12 below, we pair each reliability diagram with the corresponding confidence histogram, since reliability diagrams do not provide the underlying Figure 1).
frequencies of each bin (as in Guo al. 2017
In general we see that most models are overconfident, but our models much less so. As seen by the difference in the size of the red 'Gap' bar in the rightmost bins of Figures [0b 10c and [0d] when our FR methods predict with high confidence they are much more likely to be correct than IM-a method which works by maximizing prediction confidence. Figure shows that BUFR remains well-calibrated even when the initial shift is severe. Figure 12 shows that, even for a mild shift when all models achieve high accuracy, our methods are better-calibrater Note that the label 'Original' in Figures T0a and 10e denotes the source model the source data, while 'Source-only' in Figures
11a
11e
12a
and 12e denotes the source model on the target data.
Accurocy
Accuracy
Accuracy
Accuracy
ECE:11.7%
Accuracy
MCC: 0.1%
Accuracy
MCE:26.6%
Accuracy
MCE:26.0%
Accuracy
MCE:13.5%
08
CC 34 68
68
Confidence
Confidence
Confidence
Confidence
(a) Original
(b) SHOT-IM
(c)
(d) BUFR
Avg. Accuracy
Avg. Accuracy Avg. confidence
Avg. Accuracy Avg. confidence
Avg. Accuracy
Praction of Samales
Avg. confidence
Fraction of Samales
Frection of Samales
Fractiinn of Samales
Avg. confidence
Confidence
Confidence
Confidence
Confidence
(c) Original
(f) SHOT-IM
(g) FR
(h) BUFR
Figure 10: Reliability diagrams and confidence histograms over all EMNIST-DE DA corruptions. (a-d): Reliability diagrams showing the difference between average accuracy and average confidence for different methods. (e-h): Confidence histograms showing the frequency with which predictions made with given confidence. Each confidence histogram corresponds with the reliability diagram above it. (a & e): The source model is well-calibrated minimization leads to extreme overconfidence. (c & & h): Our methods, FR and BUFR, are much better-calibrated they do not work by making predictions more confident.
the source data. (b & f): Entropy-
23

---

Published as a conference paper at ICLR 2022
30
30
Accurocy Gap
Accurocy
Accurocy
Accurocy Gap
Gap
Gap
ECL-42.57%
ECE: 7O.07%
ECE:6
ECE: 5.3%
recuracy
rccuracy
recuracy
Co-
Aecuracy
Confidence
Confidence
Confidence
Confidence
(a) Source-only
(b) SHOT-IM
(c) FR
(d) BUFR
Avg. Accuracy Avg. confidence
Avg. Accuracy
Avg. Accuracy
Avg. Accuracy
fraction of samales
Fraction of Samples
Avg. confidence
Frection Samples
Avg. confidence
Frection Samples
Avg. conficence
14
68
60
34
64
62
64
50
54
64
Confidence
Confidence
Confidence
Confidence
(e) Source-only
(I) SHOT-IM
(g)
(h) BUFR
Figure 11: Reliability diagrams and confidence histograms for severe EMNIST-DA shift (sky) where all methods except BUFR achieve poor accuracy. Each confidence histogram corresponds with the reliability diagram above it. (a & e): Source model on the targer data achieves poor accuracy and often predicts with low confidence. (b f): SHOT-IM also achieves poor accuracy but is highly confident. (d & h): Our BUFR method achieves better ECE and MCE than all other methods.
Accurocy
Accurocy
Accuracy
Accuracy
Accuracy
MC2: 9.8%
recurscy
MCE:13. /%%
Accuracy
MCE:13.4%
Accuracy
MCCE:19.1%
433
14
68
60
14
64
34
64
Confidence
Confiderce
Confiderce
Confidence
(a) Source-only
(b) SHOT-IM
(c)
(d) BUFR
Avg. Aucuracy Avg. confidence
Avg. Ascuracy Avq. confidence
Avg. Accuracy Avg. confidence
Avg. Accuracy Avg. confidence
Practon of Samples
Fractio of Samples
Fraction of Samples
Fraction of Samples
Confidence
Confidence
Confidence
Confidence
(c) Source-only
(f) SHOT-IM
(g) FR
(h) BUFR
Figure 12: Reliability diagrams and confidence histograms for mild EMNIST-DA shift (shot noise) where all methods achieve good accuracy. Each confidence histogram corresponds with the reliability diagram above it. When highly confident, our methods (d h) are more often correct than IM f).
24

---

Published as a conference paper at ICLR 2022
I ACTIVATION DISTRIBUTIONS
EMNIST-DA (skewed). Figure 13 depicts histograms of the marginal feature and logit activation- distributions on the EMNIST-DA stripe shift. As shown, the marginal distributions on the source data (blue curve, those we wish to match) may be heavily-skewed. In contrast, the marginal distributions on the target data (before adapting, orange curve) tend to be more symmetric but have similar mean.
CIFAR-10 (bi-modal). Figure 14 depicts histograms of the marginal feature and logit activation- distributions the CIPAR-1O-C impulse-noise shift. As shown, the marginal distributions on the source data (blue curve, those we wish to match) tend to be bi-modal. In contrast, the marginal distributions on the target data (before adapting, orange curve) tend to be uni-modal but have similar mean. The two modes can be interpreted intuitively as "detected" and "not detected" or "present" and "not present" for a given feature-detector.
Alignment after adapting. Figure 15 shows histograms of the marginal feature activation- distributions on the EMNIST-DA stripe shift. This figure shows curves on the source data (blue curve, same Figure 13aj and on the target data (after adapting, orange curve) for different methods. Evidently, our FR loss causes the marginal distributions to closely align (Figure [5c). In contrast, competing methods (Figures 15a 15b) do not match the feature activation-distribuions even if they achieve high accuracy. Figure shows the same trend for CIFAR-10-C.
Zo DsK 0.8
21 Dskt 1.1
ag | DskL 0.6
a1 DsKL 1.6
Z2 DskL 1.0
23 DskL = 0.6
DsKL 0.3
a3 DsKL 0.3
Z4 DskL 0.4
Z/ DskL 0.8
a4 DskL 0.4
as DsKL
0.4
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
source
target
source
target
(a) Feature distributions
(b) Logit distributions
Figure 13: Histograms showing the first 6 marginal activation activation-distribuuine distributionss on the EMNIST-DA stripe
shift. The blue curves the saved marginal distributions under the source data (i.c. EMNIST). The orange curves are the marginal distributions under the target data before adaptation (i.e. the stripe shift). (a) Marginal feature setivation-distributio (b) Marginal logit activation-distributtoo DSKL denotes the symmetric KI. divergence.
25

---

Published as a conference paper at ICLR 2022
Zo DskL 1.7
21 1 DsKL = 0.6
8o DsKL 0.4
a1 DsKL 0.5
4.
2-
Z2 DsKL 1.3
23 DsKL 0.8
DsKL 2.6
33 D 1.2
4-
Z4 | D5KL= 1.4
25 DsKL 0.9
a4 | DsKL 0.5
a5 5KL 1.2
4-
2
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
source
target
source
target
(a) Feature distributions
(b) Logit distributions
Figure 14: Histograms showing first marginal dddivation-distributioss on the CIFAR-10-C impulse-noise shift. The blue curves are the saved marginal distributions under the source data (i.e. CIFAR-10). The orange curves are the marginal distributions under the target data before adaptation (i.e. the impulse-noise shift). (a) Marginal feature ieiiaationndisributtonn and (b) Marginal logit octivation-distributtons DSKL denotes the symmetric KI. divergence.
4
2> | Dox 0.1
2s Pse 0.3
| Dsc - 1.0
2 | Pan 0.8
2> Doc - 0.0
2 | Ese 0.0
2: Dox
2:
0.5
0.5
15
Source
target
Sourca
target
sourca
target
(a) AdaBN
(b) SHOT-IM
(c) BUFR
Figure 15: Histograms showing distribution-alignment on the EMNIST-DA stripe shift. The blue curves the saved marginal distributions under the source data (i.e. EMNIST). The orange curves are the marginal distributions under the target data after adaptation (to the stripe shift). (a.b): AdaBN and SHOT-IM do not align the marginal distributions (despite achieving reasonable accuracy Table 17 (c) BUFR matches the cediiaationdistriiuutee very closely, making DSKL very small.
26

---

Published as a conference paper at ICLR 2022
0.24
0.02
a
20 Dsg 0.22
2: Dov 0.12
20 Dsn 0.01
Dov 0.20
23 Dsc 2.00
21 Dsa 0.00
2: Dsa
0.03
0.0
0.5
15
Source
target
2ourca
target
Sourca
target
(a) AdaBN
(b) SHOT-IM
(c) BUFR
Figure 16: Histograms showing distribution allimmmmnt on the CIFAR- 10-C impulse-noise shift. The blue curves the saved marginal distributions under the source data (i.e. CIFAR-10). The orange curves are the marginal distributions under the target data after adaptation (to the impulse-noise shift). (a) AdaBN does not align the marginal distributions. (b) SHOT-IM only partially-align the marginal distributions. (c) BUFR matches the activation disttiiooisssssstiinnn very closely, making DsKL very small.
J FURTHER ANALYSIS
J.1 EFFICACY OF BOTTOM-UP TRAINING
Table reports EMNIST-DA accuracy vs. the number of (unlabelled) sxamples-per-chee available in the target domain. BUFR retains strong performance even with only 5 examples-per-cllas.
Table EMNIST-DA accuracy vs. examples-per-clase
Model
10
20
400
Marg. Gauss.
Ishii & Sugiyama 2021
49.3
49.9
50.4
50.7
50.6
Full Gauss.
55.4
59.7
61.0
63.3
68.3
PL LLc et al.
2013
45.8
46.3
46.0
46.7
49.7
BNM-IM (Ishn Sugiyama 2021
50.5
51.5
53.0
54.7
61.4
SHOT-IM (Liang et al. 2020
48.3
51.7
51.2
54.7
73.4
FR (ours)
50.8
50.5
60.1
63.1
75.6
BUFR (ours)
78.0
82.3
83.8
84.9
86.2
J.2 LOSS ABLATION STUDY
Table reports the performance our FR loss on CIFAR-10-4 and CIFAR- without: (i) aligning the logit distributions; and (ii) using the symmetric KL divergence (we instead use the asymmetric reverse KL). While these components make little difference on the easier task CIFAR-10-C, they significantly improve performance on the harder task of CIFAR-100-C.
Table 10: Ablation study of Ligt in Eq
Model
CFR-10-C
CFR-100-C
Ligt w/o logits
86.7 0.2
62.3 1.3
Ligh w/o (pl())
86.5 + 0.3
61.5 0.2
Lig
87.2 0.7
65.5 + 0.2
27

---

Published as a conference paper at ICLR 2022
J.3 WHO IS AFFECTED
We now analyse which layers are most affected by measurement shift. Figure 17 shows the (symmetric) KI. divergence between the unit-level activation distributions under the source (FMNIST) and target (EMNIST-DA crystals) data before adapting (17a) and after adapting the first layer (17b). Figure 17a shows that, before adapting, the unit-activation distributions in all layers of the network have changed significantly, as indicated by the large KI. divergences. Figure shows that, after updating just the first layer, "normality" activation distributions the target data realigning with those saved the source (shown via very low KL divergences). This indicates that measurement shifts primarily affect the first layer/block- since they can be mostly resolved by updating the first layer/block and also further motivates bottom-up training for measurement shifts.
restored all subsequent layers, with the unit-level
Convl
Conv2
Conv3
FC1
FC2
Conv1
Conv2
Conv3
FC1
FC2
0.5
0.5
:1:
0.3
0.3
:
0.2
0.2
0.1
0.0
0.0
(a) Before adapting
(b) After adapting Conv
Figure 17: Symmetric KL divergence between the unit-level activation distributions under the source (EMNIST) and target (EMNIST-DA crystals) data: (a) before adapting; and (b) after adapting only the first layer (Convl). 1). For visual clarity, we show only 10 sample units per layer.
J.4
WHO MOVES
We now analyse which layers are most updated by BUFR. Figure 18a shows that, average, FR moves the weights all layers Ir a similar distance when adapting to the target data. Figure [8b shows that BUFR primarily updates the early layers, thus preserving learnt structure in later layers.
Dist. Moved
Avg. Dist. Moved
1.0
2.0
0.8-
1.5
0.6
0.4
Avg.
00.2
0.0
0.0
Block
Block
(a) FR
(b) BUFR
Figure 18: Average distance moved by a unit in each block of 9t on the EMNIST-DA stripe shift when training (a) all layers at once and (b) in bottom-up manner. Both methods trained with the same constant learning rate.
28

---

Published as a conference paper at ICLR 2022
K FULL RESULTS
In this section we give the full results for all datasets and constituent domains.
K.1 DIGIT AND CHARACTER SUMMARY RESULTS
The simplest datasets we use are variations of the MNIST dataset (LeCun et al., 1998) Here, a model trained MNIST (source domain) before being adapted MNIST-M (Ganin et al. 2016) of
the fifteen MNIST-C (Mu & Gilmer 2019) corruptions (target domain). As mentioned Section the MNIST-based shifts can be well-resolved by number of methods.
Tables and 12 summarize the accuracy and ECEs across different models for the digit and character datasets. On MNIST-C, where source-only only accuracy is very high, all methods achieve good results (accuracy 2 -5%))-rovidin limited insight into their relative performances. On MNIST-M, our BUFR method outperforms all baselines, although SHOT is very similar performance. As discussed in Section [3] our BUFR method outperforms all baseline methods on EMNIST-DA in terms of accuracy and ECE as does not work by making predictions more confident.
Table 11: Digit and character accuracy (%) results. Shown are the mean and standard deviation. EMNIST-DA: mean performance over all 13 EMNIST-DA shifts. EMNIST-DA-S & IMNIST-DA-MLLD sample "severe" and "mild" shifts from EMNIST-DA selected based on AdaBN performance.
Model
MNIST-C
MNIST-M
EMNIST-DA
EMNIST-DA-SVR
EMNIST-DA-MEL
No corruption
99.5 = 0.1
99.5 0.1
89.4
8.14
89.4 0.1
Source-only
86.2 1.8
42.7 4.6
2.5
3.8 0.4
78.5 0.7
AdaBN (Lá
2018
94.2 0.2
59.1 1.9
46.2
0.7
84.9
PL (Lee al. 2013
96.4
4.1
0.6
2.7
83.5 0.1
SHOT-IM (Liang al. 2020
0.2
66.9
3.7
2.50
86.3
SHOT [Liang et al. 2020
9..7
94.4
80.0
55.1
86.1
FR (ours)
0.1
0.6
0.84
1.
86.4
BUFR (ours)
0.6
96.2
8..1
8.26
87.0
Turget-suprrised
99.3 0.0
98.5
86.8 0.6
85.7 1 0.6
87.3 0.7
Table 12: Digit and character ECE (%) results. Shown are the mean and standard deviation. EMNIST- DA: mean performance over EMNIST-DA shifts. EMNIST-DA-SVE & EMNIST-DA-MI.D sample "severe" and "mild" shifts from EMNIST-DA selected based on AdaBN performance.
Model
MNIST-C
MNIST-M
EMNIST-DA
EMNIST-DA-SVR
EMNIST-DA-MLD
No corruption
0.3 + 0.0
0.3 0.0
2.3
2.3
2.3
Source-only
7.2 1.4
27.0+77
30.8 + 1.6
42.6 + 3.5
4.8 + 0.5
AdaBNN (Li et al. 2018
4.1
24.4 2.8
10.
52.4
4.9+0.3
PL Lec al. 2013
3.4
56.1 2.2
49.9
97.2 0.4
16.4 0.1
SHOT-IM (Lang
2020
0.2
3.0
3.7
76.0
13.7
SHOT [Liang et al. 2020
0.2
2.9
19.7
4 23.0
14.8
FR (ours)
2.2
9.8
12.9
5..0
4.6
BUFR (ours)
3.0
2.5
4.7 0.2
5.6 + 0.3
4.2 + 0.2
Supget-supervied
0.5
1.1
7.3 0.7
0.5
8.4
29

---

Published as a conference paper at ICLR 2022
K.2
ONLINE RESULTS
Table reports the online results for CIFAR-10-C and CIFAR-100-C. FR outperforms existing SFDA methods CIFAR-10-C in terms of both accuracy and ECE. On CIFAR-100-C, our method is competitive with TENT (Wang al. 2021 method designed specifically for this online setting. As in Wang et al. (2021). these results represent the average over batches during training (i.c. single pass through the target data), rather than the average the end of training. order evaluate online performance. We omit BUFR from this table as it is not easily applicable to the online setting is difficult to set the number of steps per block without information
the total number of steps/batches (generally not available an online setting). Full per-shift results for this online setting are given in Tables 23 and 24 for CIFAR-10-C and Tables 25 and 26 for CIFAR-100-C.
Table 13: Online results. Shown are the mean and standard deviation.
CIFAR-10-C
CIFAR-100-C
Model
ACC 1
ECE
ACC f
ECE
AdaBN (Li ct al
2018
1.0
56.6 0.3
10.0 + 0.1
SHOT-IM (Liang et al.
2020
0.2
0.1
0.3
13.8 = 0.1
TENT (Wang al. 2021
81.8
11.5
14.3 6.3
FR (ours)
85.9
9.2
0.3
13.6 = 0.1
K.3 CAMELYON17 RESULTS
Table 14 reports the accuracy and ECE results for CAMELYON17. With up to 50 target examples- per-class: (i) our methods reduce the eттоr rate by approximately 20% compared the next best method; (ii) only our methods meaningfully improve upon the simple AdaBN baseline which uses the target-data BN-statistics (i.e. neither PI. SHOT-IM actually work). With up 500 target examples-pers per-class, our methods reduce the error rate by approximately 20% compared to the next best method. With over 15,000 examples-per-class
methods are competitive with existing ones.
Table 14: CAMELYON17 results for different numbers of (unlabelled) examples caammlle--per-laas per-class in the target domain.
Model
50
S00
>15k
ACC
ACC
ACC
ECE
ACC
ECF
Source-only
55.8 +: 1.6
40.8 1:
55.8
1.6
40.8 +: 2.1
55.8 +: 1.6
40.8 2.1
55.8 1.6
40.8
2.1
AdaBN
82.6
14.7 2.1
83.7
1 1.0
13.7 0.8
0.8
13.5
0.7
0.5
13.5 0.5
2.0
14.2
63.6111
13.8
1.0
8..
13.0
8.8 9.9
SHOT-IM
2.2
13.8
83.7 1 1.2
13.8 111
0.74
1..
8.2
9.7
FR (ouns)
84.6
1.5
86.0 1.1
12.1 1.1
8.6
9.7
89.5
9.8
BUFR (ours)
84.5
9.7
9.6
30

---

Published as a conference paper at ICLR 2022
K.4 MNIST-C FULL RESULTS
Tables [3 and 16 show the accuracy and ECE results for each individual corruption of the MNIST- dataset. We provide the average performance with and without the translate corruption as the assumptions behind the methods that rely on fixed classifier h no longer hold. Without the translate corruption (Avg. Mranslate) we see that all methods achieve high accuracy (2 95%)..
Table 15: MNIST-C accuracy (%) results, Shown are the mean and standard deviation.
Sre-only
AdaBN
SHOT-IM
SHOT
BUFR
Brightness
84.8
11.4
99.1 0.0
99.5 0.0
99.1 0.1
99.5
98.7 0.1
99.2 0.1
Camny Edges
72.2 0.8
91.0 0.7
96.2 1.0
98.1 0.1
9.1
97.8
9.0
Dotted
98.6 18.6::02 0.2
98.8 0.2
99.3 0.1
99.4 0.1
9..4
98.6
99.1 99.1
30.1 12.5
93.9
0.0
0.1
0.0
0.1
0.0
Glass Blur
88.9
0.3
97.8
9.1
9..
9..
9.0
Impulse Noise
0.0
Motion Blur
3.9
97.3
981
9911
9.1
98.1
98.8
Rotate
0.1
96.7 0.0
97.7
9..4
9.0
97.
9.19
Scale
0.1
97.2
98.7
9.0
9.0
9.0
98.7
Shear
08.9 0.1
98.9
0.0
9.0
0.0
9.1
9..8
Shot Noise
0.0
0.0
0.0
0.1
0.0
0.2
9
Spatter
0.1
9.1
99.0
99.0
99.1
98.1
98.8
Stripe
Translate
0.5
0.6
0.8
4.1
3.1
7..
64.5
Zigzag
91.8 138:::00 0.6
93.0 0.2
98.2 0.2
98.9
99.2 19.2:000 0.1
98.2
98.8 1 0.1
Avg.
86.2 1.8
91.2
96.6
97.7 9.2
96.
96.1
Avg Arg'transtate
1.9
98.5
98.9
99.1
9..1
Table 16: MNIST-C ECE (%) results. Shown are the mean and I standard deviation.
Sre-only
Brightness
2.4
0.3 1:3000 0.1
0.3 0.1
0.4 0.1
0.9 0.1
0.9 0.1
0.5 0.1
Canny Edges
22.20.0
6.7
0.1
Dotted Line
0.7 1: 0.1
0.1
0.1
0.4 0.1
0.9 0.0
1.0 0.1
0.6 0.1...1
Fog
26.4
0.0
1.1
Glass Blur
5.9 1.6
3.3
1810011
1.4
0.2
21
0.1
1.1
Impulse Noise
1.2 0.2
1.100.1
12101
0.1
0.7 0.1
1.5 0.1
1.0
Motion Blur
9.1
1.4
1.2
0.1
0.8
1.4
0.8
Rotate
2.0
2210011
19
12
0.6
1. 0.1
1.5
Shear
0.7 0.7
1.0
1.10
0.7 1.1.01
0.8 0.1
0.101
0.9 0.1
Noise
0.1
0.1
0.1
1.20.1
0.7
Spatter
0.6
0.1
0.7
0.7
0.9
121
0.8 0.0
Stripe
4.0
6.4
2.133.1
0.1
0.9
1.2
0.6 0.0
Translate
0.3
0.6
0.7
2..2
1..2
18.8
03.7
Zigzag
5.5
4.9
13202
0.8 0.8 0.0
0.8
14
0.8 0.800 0.1
Avg.
1.2
4.1
3.1
233 133:00
2.0
2.2
3.0
Avg Itransatee
5.9.1
2.2
1.400.
0.8 0.0
0.8 0.0
1.4
0.8 1: 0.0
31

---

Published as a conference paper at ICLR 2022
K.S EMNIST-DA FULL RESULTS
Tables 17 and 18 show the accuracy and ECE results for each individual shift of EMNIST-DA, We provide the average performance with and without the "background shifts" (bgs), where the background and digit change colour, these are often the more severe shifts.
By inspecting Table we see that the sky shift resulted the lowest AdaBN accuracy, while the shot-noise shift resulted in the highest AdaBN accuracy. Thus, deem these be the most and least severe EMNIST-DA shifts, i.c. the "severe" and "mild" shifts. We find AdaBN to be a better indicator of shift severity than source-only as some shifts with poor source-only performance can be well-resolved by simply updating the BN-statistics (no parameter updates), e.g. the fog shift.
Table 17: EMNIST-DE DA accuracy (%) results. Shown are the mean and standard deviation.
AdaBN
Marg. Cours.
Full Gaus.
BNM-IM
SHOT-M
Bricks
0.5
22.6 2.1
1.2
14:2 1.4
20.5
8.8
76.0
32.4 1
83.8
Crystals
19.7 3.1.
42.1
50.0 0.0.
61.0
1.0
47.4
61.2 2.5
71.5
38
80.1 0.0.
76.8 0.4
82.6 0.3
Dotted Line
76.2 0.7
80.8 0.4
82.3 +.4
82.6 0.5
80.7 + 0.5
86.5 +
87.1 +
87.5 +.1
87.6 0.1
88.3+00. 0.0
0.9
03.0 2.6
0.6
0.6
77.4
0.1
0..0
87.0
88.3
Grazs
2.3 0.1
0.4
52.7
3.5
6.7
14.8 8.1
12.1 40.9
61.5
38.5
79.2
84.5
Impulse Noise
36.8 1.6
76.7 0.88
81.11003
82.6
1 8.1
78.9
0.6
81.2 1
84.1
81.4 1.4...0..
81.8 1: 0.2
86.0 16.01:000
Invesee
0.5
8.1
1..12:6.2
64.4
113 2 2.0
60.4 23.4
83.2 2:
85.1 0.0.2
83.1 4: 0.7
88.3 0.1
Oranges
26.5
60.7 2.3
1.0
71.1
0.6
8.1
0.66
0.3 $0.5
82.4
82.3
0.3
Sky
33410
0.7
43:04
20.8
0.66
8.3
26.0
55.1
23.5
15.3
0.2
Stripe
15.4++.11
46.3 +.9
63.0+56
82.3
0.8
63.8 + 3.7
82.8 0.6
89.9
0.3
85.1 0.0.2
84.5 +
87.1
Zigzag
65.0++0.2
71.3+00.
73.8+00.1
76.1 0.3
72.3 + 0.2
79.7 0.5
81.0 0.5
85.8 0.2
85.7 +
87.5
0.2
M1+11
74.4
86.1
Ang-lbgs
03.3
Table 18: EMNIST-DA ECE (%) results. Shown are the mean and standard deviation.
Sec-only
AdaBNN
Marg. Ganss
Bricks
54.6 + 5.0
64.4
62.0 + 0.9
52.4
93.2 1.2
84.
79.4
22.5 + 0.3
44.1
0.4
Crystals
27.0 3.0
20.0
24.0 0.3
22.0 0.6
52.7 :
38.1
28.5 3.8
19.3 +
10.5 01:5=:00 0.3
6.3 0.3
Dotted Line
0.6
9:2 0.5
7.9
7.4
0.7
13.0
12.9
0.2
3.4 3.4
1.1
3310..1
Gaussian Blur 15.09.99
15.3
0.5
63103
21.2
15.
16.4
160103
6.4.07
4..10.1
Grass
21.6-t504
0.8
61.0
277
13.6 1.5
84
57.5 40.8
37.5 36.1
87
5.7
Impuise Noise 32.0
9.9
0.3
6.610.11
2.61
1.3
15.6 0.2
15.
5.310.11
1.10.1
14.7
3.110.1
Orangos
23.8 2.7
23.4223
10.1400
57.61.27
19.6
1946
17.4
69+05
5.:+0.3
Shot Noise
4.8 0.5
4.-0033
0.3
49+002
16.4
13.0+0.1
13.7
14.8
4.6+03
4.2-022
Sky
42.6
52.4
$.6
4..8
0.4
00.2+11...
76.0 7.5
42.7 + 23.0
68.0
5.=
Stripe
6..
31.6
4..2
6.8.1..4
3..2
16.
1.3
15.0
5.4
4.10.1
Zigzag
Avg.
30.8 &.
30.3 1.1
16.7
174
49.9
35.6
29.6
1..7
129209
47202
Ang.
28.9
20.3
18.
83+
318
18.2
15.6 0.1
14.6
5.0
4.0
32

---

Published as a conference paper at ICLR 2022
K.6 CIFAR-1O-C FULL RESULTS
Tables 19 and 20 show the accuracy and ECE results for each individual corruption of CIFAR-10-C. It is worth noting that BUFR achieves the biggest wins on the more severe shifts, i.e. those on which AdaBN (Li al. 2017) performs poorly.
Table 19: CIFAR-1O-C accuracy (%) results, Shown are the mean and standard deviation.
Sro-only
AdaBN
PL
SHOT-IM
TENT
FR
BUFR
Brightness
91.4 0.4
91.5 0.3
91.9
0.3
93.2 + 0.3
+ 0.1
93.3 + 0.3
Conirast
32.3 1: 1.3
87.1 0.3
86.6
3.2
90.8 0.9
91.3 1.7
90.9 0.9
92.9 0.7
Defoems blr
53.1 6.4
8.4
89.3 0.5
0.4
90.9
90.9
91.5 0.5
Elastic
77.6 0.6
7.42
0.8
8.5
8.5
82.7
84.2 0.3
Fog
89.5
0.5
Frost
64.4 2.4
0.7
1.2
0.6
0.7
0.6
8.9
Gauss.
3
8..
0.7
9..5
0.
0.6
9..
Gauss.
27.7
69.2
0.6
0.9
0.3
8..9
0.4
Glass
51.3.1:1.8
6..7
0.3
73.7
7..7
7..8
80.3
0.0
Impulse
25.9
1
0.5
0.8
0.8
7..6
89.3
Jpeg compe.
74.9
14.1
0.6
81 1.3
8..9
8..4
85.8 $5.810.88
hir
0.2
87.8
89.2
Pixelate
48.2
8.5
0.4
0.7
8..6
87.
0.6
Saturate
59.9
9.1
9.3
93.1
93.3
93.4
9.3
Shol
3..4
1.2
0.9
8.76
0.6
85
0.2
Snow
76.6 1.1
8..4
8.18
8..4
87.8
8..4
89.7
Spatter
0.8
0.5
0.33
0.2
8.3
914
0.5
Speckle mse
3.7
1..8
1.2
813
8.9
8..8
87.4.10.44
Zoom
60.5
8..
0.4
0.2
9.3
91.2
Avg.
57.8 0.7
0.1 0.1
82.5 0.3
85.4
86.6 0.3
87.2 0.777
89.4 0.2
Table 20: CIFAR-1O-C ECE (%) results. Shown the mean and standard deviation.
Sre-only
AdaBN
PL
SHOT-IM
TENT
FR
BUFR
Brightness
4.7 0.2
4=1.11
8.1 0.2
7.2 = 0.4
6.4 0.3
5.9
0.3
6.2 = 0.2
Conirast
43.5 2.8
5.7::0.44
13.2
3.1
9.8 0.9
8.4
0.4
0.7
Defoems blr
28.2 1
6.1.:00 0.4
10.7 0.5
0.4
8.6
7.8
7.9 0.4
Elastic
12.4
0.
2.8
18.6
16.5
1..1
1..
Fog
1.1
7.7
0.8
11.
1..1
8.9
Frost
22.7 1.7
1..4
1.2
0.66
0.6
10.7
0.9
Gauss.
40.7
6.140.4
0.7
9.5
8.5
7.5
7.4
Gauss.
6.7
18.5
2.6
2..9
1840.4
1.3
13.3
Glass
31.2.1.1.3 1.3
20.8
30.9
26.311
2..2
20.9
18.9
Impulse
51.2
23.
0.5
2..
2.87
2..6
1.3
Jpog compr.
14.6
1..5
2.66
18.9
1..4
14.5
0.7
blr
1.3
6.3003
1.2
8.7
Pixelate
36.9 2.5
11.1
17.9
14.5
1..9
1..7
9.5
Saturate
0.3
121011
7.3
6.9
6.100.1
5.9
6
Shot
5.9
1.
2.9
18.4
1.69
1.
1.2
Snow
0.5
9.8
16.1
13.6
11.6
10.5
9.7
Spatter
0.8
0.3
0..3
0.2
0.2
9.3
7.6
Speckle use
13.2
17.9
2..
18.71
16.1
13.9
11.904
Zoom
1140.
Avg.
28.2
11.2
17.5 0.3
14.6
12.8 0.3
11.3
1.
33

---

Published as a conference paper at ICLR 2022
K.7
CIFAR-100-C FULL RESULTS
Tables 21 and 22 show the accuracy and ECE results for each individual corruption of CIFAR-100-C. It is worth noting that BUFR achieves the biggest wins on the more severe shifts, i.e. those on which AdaBN (Li et al. 2017) performs poorly.
Table 21: CIFAR-100- accuracy (%) results. Shown are the mean and standard deviation.
Sro-only
AdaBN
PL
SHOT-IM
TENT
FR
BUFR
Brightness
63.2
66.1 0.5
0.7
72.6
72.2 0.5
71.8 0.5
73.6 + 0.2
Conirast
13.9
61.4
59.2 3.5
70.1 0.4
6.1
0.5
72.2 0.5
Defoems blr
35.9
6.1
69.3
0.3
0.5
0.1
72.2
Elastic
$ 0.7
6.2
6.4
6.2
6.2
6..9
67.1
Fog
Frost
41.1 0.9
65.3
0.81
0.4
0.9
0.5
0.7
Gauss.
28.2
0.3
0..
7.2
0.3
0.3
0.6
Gauss.
11.9 1.2
0.6
0.7
6.4
0.6
3..7
0.3
Glass bir
45.1 1: 0.9
53.3
$7.3 0.7
0.34
61.4
60.3
63.2
Impulse
7.8
0..
0.5
0.6
0..3
0.9
0.6
Jpeg compe.
48.6 0.9
4..8
55.8
6.5
60.8
5..5
62.6
hir
45.1
0.2
0.2
0.5
Pixelate
22.3 0.4
0.6
0.6
6..7
6..
68.1
7.5
Saturate
$5.8 0.4
6..7
7..2
0.26
7..
0.5
0.6
Shol
1.2
4.66
5..1
0.6
6..3
5.3
62.1
Snow
49.4 0.8
53.5
5.9
0.6
65.6
63.8
65.9
Spatter
54.8
0.6
0.3
0.4
0.5
0.5
0.2
Speckle nse
15.6 1.3
12
$4.2 1.5
6..1
59.8
5..8
62.1 1..1222 2.7
Zoom
45.1
0.3
7.3
7.81
0.6
Avg.
36.4
56.6 0.3
62.1 0.2
0.2
06..0.44
65.5 0.2
0.2
68.5 0.2
Table 22: CIFAR-100-C ECE (%) results. Shown are the mean and I standard deviation.
Sre-only
AdaBN
PL
SHOT-IM
TENT
FR
BUFR
Brightness
6.3 0.3
9.4 0.3
30.2
0.1
20.7 0.1
0.3
12 0.6
Conirast
37.8 3..8
1..
40.5 3.4
29.6
29.5
3.5
1.2
12.8 0.5
Defoems blr
16
9.7
0.2
0.4
0.3
1.3
1.2
Elastic
8
0.2
3..9
0.3
25.8
13.2
1..3
Fog
0.6
1.3
0.6
0.7
2..
13.
0.6
Frost
14.1
1..3
0.87
0.4
2..1
1..
13.3
Gauss.
2.4
10
01
0.2
2..
13.
12.5
Gauss.
39.3
16.7
46.8
3.5
3.7
1.5
0.5
Glass
1.1
13.4
0.7
3..6
29.1
17.9
17.6
Impulse
2.6
1.24
0.6
4.7
0.8
20.
0.2
Jpog compr.
8.6 8.2
1.4
4..
08.
29.6
19.1
18.2
blr
0.2
1.34
0.2
23.2
14.
13.
Pixelate
27.5
11.60.4
0.6
3..3
0.3
14.2
13.6
Saturate
8.8
9.3
29.6
27.
21.2
12.7
1.T
Shot
5.9
162
4.88
3..1
3..2
18.6
0.6
Snow
8.3
1..4
40.9
3..9
2..7
171.0.5
14.9
Spatter
0.3
0.1
0.3
2..2
2.
12
1.4
Speckle ase
31.55:..1
1..2
1..7
0..
30.6
18.7
16.8
Zoom
1..1
0.77
Avg.
19.4 0.9
12.5 0.1.
37.7 0.2
0.22
25.7 1.44
1..1
14.5 0.3
34

---

Published as a conference paper at ICLR 2022
K.8 CIFAR-1O-C FULL ONLINE RESULTS
Tables 23 and 24 show the accuracy and ECE results for each individual corruption CIFAR-10-C when adapting in an online fashion (see Appendix K.2 It is worth noting that FR achieves the
biggest wins on the more severe shifts, i.c. those on which AdaBN (Li al. 2017 performs poorly.
Table 23: CIFAR-1O-C online accuracy (%) results. Shown are the mean and standard deviation.
Sre-only
AdaBN
SHOT-IM
TENT
FR
Brightness
91.4 21.40..1
91.6 1..60.22
9.1
91.8 +
92.8
0.3
Contrast
32.3 1: 1.3
87.1 0.4
0.5
87.8
0.6
89.8 0.6
Defoeus bir
53.1 6.4
0.5
0.5
89.1
0.5
Elastic
0.6
0.3
80.3
79.2
82 0.4
Fog
1.1
Frost
2.44
80.7
0.8
0.8
0.7
Gauss.
8
8.7
0.6
8.5
90.8
Gauss.
5.1
0.9
0.8
0.7
0.6
Glass
1.8
6..7
70.6
6.66
74.7
Impulse
3.8
1.2
0.8
0.7
0.4
Jpeg compe.
1:
1.2
1.94
7.9
0.5
hlr
66.1
0.1
8..6
0.2
Pixelate
48.2
8.4
0.7
0.5
8.7
Saturate
0.4
91.9
9.A
0.2
0.2
Shol
4.9
1.2
1.
1.3
0.6
Snow
7..
82.6
8..9
8..4
86.8
Spatter
0.8
0.5
0.2
0.2
0.2
Speckle mse
3..7
1.7
0.66
0.6
0.2
Zoom
6.1
0.24
88.6
90.7
Avg.
0.7
80.3
83.2
1.2
85.9 1. 0.3
Table 24: CIFAR-10-C online ECE (%) results. Shown are the mean and standard deviation.
Sre-only
AdaBN
SHOT-IM
TENT
FR
Brightness
4.7 0.2
5.4 0.2
5.1
5.1 0.2
1.9 0.3
Contrast
43.5 2.8
1 0.4
8.100.
7.6
6.1
0.4
Defocus bir
28.2 1.4
7.1 0.4
0.3
71:033
6.4 0.3
Elastic
12.4
0.3
12.7
0.5
12.3
Fog
2.1
8.4
8.3
8.3
7.3
Frost
22.7 1.7
1.22
1..9
1.
9
Gauss.
40.7
7.4
6.8
7
6.3
Gauss.
6.7
19.2
1.
17.6
1.6
Glaes
1..2.1...
2..6
19.6
2..7
17.9
Impulse
51.2
2.9
20.6
2..2
1..9
Jpgg compr.
14.6
1.93
1.5
1..1
0..2
blr
1.3
7.6
Pixelate
2.6
12
1..8
1..5
8.9 0.5
Saturate
5.3
5.1
5-0.1
5.1
1.2
Shot
5.9
1..9
14.3
16
0.5
Snow
14.3
10.3
9.9 9.9
10.4
8.9 0.5
Spatter
1..9
1..2
9.1
9.6
7.6
Speekle nse
4..2
1..
1..
16.3
11.9
Zoom
Avg.
28.2 0.0.
12.1 10
1..
11.5 1.11
9.5 0.2
35

---

Published as a conference paper at ICLR 2022
K.9
CIFAR-100-C FULL ONLINE RESULTS
Tables 25 and 26 show the accuracy and ECE results each individual corruption of CIFAR-100-6 when adapting in an online fashion (see Appendix K.2 It is worth noting that FR achieves the
biggest wins on the more severe shifts, i.c. those on which AdaBN (Li et al. 2017 performs poorly.
Table 25: CIFAR-100-C online accuracy (%) results. Shown the mean and standard deviation.
Sre-only
AdaBN
SHOT-IM
TENT
FR
Brightness
63.2
66.1 0.4
0.9
0.7
69.4 0.4
Contrast
13.9 1:99::0.0 0.6
61.4 0.5
6..
66.6 66.6
64.5 0.3
Defoeus bir
35.9
65.6
0.1
0.3
0.2
Elastic
0.7
60.4
63.3
63.7
6.3
Fog
0.5
Frost
41.1
0.6
60.5
0.6
0.8
Gauss.
28.2
0.3
0.2
0.6
6..1
Gauss.
11.9 1.2
0.6
0.2
0.5
0.3
Glass
45. 0.9
0.6
57.8
5..2
5..9
Impulse
7:2 0.8
0.5
0.4
0.7
0.8
Jpeg compe.
4.96
4.78
36
1.2
5.2
56.6
hlr
45.1
0.2
0.67
0.3
Pixelate
2.4
0.6
0.71
67.1
0.66
Saturate
0..
6..7
0.6
6.6
6..3
Shot
14.1
4.66
5..9
0.4
0.34
Snow
4..4
53.5
5..7
61.8
6..5
Spatter
1.1
0.6
7..
0.6
0.7
Speckle mse
1.3
11
5..2
5..9
55.8 0.6
Zoom
0.7
65.9 0.3
6.6
0.4
68.8
Avg.
3.5
56.6 1: 0.3
62.3 0.3
63.1 0.3
62.7 0.3
Table 26: CIFAR-100-C online ECE (%) results. Shown are the mean and standard deviation.
Sre-only
AdaBN
SHOT-IM
TENT
FR
Brightness
6.3 0.3
1.1
1..6
0.2
10.9 0.2
Contrast
37.8 2.2
12.5 0.2
14.6
1..1
12.5
0.2
Defocus bir
161:0.8
1.2
1.2
1..
1.3
Elastic
0.1
12.7
12.9
13.4
13
Fog
0.6
1.2
1.2
14.
13.
Frost
1.1
0.2
0.66
0.3
13.9
Gauss.
2.4
1..9
11.6
11.9
11.9
Gauss.
5.5
0.3
1..3
17.2
1.33
Glass
1..
1.110.11
1..5
16.1
15.1
Impulse
2.6
1.2
1..1
1.54
0.3
Jpgg compr.
8.6 8.6
1..
15.9
16A4
16.2
blr
0.2
12.2
12
0.22
Pixelate
27.5
1.+
1.3
12.4
0.2
Saturate
8.8
11.4
11.3
11.7
11.3
Shot
5.9
1.3
16.4
17.
1.3
Snow
8.3
1..2
1.410.
14.7
14.8
Spatter
0.3
1.24
1.2
11.3+0.3
0.3
Speekle nse
3..55..11
18.2
16.5
17.5
16.1
Zoom
1.2
Avg.
19.4
1.11
13.8
1..1
1..6
36

---

Published as a conference paper at ICLR 2022
L
NOTATIONS
Table 27
summarizes the notations used in the paper.
Table 27: Notations.
Symbol
Description
Ps
Source feature distribution
Target feature distribution
Psa
Source d-th marginal feature distribution
Distributions
4sg
Target d-th marginal feature distribution
Source approx. marginal feature distributions
Target approx. marginal feature distributions
T
Source d-th approx. marginal feature distribution
Target d-th approx. marginal feature distribution
Wa
Source approx. marginal logit distributions
Target approx. marginal logit distributions
Tuk
Source k-th approx. marginal logit distribution
Tay
Target k-th approx. marginal logit distribution
D.
Labelled source dataset Unlabelled target dataset Input-set of the source domain Input-set the target domain Label-set
Dr
Sets
X.
X.
y.
target domain
34
Label-set of the target domain
f.
Source model, f. = l-g=(-)
Network
ft
Target model, ft = (g:(()) Source feature-extractor
9-
9:
Target feature-extractor
Classificr (or regressor)
Soft-binning function
Other
2d
min
Minimuma value of feature d (on the source data)
2d
Maximum value of feature (on the source data)
Temperature parameter for soft binning
37


Reviews:
[{"review_id": "WB3Xc6IA_n3", "reviewer": "Reviewer_uMHx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### *Strengths:*\n- The paper is well structured and easy to follow. It is well written and the method is easy to understand.\n- The experimental analysis is very extensive. Most of the claims and arguments made in the earlier sections are validated with experimental analysis. Also, the proposed method for the most part is novel and quite interesting.\n- In my opinion, the major strength of the paper comes from its experimental analysis section. Specifically, extensive comparisons are provided with the most of the methods available in the literature. The paper considers many benchmarking datasets such as cifar-10/100, camelyon17 and also develops a new benchmark with character recognition dataset emnist. However, I would suggest omniglot [1] would be a more challenging dataset than emnist and could be considered in future updates of the work.\n- The paper also provides nice analysis of the activation statistics under the *measurement shift*, ablation analysis on which components are useful and discussions on the observations. Overall it provides great insights on the affects of *measurement shift* on activation statistics.\n\n\n### *Concerns:*\n- Though the paper has reasonable novelty and an extensive experimental analysis, there are some aspects of the work which are either not clearly explained or is not validated through experiments.\n- In the earlier sections, it is argued that methods such as [2] [3] [4] etc are not comparable to the proposed method as they \" are still classification-specific and rely on good initial feature-space class-separation for entropy minimization\". However, it is not a strong argument to avoid comparison with these methods which have previously addressed the problem of source-free adaptation. Furthermore, they are very closely related to the proposed method idea of activation distribution matching as compared to their approach of feature-distribution matching. Comparing with these methods would provide key insights into which type of feature-matching is better for the cases considered in the paper. Also, the argument that these methods are classification-specific is not properly evaluated, since the proposed method is also evaluated for only the task of classification.\n- The considered *measurement shift* case is widely established as performance under *common corruptions* in the literature, which is what the paper uses for performance evaluation. It is not entirely clear if they are one of the same, which it seems to be. Then why is there a need to define it as *measurement shift*. It would be helpful to get more clarification on the differences between *common corruptions* vs proposed *measurement shift*.\n- Also, there are methods like [5], [6] which consider generalization by improving training on source/clean data and without the need for any target/corrupted data. Adding those comparisons (I am assuming in most cases proposed FR/BURF would be better) would also be helpful for benchmarking results. An interesting addition in the experiment would be to consider these improved models [5] [6] as initialization for SFDA and comparing relevant methods.\n- I understand comparison with [6] would not have been possible as its a very recent work. However, some of the performances in [6] matches the SFDA method performance, which leads to a more general question about the problem, i.e., Is corrupted data required for model to generalize to those conditions? Should we consider common corruptions as a DA problem at all or is it better modeled as a generalization issue and the focus should be to improve source/clean data training?\n\n\n\n\n\n\n[1] https://github.com/brendenlake/omniglot\n\n[2] Li, Rui, et al. \"Model adaptation: Unsupervised domain adaptation without source data.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n[3] Morerio, Pietro, et al. \"Generative pseudo-label refinement for unsupervised domain adaptation.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2020.\n\n[4] Kurmi, Vinod K., Venkatesh K. Subramanian, and Vinay P. Namboodiri. \"Domain Impression: A Source Data Free Domain Adaptation Method.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021.\n\n[5] Hendrycks, Dan, et al. \"AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty.\" International Conference on Learning Representations. 2019.\n\n[6] Wang, Haotao, et al. \"AugMax: Adversarial Composition of Random Augmentations for Robust Training.\" arXiv e-prints (2021): arXiv-2110.\n\n\n", "summary_of_the_paper": "The paper tackles a variation of domain adaptation problem where the model is pre-trained on the source data and then deployed in the target domain for adaptation. The major challenge being the absence of source domain data during the adaptation process and hence generally used unsupervised domain adaptation methods are not useful in the case. Furthermore, the paper tackles a sub-set within the ''Source-Free Domain Adaptation'' setting, termed in the paper as *measurement shift*. The proposed method addresses this by aligning the target domain activation statistics with stored statistics of activations measured during development stage (i.e. source domain training). The experimental analysis provided in the paper and supplementary material shows that proposed method is able to outperform all existing methods for the *measurement shift* case.", "summary_of_the_review": "Overall, I think the paper has some interesting ideas and has a compelling experimental analysis. However, there still remain few concerns/confusions which require further explanation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CYOxMj93Zr", "reviewer": "Reviewer_Fzmz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This paper addresses the interesting novel problem of measurement shifts, a subset of general domain shifts that occur due to changes in the measurement systems.\n- To approximate the source feature distributions, this work uses a softly-binned histogram and provides a novel differentiable implementation that is used in a loss function to adapt the model to the target domain.\n- The improvements of BUFR are demonstrated against prior SFDA works on the proposed EMNIST-DA dataset, CIFAR-10-C/100-C and Camelyon17 datasets. Extensive ablation studies are performed on EMNIST-DA to study the proposed components.\n\nWeaknesses:\n- This work attempts to illustrate the importance of measurement shifts (a subset of domain shifts) and to propose a method specific to tackling measurement shifts. Most of the empirical justifications rely on evaluations on the proposed EMNIST-DA dataset. However, there are several issues with the arguments, which are detailed as follows.\n\t- Considering Table 1, the authors do not mention which type of shifts are actually included in the severe category, except that the severe shifts cause a large drop in performance for the AdaBN prior art. Some of the shifts indicated in Fig. 7 like zig-zag, stripe and dotted line do not seem like natural measurement shifts as they occur only locally and may change the class label as well (like in the zig-zag example of Fig. 7, N looks like P). Conversely, the Camelyon17 dataset, with real-world measurement shifts, shows that measurement shifts tend to have more global effects on the images (in Fig. 9). The authors should clearly mention the criteria used for selecting shifts to represent measurement shifts. This is also important because the EMNIST-DA dataset is one of the contributions of this work. Hence, it is not clear whether these severe shifts are a good representation of measurement shifts and thus, whether evaluation on severe shifts is a good indicator of a method successfully tackling measurement shifts.\n\t- For the Camelyon17 dataset, representing real-world measurement shifts, the performance of all prior arts is competitive w.r.t. the proposed FR and BUFR methods, even when a very small number of target images are available for training. For example, when only 5 target images per class are available, all prior arts (AdaBN, PL, SHOT-IM) get ~82.6% accuracy while FR and BUFR get ~84.6% accuracy. This implies that the advantage of having a method specifically catered to measurement shifts is small. As the number of available target samples increases, the performance gap reduces and prior arts perform the same as the proposed methods. Further, in most practical scenarios, a large number of unlabeled target samples are usually available or easily obtainable.\n\t- For the CIFAR10-C/100-C benchmarks as well, the expected calibration errors (ECE) of prior arts (AdaBN and TENT) are on par with FR and BUFR (Table 13). Significant improvements to ECE are shown only for the synthetic EMNIST-DA dataset while the ECE metrics for Camelyon17 dataset are not reported. Thus, we cannot conclude whether the improvements w.r.t. usual DA methods are significant.\n\t- The previous three points illustrate that the significant improvements of this approach are on the synthetic EMNIST-DA dataset and not on the real-world measurement shift based Camelyon17 dataset or the standard CIFAR10-C/100-C benchmarks. Thus, it is not clear whether this method will give significant improvements over simpler and more generic DA techniques like PL (pseudo-labeling) or AdaBN on realistic measurement shift datasets.\n\t- Further, the ablations are performed only for the EMNIST-DA dataset. Specifically, the ablation studying the bottom-up hypothesis i.e. effect of number of blocks unfrozen for training (in Fig. 4b) is shown for the EMNIST-DA dataset only. In Table 6, it is observed that the FR performance is sometimes better than or very close to BUFR. While the bottom-up hypothesis is intuitive, these observations from Table 6 cast a doubt on it. Thus, ablations on the Camelyon-17 dataset will help clear these doubts and make the argument stronger as they represent realistic measurement shifts.\n\nMinor issues:\n- Sec. 2 (Feature restoration): $f_s = h(g_s(X_s))$. Here, $f_s$ is defined as the output of the network rather than the network itself. A possible correction could be $f_s = h \\circ g_s$.\n- A table of contents should be added for the Appendices, with hyperlinks to the various tables, figures and subsections. This will improve the readability of the paper, given its length.\n", "summary_of_the_paper": "This paper addresses the problem of source-free domain adaptation (SFDA) under measurement shifts. The measurement shifts are a subset of general domain shifts which arise from a change in measurement system. The proposed method aims to resolve this problem by restoring the target features to the source feature distribution. Towards this, the source distribution is approximated using a lightweight and flexible approximation, namely softly-binned histograms. For the target adaptation, the feature extractor is trained to re-align the target feature distribution (of a batch) with the saved source distribution. This method is termed as Feature Restoration (FR) and is performed for the feature activations and the pre-softmax activations. An extension of this method, called Bottom-Up FR (BUFR) is introduced which performs the feature restoration in a block-wise manner starting from the early layers of the network. The improvements of BUFR and FR are demonstrated on a new proposed EMNIST-DA dataset for simulating measurement shifts as well as on standard CIFAR10-C, CIFAR100-C (common corruptions) benchmarks and the Camelyon17 dataset that contains realistic measurement shifts. They perform several ablation studies on the EMNIST-DA dataset to highlight the effectiveness of the proposed components.", "summary_of_the_review": "This work focuses on specifically tackling measurement shifts (subset of general domain shifts). Most of the justifications to highlight the importance of measurement shifts are based on evaluations on the proposed synthetic EMNIST-DA dataset. While the improvements are significant on EMNIST-DA, the simpler and more generic DA techniques obtain competitive performance on the real-world measurement shift dataset (Camelyon17) which casts doubt on the requirement of specific methods to tackle measurement shifts. See the main review for detailed comments. While the paper addresses an interesting problem with good results, I am not convinced by the paper’s argument that such a specific method is better suited for realistic measurement shift DA problems than simpler, generic methods like pseudo-labelling. Thus, I give the rating of “5: marginally below the acceptance threshold”.\n\n**Pose-rebuttal comments**\n\nI thank the authors for their efforts in the rebuttal period. Most of our concerns have been addressed by the authors. Further, having read the other reviews and considering that this work is the first that addresses measurement shift DA, I increase my score to 6: marginally above the acceptance threshold. I hope that the authors will include all the discussions from the rebuttal in their revised draft.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9qESAI91VhI", "reviewer": "Reviewer_HPoa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The paper is very well-written and easy to follow. It targets the SFDA setting but with a different type of domain shift to the common setting. The measurement shift is indeed an important piece of domain shift on many applications. \n2. The proposed method is simple and effective. At the first glance, it might be doubtful that they use source features for training. The authors give a memory storage comparison for the estimated source distributions, proving it is very lightweight. \n3. Considering the simplicity of this method, it achieves promising results on several types of datasets in terms of accuracy and calibration compared to the SOTA. The ablation and analysis are sufficient to support the claim of this paper. \n\nWeakness:\n1. The main comparison of this paper is SHOT, which is an entropy-based method and thus has calibration problems and strict assumptions. I think it is also preferable to see how this method compares to other feature restoration type SFDA methods such as [1] and [2].\n2. 2nd point is not weakness. But it will be also helpful to include performance comparison on Office-31 or home as a baseline. \n\n[1]Li, Rui, et al. \"Model adaptation: Unsupervised domain adaptation without source data.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n[2] Liu, Yuang, Wei Zhang, and Jun Wang. \"Source-free domain adaptation for semantic segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "summary_of_the_paper": "This paper tackles the specific domain shift, named measurement shift on source-free adaptation setting. They analyze the drawbacks of existing entropy-based SFDA methods and instead propose a method to retore the source feature with a lightweight approximation. The paper emphasizes the difference of measurement shift with common concept shift, and their method can achieve significant performance on measurement shift setting compared to SOTA. ", "summary_of_the_review": "This paper deals with the measurement shift in the source-free DA problem. The authors propose a simple but effective method to approximate source distribution in a lightweight manner. I recommend to accept this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7M4YkzOdH0d", "reviewer": "Reviewer_gJxM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the papers: \na) The paper is very well written, well-motivated, well compared with existing literature and it was a pleasure to read.\nb) proposed method makes sense\nc) extensive results show that the method obtains good results on multiple datasets, outperforming existing source-free DA methods. \nd) the efficiency of proposed bottom-up feature restoration is ablated (and shows large gains for some experiments).\n\nThe weaknesses:\na) The proposed method is very simple (both the feature alignment and the bottom up feature restoration). However, I do not consider this a large weakness: results show that these first steps already provide large improvements. \nb) the results on the real application (Table 6) are maybe less convincing (especially the impact of bottom-up feature restoration is small) and it would be great if another real application would be found. The authors argue this scenario is not optimal for their method. Maybe they could extend a bit more on potential applciations. \n\nMinor: \nthe sentence 'Unlike most existing SFDA methods, we make no modifications ' would be nice to see references here to what SFDA methods the authors refer.\n\nI was wondering if the method could work for extending the number of channels of the input data (For exeample source only red-green channels and target red-green-blue). \n", "summary_of_the_paper": "Paper proposes a new scenario for source-free domain adaptation where the domains undergo a measurement shift (characterized by a change in the measurement system). Other than the typical source-free DA scenario, in this case, you would not need new features but you would like to update the lower layers of the network. The paper proposes a method that is based on two ideas i) aligning the marginal feature distribution (with KL loss) and ii) performing updating in a bottom-up way (allowing lower layers to first adapt to the target domain). Results show that the method outperforms existing methods on this setting. ", "summary_of_the_review": "The authors propose a relevant new scenario for SFDA and propose a new method to address this setting. Results show that the existing methods struggle for this setting. The proposed method outperforms existing methods. I think the proposed setting is relevant and I would therefore recommend acceptance. The novelty of the proposed solution is not that large though. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6woxqTpcRPY", "reviewer": "Reviewer_V9Lg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n+ The detailed framework is well explained to be easily understood, and the figures were very helpful for understanding.\n+ The state-of-the-art performance was validated well by using various types of datasets and the compared algorithms were recent and reasonable.\n+ The empirical analysis was helpful to understand the overall workflow and solve the questions about the framework.\n\nWeaknesses\n- The analysis for the intuition and assumption of the proposed algorithm is not presented. Even though the empirical analysis for the algorithm is well described, the presence of measurement shift and its importance cannot be understood just through this paper. I agree that the measurement shift can be occurred by the measurement system, but it is hard to know how the shift looks like and which part of the model is affected by the measurement shift. With the description of the weak intuition, the design of the framework was hard to be followed. Furthermore, the bottom-up feature restoration assumes that the low-level features are affected by the measurement shift, but there is no related analysis showing the effect of measurement shift for the respective layers at all. Thus, I want to see the analysis of the measurement shift to explain the reason for the proposed architecture.\n- Some sentences are repeated in the abstract and the introduction. With the limited pages of paper, repeated sentences should be avoided to explain the proposed framework as much as possible with the various aspects. I recommend the rewriting of the abstract to improve the readability of the paper. The current abstract contains too much detailed information, which can let the readers confused before understanding the intuition of this paper.", "summary_of_the_paper": "This paper focuses on the improvement of the source-free domain adaptation problem. While the previous studies destroy the model calibration to improve the domain generalization or update the source model to fit the feature distribution from the target domain, the proposed algorithm reduces the domain gap by restoring the distribution of source features at the adaptation phase. This feature restoration step is based on the assumption of the measurement shift that comes from the difference between the data-grabbing system and environments. Based on the assumption, the authors proposed the method of feature restoration (FR) and bottom-up feature restoration (BUFR) that is its extended version. The state-of-the-art performance of the proposed algorithm was validated by numerous datasets including the simple digit datasets and the real-world complicate datasets.", "summary_of_the_review": "Even though the intuition of the algorithm is not given yet, the designed framework showed the state-of-the-art performance, which means that the measurement shift would be one of the hidden keys to improve the domain generalization. When the authors present adequately the additional analysis for its intuition and revise some repeated parts, I am preparing to increase my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper aims to solve the source-free domain adaptation, specifically on measurement shift. The proposed method lowers the domain gap via restoring the source feature distribution with a lightweight approximation. The effectiveness and performance are well validated by extensive experiments on various datasets compared with other recent methods, and the ablation analysis supports the claim of the paper well. The paper is well written with clear logic to follow.
Decision: Accept (Spotlight)