Key Design Choices for Double-transfer in Source-free Unsupervised Domain Adaptation
Authors: Andrea Maracani
Raffaello Camoriano
Elisa Maiettini
Davide Talon
Lorenzo Rosasco
Lorenzo Natale
Venue: Submitted to ICLR 2023 (2023)
Keywords: Transfer Learning
Unsupervised Domain Adaptation

Abstract: Fine-tuning and Domain Adaptation emerged as effective strategies for efficiently transferring deep learning models to new target tasks. However, target domain labels are not accessible in many real-world scenarios. This led to the development of Unsupervised Domain Adaptation (UDA) methods, which only employ unlabeled target samples. Furthermore, efficiency and privacy requirements may also prevent the use of source domain data during the adaptation stage. This particularly challenging setting, known as Source-free Unsupervised Domain Adaptation (SF-UDA), is still understudied. In this paper, we systematically analyze the impact of the main design choices in SF-UDA through a large-scale empirical study on 500 models and 74 domain pairs. We identify the normalization approach, pre-training strategy, and backbone architecture as the most critical factors. Based on our observations, we propose recipes to best tackle SF-UDA scenarios. Moreover, we show that SF-UDA performs competitively also beyond standard benchmarks and backbone architectures, performing on par with UDA at a fraction of the data and computational cost. Experimental data and code will be released upon acceptance.

Under review as a conference paper at ICLR 2023
KEY DESIGN CHOICES FOR DOUBLE-TRANSFER IN SOURCE-FREE UNSUPERVISED DOMAIN ADAPTATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Fine-tuning and Domain Adaptation emerged effective strategies for efficiently transferring deep learning models to new target tasks. However, target domain la- bels are not accessible in many real-world scenarios. This led the development of Unsupervised Domain Adaptation (UDA) methods, which only employ unla- beled target samples. Furthermore, efficiency and privacy requirements may also prevent the use of source domain data during the adaptation stage. This particu- larly challenging setting, known as Source-free Unsupervised Domain Adaptation (SF-UDA), is still understudied. In this paper, we systematically analyze the im- pact of the main design choices in SF-UDA through a large-scale empirical study 500 models and 74 domain pairs. We identify the normalization approach, pre- training strategy, and backbone architecture as the most critical factors. Based on observations, we propose recipes to best tackle SF-UDA scenarios. Moreover, we show that SF-UDA performs competitively also beyond standard benchmarks and backbone architectures, performing on par with UDA at a fraction of the data and computational cost. Experimental data and code will be released.
INTRODUCTION
The recent success of deep neural networks (DNNs) in many tasks and domains often relies on the availability of large annotated datasets. This can be tackled by pre-training DNNs on a large dataset and then fine-tuning their weights with target task data (Huh et al., 2016 Yosinski et al.) 2014] Chu et al., 2016). Furthermore, fine-tuning is usually simpler to perform and faster than training the model from scratch, the dataset size can be smaller, and the final performance is typically higher (with some exceptions: see (Kornblith et al. (019)). This approach is very convenient: the model requires a single expensive pre-training and can later be re-used for multiple down-ste lown-stream tasks. This is good example of transfer learning (Zhuang et al. (02T), which leverages the information acquired from a task to improve accuracy on another task of interest.
Two relevant examples of transfer learning are Domain Adaptation (DA), that, given different-yet related-tasks, exploits source domain(s) data to improve performance on different known target do- main(s), and Domain Generalization (DG), that aims generalize unknown target(s). As opposed to fine-tuning, which the pre-training and downstream tasks can be significantly different, DA and DGen require stronger assumptions on the similarity between tasks, c.g., leveraging synthetic im- ages to improve the classification of real images that share the same label space. DA is also related to Multi-task Learning (MTL) (Caruana 1997] Ciliberto al. 2017) and Multi-domain Learning (MDL) (Joshi et al., 2012). In fact, domains can be seen as tasks in MTL or MDL. However, an explicit domain label provided and annotated examples are available for each task. particularly challenging and useful setting in practice is Unsupervised Domain Adaptation (UDA) (Tzeng et al.)
2017 Ganin et al.] 2016), which labeled samples from source domain are used together with unlabeled samples from the target domain improve performance the latter.
This work focuses on Source-Free Unsupervised Domain Adaptation (SF-UDA) (Liang et al. 2020 for the image classification task. SF-UDA is two-steps sequential version of UDA in which the labeled data is only accessible the first training phase. Adaptation to the new domain is carried out in second stage where only the unlabeled data from the target domain is available. SF-UDA nicely matches applications where continual adaptation is required with compu- tational and memory constraints, or where privacy policies prevent access to the source data. Since

---

Under review as a conference paper at ICLR 2023
these techniques are usually applied to fine-tune and adapt models with pre-trained weights, there are two different transfers into play: (1) from the base task (used for pre- pre-training) to the source domain, and (2) from the source domain to the target domain: we refer to this combined transfer as double-transsfer The objective of this work is two-fold. Firstly, we aim to study the impact of the main design choices in SF-UDA approaches, such the backbone architecture, pre-trainnn training dataset, and the way double-transfer is performed. Secondly, we aim to investigate the strengths and failure modes of SF-UDA methods, also comparing them with standard UDA approaches.
Recent works (Liang et al. 2020 Ding et al. 2022) show that SF-UDA techniques achieve compara- ble performance with state-of the art UDA methods on common benchmarks. contrast, Kim al. (202)) reports that recent UDA methods perform well on standard benchmarks because they overfir the task. Indeed, when employed other settings (i.e., non-standard architectures or datasets) they result in worse accuracy than previous methods. We investigate with targeted experiments whether overfitting affects recent SF-UDA methods as well. We pursue such objectives through large-scale systematic experiments encompassing more than 500 different architectures on 6 separate domain adaptation datasets, totaling 23 domains and 74 domain shifts. We employ different probing and SF- UDA methods to better analyze the functioning of couble-transfer methods, providing a ready-to-use recipe for effective system design. Our main findings are as follows:
The pre-training dataset choice and the resulting accuracy on the ImageNet top-1 bench- mark directly impacts the domain generalization and SF-UDA performance, for both CNNs and Vision Transformers.
Most SF-UDA methods fine-tune the model on the source domain before adaptation. How- ever, we show that in some cases this causes severe performance degradation. Specifically, we identify the type of normalization layers as having a critical role in this context.
Besides fine-tuning, the normalization strategy heavily affects the failure rate of SF-UDA in general. We present a large-scale analysis of SF-UDA methods' failure rates, comparing architectures with Layer Normalization (LN) (Ba et al., 2016) and Batch Normalization (BN) (Toffe & Szegedy (2013) and attesting the improved robustness of the former.
SF-UDA methods, like SHOT (Liang et al., 2020), SCA (see Sec. 3.2) ). and NRC (Yang et al., 2021ap, perform well also with architectures and datasets different from the usual benchmark ones and are competitive with statc-of-the thet UDA methods.
The full list of architectures, results in esv format, the code and pre-trained weights will be released
2 BACKGROUND AND RELATED WORK
Let X' be the input space, e.g., the image space, E C RR, representation space, i.e., the feature space, and y = (1,.... C) the output space for multi-class classification. A feature extractor (back- bone) is function So X + Z with parameter 0, while classifier is function with parameter & that assigns a label to any feature vector, h&s Z - y. We introduce two data distributions over X' x 3: Ms that models the source domain and pr that models the target domain.
Domain Generalization (DG). In this work, we consider DG setting as the reference task to inves- tigate transferabiliity among domains. First introduced by Blanchard al. (201T), its goal is to find a feature extractor fe and a classifier he from N i.i.d. samples of a given domain (the source ug)) that will perform well on other unseen domains (in our case just the target domain pT), that is:
min min (f v}l 0.6
given (x,, M P
(I)
where 1 [conditionn] is the indicator function. In this setting, some assumptions on the relationship between tasks (μS and pr) are needed, but only data sampled from the source domain can be used for training, while target domain data is accessible at test time only. We remark that this is similar to the standard supervised-learning problem of generalization (Gen), with the exception that, here, the training and test distributions are different. Several methods specifically target this problem, e.g., (Volpi ct al. 2018 Arjovsky et al. 2019 Isse et al. 2020 see Wang et al. (202T) for a review.
https::///
2

---

Under review as a conference paper at ICLR 2023
Usupervised Domain Adaptation (UDA). As for DG, the final goal of UDA is to learn model that performs well on the target domain. However, differently from DG, more information is available: unlabeled samples from the target (marginal) distribution data from the source. Since the seminal theoretical works (Ben-David et al. 2006; 2010: Mansour et al. 2009), many UDA methods have been proposed in the literature for image classification, including adversarial training (Ganin et al.] (016), bidirectional matching (Na et al.) (022)) per- class kernel mean discrepancy minimization (Kang et al. 2019). and also for other tasks such as object detection (Oza et al. 2021) and semantic segmentation (Toldo et (020).
accessible together with the labeled
Source-Free Unsupervised Domain Adaptation (SF-UDA). SF-UDA similar to UDA, but is more constrained. The learning process is divided into 2 phases: the (labeled) source data is available only in the first training step. Then, adaptation to the new domain occurs in the second stage, where only the unlabeled target data is available. The problem can be formulated as:
min QuinE(eaaa (eww))rr [1 (ba (Jo(()) + yy}l
0.4
given (ze, (.), S (only at step 1) and (2z))²=1 ~ pT(x) (only at step 2).
(3)
where PT(X) is the marginal of jer over the input space X' and M is the number of available target samples. We remark that, even if it is not possible to share data among the 2 steps, model parameters O' and ø' found at step are accessible at step 2. Recently SF-UDA has gained sigmificant interest: since the work by [Liang al. (2020), many other methods have been proposed and they achieved
remarkable results on standard UDA benchmarks (Li et al.] 22200] Yang et al. 2021b] Kundu et al.
2020 |Huan al. 2 Kurmi et al. [021 Xia et al.J 2022 ChC Chu et al. [022: Ding et al.) [2022]
Kundu et al. 2022). J. In Sec. 3.2 we will illustrate some SF-UDA methods in greater detail.
Experimental Studies. Training DNNs is demanding in terms of computational resources, time and data. Hence, there is great interest in the scientific community into understanding how to obtain representations that can be conveniently transferred to new tasks and what are the key ingredients to build more efficient architectures and training methods. For this reason, several studies on trans- fer learning (i.e., fine-tuning for classification or different vision tasks) have been conducted, such as (Chu al. (0116). (Huh et al.] 2016). and (Kornblith et al. (211)). Concerning UDA, (Zhang & Davison 2020 studies model selection for classical methods (and for CNNs) based on the accuracy achieved on ImageNet, while (Kim et al. 2022) provides an analysis of different pre-training tech- niques for DG and UDA. In this work, instead, we target SF-UDA approaches. SF-UDA is highly relevant for applications, since it allows the design of efficient algorithms while,
show in our experiments, achieving comparable performance to UDA. In our analysis, we decouple the effects on the final result of the two adaptations into play in double-transfer: from pre-training to source domain and from source to target domain. With our results, we provide best practices for robust SF-UDA pipelines. Our empirical analysis includes more than 500 architectures (including both CNNs and Vision Transformers). We test them on 6 datasets for domain adaptation (for a total of 74 domain shifts). To the best of our knowledge, this is the most extensive study on SF-UDA
METHODS
3.1
PROBING
For our initial experiments we adopt two probing methods to evaluate the quality of features ex- tracted from the models: linear probing and cluster probing.
Linear Probing (LP). The pre-trained feature extractor is fixed and employed to compute features from training images. These are employed train a linear classifier (i.e., multinomial regressor). Finally, classification accuracy is evaluated on the test set. This method is commonly used in the literature to evaluate feature extractors pre-trained with self-supervision (Chen al.) 2021).
Cluster Probing (CP). As in LP, we fix the feature extractor and compute features from the training images. Then, for each class, the class prototype is computed as the average feature vector of the examples from that class. At prediction time, new samples are classified based on the class of the closest prototype (in our case, we use the cosine dissimilarity). As in LP, we evaluate generalization capabilities on the test set. CP allows inspecting the properties of the learned representation, pro- viding useful insights for SF-UDA which exploits the underlying structure of the feature space to overcome the lack of ground truth the target domain. Refer to App.AAfor more details. Remark. In next sections, we append "Gen" or "DGen" to the method name to indicate the test accuracy on the source or target domain, respectively. If nothing is indicated use LPP as default.
3

---

Under review as a conference paper at ICLR 2023
3.2 SF-UDA
Simple Class Alignment (SCA). This algorithm adapts the classifier to the target domain without altering the feature extractor. As in cluster probing, a first phase computes class prototypes of labeled data of the source domain. In the second step, the prototypes are used as initialization centroids in spherical k-means (Homik et al. 2012), which is executed on the target unlabeled data. Hence, the final centroids are adapted prototypes that account for the domain shift. The resulting classifier assigns the class of the closest centroid to new inputs (based on cosine dissimilarity). This method was presented in (Kagg et al. 2019) to compute pseudo-labels. Variants are discussed in App. B Source HypOthesis Transfer (SHOT). This algorithm (Liang et al., 2020 2021) is considered the state-of-thee arr in SF-UDA for its efficiency and is solid baseline for novel SF-ÜDA methods. The first transfer requires fine-tuning the model to the source domain. The second transfer alternates two steps iteratively: (1) pseudo-labels computation for target samples, and (2) feature extractor fine-tuning using the Information Maximization loss on previously computed pseudo-labels, while keeping the classifier fixed. Interestingly, the latter proved crucial for achieving high performance. For pseudo-labels computation, SHOT builds on a modified version of SCA. The feature extractor (fe) and the classifier (ha), trained on the source domain, are used to predict probabilities for each target sample for all the C' classes p; (pa..... ..c)) Then, the initialization prototype for class c, c = I...., C' is computed as a weighted average of the target features:
kc Exvesige (a)))(
(4)
Ex.es Pie
The prototypes ,1, .2..... I-NN classifier computes the pseudo-labels of the target dataset. Remark. We select SCA and SHOT for our experiments since they are recent representative SF- UDA approaches that tackle the second transfer in two complementary ways. Indeed, while SHOT keeps the classifier fixed and adapts the feature extractor, SCA does exactly the opposite. Further experiments with the NRC method (Yang et al. 2021a) corroborate our results.
.........k kc are used to initialize spherical k-means (as in SCA), and the final
4 EXPERIMENTS
4.1 SETUP
Models. We rely on the Pylorch Image Models Python library (timm) (Wightman) 2019) for the implementation of the various models considered in our experiments. It provides access to a re- markably large number of different architectures and pre-trained weights. In our evaluation of the probing approaches and SCA (without fine-tuning) we used 500 models taken from more than 25 different families of architectures (e.g.. VGG (Simonyan
Zisserman (015), ResNet (He et al., 2016), EfficientNet (Tan & Le| 2019). ConvNext (Liu et al.) (022). ViT (Dosovitskiy et al.] 2020). SWIN (Liu et al., 2021). Deit (Touvron et al. (021) and XCIT (Ali et al.) (021)). Instead, for the experiments with SHOT and with networks fine-tuning, we sampled subset of 59 models, taken from more than 12 familics of architectures. Notably, our analysis comprises both modern Vision Transfomers and more traditional CNNs. More details are available in App. C
Pre-training. For the first transfer, we consider two datasets: ImageNet (ILSVRC-2012), com- posed of 1.2M images for 1000 mutually exclusive classes, and the superset ImageNet21k (Deng) et al. 2009), composed of 14M images for 21, 1.841 not mutually exclusive classes. Specifically, we either consider models pre-trained on ImageNet (/N) on ImageNet2 and then fine-tuned on the 1000 classes of ImageNet (IN2IK).
Domain Adaptation Image Datasets. DA datasets for image classification typically have two or more sub-datasets corresponding to different domains sharing the same classes. In our experiments, we considered: DomainNer (Peng et al., 2019). a large dataset with domains of common objects, divided into 345 categories (596 006 images): ImageClef-DA (Long et al., 2017). small dataset with 4 domains and 12 classes (2 400 images); Modern Office-31 (Ringwald & Stiefelhagen 202T). a novel version of the Office-31 dataset (Saenko et al. 2010)) with an additional synthetic domain (for a total of 4) and 31 classes (7 210 images); Visda-2017 20ee200017Peeeeeeee et al. 2017). )) a large sim-to-ree co-real dataset with 12 categories and 2 domains (280 000 images): Office-Home (Venkateswara et al.] (017), 7), with domains and 65 categories (15 588 images); Adaptiope (Ringwald & Stiefelhagen 2021). ), having
3 domains (synthetic, product and life) and 123 different classes (36 900 images). We end up with
4

---

Under review as a conference paper at ICLR 2023
ImageNet
ImageNe/21k
89
74
Accuracy
59
ImageNet top1 accuracy
We compare performance of different feature extractors (which have been pre-trained either on ImageNet or Imagene(2Ik) for, respectively: (i) Generalization LP, (ii) Domain generalization LP, (iii) SCA and (iv) FT + SHOT. In the first sub-plot (LP Gen) the test accuracy on the source is considered, while on other subplots (LP DGen, SCA and FT+SHOT) the test is performed on the target domain.
Figure 1:
Table 1: Adjusted R' values for linear (only top1 accuracy considered) and multi-linear (topl and pre-training) explanatory models for linear probing (LP), cluster probing (CP), SHOT, SCA and fine tuning (FT).
CP
LP
SCA
SHOT
FT
Gen
Gen
DGen
DGen
Gen
DGen
SCA
SHOT
Linear
0.698
Multi-Linear
0.851
0.908
0.935
0,918
0.902
0,890
0.878
0.822
0.792
0.932
23 different domains and 74 domain pairs in tota Experimental Details. We use the aforementioned datasets conduct a systematic study on the main design choices in double-transfer. Specifically, to evaluate the generalization capability (Gen) of the various feature extractors (first transfer), we randomly split the images from each domain into training (80%) and test set (20%). We use the former to either train the classifier (LP or CP) or to fine-tune the model and we evaluate the accuracy on the test set. The final performance of one feature extractor is given by the average of all the accuracy values obtained on the 23 domains. Similarly, for the second transfer (DGen and SF-UDA settings), we consider each domain pair and the final performance is obtained by averaging consider different combinations for the two transfers. For the first one, we either train a classifier on the source domain, keeping the feature extractor fixed (as in LP and CP), while training the classifier. Instead, for the second phase we mainly consider SHOT and SCA. This leads to 4 different combinations: SCA, FT+SCA, SHOT and FT+SHOT. Finally, in SF-UDA, is common in the literature and in UDA benchmarks, we consider the transductive setting (Kouw & Loog 2019) where the accuracy evaluated on the same images used for adaptation (albeit without labels). For technical details on the experiments refer to App. D
all the 74 pairs. Note that for SF-UDA
fine-tune (FT) it
4.2 RELEVANCE OF PRE-TRAINING AND IMAGENET ACCURACY IN SF-UDA
How to choose the best feature extractor for SF-UDA?
In Fig. [ we compare the LP performance of different feature extractors for Generalization and Do- main Generalization. Furthermore, we present results for SCA and FT+SHOT. Both ImageNet and ImageNet21k pre-trainings are considered. We plot the relationship between the accuracy obtained on ImageNet and the one achieved on the task of interest. As it can be noticed, the latter is mainly influenced by two factors: (i) the ImageNet top-1 accuracy, and (ii) the pre-training dataset. Specif- ically, in all cases the performance the task of interest depends linearly the ImageNet top-1 accuracy of the backbone. For instance, the accuracy of backbone can be modeled as:
accuracy(B)) m -m tB) +
(5)
II Visda-2017, differentt from the common benchmark, consider both experiments (synthetic - real and real + synthetic).
5

---

Under review as a conference paper at ICLR 2023
Table 2: ImageNet ImageNet21k pre-training. The accuracy reported are the average of 74 domain shifts.
Backbone
VGG19
ResNet50
W-ResNet50
DenseNeti61
ConvNext B
params
143.7
28.7
LP DGen
IN2IK
52.5
53.3
SCA
IN2IK
56.0
IN2IK
55.7
62.0 64.0
65.3
72.7
where m and q are the parameters of the linear model (specific to each experiment), while e is a random variable that accounts for the variance in the data, not explained by the model.
Instead, regarding the dataset choice for pre-training, it is known from the literature (Dosovitskiy boost the accuracy Equation this should also improve the accuracy for all the downstream tasks. This is confirmed by our experiments. Notably, by taking two models with the same ImageNet top-1 accuracy but pre- trained on different datasets, it is evident from Fig. that the model pre-trained on ImageNet21 Ik has better performance (on average). This implies that the ImageNet2 pre-training yields an additional improvement in the considered transfer tasks that cannot be explained by the increased ImageNet accuracy of the model. To account for this, we introduce the pre-training into the linear statistical model and describe the data through multi-linear model with interaction. Let pretrain be equal to for ImageNet21k backbones and to 0 for ImageNet backbones, then the model becomes:
al. 2020 that using ImageNer21k
ImageNet. Thus, according to
accuracy( (B) = [m + Am pretrain(B)] tBp)( + + Âq pretrain(B) t.
(6)
where Am and Ag are the newly introduced parameters in the model.
In Tab. [ we compare the goodness-off of-fit (adjusted R²) of the multi-linear model with the linear one. The difference is significant for all experiments with probing and SF-UDA methods. App. reports the coefficients of the statistical models and the scatter plots of further experiments. In addition, in App. F we compare the performance of the same architecture (ResNet50) pre- rreitrained on ImageNet2 lk in ways: (i) with the standard training process (used for all experiments in this work), and (ii) with the technique proposed by Ridnik et al. (2021) which employs a specific semantic pre- training on filtered version of ImageNet21k to reduce the strong class imbalance and tackle the multi-label nature of the dataset. We observe that semantic pre-training significantly improves domain generalization, while when applying SF-UDA methods the gap recovered and the model performs equally well with both pre-trainings. In App. we compare supervised and self-supervised pre-training strategies. Despite the promising results, the former still outperform the second.
Does the size of the model count? One may expect that only large models (in terms of number of parameters) could benefit from such a large pre- pre-training dataset as ImageNet2lk, However, Tab show that this also applies to smaller models.
4.3 FINE-TUNING ON THE SOURCE DOMAIN
In this section, we study the impact on downstream tasks of fine-tuning the model on the source data (first transfer). Results for 59 models are reported in Fig. averaged over 23 domains in case of experiments with Gen tasks and on the domain pairs all other cases. The reported plots present accuracy differences between each method and its baseline (as green or red arrows). For instance, the lop-left plot reports the difference in source accuracy between no fine-tuning (i.e., LP) and fine-tuning the source domain.
Firstly, can be noticed that in cases where no fine-tuning is applied (namely SHOT and SCA plots), remarkable accuracy gain is still obtained with respect to LP DGen. Interestingly, Tab. B highlights how SHOT (which adapts the feature extractor on the target domain) obtains an accuracy gain of 6.97%(+2.25 on average the target domain. In comparison, SCA gains average 4.21%((1..5) while being 50-60 times faster than SHOT (see App.
Then, consider the where fine-tuning is applied on the source domain. Contrary to the Gen task case where fine-tuning the feature extractor on one domain increases its However, this does not always improve performance the second transfer, can be noticed in the second column of

---

Under review as a conference paper at ICLR 2023
Batch Norm.
Layer Norm.
X ImageNet
ImageNe121k
Source accuracy: FT
Target accuracy: FT
Target accuracy: SHOT
Target accuracy: ST+S
Accuracy
73
hultml in Aoalin
Targei accurncy: SCA
Targel accuracy: FT+SCA
73
doogt
IN rank
Figure Architectures sorted on the x-axis according IN topl accuracy (best on the right). We dis- tinguish models based on the employed pre-training and normalization layers. The markers' y-value represents the LP accuracy of the model, while the green/red arrows represent the accuracy improvenenttdderaaatioo when a method is applied. FT and LP are always performed on the source with labels, while adaptation meth- ods are always performed on the target without labels. Top-left: source accuracy improves with fine-tuning (average over 23 domains). Top-right: target accuracy slightly increases for layer-norm models but it is de- graded for batch-norm models. Bottom: SCA and SHOT improve after fine- linettuning In both cases we average results over 74 domain pairs (we report subset of the models for visualization purposes).
Specifically, while for SHOT fine-tuning during the first transfer always yields subsequent performance gain, models with BN layers do not benefit from this adaptation on the source for SCA and especially for the FT-only target accuracy. In some cases, fine-tuning on the source even degrades performance (see ffr further results).
Fig. 2
We thoroughly analyze this matter in Fig. 3 We report results for domain pairs from ModernOf- fice31 (namely, Synthetic and DSLR), for which this phenomenon is most evident. However, it occurs also large number of other domain pairs. In the first row, we experiment in the Syn- thetic - DSLR case. As evidenced by the first-column plots, BN layers in this case lead to a very significant target accuracy degradation after fine-tuning compared to naive LP. Nonetheless, in the second column we show that using ADABN (Li et al. (217)) to adapt the BN layers the target domain compensates for the initial loss. Concerning SF-UDA, note that while SCA is strongly af- fected by the performance degradation brought by BN layers adaptation (third column), SHOT is able to nicely recover (fourth column), since it employs BN statistics computed on the target domain during the iterative feature extractor adaptation. Finally, in the second row report results for the DSLR - Synthetic transfer. Notably, in this case ADABN BN does not help recover performance, while both SCA and SHOT allow to compensate for the initial loss. Note that there may exist methods to improve the target accuracy after fine-tuning of models with BN (c.g., fixing BN statistics, using augmentations, etc.), but the study of these techniques is out of the scope of our analysis.
Remark 1.
One could ascribe the performance drop to architectural properties other than the BN layers, but with the previous examples (i.e., column in Fig. we show that the main cause actually lies in the BN statistics. Moreover, the issue never occurs for models with LN layers.
Remark The performance drop when fine-tuning models with BN layers is especially evident for pairs with shifts between real and synthetic domains, but it is not limited to such cases (see, e.g.,
Clipart > Art (OfficeHome) and Webcam + Amazon (ModernOffice31) In App. G we show a more extensive comparison between models with BN and LN layers.
Remark 3. Tab. 3 reports the average difference in accuracy with respect to LP DGen, grouping the models used for this experiment by pre-training dataset and normalization layers. We compare performance for FT DGen, SCA, FT + SCA, SHOT, and FT + SHOT.

---

Under review as a conference paper at ICLR 2023
Batch Norm.
Layer Norm.
X ImageNet
ImageNet21k
DGen FT DGen
ADABN
FT SCA
SNOT
.000
coca
Og
62
Accuracy
a7
0-8
12
IN rank
Figure 3: Rows represent different domain pairs (Sythetic- + DSLR and DSLR + Synthetic from Modemn Office31). In column 1, arrows indicate the effect of FT on the target accuracy with respect to the LP DGen baseline (the markers). In the other columns, the markers represent the target domain accuracy after fine-tuning and arrows represent the performance difference using ADABN, SCA SHOT (after the fine-tuning).
Table 3: Average accuracy change (%) with respect to DGen with LP. For each architecture, the accuracy change the average over 74 different domain-shifts. Each row, shows the performance obtained by models grouped by different combinations of their pre-training and normalization layers. The number of considered models reported in parenthesis. each column, we report results for different downstream tasks.
Models
FT DGen
SCA
SHOT
FT+SCA
FT+SHOT
All (59)
0.46 3:71
4,21 + 1.39
6.97 = 2.24
4,90 +: 3,36
9.66 1.8
IN (32)
-0.39 + 3,23
3,97 + 1.40
6.57 1,95
4,20 4 3.62
9.68 1,76
IN2IK (27)
4.0
4.49 1.35
7.45 2.51
5.72 4:00
9.63 1,93
(32)
-2.11 +: 2.75
4.08 + 1.55
6.21 2.39
2.16 4: 2.94
9.68 1.96
LN (27)
3.62 +: 1.57
4,36 + 1.19
7.88 1.68
8.14 + 1.06
9.63 + 1.69
IN+BN
-1.61
3.75
6.29 2.10
2.74 2:91
9.67 1,90
IN+LN (8)
3.28
4.64 0.66
7.42 = 1.12
1: 0.98
9.71 :: 1,40
INZIK+EN (8)
-4.02 + 2.30
5.07 + 1:27
5.97 = 3.29
0.44 +: 2:45
9.72 # 2.28
IN2IK+LN (19)
3,77 4.1.1
4,25 +: 1.35
8.07 1.86
7.95 4: 1,56
9.60 1.84
4.4 ROBUSTNESS AND FAILURES ANALYSIS
In SF-UDA, target annotations are unavailable, precluding performance evaluation after adaptation in real scenarios. Therefore, it would be beneficial in practice to have an estimate of how likely SF- UDA methods are to degrade performance on the target domain and how severe such degradation could be. To this aim, extensively evaluate the robustness of SF-UDA Given model and DA task, we fix LP DGen as baseline and consider an SF-UDA approach to fail if its target accuracy after adaptation is lower than the baseline's. Tab. 4 groups 59 models by pre-training dataset and normalization layers and reports the average failure rates of each group on the 74 considered domain pairs. BN models always show remarkably higher failure rate: FT degrades the target accuracy 51.5% of the times for BN models, while just 28. 18.1% 1% for LN ones. Applying SCA or SHOT after fine-tuning can reduce failures, but LN models are still more robust and fail less frequently. Pre- training also impacts the failure rate: LN models pre-trained on ImageNet2 lk have lower rate than those pre- traitre on ImageNet. However, BN models deteriorate more often when traited on ImageNet21k. Indeed, ImageNet21k models, as shown in Sec. 4.2 achieve a better I.P DGen, and therefore a stronger baseline value. For this reason, the instability of BN layers might result in more apparent degradation. Finally, note that SCA is very sensitive to the normalization layers choice and that FT+SHOT the method with the lowest failure rate overall.
Performance Degradation in Case of Failure. As before, degradation magnitude mostly depends on normalization: in case of failure, performance on target domain after FT (on source) decreases by 10.5% for BN and by just 2.10% for LN. SCA is unable to recover the FT degradation: in fact, the accuracy of FT+SCA decreases by 12.7% for BN and only 1.4% for LN models. The full table with average performance degradation is reported in App.
8

---

Under review as a conference paper at ICLR 2023
Table 4: Average Failure Rate (%) on 74 domain shifts. Each row, shows the performance obtained by models grouped by different combinations of their pre- gre-trannin and normalization layers. The number of considered models is reported in parenthesis. In each column, we report results for different downstream tasks.
Models
FT DGen
SCA
SHOT
FT+SCA
FT+SHOT
All (59)
40.82 = 16.14
21.05 + 11:92
11.18 + 8.75
19.08 # 14.57
4.47 + 3,11
IN (32)
45,86 13.19
25.84
12:43
13.13 8.97
23.35 13,89
5,15 3,62
IN2IK (27)
34.83 17,48
15.37 843
8.86 8.03
14.01 13.93
3,65 3.49
BN (32)
51,52 = 11.42
27.24 + 11.64
14.82 9.33
29.77 = 11.05
5,83 +: 3.77
LN (27)
28.13 10.85
13.71 + 7:25
6.86 =: 5.61
6.41 +: 4.52
2,85 2.69
BN+BN (24)
50.51
28.89 12:78
14.13
29:00 11.19
5.52 3:74
IN+LN (8)
31.93 7.61
16.72 +: 4.63
10.14
2.58
4.05 +: 3,33
IN21K+BN (8)
$4.56
12.11
22.30 + 5.11
16.89 +: 10.27
32.09 #: 11.00
6.76 3.96
IN2IK=LN (19)
26.53 11.77
12.45 + 7,36
5.48
3.39
6.40 5.19
2.35 2.33
Table S: Comparison, using modern architectures (SWIN and ConvNext) SF-UDA (SCA, SHOT and NRC)
with not Source-Free methods: DANN (Ganin et al. 2016). CDAN (Long et al. 2016). AFN (Xu et al. 2019).
MDD Zhang et al. 2019) and MCC (Jin et al. 2020
Backhone
Datasel
DANN
CDAN
AFN
MDD
MCC²
FT+SCA
FT+SHOT
FT+NRC
Officc-Home
86.6
88.4
85.7
86.5
88.3
86.4
89.3
89.5
LWIN
DomainNet
49.4
50.4
46.4
41.5
47.1
49.0
51.2
48.4
Office Ofoe
86.7
89.5
85.5
86.4
88.9
87.4
88.8
89.2
ConNNext XI.
Net
48.8
51,2
46.7
42.8
45.6
49.8
51.0
47.1
Roquires source, reproduced by Kim ct 2022
Source-Free. reproduced
Trained and evaluated on random 15% subset of DomainNet (official distributed implementation not available).
4.5 COMPARISON WITH STANDARD UDA
Recent results indicate overfitting to the benehmark for UDA approaches (Kim et 2022): with modern backbones, classic algorithms outperform newer ones on uncommon datasets. In constrast, SF-UDA techniques demonstrate a performance gain with respect to the LP DGen baseline.
In Tab. 5 we report SCA, SHOT, and NRC in a setting different from the standard evaluation pro- tocol and compare them to both established and novel UDA solutions. We consider two modern architectures: SWIN (Liu et al.] 2021 and ConvNext (Liu et al.) (022).. Three main observations arise: (i) despite the stricter constraints on the learning setting, SF-UDA algorithms are competi- tive with standard UDA; (ii) SCA performs par (or even outperforms) common UDA algorithms while being more efficient (see Tab. 12 in App. and, (iii) although FT+NRC has been trained on subset of DomainNet, this proved sufficient for it to be competitive with UDA counterparts.
5 CONCLUSIONS
The lack of supervision in UDA hinders model selection when designing and deploying learning systems. In this challenging setting, large-scale experiments can provide empirical insights into best practices to tackle real-world UDA problems. In this work, we shed light on the role of the key design choices underlying SF-UDA algorithms: pre-training dataset, architecture, fine-tuning, and normalization strategy. To this end, we present extensive experiments evaluating the impact of such choices. Our results consistently demonstrate that pre-training accuracy on the standard ImageNet strongly correlates with domain adaptation performance. Moreover, despite similar performance on the standard ImageNet test set, models pre-trained on ImageNet21k adapt more effectively. Our results challenge the common practice of fine-tuning to the task at hand: blind adaptation might lead to catastrophic failures, mainly when BN is applied and normalization statistics of the source data are distant from the target ones. Our robustness analysis confirms the critical role of the normalization layers. On average, BN architectures suffer from a higher failure rate, i.c., adaptation deteriorates performances. Despite being more constrained in terms of source data availability and computational cost, SF-UDA proves competitive with UDA approaches. Our analysis of uncommon architectures and domain shifts demonstrates the robustness of SF-UDA beyond standard benchmarks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, and Herve Jegou. Xcit: Cross- covariance image transformers. In Advances in Neural Information Processing Systems, 2021.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:190..02893, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.06650 2016.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances in Neural Information Processing Systems, 2006.
Shai Ben-David John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 79(1):151-175,
2010.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. In Advances in Neural Information Processing Systems, 2011.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003, 04297, 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-9649, 2021.
Brian Chu, Vashisht Madhavan, Oscar Beijbom, Judy Hoffman, and Trevor Darrell. Best practices for fine-tuning visual classifiers to new domains. In European Conference on Computer Vision, pp. 435-442 2016.
Tong Chu, Yahao Liu, Jinhong Deng, Wen Li, and Lixin Duan. Denoised maximum classifier discrepancy for sourcefree unsupervised domain adaptation. In Thirty-Sixth AAAI Conference Artificial Intelligence (AAAI-22), volume 2, 2022.
Carlo Ciliberto, Alessandro Rudi, Lorenzo Rosasco, and Massimiliano Pontil. Consistent multitask learning with nonlinear output relations. Advances in Neural Information Processing Systems, 30, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 248-255, 2009.
Ning Ding, Yixing Xu, Yehui Tang, Chao Xu, Yunhe Wang, and Dacheng Tao. Source-free domain adaptation via distribution estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7212-7222, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition scale. In International Confer- ence on Learning Representations, 2020.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. works. Journal of Machine Learning Research, 17(1) (:20:::2000000 2016.
training of neural net-
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Kurt Hornik, Ingo Feinerer, Martin Kober, and Christian Buchta. Spherical k-means clustering. Journal of statistical software, 50:1-22, 2012.
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. Advances in Neural Informa- tion Processing Systems,
2021.
Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learn- ing? arXiv preprint arXiv:1608.086// 2016.
Maximilian Isse, Jakub M Tomezak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders. In Proceedings of the Third Conference on Medical Imaging with Deep Learning, pp. 322-348. PMLR, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456, 2015.
Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. In European Conference on Computer Vision, pp. 464-480, 2020.
Mahesh Joshi, Mark Dredze, William Cohen, and Carolyn Rose. Multi-domain learning: when do
domains matter? In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 1302-1312, 2012.
Guoliang Kang, Lu Jiang, Yi Yang, and Alexander Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4893-4902 2019.
Donghyun Kim, Kaihong Wang, Stan Sclaroff, and Kate Saenko. A broad study of pre-training for domain generalization and adaptation. arXiv preprint TrXiv:22031//8/9 2022.
Simon Korbllith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2661-2671, 2019.
Wouter M Kouw and Marco Loog. A review of domain adaptation without target labels. 1EEE Transactions on Pattern Analysis and Machine Intelligence, 43(3):766-785, 2019.
Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al. Universal source- domain adap- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4544-4553, 2020.
Jogendra Nath Kundu, Akshay R Kulkarni, Suvaansh Bhambri, Deepesh Mehta, Shreyas Anand Kulkarni, Varun Jampani, and Venkatesh Babu Radhakrishnan. Balancing discriminability and transferability for fource-free domain adaptation. In International Conference Machine Learn- ing. pp. 11710-11728. PMLR, 2022.
Vinod K Kurmi, Venkatesh K Subramanian, and Vinay P Namboodiri. Domain impression: A the IEEE/CVF Winter Conference
source data free domain adaptation method. In Proceedings the on Applications of Computer Vision, pp. 615-625, 2021.
Rui Li, Qianfen Jiso, Wenming Cao. Hau-San Wong, and Si Wu. Model adaptation: Unsuper- vised domain adaptation without source data. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9641-9650, 2020.

---

Under review as a conference paper at ICLR 2023
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normaliza- tion for practical domain adaptation. In International Conference on Learning Representations, 2017.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pp. 6028-6039, 2020.
Jian Liang, Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 2021.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976-11986 2022.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. In Advances in Neural Information Processing Systems, 2016.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International Conference on Machine Learning, pp. 2208-2217, 2017.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.34300, 2009.
Jaemin Na, Heechul Jung. Hyung Jin Chang, and Wonjun Hwang. Fixbi: Bridging domain spaces for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1094-1 1103, 2021.
Poojan Oza, Vishwanath A Sindagi, Vibashan VS, and Vishal M Patel. Unsupervised domain adap- tation of object detectors: A survey. arXiv preprint arXiv:2 13500, 2021.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv: 2017.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 1406-1415 1415, 2019.
Tal Ridnik, Emanuel Ben-Barute Asaf Noy, and Lihi Zelnik-Manon Imagenet-21k pretraining for the masses. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Tobias Ringwald and Rainer Stiefelhagen. Adaptiope: A modern benchmark for unsupervised do- main adaptation. Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pp. 101-110. 2021.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European Conference Computer Vision, pp. 213-226, 2010.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-6114, 2019.
Marco Toldo, Andrea Maracani. Umberto Michieli, and Pietro Zanuttigh. Unsupervised domain adaptation in semantic segmentation: review. Technologies, 8(2):35, 2020.
12

---

Under review as a conference paper at ICLR 2023
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. 2021.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 7167-7176, 2017.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 5018-5027, 2017.
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In Advances in Neural Information Processing Systems, 2018.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, and Tao Qin. Generalizing to unseen domains: survey on domain generalization. In International Joint Conference on Artificial Intelligence, pp. 4627-4635, 8 2021.
Ross Wightman. Pytorch image models.
pytorch- image models 2019.
Haifeng Xia, Handong Zhao, and Zhengming Ding. Adaptive adversarial network for source-free domain adaptation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 9010-9019, 2021.
Ruijia Xu, Guanbin Li. Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 1426-14255 1435, 2019.
Shiqi Yang. Joost van de Weijer, Luis Herranz, Shangling Jui, al. Exploiting the intrinsic neighbor- hood structure for source- free domain adaptation. In Advances in Neural Information Processing Systems, 2021a.
Shiqi Yang, Yaxing Wang, Joost de Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8978-888 888888997440 2021b.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems, 2014.
Youshan Zhang and Brian D Davison. Impact of imagenet model selection domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, pp. 173-182, 2020.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pp. 7404-7413, 2019.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing Hc. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1); 43-76, 2021.
13

---

Under review as a conference paper at ICLR 2023
A PROBING METHODS
In this section, we discuss the probing methods employed to inspect and assess learnt models.
We follow the the same notation introduced in Sec. Let fe be given fixed feature extractor and C' the number of classes. We introduce the training set STRAIN = {(z, v)) where 2; = fo( and (1) ~ us i.i.d. samples from the source domain, for i 1,...,N.
Similarly, we define STEST to be a set of M i.i.d. evaluation samples. Based on the experiment, evaluation samples can come from the same distribution of the training (generalization) from different one Soomain--gnnralize
and domain adaptation).
Linear Probing (LP) trains and evaluates linear classifier (Multinomial Regression) the features extracted from a fixed model. During training of the linear classifier, we seek for function that maps the features to the C simplex minimizing the (L2-regularized) log loss:
g - A C-1
2+ o (Wz). 2),
where is the softmax function and W R CxD is found as:
W min W||
Σ
- llog(a(W'ss
W'EReeD
STRAIN|
(z.g)ESmAIn
with À > O regularization hyperparameter. Then, we can convert this function to classifier hMR:
hMR + V,
+ arg max g(z)
Hence, linear probing accuracy is evaluated on the test set:
LP accuracy
Σ (R.D)
(hum((z)= v).
=
STEST
Cluster Probing (CP): trains and evaluates a Nearest Neighbour (I-NN) classifier on the features extracted from fixed model. For every class, prototype is found averaging the feature vectors of that class and, then, the prototypes are used as I-NN classifier. In particular, for each class c the prototype is evaluated as:
E(.,)ss 1/y = c)2
ke E(E,)) 1/y = c}
where c= 1,...,
...,.. C. Hence, the associated classifier is:
IINN 2 =.
[1 (ke,
ト arg min eEy
2
2
k |
where we use the cosine dissimilarity to quantify the distance between features and prototypes.
The accuracy is evaluated for cluster probing in equation 7 using hINN:
CP accuracy
Σ
1hINN(*)= = y).
STEST
(2.9)
Intuitively, high accuracy in the cluster probing means that the features of the same class are well clustered (spherically). while clusters of different classes are nicely separated (always spherically). This is very desirable property for unsupervised domain adaptation where need leverage on the underlying structure, since labels of the target domain are not available.
14

---

Under review as a conference paper at ICLR 2023
B SIMPLE CLASS ALIGNMENT
The Simple Class Alignment (SCA) method is part of many domain adaptation algorithms like CAN (Kang et al. 2019) and SHOT (Ling et al. 2020). Nevertheless, literature, the method is never employed alone and its contributions, as part of the overall DA methods, have not been explored extensively.
The algorithm stands out for its simplicity: (1) find a prototype for every class of the source domain, (2) use these prototypes as initialization for K-Means on the unlabeled target domain and finally, (3) the resulting prototypes
1-NN classifier for the target domain, i.e, classify every sample based on the class of the closest prototype. As in previous works, we used cosine dissimilarity (spherical K-Means) to evaluate how close feature vectors are.
We highlight that this method finds a new classifier for the new domain by aligning the prototypes the target distribution. Hence, it can be effective with features extracted from frozen model without fine-tuning. Consequently, SCA is highly efficient Domain Adaptation method that can achieve results comparable with the state-of-the-art.
But, how to find the initialization prototypes? We compare 4 different techniques considering problem where we have classes and fixed feature extractor that can be used to extract, from images, feature vectors in R& (we indicate feature vectors with letter z).
From source labels. Leverage the available source labels to partition the source feature vectors into the sets CI...., Cc, where all features of class j are in Cj. Then, compute the prototype for class as the average of features of that class:
wj
(8)
Cg
=EC,
This initialization has been used in CAN (Kang al. 2019) method.
From Multinomial Regression weights. Learn a Multinomial Regression (MR) classifier h + o( (Wz), 2), where a is the Softmax function, by training on the labeled source domain. Then, initialize the prototypes as the rows of the matrix W E RCxd Algorithm this initialization.
shows all the steps of SCA with
From target hard predictions (pseudo-labels). Train MR classifier on the source domain and pseudo-label the target accordingly. Initialize each prototype with the average feature samples as- sociated to the same pseudo-label. We note that this initialization, followed by spherical K-Means is very similar to the previous one: the only difference is that, this case, in the first iteration to com- normalized.
pute the distances for the assignments the weights of matrix W and features As we will see, this method do not perform as good as the others.
source data and compute prototypes as weighted averages of all samples where weights are given by predicted class probabilities. More precisely, for each feature target vector z; the output of the MR classifier (after the Softmax) is P, = (Pil...., P.c) where is the predicted probability of sample i to belong to class c. Then, the prototype of class is evaluated as:
From target soft predictions. Fit MR classifier
EiPiozi EiPio
k.
This method has been used in SHOT algorithm.
We remark that these methods can also be applied a fine-tuning of the feature extractor has been performed or linear layer is employed in place of the MR classifier.
Which is the best method to initialize prototypes? In Tab. d we report the average gain on all mod- els of SCA (without fine-tuning), the failure rate, the average gain in of successful adaptation and the average degradation in case of negative transfer for the 4 types of initialization. The SCA gain/degradation is considered with respect naive domain generalization. The method that per-
15

---

Under review as a conference paper at ICLR 2023
Algorithm 1 SCA from Multinomial Regression weights
Require:
Multinomial regressor for source domain with weights W (wi, W2,... we) Roxd = (21, 22, ,2,,..,,2.. E RNdx unlabeled target samples (features).
1. - (l,.... 1)a X - diag( (Z
Normalize features to unit length.
while not converged do
W c diag (W W)12)-1/2W
Normalize weights to unit length.
for i in I.....N
N do 4 + arg min je(,.. (Zi,, W
Assign every sample to the most similar weight vector.
end for
for j in 1, .,....,............. C do
D Update the weight vectors.
N 1.
W,. E [ar=])
end for
end while
return W
forms best is the initialization from the Multinomial Regressor weigths: it is robust (about 20% failure rate) and the degradation, in case of failure, is modest (1.8%).
The method of hard predictions is the only that performs poorly, when it succeed it can gain more than other methods (7.2%), but it has a large failure rate and it leads to large performance impairment in case of failure.
Table 6: Comparison of the four different SCA initializations. The results are the average of 548 architectures and 74 different domain pairs. The A accuracy the difference between the target accuracy after SCA and the naive domain generalization of MR classifier trained on source. The failure rate the percentage experiments where SCA degrades the naive domain generalization. In the last two rows are reported the accuracy gain and degradation considering the successful and failure adaptation, respectively. These result differ from the ones in the main text, since here we consider the full set of 548 architectures.
Source Labels
MR Weights
Hard Pred.
Soft. Pred.
A Accuracy
3.8 +
4.5
-0.3 +
3.7 +
Failure Rate
25.1 + 102
20.1 =
46.2 +
29.3 + 102
A Accuracy Success
5.8 +
6.2
7.2 1
6.4 1:
A Accuracy Failure
-1.8 + 06
-1.8 + 1.0
-15.4 4:43
-2.4 + 16
C ARCHITECTURES
In Fig. we present some plots to help the localization of architecture families on the scatter plots. The architectures of the same family differs for size, e.g., Resnet50 and Resnet101, training proce- dure (even if the training is always performed on ImageNet or ImageNet21k) or for minor architec- tural details.
Subset of 59 models. For fine-tuning and SHOT experiments we used a subset of mod- els taken from the timm library (with the exception of (onet50_in21k resnext101 _2x22d_______111111100 in21k,
16

---

Under review as a conference paper at ICLR 2023
ImageNet
ImageNet21K
vit
densenet
inception
regnet
convnext
vgg
mixer b
xcit
levit
resnest
nfnet
hardcorenas
hrnet
deit
resnet
cait
fbnet
beit
pit
Icnet
swin
mobilenet
efficientnet
tinynet
Figure 4: Families of architectures: on the x-axis we have the ImageNet topl accuracy, on the y-axis the accuracy after applying SCA (without fine-tuning) averaged across 74 domain shifts.
densenet161_in211 wide_resnet50_221 in2kk, vgg19_ aen19_nn__i__ bn_in21k, that were trained by us). The mod- els are:
beit best_ cass__patchl__30 beit large_ppachh6 224, beit large patch16 6_84,,3444,_ convnext base, con- mnext_base_ masse_nn2ftllk convnext large, convnext xlarge_384 in22fin2 convnext xlarge_ in2fftkk. ciistilled_patchhte
deit deit_ doaallddsttlled___ee distilled_patchh16 wtt_ppaccch_21111 224,,,22422_66 dla169, ecaresnetlOld, ef- ficientnetv2_reg Bacitt22_______________ efficientnet b4, efficientnet_ el, efficientnet em, efficientnet.er es, fbnetv3 ghostnet_100,, gluon_resnet50_ve (ixee_bl6_224..0
gluon_resnext101_33333 101__x0ddd,__,3 gmixer_24 24_224, hrnet_w44, mixnet_m, bit_ti_224, resnet152d, resnet18, resnet26, resnet50, selecsls60, swin swin_base Baa___as__aacc________ 2etchh4_winowl1222 384, swin base Looee__atcc___a window7 224. swin large patch4 window7 224, efficienttetv2 in21ftlk, tf efficienntetv22 in21ftlk, tf efficientnetv2 xl in2litlk. tf mobilenetv3. mobilenetv3_small 100, vgg19 bn, visformer_smalle vit vit_base__paa base_patch16. 224 miil, vit_basee_atthe 224, vit_ vit_base_r50__ 384, vit Largge_patchi 224, vit vit wwsmllllpacchh 224, vit_tiny vit_tiny_patchl 384, xception65, xception71, 1ennet50_i2210 resnext101_3220 _2xdd in21k, densenet16l_in22 hide_resnet50_22 in222 in_2kk...
jx_nest_ting
(r_xs_distilled. 224, repvgg repvgg_b3g4
384, vit
vit_small_patch 384,
17

---

Under review as a conference paper at ICLR 2023
D TECHNICAL DETAILS
For all experiments we used Python 3.7.6 with PyTorch 1.12.1 and CUDA 10.2.
We followed the common procedure used in SF-UDA (see for example SHOT Liang al. (2020))
with some adjustments to face the large number of experiments and the great variety and diversity of datasets and models.
Model. For every pre-trained model we remove the last linear layer (ImageNet classifier). Then, we add randomly initialized bottleneck followed by linear layer (classifier for the given task). The bottleneck is composed by linear layer that maps the features of the backbone to 256 dimen- sions, followed by normalization layer and a ReLU activation. The bottleneck normalization layer depends on the backbone, i.e., Batch Normalization if the backbone contains Batch Normalization layers, otherwise it Layer Normalization layer.
The distributed training is performed on variable number of GPUs depending on the model size (between 2 and Nvidia V100 16GB) and automatic mixed precision is used by de- fault. The batch normalization statistics are always synchronized between different processes. The global batch size is kept fixed at 64 and, in the cases where the memory of 16 GPUs is not enough, gradient accumulation is used. SGD with Nesterov momentum (0.9) is used for the optimization. The initial learning rate for the classifier and the bottleneck 0.01, while the initial learning rate of the pre-trained backbone is 0.001. The learning rates decreased following the same exponential scheduling used in SHOT. During training, the 15% of the training set is kept as validation (used just determine the stopping iteration). To face the great diversity of dataset sizes limit the number of optimization steps of each epoch to 100 (in practise we considered one epoch just the interval between two validations and not, as is usual, an iteration of the full dataset). We fixed the minimum number of epochs at 10 and the maximum number at 100. After each epoch the val- idation accuracy evaluated and if there are 5 epochs in row where the validation accuracy does not increase, the training is stopped and the model weights with the highest validation accuracy are returned. As regularization we used weight decay (0.001) and label smoothing (0.1) 1) on the standard crossentropy loss.
Training.
SHOT. For the adaptation with SHOT we limited the number of epochs at (always with a max- imum of 100 optimization steps) and used the same hyperparameters of the original paper. We always used distributed training and automatic mixed precision.
NRC. For NRC (Yang al., 202 method we used the official code (not distributed) with the official hyperparameters. We added just automatic mixed precision to allow the training of larger models.
ImageNet21k training. To train ResNet50, VGG19, Wide- Resnet50, Resnext101 32x8d and DenseNet161 ImageNet21k used the official torchvision training script with the default hy- perparamters, with the exception of:
initial learning rate: 0.8
batch size: 2048
Ir schedule: constant decay every of 0.1 every 15 epochs
. epochs: 50
No augmentations are used and neither a validation set, the final weights are tested on the down- stream SF-UDA tasks.
18

---

Under review as a conference paper at ICLR 2023
E STATISTICAL ANALYSIS OF RESULTS
Are the pre-training differences that we observed relevant? Even if, from the seatter plots presented in Fig. iit is possible to see, visually, that there is interesting gap between models pre-trained on ImageNet and ImageNet21k we fit a linear model to understand the benefits of introducing pre- training independent variable. In particular, for any experiment, first we try to associate the result accuracy that it is possible to obtain with given backbone B to its ImageNet topl accuracy:
accuracy (B) m top (B) +
(10)
possible to find the parameters m and & with the ordinary least squares estimator, Then, we en- code the two pre-training into a binary function such that if the backbone is pre-trained on ImageNet pretrain ()) = 0, while it is pre-trained on ImageNet2Ik pretrain(B) and we introduce into the model. In particular
consider the following:
Different intercept model
accuracy(B) = m (opi(B) + (4 + Aq pretrain(B))
(11)
Different slope model (interaction)
accurocy(B)) (m + Am Am.prett pretrain(B)) topi (B)
(12)
Different slope and intercept
accuracy(B)) (m + Am (om-rrrtrain(n)) t (B) + (q + Ωq. pretrain(B)) (13)
To compare the different models
the goodness-of-fi (R²) that represents the fraction of explained variance. Given the observed values (x1, /1 y1),..., NI DNv (where z; is vector of independent variables, while y; is the associated dependent variable) and the values predicted by the model f: Ø1 f(x))...... UN = [(EN) we can compute the mean of the observed data as:
N
V
(14)
N
the total of squares is defined as:
SS
(ys 7)
(15)
while the residual sum of squares:
N SS Σ Uts - 02)2
(16)
i=1
then the coefficient of determination can be defined as:
SSres
R² =
(17)
SS..t
Then, even if in our case the difference is marginal, we use the adjusted R2 (indicated as R22) to account for the different number of independent variables in the models, that is defined as:
= N
R²)
(18)
df
where df are the Pegrees-of-freedde of the model.
We report all the coefficients of the linear and multi- models and the R. values in Tab. 7 An illustration is given in Figs. 5 14 In some cases (for example in SHOT), we get the p-values for the Am and Aq parameters higher than the statistical significance of 0.01. In this case, we just need to remove one of the two parameters to make remaining ones all significant. This phenomenon is due to the fact that the two lines, in these experiments, are almost parallel and so, Am is approximately 0 (see Fig. [0) and can be safely be removed from the statistical model without decreasing the goodness-of-fit.
19

---

Under review as a conference paper at ICLR 2023
Table Coefficients and R² obtained with single and multi linear models. Parameters of the multi-linear model with p-value higher than 0.01 are removed from the model, since their contribution not statistically significant.
Experiment
model
m
Am
Aq
R
single reg.
0.61
0.29
0.74
LP Gen
multiple reg.
0.51
0.31
-0.23
0.36
0.85
single reg.
0.72
0.16
0.76
CP Gen
multiple reg.
0.59
0.23
-0.14
0.26
0.91
single reg.
1.14
0..4
0.81
LP DGen
multiple reg.
0.95
0.62
-0.45
-0.26
0.93
single reg.
1.14
00.41
0.75
CP DGen
multiple reg.
0.92
0.54
-0.38
-0.25
0.92
single reg.
1.08
-0.32
0.73
SCA
multiple reg.
0.87
0.42
0..7
0.16
0.90
single reg.
1.62
0.71
0.79
SHOT
multiple reg.
1.21
0.07
0..0
0.89
single reg-
0.62
0.35
0.80
FT Gen
multiple reg.
0.48
0.02
0.46
0.88
single reg.
1.68
-0 -0.82
0.70
FT DGen
multiple reg.
1.16
0.09
- 0..43
0.82
single reg.
1.62
-0.72
0.67
FT+SCA
multiple reg.
1.10
0.08
-0.3
0.80
single reg. multiple reg.
1.51
- 0.59
0.84
FT+SHOT
1.15
0.06
0.32
0.93
Single: R = 0.736
Multi: R 0.851
85
LP Gen accuracy
88
65
56
76
86
Figure Average Linear Probing generalization accuracy (over domains).
20

---

Under review as a conference paper at ICLR 2023
Single: R² 0.756
Multi: R 0.908
CP Gen accuracy
77
38
57
56
78
86
top1
Figure Average Cluster Probing generalization accuracy (over domains).
Single: R² 0.810
Multi: R
0.935
70
LP DGen accuracy
50
4
30
56
76
86
top1
Figure 7: Average Linear Probing domain generalization accuracy (over 74 domain shifts).
Single: R² = 0.755
Multi: R² = 0.918
CP Dgen accuracy
60
40
*
30
56
67
86
67
76
86
top1
Figure 8: Average Cluster Probing domain generalization accuracy (over domain shifts).
21

---

Under review as a conference paper at ICLR 2023
Single: R² 0.731
Multi: R 0.902
SCA accuracy
65
35
56
72
87
top1
Figure 9: Average accuracy (over 74 domain shifts) of models after applying SCA (without fine-tuning).
Single: R = 0.792
Multi: R 0.890
SHOT accuracy
55
40
65
80
65
80
top1
Figure 10: Average accuracy (over 74 domain shifts) of models after applying SHOT (without fine-tuning).
Single: R² 0.803
Multi: R 0.878
FT Gen accuracy
75
56
76
86
Figure 11: Average generalization accuracy (over 23 domains) of models after applying FT.
22

---

Under review as a conference paper at ICLR 2023
Single: A = 0.698
Multi: R 0.822
FT DGen accuracy
38
56
72
87
56
T2
87
top1
Figure 12: Average domain generalization accuracy (over 74 domain shifts) of models after applying the FT.
Single: R
0.668
Multi: R² 0.792
FT+SCA accuracy
50
56
72
87
56
72
87
top1
Figure 13: Average accuracy (over 74 domain shifts) of models after applying the FT and SCA.
Single: R = 0.838
Multi: R² 0.932
FT+SHOT accuracy
70
55
65
80
65
80
top1
Figure 14: Average accuracy (over 74 domain shifts) of models after applying the FT and SHOT, this case the two lines of the statistical multi-linear model are parallel (Am was not statistically
23

---

Under review as a conference paper at ICLR 2023
F COMPARISON OF DOMAIN GENERALIZATION AND SCA WITH SEMANTIC TRAINININ
We further investigate the role of pre-training strategy. Tab. g contrasts the domain generalization accuracy for miil (Ridnik et al. 2021) and naive pre-training on different domain generalization datasets. Tab. 9 shows results for different initializations of ResNet50 on Office31.
Table 8: Comparison results (accuracies %) semantic pre-training (miil) of Ridnik ct al. 2021) and naive pre-training of ResNet50, for linear probing domain generalization and SCA. The results are avaraged across the domains of the datasets.
Experiment
Pre-train
031
Visda
O.Home
Adapt.
I-CLEF
D.Net
naive
86.6
63.1
68.5
59.1
79.8
19.9
LP DGen
miil
88.8
67.3
71.4
68.2
81.9
25.3
naive?
90.3
80.7
74.4
73.8
85.1
23.6
SCA
miil
90.4
79.8
72.2
73.3
84.8
21.3
Performed by us.
Table 9: Comparison of results (accuracies %) on Office31 benchmark using different pre-trained weights of Resnet50. We included the following pre-training: timm (Wightman, 2019). the semantic training of Ridnik al. 202T) (miil) and naive pretrain. Office31 domains Amazon, Webcam and DSLR and the experi- ments are indicated with source + target. For these experiments the official code (not distributed and without Automatic Mixed Precision) of SHOT (Liang et al. (2020) and NRC (Yang et al. 2021a)) have been used, so the results may be slightly different from the ones reproduced with our implementation.
Office31 (ResNet50)
Method
Init
IN21k
A
W
A
D
W
A
W W- D
D D A
D
W
Avg
timm
77.5
78.9
68.3
97.8
64.6
96.6
80.6
LP DGen
miil
89.4
89.1
78.3
99.4
78.7
98.1
88.8
naive
86.2
83.3
77.3
99.4
76.4
97.1
86.6
timm
89.2
92.5
73.4
97.2
73.9
90.1
86.0
SCA
miil
93.8
97.4
79.0
98.2
78.8
95.2
90.4
naive
/
93.3
94.6
80.1
98.2
80.0
95.7
90.3
timm
89.3
87.1
68.9
99.8
67.1
85.5
FT+SHOT
miil
93.2
92.4
75.3
99.8
76.8
98.8
89.4
naive
V
91.1
94.0
75.3
99.6
73.5
98.8
88.7
timm
94.1
93.0
72.2
99.8
69.8
98.2
87.7
FT+NRC
miil
96.0
89.6
76.7
99.6
75.0
98.1
89.2
naive
92.6
96.0
80.4
99.8
78.5
98.4
90.9
24

---

Under review as a conference paper at ICLR 2023
G BATCH NORMALIZATION VS LAYER NORMALIZATION
Fixing a source-target domain pair, we compute the average domain generalization improve- ment/degradatioo over the different architectures. particular, split all 59 architectures consid- ered into 4 groups based on normalization layers and pre-training: BN+IN (24 models), BN+IN2IK (8 models), LN+IN (8 models) and LN+IN2IK (19 models). In Fig. T5 we show the average im- provement/degradatton for each domain pair and for every model group. As it is possible to see, there are some domain pairs where the models with BN degrades lot the domain generalization, while all models with LN are more stable.
modernofiiee3
adaptiope
visda
officehome
imageclef
domainnet
Batch Norm. IN
20
0
-99
-40
-60
Batch Norm. IN21k
0
-19
.40
A DGen
-60
Layer Norm. IN
20
0
-19
.40
-60
Layer Norm. IN21k
0
-19
-40
-60
Domain Pairs
Figure 15: Fach bar represent different domain-shift experiment, i.e., and target. Bars height the average gain/degradation domain generalization (always w.r.t. the haseline I.P domain generalization) for the given domain pair.
In Tab. [Q]are reported the numerical results for the domain pairs with the highest degradation caused by the fine-tuning with Batch Normalization layers. In particular, the domain pairs correspond to the 10 left-most bars of Fig. 15
Tab. [ we report the average degradation case of failure on all domain pairs for different model groups. Also here it is possible to see that the models with Batch Normalization are affected from high degradation of the domain generalization with the fine-ture tuning.
25

---

Under review as a conference paper at ICLR 2023
Table 10: Domain pairs with the highest fine-tuning domain generalization degradation caused by Bacth Nor- malization layers.
dataset
source synthetic
target
bn in
bn in2
In in
In in21k
1
wehcam
-48.6
-66.1
-0.3
1.2
modernoffice31
synthetic
dslr
-46.8
-65.4
-1.3
1.0
modernoffice31
synthetic
amazon
-39.5
-5.44
-1.3
0.6
adaptiope
synthetic
real life
-33.1
-5..0
-2.6
3.2
adaptiope
synthetic
product_imeg images
-31.2
-45.7
-0.3
visda
train validation
-17.6
-8.7
-2.4
modernoffice31
dsIr Clipart
synthetic
-10.3
-18.5
-5.3
-0.9
officehome
Art
-10.0
-10.2
2.3
2.7
1
webcam
synthetic
-10.0
-10.0
-3.9
-2.2
modernofiie31
webcam amazon
-9.6
-11.0
-2.7
-0.5
Table 11: Average dogradation (%) in case of failure (averaged across 74 domain pairs). Every row select certain type of model to make full comparison. Between parethesis in reported the number of model in that group.
Models
FT DGen
SCA
SHOT
FT+SCA
FT+SHOT
All (59)
-6.67 4.55
-1,55 +
-3.11
2.76
-7.88 6675
-1.58 +
IN (32)
-8.05
-1.79
-2.66
-8.79
-1.36
IN2IK (27)
-5.02
-1.27
-3.64
-6.65
-1.89
BN (32)
-10.51 4: 2:22
-1.92
-2.83 1.34
-12.72
-1.82 2.16
LN (27)
-2.10 0.6
-1.12 0.40
-3.43 3,82
-1.41 0.77
-1.27 4: 1:26
IN+BN (24)
-9.82 2:101
-1.95 0:70
-2.56
-11.26 4.33
-1.27 4 0.72
IN+LN (8)
-2.75 1.063
-1.32 1 0.3
-2.96 2.26
-1.39 0.57
-1.60 4: 1.33
IN2IK+BN (8)
-12.60 + 1.40
-1.82 0.400
-3.65 1.94
-17.10 +335
-3.31 4 4.47
IN21K+LN (19)
-1.83 0.443
-1.03 + 0.40
-3.63
4.35
-1.42+ 0.87
-1.08 + 1.22
26

---

Under review as a conference paper at ICLR 2023
H COMPUTATIONAL TIMES.
In Tab. 12 we report the average number of seconds to perform SCA, one epoch of fine-tuning and epoch of SHOT adaptation for Office31 and Visda datasets (averaged across all different domains). We remark that SCA is performed just once, while the fine-tuning (in our experiments) uses between 15 (10 5, see technical details in App. D) and 100 epochs, while SHOT uses always 15 epochs.
Dutaset
Model
FT
SCA
SHOT
DANN
Table 12: Time required (in seconds) fine-tuning and adaptation. The times are averaged on the do- mains of the used dataset. For fine-tuning and SHOT the time reported refers that according to standard training, SHOT requires epochs adapt while DANN (Ganin et al. 2016 requires 100 epochs. Models are in mixed precision training with batch size 64 split into 2 x NVIDIA RTX 2070 SUPER single machine with AMD Ryzen 3900x.
ResNer50
1.7
Office31
VTT Base
5.7
4,5
14.8
10.8
epoch. We remark
ResNet50
211.1
92.1
425.3
404.1
Visda
381.6
327.1
1068.5
790.3
I ADDITIONAL RESULTS: BEST PERFORMING MODELS.
In Tab. E we provide domain generalization results on large architectures different DA datasets when fine-tuning varying on the fine-tuning choice.
Domain generalization accuracy for large backbones with different fine-tuning decisions. Both the fine-tumie tunitt and non fine-tuning cases are considered. For each architecture, the accuracy change the average over the domains of the considered dataset.
Table 13:
Model
Method
Office31
OfficeHome
Adapt.
Mod.Ofiie3
Image-C1.FF
DomsinNet
SCA
x
94.8
85.1
89.0
93.1
97.4
88.6
45.3
BEIT 384
94.2
79.6
88.9
90.7
95.5
86.4
45.0
SHOT
95.8
SCA
1.
SHOT
89.3
43.4
VIT
SHOT
93.9
95.4
93.9
83.9
92.1
95.8
89.5
54.1
95.5
$4.9
86.9
90.0
96.5
88.8
ConvNest XL. 384
94.8
84.1
87.4
90.8
93.8
83.2
49.8
SHOT
94.0
89.2
90.3
93.9
94.0
89.0
48.5
95.1
88.1
88.8
93.3
92.2
39.4
$1.0
J SELF-SUPERVISED PRE-TRAINING
In all previous experiments we considered only supervised pre-training. However, recently, the community shows a great interest on self supervised learning (SSL) techniques, which proved to achieve remarkable performance. Thus in Tab. [4] we present the results of some experiments on ResNet50 trained with three different SSL. methods, namely, DINO (Caron et al. 2021), MOCO vl (He et al. 2020) and MOCO v2 (Chen et al. 2020). As can be noticed, after performing a fine- tuning on the source domain (and eventually applying SCA) MOCO v2 performs very similarly (on average) with a supervised pre-training on ImageNet, while DINO and MOCO v1 achieve lower accuracy. When FT+SHOT is applied MOCO v1 still underperforms other methods, while DINO and MOCO v2 achieve similar and promising results which, however, are still marginally below the performance of the supervised pre-training, which gets the best accuracy (66.1%)
From our findings and extensive experiments reported the main text, we know that the fine-tuning on the source domain of model with Batch Normalization layers (like ResNet50) can lead to
27

---

Under review as a conference paper at ICLR 2023
large drop in performance in some scenarios and the right way adapt these models after the fine- tuning is to adapt the feature extractor (with SHOT, for example). Moreover, although the results promising for SSL pre-training and even if MOCO-v2 performed similarly to supervised pre- training on FT and FT+SCA, the drop in accuracy for FT+SHOT highlights that SSL. pre-training still underperforms supervised pre-training (on ImageNerlk) SF-UDA, when the right adaptation choices are made.
Table 14: Target accuracy (%) using different initialization weights of ResNet50 after performing the following experiments: fine-tuning the source domain (FT), FT+SCA and FT+SHOT.
Experiment
Pre-train
M031
Visda
O.Home
Adapt.
I-CLEF
D.Net
Avg
DINO
42.7
45.6
45.7
30,9
71.3
19.7
42.7
MOCO
38.5
44.6
41.7
28.6
20.1
40.0
FT
MOCO v2
54.7
54.2
55.1
39.5
76.9
25.5
51.0
Supervised (INIK)
56.4
49.6
55.5
43.3
75.9
21.6
50.4
DINO
50.8
54.5
49.8
35.9
75.0
21.9
48.0
MOCO vl
49.5
52.3
43.9
34.3
70.4
23.9
45.7
FT+SCA
MOCO v2
63.4
56.9
55.8
45.3
79,5
28.0
54.8
Supervised (INIk)
62.7
55.5
59.4
49.6
79.3
23.6
55.0
DINO
71.8
67.3
58.0
55.4
77.5
24.1
59.0
MOCO vl
56.6
58.3
52.9
37.5
71.7
19.2
49.4
FT+SHOT
MOCO v2
73.2
65.7
66.9
51.9
81.0
26.2
60.8
Supervised (INIK)
68.9
27.3
Table we we report the results on datasets to compare the DINO SSL pre-training to a supervised pre-training on ImageNet2 on a larger architecture that uses Layer Normalization (VIT Base), The results confirm that, in most cases, the gap between SSL and supervised pre-training is still quite large. We also report some instabilities of SHOT using the DINO pre-training (for example VisDA dataset) that very rarely happened with supervised pre-training.
Table 15: Target accuracy (%) using different initialization weights of Base after performing the following experiments: fine-tuning on the source domain (FT), FT+SCA and FT+SHOT.
Experiment
Pre-train
031
MO31
Visda
O.Home
DINO
80.7
72.4
65.3
63.6
FT
Supervised (IN21K)
89.7
86.9
77.9
79.9
DINO
85.2
81.1
73.6
67.9
FT+SCA
Supervised (IN21k)
92.7
91.9
82.6
82.5
DINO
87.5
75.8
56.9
74.2
FT+SHOT
Supervised (IN21K)
93.7
93.9
87.5
84.4
28


Reviews:
[{"review_id": "_-_xTXW1kQN", "reviewer": "Reviewer_1ukJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work explored source-free domain adaptation setting (SF-UDA). Different from conventional UDA scenario, SF-UDA assumes that model adaptation of target domain only accesses well-trained source model without well-labeled source data.", "summary_of_the_review": "This work explored source-free domain adaptation setting (SF-UDA). Different from conventional UDA scenario, SF-UDA assumes that model adaptation of target domain only accesses well-trained source model without well-labeled source data. Overall, in my opinion, this work is only an experimental report instead of a complete research paper due to two main reasons.\n\n1. In Section “Method”, the current manuscript only provided many details of the existing SF-UDA work (SHOT) without description of new proposed algorithm. Thus, this current version lacks a complete novel algorithm.\n\n2. Currently, there are a lot of explorations on SF-UDA topic [1, 2, 3]. However, the current work does not make fair comparisons with them. Thus, the experiments are not enough.\n\n[1] Chu, Tong, et al. \"Denoised maximum classifier discrepancy for source free unsupervised domain adaptation.\" Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-22). Vol. 2. 2022.\n\n[2] Ding, Ning, et al. \"Source-Free Domain Adaptation via Distribution Estimation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Kundu, Jogendra Nath, et al. \"Balancing discriminability and transferability for source-free domain adaptation.\" International Conference on Machine Learning. PMLR, 2022.\n", "strength_and_weaknesses": "This work explored source-free domain adaptation setting (SF-UDA). Different from conventional UDA scenario, SF-UDA assumes that model adaptation of target domain only accesses well-trained source model without well-labeled source data. Overall, in my opinion, this work is only an experimental report instead of a complete research paper due to two main reasons.\n\n1. In Section “Method”, the current manuscript only provided many details of the existing SF-UDA work (SHOT) without description of new proposed algorithm. Thus, this current version lacks a complete novel algorithm.\n\n2. Currently, there are a lot of explorations on SF-UDA topic [1, 2, 3]. However, the current work does not make fair comparisons with them. Thus, the experiments are not enough.\n\n[1] Chu, Tong, et al. \"Denoised maximum classifier discrepancy for source free unsupervised domain adaptation.\" Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-22). Vol. 2. 2022.\n\n[2] Ding, Ning, et al. \"Source-Free Domain Adaptation via Distribution Estimation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Kundu, Jogendra Nath, et al. \"Balancing discriminability and transferability for source-free domain adaptation.\" International Conference on Machine Learning. PMLR, 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "U8XCjXgcUF", "reviewer": "Reviewer_GrZy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about Source-free Unsupervised Domain Adaptation (SF-UDA).  The authors provide an analysis of the impact of main design choices in SF-UDA through a large-scale empirical study with several CNN and ViT models, methods, and datasets for image classification. In particular, they observe the normalization approach, pretraining strategy, and backbone architecture are the most impactful design choices, and propose best practices for robust SF-UDA methods. The authors also seek to study the strengths and failure modes of SF-UDA methods and compare them with UDA approaches. Empirical results show that SF-UDA methods like SHOT can provide accuracy comparable with that of state-of-the-art UDA methods (at much lower data and computational cost, and better than standard benchmarks and backbone architectures. ", "summary_of_the_review": "Overall this is good quality submission.  The proposed study is well-motivated, although the experimental validation could be improved. ", "strength_and_weaknesses": "Strengths:\n+ The authors provide an in-depth empirical analysis of design choices for SF-UDA with several UDA and SF-UDA methods, backbone architectures, and datasets.  This SF-UDA  setting is particularly challenging and still remains understudied. I anticipate that this study and its results should be useful to researchers and practitioners working in this area.  \n+ The methods and experimental methodology are described in enough detail to understand the paper.     \n+ The conclusions of this study are interesting. For instance, it is useful to know that State-of-art SF-UDA methods, like SHOT, are competitive with state-of-the-art UDA methods, but can be implemented much more efficiently. \n+ The supplementary material provides additional algorithmic and implementation details on baseline methods, and experimental results that help support the paper.  \n\nWeaknesses:\n- Some definitions are not very clear at the beginning: DA, DG, etc., and how they relate to SOA methods for multi-task and multi-domain learning.\n- The authors do not provide a critical analysis of methods, or highlight from the outset the key challenges facing practitioners or researchers working on SF-UDA methods and problems. \n- The empirical results are not fully convincing. The scope of this study is limited to standard image classification problems, using common datasets. There is no results or discussion on the results that would be obtained from real-world data captured in the wild.  The authors should explore the impact on the performance of, e.g., class imbalance, noisy image data, etc. \n- The authors should measure domain shift empirically in order to characterize the difficulty of an SF-UDA problem.  \n- Results on FT+NRC should not be included in the results if it is trained using only trained on 15% of the DomainNet data.  \n- Finally, the authors should asses and compare the computational complexity of different SF-UDA methods, and show the differences w.r.t. UDA methods. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TGKnxiAnUtq", "reviewer": "Reviewer_R8P1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the source-free unsupervised domain adaptation problem and analyzes the impact of several design choices (e.g., the pre-training dataset, the pre-trained network, the normalization strategy, and the adaptation strategy) through a large-scale empirical study.", "summary_of_the_review": "This paper conducts an extensive study on the source data-free unsupervised domain adaptation problem, which is interesting and important in this field. However, some more studies are required to make this study more valuable. ", "strength_and_weaknesses": "Pros\n\n1. this studied problem is interesting and receives increasing attention in the transfer learning field, making a large-scale empirical study valuable\n\n2. the empirical analysis includes more than 500 architectures (including both CNNs and Vision Transformers), 6 datasets for domain adaptation, which is very extensive\n\nCons\n\n1. the selected source-free domain adaptation (SFDA) methods are relatively less, only a classifier adjustment method (SCA) and a feature encoder fine-tuning method (SHOT) is validated throughout the study. More SFDA methods like Li et al, CVPR 2020 could be used in an extensive study. \n\n2. generally speaking, the observations would not inspire more ideas in the SFDA topic as most conclusions have already been known.\n\n3. to study SFDA, these choices (the pre-training dataset, the pre-trained network, the normalization strategy) are vital but the authors may focus more on the adaptation strategy, like whether to recover the source data and the robustness to different label distributions (e.g., open-set, partial-set) \n\nSome missing references:\n\n1. Liang, Jian, et al. \"Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).\n\n2. Huang, Jiaxing, et al. \"Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data.\" Advances in Neural Information Processing Systems 34 (2021): 3635-3649.\n\n3. Kurmi, Vinod K., Venkatesh K. Subramanian, and Vinay P. Namboodiri. \"Domain impression: A source data free domain adaptation method.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021.\n\n4. Xia, Haifeng, Handong Zhao, and Zhengming Ding. \"Adaptive adversarial network for source-free domain adaptation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\nTypos:\nsiCP -> CP in section 3.1", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EQkqbOFxV6y", "reviewer": "Reviewer_cjNK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper reports the performance of SFDA methods under the new setting, called double-transfer, via a large-scale empirical evaluation. The double-transfer represents two-stage transfer learning setting in which the pre-trained model is firstly fine-tuned with a source domain dataset and is then adapted to the target domain. Under this setting, the results shown in this paper reveal several critical factors to achieve better performance after the adaptation. ", "summary_of_the_review": "This paper reports some interesting findings obtained through the empirical studies with a new setting called double-transfer. However, I have several concerns on clarity and novelty as stated above, which makes my score conservative. I vote for \"weak reject.\"", "strength_and_weaknesses": "\n\n<strength>\n\n- The double-transfer setting studied in this paper seems to be quite practical but has not been well explored in the literature. I love the motivation of this work.\n- The experiments cover a very diverse set of datasets as well as model architectures and are done on a large scale.\n\n<weakness>\n\n- Not so surprising results. Some findings in this study are closely related to those shown in the literature; for example, using a large-scale dataset is beneficial in pre-training [Dosovitskiy et al., 2020], and BN has a crucial role when conducting domain adaptation [Li et al., 2017].\n- In the experiments, only supervised pre-training is considered. Since self-supervised representation learning has recently become a popular way to pre-train the model, it would be great if it is included in the comparison to reveal the best strategy of pre-training in double-transfer setting.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject