On-Target Adaptation
Authors: Dequan Wang
Shaoteng Liu
Sayna Ebrahimi
Evan Shelhamer
Trevor Darrell
Venue: ICLR 2022 Submitted (2022)
Keywords: domain adaptation
source-free adaptation
unsupervised domain adaptation

Abstract: Domain adaptation seeks to mitigate the shift between training on the source data and testing on the target data. Most adaptation methods rely on the source data by joint optimization over source and target. Source-free methods replace the source data with source parameters by fine-tuning the model on target. Either way, the majority of the parameter updates for the model representation and the classifier are derived from the source, and not the target. However, target accuracy is the goal, and so we argue for optimizing as much as possible on target. We show significant improvement by on-target adaptation, which learns the representation purely on target data, with only source predictions for supervision (without source data or parameter fine-tuning). In the long-tailed classification setting, we demonstrate on-target class distribution learning, which learns the (im)balance of classes on target data. On-target adaptation achieves state-of-the-art accuracy and computational efficiency on VisDA-C and ImageNet-Sketch. Learning more on target can deliver better models for target.


Under review as a conference paper at ICLR 2022
ON-TARGET ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Domain adaptation seeks mitigate the shift between training on the source data and testing on the larget data. Most adaptation methods rely on the source data by joint optimization over source and target. Source-free methods replace the source data with source parameters by fine-tuning the model on target. Either way. the majority of the parameter updates for the model representation and the classifier are derived from the source, and not the target. However, target accuracy is the goal, and so we argue for optimizing as much as possible on target. We show significant improvement by on-target adaptation, which learns the representation purely on target data, with only source predictions for supervision (without source data parameter fine-tuming) In the long-tailed classification setting, we demon- strate on- on-target class distribution learning, which learns the (im)balance of classes target data. On-target adaptation achieves state-of-the-art accuracy and com- putational efficiency on VisDA-C and ImageNet-Skete Learning more on target can deliver better models for target.
INTRODUCTION
Deep networks achieve tremendous success on various visual tasks at the expense of massive data collection and annotation efforts. Even more data is needed when training (source) and testing (target) data differ, as the model must be adapted on the new data to maintain accuracy. To re- duce the annotation effort on new data, unsupervised domain adaptation (UDA) approaches transfer knowledge from labeled source data to unlabeled target data. Standard UDA requires simultaneous optimization on the source and target data to do so. However, this requirement may not be entirely practical, in that shifted or future target data may not be available during training. Furthermore, (re-) processenn source data during testing may be limited by computation, bandwidth, and privacy. Most importantly, it is the target data that ultimately matters for testing. In this work, we therefore turn our attention from source to target, and how to learn more from it.
Recent work adapts to the target data without the source data or even adapts during testing. However, these "source-free" and "test-time" approaches still rely heavily on the source parameters for fine- tuning. Source-free adaptation initializes from source parameters then optimizes without the joint use of source data (Liang et al., 2020; Li et al., 2020; Kundu et al., 2020). Test- time adaptation partially updates source parameters the target data while testing (Sun et al., 2019; Schneider et al., 2020; Wang et al., 2021). 1)21..... Such approaches reduce reliance on the source data, and can even improve accuracy, but have they made full use of the target data? Many of the model parameters are fixed (Liang et al., 2020: Schneider et al., 2020; Wang et al., 2021) or regularized toward the source parameters (Li al., 2020; Kundu et al., 2020). We investigate whether more can be learned from target, and more accuracy gained, by not transferring the source parameters.
target data
We propose on-target adaptation to unshackle the target representation from the source representa- tion. To do so, we (1) factorize the representation from the classifier and (2) separate the source parameters from the source predictions. By factorizing the representation from the classifier, we can train the representation entirely on the target data by self-supervision. Given this on-target rep- resentation, we can then supervise a new classifier from source predictions by distillation (Hinton et al., 2015), without transferring the source parameters. Not transferring parameters frees our target model from the constraints of the source architecture, so that we can experiment with distinct target architectures. In this way, that is more accurate and more efficient. In contrast to prior work on adaptation, this uniquely allows for learning 100% of the target model parameters target data, as illustrated by Figure 1.
even change the model size to optimize target-speeiff model

---

Under review as a conference paper at ICLR 2022
source
source
target
target
target
(a) UDA
(b) Source-Free
(c) On- On-Tat (ours)
Figure 1: Domain adaptation adjusts a model trained on source data for testing on target data. We contrast methods by their updates source and target. Unsupervised domain adaptation (UDA) jointly learns 50/50 on source/target. Source-free adaptation transfers source parameters, then se- lectively learns target. Our on-target approach learns 100% of the testing model parameters on target by neither sharing nor transferring source parameters, but instead distilling source predictions.
To realize our proposed factorization and separation, we employ contrastive learning, source- free adaptation, and teacher-student distillation. We initialize the target representation by self- supervision with contrastive learning. We turn the source model into a teacher model by source- supervise distillation. We lastly train the student
free adaptation, and then generate pseudo-labele model on the teacher supervision, starting from the target representation and new classifier param- eters, and repeat this teacher-student cycle by resetting the student classifier parameters between epochs. Contrastive learning has recently enabled self-supervised representations to compete with or even surpass supervised representations (Chen & He, 2020; Caron et al., 2020; He et al., 2020; Chen et al., 2020; Grill et al., 2020; Zbontar et al., 2021). We show it provides sufficient target representation.
Our experiments show on- on-target adaptation achieves Cfate-of-the-at accuracy and computational ef- ficiency on common domain adaptation benchmarks. For model accuracy, our method brings ~3% absolute improvement compared state-of-the-ar unsupervised and source-free domain adaptation methods on VisDA-C (Peng et al., 2017) and ImageNet Sketch (Wang et al., 2019a) while reducing 50%+ of parameters. For computation, our method reduces FLOPs by 50+% % and memory by 75+% for each forward pass of the target model. In the long-tailed classification setting, on-target class distribution learning equals the staeeoffthheaa learnable weight scaling (Kang et al., 2019) with- out needing source data. Ablation experiments support the generality of on-target representation learning across architectures, contrastive learning methods, losses, and amount of optimization.
Our contribution is to investigate whether the source data should be the primary source of target model parameters, and to propose an alternative: on-target adaptation. Our insight is that the source representation can be fully decoupled from source supervision. Domain adaptation normally empha- sizes the representation of source data, by either jointly optimizing on source data or transferring source parameters. On-target adaptation emphasizes the representation of target data instead, by distilling source predictions into a self-supervised target representation. We are the first to show this is feasible, new kind of source-free adaptation. Furthermore we show it improves accuracy and reduces computation on standard benchmarks like VisDA-C.
RELATED WORK
Adaptation On-target adaptation is unique in its decoupling of the target representation from the source representation. Prior adaptation approaches transfer the source representation the target, either by joint optimization or by initialization. To transfer the source model to a visually different target domain, unsupervised domain adaptation (UDA) learns a joint representation for both domains for visual recognition tasks, such as image classification (Tzeng et al., 2014), object detection (Chen al., 2018), semantic segmentation (Hoffman et al., 2016). Some of the most representative unsu- pervised domain adaptation ideas are 1) maximum mean discrepancy (Long et al., 2015; 2017); 2) moment/correlation matching (Sun et al., 2016; Zellinger et al., 2017); 3) domain confusion (Ganin & Lempitsky, 2015; Tzeng et al., 2017); 4) GAN-based alignment (Liu et al., 2017; Hoffman et al., 2018). All these UDA methods need simultaneous access to both source and target data. In practice, it might be impossible to meet this requirement due to limited bandwidth. computational power, or privacy concerns. Therefore, test-time training (Sun et al., 2019), source-free adaptation (Liang al., 2020), and fully test-time adaptation (Wang al., 2021) settings focus on adapting source
2

---

Under review as a conference paper at ICLR 2022
Sage St asaree model with around buch
Suge tew-timee cdletation os seuace moddl
Stage contruxtvve louming target dotain
Stage lagel with picudd-label
IslcHeatyy1
Mot',, Sirtlaar SWAY
Figure 2: On-target adaptation proceeds in four stages. Source data is colored in orange while target data is colored in blue. Stage 0 is the only stage to source data. Stage 3 is the stage that connects source and target: the target representation from stage is fine-tuned as a student model on the predictions of the teacher model from stage 1. Note that parameters are shared or transferred from source to target, so the target parameters are fully learned on target data.
model by fine-tuning on the target data without source data. Exciting concurrent work even adapts without the source model by only using source predictions (Liang al., 2021; Zhang et al., 2021; Wu et al., 2021). These "black-box" adaptation methods exclusively optimize teacher and student predictions, with distillation losses and output regularizers. While we likewise apply teacher-student learning, our work is complementary in using contrastive learning as a loss on the input for the stu- dent representation.
Semi-supervised learning Many UDA methods follow the practice of semi-superrisee learning, es- pecially pseudo labeling (Lee, 2013) which is to utilize the model prediction to generate supervision for the unlabeled images. The typical setup of unsupervised domain adaptation methods is to jointly optimize with ground truths on the source and pseudo labels on the target (Zhang et al., 2018; Choi al., 2019; Long et al., 2017: Zou et al., 2018). When source data annotations are not available, DeepCluster (Caron et al., 2018) and SHOT (Liang et al., 2020) further leverage weighted k-means clustering to reduce the side effects on noisy pseudo labels. Similarly, our method does not require access to labeled source data, while only relying on the target images with generated pseudo labels. In addition, our method heavily benefits from the contrastive learned target domain representation, which is treated as initialization to overcome the misleading of noisy pseudo labels.
Long-tailed recognition Long-tailed recognition tackles imbalanced class distributions in real- world data. Existing work divides into three groups: 1) re-balancing the data distribution (Chawla et al., 2002; Han et al., 2005; Shen et al., 2016; Mahajan et al., 2018); ; 2) designing class-balanced losses (Cui et al., 2019; Khan et al., 2017: Cao et al., 2019; Khan et al., 2019; Huang et al., 2019; Lin et al., 2017: Shu et al., 2019; Ren et al., 2018; Hayat et al., 2019); 3) transfer learning across classes (Yin et al., 2019; Liu et al., 2019). All of these methods address imbalance by altering train- ing, so that the model may learn more balanced features, and a classifier that covers common (head) and rare (tail) classes. We instead adapt the classifier for long-tailed recognition during testing.
METHOD: ON-TARGET ADAPTATION
The goal of the proposed on-target adaptation is to tackle domain shift during the test-time with only source model, without the access of annotation and source data. Specifically, the supervised model with source parameter ((::)) trained on source images x and labels y needs to generalize on unlabeled target data z' when an unneglectable domain shift happened. Our on-target adaptation (Figure 2) is proposed to obtain target model parameter 0c purely during test-time.
Stage 0 (source): train model with labeled source data We train a deep ConvNet and learn source parameter H by minimizing vanilla cross-entropy loss L(j", yy) on labeled source data (a* (x).y"). Specifically, L(Ø"; y²)
(v)( log(p(ū2) for the predicted probability ø of class c, where target probability yde is 1 for the ground truth class gt and 0 for the rest.
Stage (teacher): adapt source model without source data We update the source parameter 0* during testing to minimize information maximization (InfoMax) loss (Gomes et al., 2010). Specif- ically, InfoMax loss augment entropy loss Lent -E.((E) log(p(α)) with diversity objective Laii = D I| [c)) - log(C). where DxL. indicates the KullbackL divergence, Ic is an all-one vector with C' dimensions. Here Alc indicates the target label vector with evenly distributed y probabilities, where Laiv is propose to enforce the global diversity over classes.
3

---

Under review as a conference paper at ICLR 2022
"Weak"
"Suay" Augrecaiiics
Stage 0
Stage
Stage 2
Source Model
Afxpusion
The tire
Teacher Model
Student Model
Commetu Laaming
Scratch Model
Figure 3: Teacher-student (stage 3) learning in method. Transfer learning between the teacher (orange) and the student (blue), where pseudo labels are generated on the The model is trained the strongly-and -aungggedd target data match the pseudo labels.
images.
As for the parameters to optimize over, we follow the motivation of decoupling the representation and classifier. When the classifier frozen, the goal of optimization deriving proper target features from the source model. In particular, keep the classifier the same on both source and target domain, and obtain A by the gradient of the test-time objective (InfoMax), to update the representation part of model parameter 0"
mitigate domain shift by
Stage 2 (student): initialize target model with contrastive learning Instead of fine-tuning from source model, we choose to initialize the target feature purely from target data. Benefiting from the recent advances contrastive learning methods, we train an unsupervised model with purely unlabeled target images. Specifically, we initialized target representation via improved momentum contrast learning (MoCo v2) (He et al., 2020; Chen et al., 2020). It is worth noting that our method does not require a specific contrastive learning method. In other words, the default MoCo v2 could be easily replaced by a more recent self-supervised learning model, such as SWAV (Caron et al., 2020), SimSiam (Chen He, 2020), Barlow Twins (Zbontar et al., 2021). Such modular design makes it easier to benefit from the latest advance in contrastive learning. We have performed an ablation study on the choice of contrastive learning method Section 4.5.
Stage 3 (teacher-student): transfer knowledge from teacher to student We use the adapted source model [(:; + Δ) as the initial teacher model to generate pseudo labels y" on unannotated target images aff Then we fine-tune the student model [:; 0*) initialized by contrastive learning on target data with cross-entropy loss C(j', y²*) = _ log log(p((()). Specifically, we use normal distribution with a mean of zero and standard deviation of 0.01 for the classification head, since contrastive learned model does not contain classifier. The teacher would be replaced with the latest student to gradually denoise pseudo labels for the subsequent phase. Meanwhile, the contrastive learned model would re-initialize the student feature to eliminate the accumulated errors from im- perfect pseudo labels. In other words, the student model would start over one more time with only newer pseudo labels for the next transferring phase.
Figure 3 illustrates the procedure of transferring the knowledge from teacher student. Specifi- cally, the interaction between teacher and student models benefits from consistency regularization and pseudo-labeling, inspired by recent semi- superiiiee learning approach called FixMatch (Sohn et al., 2020). During the transferring, we augment the target images with random cropping, ran- dom flipping, and AutoAugment with ImageNet policy, as "strong" augmentation,, while the "weak" augmentation is the combination of resizing and center cropping when generating pseudo labels. Relying on the assumption that the model should generate similar predictions on data-augmented versions of the same image (Bachman et al., 2014; Sajjadi et al., 2016; Laine & Aila, 2016), consis- tency regularization enforces the cross-entropy loss between student output on strongly- augmentd images and teacher output augmentd images.
4 EXPERIMENTS
4.1 SETUP
Datasets We evaluate our method on both domain adaptation and long-tailed recognition bench- marks, including VisDA-C (Peng et al., 2017), Office Home (Venkateswara et al., 2017), Sketch (Wang et al., 2019a), ImageNet-LT (Liu et al., 2019), and iNaturalist-18 (Van Hom et al., 2018). Figure 5 presents some of example images illustrate domain shifts.
4

---

Under review as a conference paper at ICLR 2022
Metric We report top-1 accuracy (denoted acc.) on the whole dataset for all datasets. On VisDA- C, we additionally report the percentage accuracy of each category and the corresponding average of all categorical accuracies (denoted as avg.), due to the imbalanced distributed label space. On Office Home, we calculate the average for all kinds of domain shifts as the summary number for each method. On long-tailed recognition benchmarks, on many-shot (more than 100 samples), medium-shot (20-100 samples), and few-shot (less than 20 samples) following the evaluation protocol from Liu et al. (2019); Kang et al. (2019).
additionally report percentage accuracy
Baselines We choose the most recent fully test-time adaptation method, TENT (Wang et al., 2021), and source-free adaptation framework. SHOT (Liang et al., 2020), as our "online" and "offline" adaptation baselines. TENT does not alter training. while SHOT minimally customizes the source architecture and training. Entropy minimization the target optimization objective for both TENT and SHOT. SHOT additionally regularizes optimization by the information maximiza- tion (InfoMax) loss (Gomes et al., 2010; Shi & Sha, 2012; Hu et al., 2017) to augment entropy minimization each sample with diversity maximization across samples. Following SHOT, we therefore augment TENT into TENT-IM by including this regularization. Alongside their role as baselines, these methods can serve teacher models for stage 1. We also compare with unsupervised domain adaptation (UDA) baselines, including DANN (Ganin & Lempitsky, 2015), DAN (Long al., 2015), ADR (Saito et al., 2018), CDAN+E (Long et al., 2018), CDAN+BSP (Chen et al., 2019), CDAN+TN (Wang et al., 2019h), SAFN (Xu et al., 2019), SWD (Lee et al., 2019), DSBN+MSTN (Chang et al., 2019). STAR (Lu et al., 2020). It worth noting that all these UDA methods are fine-tuning from the ImageNet pretrained ResNet-101 source model, with access to both source and larget data. TENT-IM and SHOT likewise initialize their representation from the source model. In contrast, our method trains ResNet- 18 from scratch, and entirely on the target data, by contrastive learning for the representation and teacher-student learning for the classification.
For long-tailed recognition, we choose learnable weight scaling (LWS) (Kang et al., 2019) as the train-time baseline. LWS first decouples the full model into representation and classification, and then only re-scales the classifier parameters with class-balanced sampling. Our on-target class dis- tribution learning extends LWS to test-time, by re-scaling the parameters on unlabeled target data.
Architecture For comparability with state-of-the-art models, we choose 18/50/101-layer ResNet models (He et al., 2016) for both main results and ablation studies. When reproducing the prior works, keep the architecture the same, for example, the cright-normalizatton (Salimans & Kingma, 2016) augmented ImageNet pretrained ResNet-101 for SHOT (Liang et al., 2020).
4.2 IMPLEMENTATION
Our implementation is in PyTorch (Paszke et al., 2019) and depends the VISSL (Goyal et al., 2021), MMClassification (Contributors, 2020), and Weights & Biases (Biewald, 2020) libraries. The code is attached and will be released for publication.
Stage 0 (source) We train residual networks (He et al., 2016) with various depths (including 18, 50, 101), and initializations (ImageNet pretraining Kaiming init (He al., 2015) when training from scratch). We optimize cross- cooso-y loss by SGD with an initial learning rate 0.1, momentum 0.9, weight decay 0.0001, batch size 256. We do not apply label smoothing (Müller et al., 2019) except for SHOT (Liang et al., 2020), as it specifically includes it. We adopt the standard data augmentation pipeline from ImageNet training, such as random cropping, random flipping, and color jitter. We choose ImageNet statistics as the default input mean and variance for all models.
Stage 1 (teacher) We experiment with three types of teacher models: 1) source-only. 2) TENT- IM (Wang et al., 2021), 3) SHOT (Liang et al., 2020). To optimize this altered loss, we choose SGD with learning rate 0.0001, momentum 0.9, and weight decay 0.0001. addition to batch normaliza- tion (Ioffe& & Szegedy, 2015), we also update convolutional layers except the final classification layer. As for SHOT, We execute the authors' open-sourced codebase with the same arper-parameters for various architectures (ResNet-50 and ResNet-101) 101), initialization (from scratch and ImageNet pre- train), and domain shifts (train to val/test splits on VisDA-C).
Stage 2 (student) We experiment with two designs as students: 1) source-only, 2) contrastive learning. Specifically, we leverage some of off-the- the-hhf contrastive learning methods to initialize target-domain representation, such MoCo v2 (Chen al., 2020), SimSiam (Chen & He, 2020),
5

---

Under review as a conference paper at ICLR 2022
#1 source-free #2 contrastive
#3 teacher VisDA-C
Require: model A source model, for predictions
method
adaptation
learning
student
-Vval
-Rest
Require: image transform fu weak augmentation
Require: model fr 8 lurget model, for representation
source-only free iterated distillation with adaptation
21.8
23.9
Require: image transform (, # strong augmentation
31,4
34.3
fo fr ft source model for the first teacher
28.3
31.8
for 4- to N do # for each epoch
43.3
46.0
f. + Js # initialize by contrastive learning
on-target adaptation
29.1
33.9
for b to B do 4 for each mini-batch
49.9
512
y + fi 1 (f (a)) teacher
(t,t,))) # student
Table I: Each stage of our on-target adaptation improves target accuracy. Contrastive learning (stage 2), for fitting the representation on target data alone, helps whether or not the teacher is adapted (stage 1).
i - Lossty. y ) hard-label cros-entropy
end
end for
return fN # student model for lesting
Figure 4: On-target pseudo-code.
method
network
plane
beyel
bus
borse
knife
meyel
person
plant
skibrd
train
trunk
g.g.
ADR (Sailo al., 2018)
R1GIP
94.2
48.5
84.0
72.9
90.1
74.9
92.6
72.5
80.8
61.8
82.2
28.8
73.6
73.8
CDAN+E (1 Long et al., 2018)
R10IP
85.2
66.9
83.0
50.8
84,2
74.9
88.1
74.5
83.4
76.0
81.9
38.0
73.9
71.0
CDAN-BSP (Chen et al., 2019)
R101P
92.4
61.0
81.0
57.5
89.0
80.6
90.1
77.0
84.2
77.9
82.1
38.4
75.9
73.4
SAP al., 2019)
R10IP
93.6
61.3
84
70.6
94,1
79.0
91.8
79.6
89.9
55.6
89.0
24.4
76.1
75.6
SWD(Lee al., 2019)
R101P
90.8
82.5
81.7
70.5
91.7
69.5
86.3
77.5
87.4
63.6
85.6
29.2
76.4
75.6
DSBN-MSTN (Chang et al., 2019)
RIOIP
94.7
86.7
76.0
72.0
95.2
75.1
87.9
81.3
91.1
68.9
88.3
45.5
80.2
79.2
STAR (Lu et al., 2020)
R101P
95.0
84.0
84.6
91.6
91.8
85.9
78.4
94.4
84.7
87.0
42.2
82.7
80.4
TENT IM (Wang al., 2023)
R50S
58.9
40.1
50.2
23.6
22.6
25.3
29.8
24.8
22.9
30.2
45.1
20.1
32.8
31.4
TENT-IM Ours
R508
90.1
66.0
75.2
41.3
29.2
11.2
$7.0
60.8
40.1
51.1
73.6
23.8
51.6
50.9
TENT-IM Ours
R18S
90.5
65.2
26.7
12.9
51.6
44.2
46.0
71.
24.0
50.9
49.9
SHOT (Lin et 2020)
R10IP
94.6
86.6
79.5
55.6
93.6
96.1
79.8
80.7
89.2
89.0
86,
57.1
82.3
77.8
SHOT Ours
R18S
96.0 89.5
95.9
94.2
93.8
89.9
89.1
58.2
85.9
82.8
Table 2: Classification accuracy of on-target adaptation on VisDA-C (validation) across all cate-
gories and averaged over classes (avg.) and images (acc.). R18/50/101S denotes Res/ett-18/50/10 randomly initialized from scratch and R R18/50/101P denotes ResNet-18/50/101 pretrained on Ima- geNet.
SWAV (Caron et al., 2020), Barlow Twins (Zbontar et al., 2021). Compared to their training recipes on ImageNet, we have more epochs on VisDA-C val/test with the same batch size, learning rate, data augmentation, and model architecture, to make the training procedure longer with the smaller amount of images.
Stage 3 (teacher-student) By default, the whole knowledge distillation consists of three phases, where each phase has 10 epochs to train the student model with the hard pseudo label. The student would be reset to the contrastive model to avoid error accumulation at the beginning of every phase. The teacher would be replaced with the latest student before starting the next phase, so that the qual- ity of pseudo-labeling could be improved gradually. We utilize SGD with an initial learning rate 0.01, momentum 0.9, weight decay 0,0001, batch size 256, and cosine annealing scheduler (Loshchilov & Hutter, 2016).
4.3 ON-TARGET ADAPTATION
Table I reports reports the change in target accuracy for each stage of on-target adaptation on VisDA- C. The source model is ResNet-50 trained from scratch. The target model is ResNet-18. 18.. Source-free adaptation is done by TENT-IM. In teacher-student learning, the student, with either the source (teacher) or our on-target (contrastive) representation, is trained on the predictions of the teacher. This is repeated for multiple epochs. The on-target representation (stage #2) improves accuracy with and without source-free adaptation of the teacher, as it only depends on the target data.
VisDA train + val Table 2 compares our method with state-of -fh--e--rttt-ttt- unsupervised (upper part) and test-time (lower part) domain adaptation approaches from VisDA-C train to val splits. The proposed on-target adaptation significantly improves the existing test-time adaptation methods. It is worth noting that all these existing methods need to keep the same architecture when joint-training or fine- tuning the source model. On the contrary. our method could utilize a much more lightweight model, such as ResNet-18 18 as shown in this table. For example, our method brings 18+ points im- provement compared to source-free adapted teacher TENT-IM, while reducing over 50% parameters and runtime flops, and 75% memory consumption at each feed- forward of target model.
VisDA train + test Table 3 compares method with -ftte-tf-the-ar unsupervised (upper part) and test-time (lower part) domain adaptation approaches from VisDA-C train to test splits. Similar to table 2, our method dramatically improves the performance of all kinds of teacher models.

---

Under review as a conference paper at ICLR 2022
source-only ours
TENT-IM
SHOT
network avg. acc. avg. acc. R50S
network
avg. acc. avg. acc.
network avg. acc. avg. acc.
22.1 23.9 30.9 33.9
R50S
34.2 34.3 49.3 51.2
RIOIP 89.3 88.4 91.7 91.6
R50P
34.4 37.7 39.8 43.5
R50P
60.4 62.4 74.8 77.7
R50P 76.4 78.0 81.1 83.0
Table 3: Classification accuracy of our method supervised by three teachers: source-only, SHOT, and TENT-IM on VisDA-C (test). R50/101S denotes ResNet-50/101 randomly initialized from scratch and R50/101P denotes ResNet-50/101 pretrained on ImageNet.
method
network
accuracy
methed
netwerk
accuracy
Anisotropic (Mishra al., 2020)
ResNet-50
24.5
DANN (Ganin & Lempitsky, 2015)
ResNet-50
$7.6
Debiased (Li al., 2021)
ResNet-50
28.4
DAN (Long at., 2015)
ResNet-50
56.3
Crop (Hermann et al., 2020)
ResNet-50
30,9
CDAN+BSP (Chen et 2019)
CDAN+E (Long et al., 2018)
ResNet-50
ResNet-50
65.8
66.3
RVT (Mao et al., 2021)
DeïT-B
36.0
SAFN (Xu al.., 2019)
ResNet-50
67.3
TENT-IM (Wang et ul., 2021)
ResNet-50
35.6
CDAN+TN (Wang al., 2019b)
ResNet-50
67.6
TENT-IM + Ours
ResNet-18
37.5
TENT-IM (Wang et al., 2021)
ResNet-50
TENT-IM Ours
ResNet-50
40.5
TENT-IM Ours
ResNet-18
56.8
Table 4: Adaptation on ImageNet-Sketch.
Table 5: Adaptation on Office-
In the following two paragraphs, we discuss the potential application of on-target adaptation without contrastive learming. When test-time data is not sufficient enough to finish the contrastive learning, we could skip contrastive learning on target data (stage 2) of the proposed method. In other words, we directly fine-tune the target model initialized by the source model. We believe that adaptation performance could be further improved once the contrastive learning could no longer be data-hungry or target domain data could be abundant.
ImageNet + Sketch Table 4 reports the empirical results on generalization regarding Ima- geNet/Sketch as source/target domain. For on-target adaptation, we try two student models: model as same as the teacher model (ResNet-50) and small supervised model pretrained on ImageNet (ResNet-18). Our method additionally brings ~5/3% improvements compared the teacher model with the same/shallower student models, where the teacher model, TENT-IM, already outperforms the previous state-of-the-art by ~5%
Office Home Table 5 compares our method with state-of-the-arr unsupervised (upper part) and test- time (lower part) domain adaptation approaches for the various domain shifts in Office Home, Our method vanccs the average accuracy over all domain shifts of teacher model (TENT-IM) by 3%.
4.4 ON-TARGET CLASS DISTRIBUTION LEARNING
In this section, we argue for calibrating classifier during the test-time, without any modification on training procedure, aiming at long-tailed recognition task. Here we treat the long-tailed data as the source domain while regarding the class-balanced data as the target domain. During training, instance-balanced sampling provides generalizable representation to start with. Then the classifier is re-scaled during the test-time on the class balanced data.
First, we train the source domain model with the balance sampling, which samples each sample with the same probability. In this way, the learned classifier has a higher prior probability on the head compared the tail. Then we calibrate the parameters of the classifier while freezing the feature with test images and pseudo labels, following the practice of our on-target adaptation. It worth mentioning that we do not utilize contrastive learning to re-initialize the representation on target data or tune the feature part in teacher-student (stage 3). The major reason for such choice is to follow the practice of LWS (Kang al., 2019), which points out that the domain shift only exists within class distribution that only classifier needs to be calibrated.
The empirical results indicate that our method could automatically calibrate the categorical prior and adaptive fit the test data distribution without the access of training data. Table 6 demonstrate that our test-time on on-target adaptation could achieve comparable performance compared to of the-art training-time time methods on ImageNet-LT and iNaturalist-18 18 datasets. We report results for two popular ConvNets, ResNet-50 and ResNet-100 101, as teacher models trained on long-tailed data. Our method achieves comparable overall performance with the train-time method (LWS) on both

---

Under review as a conference paper at ICLR 2022
|Naturafis188
ImageNet-LT
method
muny
medium
few
acc.
many
medium
few
acc.
ResNet-50
72.2
63.0
57.2
61.7
64.0
33.8
5.8
41.6
+ LWS (Kang et al., 2019)
65.0
66.3
65.9
57.1
45.2
29.3
47.7
+ On Target Class Distribution
64.2
66.3
65.9
65.9
55.7
46.0
28.6
47.4
ResNet- 101
75.9
66.0
59.9
64.6
66.6
36.8
7.1
44.2
+ LWS (Kang et al., 2019)
69.6
69.1
67.9
68.7
60.1
47.6
31.2
50.2
+ On- Targtt Class Distribution
66.5
69.1
68.3
68.5
58.9
48.7
31.8
50.3
Table 6: Comparing our method performance with Iearnable weight scaling (LWS) on long-tailed benchmarks including 1Nauuralis188 and ImageNet-LT. Note that LWS adapts during training while our method can adapt during testing, which is more efficient.
imagenet source
SHOT
ours
imagenet source
TENT
ours
network
pretrain
only
avg.
acc, avg.
acc.
network
pretrain only
acc. avg. acc.
ResNet-50
x
X
49.7
48.3 69.1 67.3
ResNet-50
x
X
32.8 31.4 50.9 49.9
ResNet-50
/
x
75.0
74.5 77.8 78.6
ResNet-50
/
x
60.9
60.2 75.1 73.8
ResNet-101
/
x
82.3
77.8 85.9 82.8
ResNet-18
x
x
34.0
33.1 51.4 51.4
ResNet-101
49.9
55.5
60.0 65.6
ResNet-50
x
17.8 21.8 22.3 29.1
Table 7: Classification accuracy on VisDA-C (validation). "Imagenet pretrain" indicates whether we utilize ResNet pretrained on ImageNet at stage O. "Source only" indicates whether we skip test- time adaptation (stage I) and directly use the source model to generate pseudo labels at stage
datasets. Compared to the vanilla ResNet-50 and ResNet-101, our fully test-time method signifi- cantly improves the overall performance by a large margin. Considering the accuracy of few-shot (less than 20 samples) categories, our method outperforms the train-time practice three out of four cases, extending the usage scenarios train-time long-tailed recognition methods.
4.5 ABLATION STUDIES
Stage O: network & initialization The upper part of Table 7 presents the numbers of SHOT/TENT- IM with ResNet in various depths (18, 50, 101) and initializations (from scratch, ImageNet pretrain). We observe that our method consistently improves the teacher models with various model archi- tectures and initializations, which indicates the usability and versatility of the proposed framework. When adapting from synthetie to real-world domains, the ImageNet pretrained model should not be utilized to start with, due to the learned inductive bias of its parameters. Therefore we also experiment on training from scratch for teacher models. Larger capacity does not lead to better generalization when training from seratch. On the contrary, We observe that deeper ImageNet pretrained ConvNet provides a stronger inductive bias from the beginning.
Stage 1: test-time adaptation The lower part of Table 7 presents the numbers of source-only models as teacher, without any source-free adaptation (TENT-IM/SHOT). The final accuracy after teacher- student suffers from the poorer quality of the initial pseudo label. ImageNet pretraining could al- leviate such phenomenon, but the numbers with test-time adapted teachers are still significantly better than the source-only ones. Comparing these empirical results with table 2, TENT-IM boosts the initial/final accuracy by 15.0/28.6 points, while SHOT brings 32.4/25.9 points improvement. In a word, test-time adaptation should be leveraged in preparation for trustworthy pseudo labels.
Stage 2: on-target feature The left part of Table 8 presents the ablation study of student architecture and initialization. When the student model not initialized on target data, such as source data (VisDA-C train) or the external large dataset (ImageNet), the overall accuracy drops 4+ points, which indicates the necessity of on-target feature learning. We also ablate the same contrastive learning algorithm (MoCo v2) different data source, such as VisDA-C val and ImageNet. The empirical results indicate that more external data does not bring any advantage, which echos our statement on the target-specific representation learning.
Stage 2: contrastive learning learning frameworks, including SwAV, SimSiam, Barlow Twins. We train these contrastive learned models with the same number of epochs to have a fair comparison. Compared to the performance of the teacher model, all these learned features bring a noticeable improvement, achieving the com-
The right part of Table 8 presents the results with various contrastive
8

---

Under review as a conference paper at ICLR 2022
method
avg.
acc.
method
avg. acc.
TENT
32.8 31.4
Ours (MoCo) 50.9 49.9
VisDA main
VisDA-C train
45.3 43.3
SwAV
50.5 49,4
ImageNa
ImageNet
47.3
46.2
SimSiam
48.7 47.6
ImageNet (MoCo)
46.6
45.5
Barlow Twins 46.3 44.8
ViLDA Icst
Stetch
Offficc Home
Table 8: Classification accuracy on VisDA-C (validation). Left side: Ablation results on the student model with various initial- ization. Right side: Ablation results the contrastive learning method using MoCo, SwAV, SimSiam, and Barlow Twins.
Figure 5: Example images from VisDA-C, ImageNet, ImageNet- Sketch, and Office-Home.
teacher soft phase 0 phase
phase 2
phase 3 phase 4 phase 5
phase 6 phase 7 phase 8 phase 9
X
32.8
44.2
48.2
50.9
52.7
54.3
56.0
57.0
58.1
58.9
TENT
/ 32.8
44.3
53.5
56.4
59.9
61.4
63.6
64.2
65.1
65.6
X
82.3
84.8
85.5
85.9
86.2
86.3
86.3
86.3
86.2
86.3
SHOT
82.3
84.7
85.2
85.6
85.2
85.7
85.0
85,4
84.9
85,2
Table 9: Classification accuracy of our method on VisDA-C (validation). "Soft" indicates that the hard-label cross-entropy loss is replaced with the soft-label KL divergence loss for each even- numbered phase. Note that our default number of phases (phase 3) is highlighted.
parable numbers with the reference performance of MoCo v2. We observe that our method is not sensitive to the choice of contrastive learning method. this way, our on-target adaptation could be further improved by introducing more advanced contrastive learning approach in the future.
Stage 3: more phases Table 9 presents the detailed numbers for each phase during sncceer-ttuennt We observe that the first phase already significantly outperforms the test-time adapted teacher model, which is also the state-of-the-an practice. The following several phases gradually improve the re- sults, taking the last generation student the next teacher. Considering the pecuraccuracy trade-off, we choose to have three phases as our default setup, even though more phases could lead to better result. For example, 9-phase (3x) optimization brings up around 9 points improvement compared to our default 3-phase (1 xx) one with TENT-IM as the initial teacher model.
Stage 3: soft label Table 9 presents the ablation study the design choice of loss function. Our default setup only utilizes hard labels with cross-entropy loss. Actually, our framework also benefits from the soft label with KullbackLeibler divergence loss, following the popular practice of knowl- edge distillation (Hinton et al., 2015). We observe that the mix of both hard and soft label bring up the best performance. we replace the cross-entropy loss (hard label) with KullbackLeibler diver- gence (soft label) for the even number of phases. We set the number of epochs as one for all these interpolated soft label phases for a more computational-frit -rienndlyly practice. The known drawback is that the soft label part typically needs specific tuning learning rate, loss weight, temperature, and so on. Existing works (Berthelot et al., 2019b;a; Sohn et al., 2020) discuss sharpening (temperature) and thresholding (confidence threshold) to improve the performance of semi-supervised learning. divergence loss without bells and whistles like
Instead, we only ablate the default KullbackL temperature and confidence threshold. Our default training objective chooses be the most robust hard label with cross-entropy criterion for all the other experiments.
5 CONCLUSION
Domain adaptation is itself adapted to many different needs: unsupervised domain adaptation jointly optimizes over labeled source and unlabeled target data, source-free adaptation adapts target given source parameters instead of source data, and test-time adaptation even adapts while making predic- tions. Across each of these varieties. the source comes first. The target representation either aligned to the source representation or it is initialized from it by transfer learning. On-target adap- tation departs from this standard practice by transferring the source predictions without the source representation. This decoupling is unconventional, but useful, because it enables learning all of the target model parameters on the target data. Given enough target data, on-target adaptation improves accuracy by learning the model for target data on target data.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. arXiv preprint arXiv: 1412.4844 2014.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmenta- tion anchoring. arXiv preprint arXiv:1911.09778 2019a.
David Berthelot, Nicholas Carlini, lan Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. NeurIPS, 2019b.
Lukas Biewald. Experiment tracking with weights and biases. https: wwwwwwww..bb.... wandb. com/, 2020.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. arXiv preprint arXiv: (201::9900.071111 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In ECCV, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 09882, 2020.
Woong-Gi Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. Domain-speciin batch normalization for unsupervised domain adaptation. CVPR, 2019.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. JAIR, 2002.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint 1..i::2011.105660 2020.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint 2xXiv:2003.04297,, 2020.
Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs. discriminabil- ity: Batch spectral penalization for adversarial domain adaptation. In /CML,2015
Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Domain adaptive faster r-cnn for object detection in the wild. In CVPR, 2018.
Jaehoon Choi, Minki Jeong. Taekyung Kim, and Changick Kim. Pseudo-labeling curriculum for unsupervised domain adaptation. arXiv preprint arXiv: 1908.0020.. 2019.
MMClassificction Contributors. Openmmlab's image classification toolbox and benchmark.
https: htttps://githin /tttttb..ttttt////// com/ oppn mmlab/ haillabmmmccie http///maaassiaaaaaaaiaaaaaaaaaa 2020.
Yin Cui, Menglin Jia, Tsung-Y Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In CVPR, 2019.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015.
Ryan Gomes, Andreas Krause, and Pietro Perona. Diseriminative clustering by regularized informa- tion maximization. In NeurlPS, 2010.
Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. 2021.
Vissl. https:: /////tttt/tg htw..bb...uugggggggg//ggg..bebb. http///aceebookeseerrcc//tttttia
Bran-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07770 07733, 2020.
10

---

Under review as a conference paper at ICLR 2022
Hui Han, Wen- Yuan Wang, and Bing-Huan Mao. Borderline-smote; new over-sampling method in imbalanced data sets learning. In ICIC, 2005.
Munawar Hayat, Salman Khan, Waqas Zamir, Jianbing Shen, and Ling Shao. Max-margin class imbalanced learning with gaussian affinity. arXiv preprint arXiv: 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, June 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsu- pervised visual representation learning. In CVPR, 2020.
Katherine L Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. In NeurlPS, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1508 150.00053, 2015.
Judy Hoffman, Dequan Wang, Fisher Yu, and Trevor Darrell. Fens in the wild: Pixel-level adversar- ial and constraint-basee adaptation. arXiv preprint arXiv:1612.026499 2016.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In ICML, 2018.
Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In ICML, 2017.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Deep imbalanced learning for face recognition and attribute prediction. TPAMI, 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR, 2019.
Salman Khan, Munawar Hayat, Syed Waqas Zamir, Jianbing Shen, and Ling Shao. Striking the right balance with uncertainty. In CVPR, 2019.
Salman H Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A Sohel, and Roberto Togneri. Cost-sensitive learning of deep feature representations from imbalanced data. TNNLS, 2017.
Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al. Universal source-free domain adaptation. In CVPR, 2020.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint 161.022610 2016.
Chen-Yu Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. In CVPR, 2019.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep
neural networks. In ICML Workshop on challenges representation learning. 2013.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In CVPR, 2020.
Yingwei Li, Qihang Yu, Mingxing Tan, Jieru Mei, Peng Tang, Wei Shen, Alan Yuille, and Cihang Xie. Shape-texture debiased neural network training. ICLR, 2021.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In /CMI. 2020.
11

---

Under review as a conference paper at ICLR 2022
Jian Liang, Dapeng Hu, Ran He, and Jiashi Feng. Distill and fine-tune: Effective adaptation from black-box source model. arXiv preprint arXiv:2104.01539 2021.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
Ming-Y Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. arXiv preprint arXiv: S.53v:17030008800 00848, 2017.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In CVPR, 2019.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In ICML, 2015.
Mingsheng Long, Han Zhu. Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In ICML, 2017.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial
domain adaptation. In NeurIPS, 2018.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.000 03983, 2016.
Zhihe Lu, Yongxin Yang, Xiatian Zhu, Cong Liu, Yi-Zhe Song, and Tao Xiang. Stochastic classifiers for unsupervised domain adaptation. In CVPR, 2020.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018.
Xiaofeng Mao. Gege Qi, Yuefeng Chen, Xiaodan Li, Shaokai Ye. Yuan He, and Hui Xue. Rethinking the design principles robust vision transformer. arXiv preprint arXiv:2105.07926, 2021.
Shlok Mishra, Anshul Shah, Ankan Bansal, Jonghyun Choi, Abhinav Shrivastava, Abhishek Sharma, and David Jacobs. Learning visual representations for transfer learning by suppressing texture. arXiv preprint arXiv:2011.01900 2020.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? arXiv preprint arXiv: 1rXXv::906.00222. 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurlPS, 2019.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. VisDA: The visual domain adaptation challenge. arXiv preprint 170:v:1710006924, 2017.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, 2018.
Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Adversarial dropout regular- ization. In ICLR, 2018.
Mchdi Sajjadi. Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. arXiv preprint arXiv:1606.04500 045866 2016.
Tim Salimans and Diederik Kingma. Weight normalization: A simple reparaattrrriinntioonn to accel- erate training of deep neural networks. arXiv preprint arXiv: 1602.0880, 2016.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. arXiv preprint 2006v:2006. 1697)) 2020.
12

---

Under review as a conference paper at ICLR 2022
Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In ECCV, 2016.
Yuan Shi and Fei Sha. rnformation-theorriin learning of discriminative clusters for unsupervised domain adaptation. In ICML, 2012.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight- net: Learning an explicit mapping for sample weighting. arXiv preprint arXiv:1902.0773 2019.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervisee learning with consistency and confidence. arXiv preprint orXiv:2001.07688 2020.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In AAAI, 2016.
Yu Sun, Xiaolong Wang. Zhuang Liu, John Miller, Alexei A Efros, and Moritz Hardt. Test-time training for out-of-distribttion generalization. arXiv preprint arXiv: 13X2:11 2019.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, 2017.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In NeurlPS, 2019a.
Ximei Wang, Ying Jin, Mingsheng Long, Jianmin Wang, and Michael I Jordan. Transferable nor- malization: Towards improving transferability of deep neural networks. In NeurlPS, 2019h.
Kunhong Wu, Yucheng Shi, Yahong Han, Yunfeng Shao, Bingshuai Li, and Qi Tian. Domain adaptation without model transferring. arXiv preprint arXiv:2107. 10/74, 2021.
Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In ICCV, 2019.
Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learn- ing for face recognition with under-represented data. In CVPR, 2019.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.032 2021.
Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger-Plate Central moment discrepancy (cmd) for domain-invariant representation learn- ing. arXiv preprint arXiv:/702.088) 2017.
Haojian Zhang, Yabin Zhang, Kui Jia, and Lei Zhang, Unsupervised domain adaptation of black-box source models. arXiv preprint arXiv:2101.02839 2021.
Weichen Zhang, Wanli Ouyang, Wen Li, and Dong Xu. Collaborative and adversarial network for unsupervised domain adaptation. In CVPR, 2018.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via
self-training. In ECCV, 2018.
13


Reviews:
[{"review_id": "q3LaJWbSDYU", "reviewer": "Reviewer_QwsH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* I like the source-free adaptation problem setting, I think it is very practical\n* Combining model fine-tuning on target (or invoking a SOTA source free method) generating a teacher for distillation and using contrastive self-supervised student is a natural idea to try and I think the community should be aware of the good results\n* SOTA improvements in some cases are welcomed\n* extensive experiments and ablations\n* code included\n\nCons:\n* novelty seems a bit limited, the paper combines some standard ideas (target adaptation, distillation, contrastive pre-train on target, fix-match), the main novelty seems to be in the system - combining these for source free adaptation\n* writing could be improved, sometimes it is hard to follow the mass of details and some conclusions become less clear\n* why in Table 1 some numbers are missing and X is drawn? did not find an explanation in the text\n* in Table 2 \"Ours\" is only shown on top of TENT or SHOT, how about without source free sota methods at play? is it in table 3? but there the numbers are a bit low and disappointing...\n* Tables 4 & 5 only do TENT / TENT + ours, I looked briefly in SHOT and saw higher numbers for Table 5 as it seems, though there the proposed method did not hit sota anyway, how about adding SHOT also to table 4 (and 5)?\n* Presentation (more minor): ablation tables are better re-designed, e.g. look at table 7 the message there was to show that there is improvement in every row reading from left to right, but the V and the X leads the reader to wonder if adding more things improves or not in columns and it is actually opposite in that table (but was so in earlier tables). In short - please make the style consistent, it is really hard to follow. This was just an example.", "summary_of_the_paper": "The paper proposes a multi-stage approach for source-free domain adaptation, namely when source data is not available and one can only use the model pre-trained on the source and adapt it based on unlabelled target domain data. The paper advocates for learning on target not via fine-tuning the source model, but rather distill from it initializing on source using contrastive learning (the authors use Moco V2). The source model is also adapted on target via InfoMax loss before being used as the teacher for the distillation. The authors propose to use the FixMatch strategy (with strong and weak aug taken from AutoAugment) for the distillation. The authors show some gains when combining with SOTA source-free adaptation methods: TENT and SHOT. The authors also show that improved accuracy can be obtained with much smaller models trained on target using their method (e.g. replacing Res50 or Res101 with Res18 on Visda-C. Extensive experiments, ablations, and results are provided.", "summary_of_the_review": "I see a balance between pros and cons in this paper, yet I do have some positive tendency as the method itself is clear and intuitive to me and can boost performance in some interesting cases (not always, but still). Therefore, at least until I see some more clarifications from the authors and other reviews, I prefer to rank it slightly above borderline, with an intention to revisit and yet in a positive spirit of leaning towards acceptance :-)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fYaFiOy7Tit", "reviewer": "Reviewer_pNRq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:** Interesting and intuitive idea that (i) is well motivated; and (ii) seems to perform well on a variety of datasets.\n\n**Weaknesses:**\n  - **Missing critical details:**\n    - *Loss function:* It is not clear what is actually being done in stage 3 (teacher-student) due to the lack of an equation for the final loss function, e.g. soft vs. hard losses, \"consistency regularization\" vs. pseudo-labelling, etc. This should be added to Section 3 to better describe the method.\n    - *Algorithm:* It is not clear what is actually being done in stage 3 (teacher-student) due to the lack of an algorithm, e.g. the replacing student and teacher models at certain phases, the swapping between soft and hard losses and certain phases, the re-initializing of student features at certain phases (to correct for incorrect pseudo-labels), etc. This should be added to Section 3 (or Suppl. Material, and pointed to) to better describe the method.\n  - **Lack of uniformity in the evaluation (cherry picking?)**:\n    - *Table 1:* Why are the 4 entries in the bottom right corner not filled in?\n    - *Table 3:* Why does the right column with SHOT not contain the same rows as the other two columns, namely R50S and R50P? Why has R50S been replaced with R101P? Do we not want to see the performance of SHOT with R50S?\n    - *Table 4*: Why are SHOT and SHOT + ours omitted from this table?\n    - *Table 5*: Why are SHOT and SHOT + ours omitted from this table?\n    - *Table 7:* Why have different networks (“network”), pre-training strategies (“imagenet pretrain”) and adaptation strategies (“source only”) been used for SHOT (left) and TENT (right)? Are these the combinations where your method achieved the biggest improvement?\n  - **Some prior/related work is not properly cited**:\n    - *Not transferring source parameters:* Prior work has already *not* transferred source parameters in the setting of *black-box* source-free domain adaptation [2,3,4]. In fact, Fig. 1c exactly depicts this setting of black-box source-free domain adaptation. This should be discussed.\n    - *Factorizing the representation and classifier:* SHOT[1] does exactly this, training the representation while holding fixed the classifier. I think you should cite them in Section 3 (last paragraph of page 3) and perhaps the introduction. Otherwise it (wrongly) comes across as a contribution of this paper.\n    - *L_{ent} and L_{div}:* the use of both L_{ent} and L_{div} losses was introduced by SHOT[1] (i.e. their SHOT-IM loss), and should be cited accordingly in Section 3, Stage 1 (teacher).\n    - *Using SHOT-IM loss with TENT (Section 4.2, Stage 1):* Since you replace the loss of TENT (L_{ent}) with that of SHOT-IM (L_{ent} and L_{div}), I don’t think you can refer to this method as TENT anymore because it is no longer TENT. Rather, it is a mix between TENT and SHOT-IM and should be labelled as such, e.g. TENT-IM.\n\n**Further comments**:\n - **\"Test-time\":** I find the use of \"test-time\" a bit misleading for the *adaptation phase* of source-free domain adaptation. One is not really \"testing\" at that time, but rather adapting a model to the target data for many epochs (could be several days). Instead, it could simply be called adaptation, target adaptation, or on-target adaptation! The addition of “offline” and “online” test-time approaches further added to my confusion. \n - **\"Label smoothing *could* be leveraged\":** Was label smoothing used or not? (Section 4.2, Stage 0).\n\n\n[1] Liang, J., Hu, D., & Feng, J. (2020). Do we really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation. In *International Conference on Machine Learning* (pp. 6028-6039).\n\n[2] Liang, J., Hu, D., He, R., & Feng, J. (2021). Distill and Fine-tune: Effective Adaptation from a Black-box Source Model. *arXiv preprint arXiv:2104.01539*.\n\n[3] Zhang, H., Zhang, Y., Jia, K., & Zhang, L. (2021). Unsupervised Domain Adaptation of Black-Box Source Models. *arXiv preprint arXiv:2101.02839*\n\n[4] Wu, K., Shi, Y., Han, Y., Shao, Y., & Li, B. (2021). Black-box Probe for Unsupervised Domain Adaptation without Model Transferring. *arXiv preprint arXiv:2107.10174*.\n", "summary_of_the_paper": "This work addresses the problem of source-free domain adaptation. Instead of fine-tuning *the source model* on the target data, this work proposes to fine-tune *a representation learned on the target data alone* using e.g. self-supervised contrastive learning. This permits distinct target architectures that can improve performance and/or reduce memory and computational cost.", "summary_of_the_review": "Overall, this seems to be a good paper. A well-motivated and intuitive idea is proposed, and it appears to work well across a variety of datasets. However, critical details are omitted (e.g. what is the complete loss function and algorithm?), there is a lack of uniformity in the evaluation (raising suspicions of cherry picking), and related works are not cited satisfactorily (e.g. black-box source-free domain adaptation). As a result, I believe this paper is only marginally above the acceptance threshold in its current form. If the aforementioned concerns can be addressed, I will improve my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9EZULyUBe8I", "reviewer": "Reviewer_Lx65", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:  The paper discusses a direct target adaptation to improve the performance of target data ( which is the partial objective of the traditional domain adaptation method). The paper shows a different variation of each component ( initialization, self-supervised, etc.). The paper is very easy to follow.\n\nWeakness: The paper needs to improve significantly in terms of proper evaluations ( fair evaluations). Some of the points are listed here: \n The paper's novelty is limited; it uses most of the well-studied techniques such as Infomax-based entropy loss, self-supervised methods, and distillation. The paper seems it is only combinations of those methods without proper justification ( motivation) why should these particular techniques are suitable for the current problem. Although they showed it only based on empirical performance, it should not be the motivation or intuition to use particular techniques.\nIn stage 1, the parameters of the feature extractor are updated, not the classifier; I wonder what would be effective if we update the classifier too. This will help the feature to be class discriminative for the target domain rather than the source domain.\nRegarding comparing state-of-the-art methods, the performance of domain adaption methods on the source domain does not degrade while adapting the target domain. In this case, we concern only with target performance, so I suspect the model will perform poorly on the source domain. So the comparison in terms of the target domain is not fair enough.\nIn Table 2, the performance reported on TENT ( baseline)  with network  R50S but the proposed model trained on R18S; what is the reason to choose different base networks. For the fair comparing the base network must be the same for all the experiments. The same thing for the SHOT baseline.\nIn Table 5, again, the base network is different from the baseline or other state-of-art methods. In fact, authors should include the source-only models' performance in each table, which is the super baseline for all the source-free or ( not source-free) domain adaption methods.\nOverall due to the missing proper fair comparison, the results do not show the effectiveness of the proposed methods. \nThus I would like to see the results with fair base network.\n\n", "summary_of_the_paper": "The paper proposes a direct target adaptation from the source trained model (without the source data).  It applied self-supervised and distillation methods to learn from unlabeled target data.", "summary_of_the_review": "The paper needs a proper evaluation on the fair baseline. The weaknesses are discussed above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work proposed an interesting source free adaptation setting, where one is asked to adapt a pre-trained source model to a target domain without accessing data from the source domain. While reviewers find the setup interesting and the initial results encouraging, they expressed concerns on the limited novelty of the work as well as incomplete evaluation. Multiple reviewers (reviewer Lx65  and pNRq) raised concerns on the fairness of the evaluation, which was not fully addressed by the authors during rebuttal. Please consider addressing these comments in your draft.
Decision: Reject