Attracting and Dispersing: A Simple Approach for Source-free Domain Adaptation
Authors: Shiqi Yang
Yaxing Wang
Kai Wang
SHANGLING JUI
Joost van de weijer
Venue: NeurIPS 2022 Accept (2022)
Keywords: source-free domain adaptation

Abstract: We propose a simple but effective source-free domain adaptation (SFDA) method. Treating SFDA as an unsupervised clustering problem and following the intuition that local neighbors in feature space should have more similar predictions than other features, we propose to optimize an objective of prediction consistency. This objective encourages local neighborhood features in feature space to have similar predictions while features farther away in feature space have dissimilar predictions, leading to efficient feature clustering and cluster assignment simultaneously. For efficient training, we seek to optimize an upper-bound of the objective resulting in two simple terms. Furthermore, we relate popular existing methods in domain adaptation, source-free domain adaptation and contrastive learning via the perspective of discriminability and diversity. The experimental results prove the superiority of our method, and our method can be adopted as a simple but strong baseline for future research in SFDA. Our method can be also adapted to source-free open-set and partial-set DA which further shows the generalization ability of our method. Code is available in https://github.com/Albert0147/AaD_SFDA.

Attracting and Dispersing: A Simple Approach for Source-free Domain Adaptation
Shiqi Yang', Yaxing ; Kai Wang', Shangling Jui Joost van de Weijer'
Computer Vision Center, Universitat Autonoma de Barcelona, Barcelona, Spain
2 Nankai University, Tianjin, China
Huawei Kirin Solution, Shanghai, China
{syang, kwang, joost eev.. yaxing@nankai .ed.. en, ariishhagll ingghhawei..ccommm com
Abstract
We propose simple but effective source-free domain adaptation (SFDA) method. Treating SFDA as an unsupervised clustering problem and following the intuition that local neighbors in feature space should have more similar predictions than other features, we propose to optimize an objective of prediction consistency. This objective encourages local neighborhood features in feature space to have similar predictions while features farther away in feature space have dissimilar predictions, leading to efficient feature clustering and cluster assignment simultaneously. For efficient training, we seek to optimize an upper-bound the objective resulting in two simple terms. Furthermore, adaptation, source-free domain adaptation and contrastive learning via the perspec- tive of discriminability and diversity. The experimental results prove the superiority of our method, and our method can be adopted as a simple but strong baseline for future research in SFDA. Our method can be also adapted source-free open-set and partial-set DA which further shows the generalization ability of our method. Code available in ht httpp::/ /ttp://ggttt..t//e wtw// /AaD_SFDD
relate popular existing methods in domain
Introduction
Supervised learning methods which are based on training with huge amounts of labeled data are advancing almost all fields of computer vision. However, the learned models typically perform decently test data which have similar distribution with the training set. Significant performance degradation will occur if directly applying those models to a new domain different from the training set, where the data distribution (such as variation of background, styles camera parameter) is considerably different. This kind of distribution shift is formally denoted as domain/distribution shift. It limits the generalization of the model to unseen domains which is important in real-world applications. There are several research fields trying to tackle this problem. One of them is Domain Adaptation (DA), which aims reduce the domain shift between the labeled source domain and unlabeled target domain. Typical works [1] 38 resort to learn domain-invarian features, thus improving generalization ability of the model between different domains. And in the past few years, the main research line of domain adaptation is either trying to minimize the distribution discrepancy
between two domains [32] 33 35). or deploying adversarial training on features to learn domain invariant representation [52 68 36). Some methods also tackle domain shift from the view of
semi-supervised learning 67 Or clustering 50 5).
focus on source-free domain adaptation (SFDA), where source data unavailable during target adaptation, due to data privacy and intellectual
Many recent methods 24
64 66 55
15
CCoresponding Author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Table Detailed comparison of SFDA methods on VIsDA. 'ODA/DD'' means whether the method
reports the results for open-set or partial-set DA. |C| means number of training objective terms,
Method
Extra Modules/Processsing Preeeessnooss Access all target data for pscudo labeling
ODA/PDA
Per-class
SHOT 26
V
82.9
3C-GAN 24
Data generation by conditional GAN Self-supervised learning with extra classifiers
X
5
81.6
Not
[6
X
84.3 5
G-SFDA
66
Store features for nearest neighbor retrieval
X
2
85.4
NRC
64
Store features for 2-hop nearest neighbor retrieval
X
4
85.9
HCL
Store historical models
83.5
Ours
Store features for nearest neighbor retricval
2
88.0
property concerns of both users and businesses. Some SFDA methods resort to neighborhood
clustering and pseudo labeling. However, pseudo labeling methods [29] may suffer from negative impact from noisy labels [28], and neighborhood clustering methods [66] 64) fail to investigate
the potential information from dissimilar samples. Other methods either demand complex extra
modules/processing [24]]]]] 61 or the storing of historical models for contrastive learning [5]
Based on the fact that target features from the source model already form some semantic structure and following the intuition that for a target feature from (source- preerccedd)) model, similar features should have closer predictions than dissimilar ones, we propose a new objective dubbed as Attracting- and-Dispersing (AaD) to achieve it. we upperbound this objective, resulting in a simple final objective which only contains two types of terms, which encourage discriminability and diversity respectively, Further, we unify several popular domain adaptation, source-free domain adaptation and contrastive learning methods from the perspective of discriminability and diversity. Experimental results on several benchmarks prove the superiority of our proposed method. Our simple method improves the state-of-the -frte--r- the challenging VisDA with 2.1 1% to 88.0%. Additionally, extra experiments open-set and partial-set DA further prove the effectiveness of our method. A preliminary comparison between different SFDA method is shown in Tab. [ which shows the simplicity and generalization ability of our method: it only requires the storing of features and a few nearest neighbors searches without any additional module like a generator (24] or classifier [6].
We summary our contributions as follows:
We propose tackle source-free domain adaptation by optimizing an upperbound of the proposed clustering objective, which is surprisingly simple.
We relate several popular existing methods in domain adaptation. source-free domain adaptation and contrastive learning via the perspective of discriminability and diversity, which is helpful to understand existing methods and beneficial for future improvement.
The experimental results prove the efficacy of our method, especially we achieve new state-of-the- art on the challenging VisDA, and the method can be also extended to source-free open-set and partial-set domain adaptation.
2 Related Work
Domain Adaptation. Early DA methods such as 33 49 53 adopt moment matching to align feature distributions. For adversarial learning methods, DANN 9 formulates domain adaptation as an adversarial two-player game. The adversarial training of CDAN 34 is conditioned on several
sources of information. DIRT-T 47] performs domain adversarial training with an added term that
penalizes violations of the cluster assumption. Additionally, [22 36 adopts prediction diversity between multiple learnable classifiers achieve local category-level feature alignment between source and target domains. SRDC (50) proposes to directly uncover the intrinsic target discrimination via discriminative clustering achieve adaptation. CST [3] proposes simple self-training strategy to improve the rough pseudo label under domain shift.
Source-free Domain Adaptation. The above-mentioned normal domain adaptation methods need to access source domain data at all time during adaptation. In recent years plenty of methods emerge trying to tackle source-free domain adaptation. USFDA 20) and FS (2I) resort to synthesize extra training samples in order to get compact decision boundaries, which is beneficial for both

---

Algorithm/ Attracting and Dispersing for SFDA
Require: Source-pretrained model and target data D,
I: Build memory bank storing all target features and predictions
2: while Adaptation do
3: Sample batch T from Dr and Update memory bank
4: For each feature 2; T, retrieve K --eerrstt neighbors (C.) and their predictions from memory bank
5: Update model by minimizing Eq.
6:
end while
the detection of open classes and also target adaptation. SHOT 26 proposes to freeze the source classifier and it clusters target features by maximizing mutual information along with pseudo labeling
for extra supervision. 3C-GAN (4 synthesizes labeled target-style training images. It is based on
conditional GAN to provide supervision for adaptation. BAIT [65] extends MCD [44] to source-free
setting. Net [61] proposes to learn an additional target-specific classifier for hard samples and adopts contrastive category-wise matching module to cluster target features. HCL [53 adopts Instance Discrimination [60] for features from current and historical models to cluster features, along with generated pseudo label conditioned historical consistency. G-SFDA [66] and NRC [64] propose neighborhood clustering which enforces prediction consistency between local neighbors.
Deep Clustering and Contrastive Learning. Recent Deep Clustering methods can be roughly divided into two groups, they the differ in how they learn the feature representation and cluster as- signments, either simultancously or alternatively. For example, DAC [2] and DCCM [58] alternately update cluster assignments and netween-sample similarity. Simultaneous clustering methods IIC [18] and ISMAT [4] are based on mutual information maximizing between samples and theirs augmenta- tions. LA [70] depends on a huge amount of nearest neighbor searches and multiple extra runs of k-means clustering to aggregate features. Recent unsupervised clustering works 255 51 46 rely on contrastive learning, where InfoNCE B7) is typically deployed. And recently NNCLR proposes to use nearest neighbors in the latent space positives in contrastive learning to cover more semantic variations than pre-defined transformations. However an inevitable problem normal contrastive learning is class collision where negative samples are from the same class. To tackle this issue, recent works 23 16 propose to estimate cluster prototypes and integrate them into contrastive learning.
start to
3 Method
For source-free domain adaptation (SFDA), we are given source-pretrained model in the beginning and an unlabeled target domain with N, samples as DD = [x]] Target domain have same C classes as source domain in this paper (known as the closed-set setting). The goal of SFDA is to adapt the model to target domain without source data. We divide the model into two parts: the feature extractor f. and the classifier g. The output of the feature extractor is denoted as feature (z, = f (x) E R²), where h is dimension of the feature space. The output of classifier is denoted as (p. = (()())) E RC where is the softmax function. We denote P E R as the prediction unsupervised clustering problem, we address
matrix in mini-batch. Regarding the SFDA SFDA problem by clustering target features based on the proposed AaD. In additionally, we relate method with several existing DA, SFDA and contrastive learning methods.
3.1 Attracting and Dispersing for Source-free Domain Adaptation
Since the source- pretrccpe model already learns a good feature representation, it can provides a decent initialization for target adaptation. We propose to achieve SFDA by attracting predictions for features that are located close feature space, while dispersing predictions of those features farther away in feature space.
We define Pij as the probability that the feature 2; E R² has similar (or the same) prediction feature
Zy: Pij =
MM pire It can be interpreted as the possibility that p5 is selected as the neighbor of p;
in the output space [10]
3

---

We then define two sets for each feature close neighbor set C, containing K K-nearest neighbors of 2, (with distances as cosine similarity), and background set B, which contains the features that are not in C; (features potentially from different classes). To retrieve nearest neighbors for training, we build two memory banks to store all targer features along with their predictions just like former works [27 66 64 42), which is efficient in both memory and computation, since only the features along with their predictions computed in each mini-batch are used to update the memory bank.
Intuitively, for each feature 2;, the features in B, should have less similar predictions than those in c
To achieve this, we first define two likelihood functions:
P Ps
e Ps
0(
II Pay JEC;
II JEC.
P(
= II
Pt = JEB;
eP p (1)
2k-1
JEB;
2k
where 9 denotes parameters of the model, for readability we omit in following equations. The probability PÜ in Eq. is the stored prediction for neighborhood feature 2j, which is retrieved from the memory bank.
We then propose to achieve target features clustering by minimizing the following negative log- likelihood, denoted as AaD (Attracting- and---nd---d and-Dispersing):
P(C.) L,,,,, B.) -og P(B.)
Noting that, if we only have P(C.), it will be similar to Instance Discrimination 60), but we also consider P(B,) and we operate on predictions instead of features. If regarding weights of the classifier g as classes prototypes, optimizing Eq i not only pulling features towards their closest neighbors and pushing them away from background features, but also towards (or away from) corresponding class prototypes. Therefore, we can achieve feature clustering and cluster assignment simultaneously.
To simplify the training, instead of manually and carefully sampling background features, we use all other features except 2; the mini-batch as B,, which can be regarded as an estimation of the distribution of the whole dataset. We can reasonably believe that overall similarity of features in C, is potentially higher than that of B,, even if B, has intersection with C, since features in C, are the closest ones to feature 2,. By optimizing Eq. 2 we are encouraging features in C, which have a higher chance of belonging to the same class, have more similar predictions to 2; than those features in B,, which have a lower chance of belonging to the same class. Note all features will show up both the first and second term; intra-cluster alignment and inter-cluster separability are expected to be achieved after training.
One problem optimizing Fq. is that all target data needed compute Eq. 1 which is infeasible in real-world situation. Here we resort to get an upper-bound of Eq. 2
Ta, BB) = log P(C,) P(B.)
Σ JEC:
Io Ps log
e
FFryy ]] +
Σ
Io Pm
log
T Pk
m mEB;
= Σ
Σ
per p Pm + (Ne. - No,) log
of Pk)
ps Ps
SEC,
meB
Since we set No. < NB,, with Jensen's inequality:
Nr pTpm +No, NNs)
L. Bd) S
Σ 3EC
pī Ps
Σ
pi pk + log N.)
N.
mEB,
2 Pa No,
T P Pm
Σ
oe ps (Nc, Nos. kgB,
+ log No)
(4)
mEB
ECC.
Σ JCC
P Pj
No,
Nc.
Σ pm
pm + (Nc, No, log N.
mEB,
For better understanding, refer to Bi and C: as index

---

Table Decomposition of methods into two terms: discriminability (dis) and diversity (div), which will be minimized for training.
Method
Task
dis term
div term
MI
SFDA&Clustering
H(Y())
HOY rank(P)) Ee=1 KL(pellac)
BNM
DA&SFDA
NC
SFDA
-9(Wapp ps)
InfoNCE
Contrastive
- f(x)* f(8)TT
logl
+ + el ( sOWl
Ours
SFDA
Ejeo,pf PI
Emeg of
where Nc. and NE, the number of features in C, and B, Note that we cannot get this upper-bound without P(B,). The approximation above in the penultimate line is to estimate the average dot product using the mini-batch data. This leads to the surprisingly simple final objective for unsupervised domain adaptation:
= E [,(C;, with L,(C,, =
ET.,, +
Σ MEB
ps Pm
(5)
jEC.
Note the gradient will come from both p, and Pm- The first term aims enforce prediction consistency between local neighbors, and the naive interpretation of second term to disperse the prediction of potential dissimilar features, which are all other features in the mini-batch. Note that the dot product between two softmaxed predictions will be maximal when two predictions have the same predicted class and are close one-hot vector. Our algorithm is illustrated in Algorithm. [
Unlike using a constant for the second term in Eq. we empirically found that using a hyperpa- rameter À to decay second term (starting from 1) works better, we will adopt SND [43 to tune this hyperparameter unsupervisedll. One reason may be that the approximation inside Eq. 3.1 is not necessarily accurate. And as training goes on, features gradually clustering, the role of the second term for dispersing should be weakened. Additionally, considering the current mini- batch with the correctly predicted features and 2m belonging to the same class. In this case the second term in both (C,, B,) and T.m.Cnn, (Cm, Bm)) tends to push pm to the wrong direction, while the first term in Tim (Cm, BBm) can potentially keep current (correct) prediction unchanged. Hence, this will suppress the negative impact of the second term. We will further deepen the understanding of these two terms in the next subsection.
3.2 Relation to Existing Works
In this section, we will relate several popular DA, SFDA and contrastive learning methods through two objectives, discriminability and diversity. This can improve understanding of domain adaptation methods, as well as improve the understanding of our method.
Mutual Information maximizing (MI). SHOT-IM (26) proposes achieve source-free domain adaptation by maximizing the mutual information, which is actually widely used in unsupervised clustering 40 14
LM H((X)) H(Y)
(6)
which contains two terms: conditional entropy term H(IIX) to encourages unambiguous cluster assignments, and marginal entropy term H(Y) encourage cluster sizes to be uniform to avoid degeneracy. In practice, H(Y) is approximated by the current mini-batch instead of using whole dataset 48 [4]
Batch Nuclear-norm Maximization (BNM). BNM 5 6 aims to increase prediction discrim- inability and diversity to tackle domain shift. It is originally achieved by maximizing F-norm (for discriminability) and rank of prediction matrix (for diversity) respectively:
L Plak-rnn((P (P)
-
In their paper, they further prove merely maximizing the nuclear norm ||||| can achieve these two
goals simultaneously. In relation to our method, if target features are well clustering during training, we can presume the K-nearest neighbors of feature 2; have the same prediction, the first term in Eq. [ can be seen as the summation of diagonal elements of matrix PPT which is actually the square
5

---

of F-norm (| (ttace(PP") then it is actually minimizing prediction entropy [5]. As for second term, we can regard it as the summation of non-diagonal element of PP" it encourages all these non-diagonal elements to be thus the rank(pP?)
rank P) is supposed to increase, which indicates larger prediction diversity [S] In a nutshell, compared to SHOT and BNM our method first considers local feature structure to cluster target features, which can be treated as an alternative way to increase discriminability at the late training stage, meanwhile as discussed above our method is also encouraging diversity.
Neighborhood Clustering (NC). G-SFDA [66 and NRC 64 are based on neighborhood clus- tering to tackle SFDA problem. Those works basically contain two major terms in their optimizing objective: a neighborhood clustering term for prediction consistency and marginal entropy term (Y) for prediction diversity. NRC [64] further introduces neighborhood reciprocity to weight the different neighbors. Their loss objective can be written as:
T, =- Σ g(W S.WSPP PJ) + Σ (Tollll))); with Pe - n4 ΣT
pf and ((==I..,CC) C
JEC,
where Way will weight the importance of neighbor and g()) is log or identity function. Although the first term of G-SFDA and NRC is the same as that of our final loss objective Eq. [ note that motivation is different as we simultaneously consider similar and dissimilar features, and Eq. 5 is deduced approximated upper-bound of our original objective Eq.
And note actually the marginal entropy term HY) - Pe. logg. = = ( log C. Although the second term of those methods are favoring prediction diversity to avoid the trivial solution where all images are only assigned to some certain classes, the margin entropy term presumes the prior that whole dataset or the mini- is class balance/uniformly distributed, which barely true for current benchmarks or in real-world environment. In conclusion, the above three types of methods are actually all to increase discriminability and meanwhile maximize diversity of the prediction, but through different ways.
which widely used in Contrastive Learning. Here we also link our method to InfoNCE contrastive learning. As recent paper (56) points out that InfoNCE loss can be decomposed into terms:
37
LinfoNCE B(z.a)),, (-f(f) f(y\/Tl +
E
[logle!/
>Ee 1" ((0)/T))
pr-Pdato {x, M Dis Piiaa
The first term is denoted as alignment term (with positive pairs) is to make positive pairs of features similar, and the second term denoted as uniformity term with negative pairs encouraging all features to roughly uniformly distributed in the feature space.
The Eq. g shares some similarity with all the above domain adaptation methods in that the first term is for the alignment with positive pairs and the second term encourage diversity. But note that the remarkable difference is that the above domain adaptation methods operate in the output (prediction) space while contrastive learning is conducted in the (spherical) feature space. Therefore, simultaneously feature representation learning and cluster assignment can be achieved for those domain adaptation methods. Note normal contrastive learning methods, extra KNN or linear learnable classifier needs to be deployed for final classification, while our model can directly give predictions.
We list all above methods in Tab. 2 Finally, returning to Eq. 5 we can also regard the second term variant of diversity loss avoid degeneration solution, but without making any category prior assumption. Intuitively, with target features forming groups during training, the second term should play less and less important role, otherwise it may destabilize the training. This is similar the class collision issue in contrastive learning. If our second term contains too many features belonging to the same class. Thus it is reasonable to decay the second term.
4 Experiments
Datasets. We conduct experiments on three benchmark datasets for image classification: Office-31, Office-Home and VisDA-C 2017. Office-31 [1 contains 3 domains (Amazon, Webcam, DSLR)

---

Table 3: Accuracies (%) on Office-Home for ResNet50-based methods. We highlight the best result and underline the second best one.
Method
STAA- --A--CAA->> -PrAr
RRCI
AArCI
:PTCI
-RwPT
>APPr
CIPT
-RwRw
AATRw
:CRRw
PrAvg
ResNet-50
13
34.9
50.0
58.0
37.4
41.9
38.5
31.2
60.4
53.9
41.2
59.9
46.
MCD 44
X
48.9
68.3
74.6
61.3
67.6
68.8
57.0
47.1
75.1
69.1
52.2
79.6
64.1
CDAN
B4
x
50.7
70.6
76.0
57.6
70.0
70.0
57.4
50.9
77.3
70.9
56.7
81.6
65.8
SAFN 63
X
52.0
71.7
76.3
64.2
69.9
71.9
63.7
51.4
77.1
70.9
57.1
81.5
67.3
MDD
X
54.9
73.7
77.8
60.0
71.4
71.8
61.2
53.6
78.1
72.5
60.2
82.3
68.1
TADA 57
X
53.1
72.3
77.2
59.1
71.2
72.1
59.7
53.1
78.4
72.4
60.0
82.9
67.6
SRDC
X
52.3
76.3
81.0
69.5
76.2
68.7
81.7
76.3
57.1
85.0
71.3
SHOT 26
57.1
78.1
81.3
68.0
78.2
78.1
67.4
54.9
82.2
73.3
58.8
84.3
71.8
Net
61
58.4
79.0
82.4
67.5
79.3
78.9
68.0
56.2
82.9
74.1
60.5
85.0
72.8
G-SFDA 666
57.9
78.6
81.0
66.7
77.2
77.2
65.6
56.0
82.2
72.0
57.8
83.4
71.3
NRC [4
57.7
82.0
68.1
79.8
65.3
71.0
58.6
72.2
BNM-S
6
57.4
77.8
81.7
67.8
77.6
79.3
67.6
55.7
82.2
73.5
59.5
84.7
72.1
Ours
59.3
79.3
82.1
68.9
79.8
79.5
67.2
57.4
83.1
72.1
58.5
85.4
72.7
Table 4: Accuracies (%) VisDA-C (Synthesis
Real) for ResNet101-baseee methods. We
highlight the best result and underline the second best one.
Method
SF plane beyel bus
horse knile meycl person plant sktbrd train truck Per-class
ResNet- 13
55.1
53.3
61.9
80.6
17.9
79.7
31.2
81.0
26.5
73.5
8.5
52,4
CDAN+BSP
92.4
61.0
81.0 57.5 89.0
80.6
77.0
84.2
77.9
82.1
38.4
75.9
MCC 19
88.7
80.3
80.5 71.5
90.1
93.2
85.0
71.6
73.8
85.0
78.8
STAR
95.0
84.0
84.6 73.0
91.6
91.8
85.9
78.4
94.4
84.7
87.0
82.7
RWOT 62
x
95.1
80.3
83.7 90.0
92.4
68.0
92.5
82.2
87.9
78.4
90.4
68.2
84.0
3C-GAN 24
V
94.8
73.4
68.8
74.8
93.1
95.4
88.6
84.7
89.1
84.7
83.5
48.1
81.6
SHOT 26
V
94.3
88.5
80.1 57.3
93.1
94.9
80.7
80.3
91.5
89.1
86.3
58.2
82.9
Net 61
/
94.0
87.8
85.6
66.8
93.7
95.1
85.8
81.2
91.6
88.2
86.5
56.0
84,3
G-SFDA [66
V
96.1
88.3
85.5
74.1
97.1
95.4
89.5
79.4
95.4
92.9
89.1
42.6
85.4
NRC 64
V
96.8
91.3
82.4
62.4
96.2
95.9
86.1
80.6
94.8
94.1
90.4
59.7
85,9
HCL. 155
V
93.3
85.4
80.7
68.5
91.0
88.1
86.0
78.6
86.6
88.8
80.0
74.7
83.5
Ours
97.4
90.5
80.8
76.2
97.3
96.1
89.8
82.9
95.5
93.0
92.0
64.7
88.0
with 31 classes and 4,652 images. Office-Home [54] contains 4 domains (Real, Clipart, Art, Product)
with 65 classes and total 15,500 images. VisDA (VisDA-C 2017) [39 is a more challenging dataset, with 12-class synthetic-to-red object recognition tasks, its source domain contains of 152k synthetic images while the target domain has 55k real object images.
Evaluation. The column SF in the tables denotes source-free. For Office-31 and Office-Home, we show the results of each task and the average accuracy over all tasks (Avg in the tables). For VisDA, show accuracy for all classes and average over those classes (Per-class in the table). All results are the average of three random runs for target adaptation.
Model details. To ensure fair comparison with related methods, we adopt the backbone of a ResNet-50 133 for Office-Home and ResNet- 101 11 for VisDA. Specifically, we use the same network architecture as SHOT 266.. BNM-S (6. G-SFDA [66] and NRC 164), i.e., the final part of the network is: fully connected layer Batch Normalization [17] fully connected layer with weight normalization [45y. We adopt SGD with momentum 0.9 and batch size of 64 for all datasets. The learning rate for Office-31 and Office-Home is set to le-3 3 for all layers, except for the last two newly added fo layers, where we apply 1e-2. Learning rates are set 10 times smaller for VisDA. We train 40 epochs for Office-31 and Office-Home while 15 epochs for VisDA.
There are two hyperparameters Ne, (number of nearest neighbors) and to ensure fair comparison we set Ne, to the same number as previous works G-SFDA (66) and NRC [64], which also resort to nearest neighbors. That is, we set Nc, to on Office-31 and Office-Home, 5 on VisDA. For X, we set
(1 10 directly apply SND (43) select 3 imsupervisedly. Based on SND we / to 0 on Office-Home, on Office-31 and 5 on VisDA.
ster max max_iter -8, ) where the decay factor controls the decaying speed. We
4.1 Results and Analysis
Quantitative Results. As shown in Tables 5 Left) where the top part shows results for the source-present methods that use source data during adaptation, and the bottom part shows results for
7

---

do Table 5: (Left) Accuracies (%) on Office-31 for ResNet550basse methods. We highlight the best result and underline the second best one. (Right) Ablation study on number of nearest neighbors
No.. We highlight the best score and underline the second best one.
Method
SHA-DDA ADAA ----D-WW---.
WD
WWD+WW WW
DD--DD--- -DW AAvv
MCD
44
X
92.2
88.6
98.5
100.0
69.5
69.7
86.5
Nc.
Avg
CDAN
94.1
98.6
100.0
71.0
69.3
87.7
Office-31
Ne.
Per-class
MDD
x
90.4
90.4
98.7
99.9
75.0
73.7 8800
89.1
VisDA
DMRL
59
X
93.4
90.8
99.0
100.0
73.0
71.2 87.9
MCC 19
X
95.6
95.4
98.6
100.0
72.6
73.9
89.4
89.5
86.7
SRDC 150
95.8
95.7
99.2
100.0
76.7
77.1 90.8
89.9
87.4
SHOT 26
94.0
90.1
98.4
99.9
74.7
74.3 88.6
Office-Home
88.0
3C-GAN
93.7
98.5
99.8
77.8 89.6
72.2
88.0
NRC 64
99.0
100.0
75.0
2
72.6
88.0
HCL 15
94.7
92.5
98.2
100.0
75.9
77.7 89.8
BNM-S
93.0
92.9
98.2
99.9
75,4
75.0
89.1
72.7
Ours
96.4
92.1
99.1
100.0
75.0
76.5
89.9
N without decay
only first term Figure 1: Visualization of decision boundary on target data with different training objective.
Source model
with decay
the source-free DA methods. On Office-31 and VisDA, method gets state-of-the-art performance compared to existing source-free domain adaptation methods, especially on VisDA our method outperforms others by a large margin (2. % compared NRC). And our method achieves similar results on Office-Home compared to the more complex Net method (which combines three classifiers and five objective functions). The reported results clearly demonstrate the efficiency of the proposed method for source-free domain adaptation. It also achieves similar or better results compared to domain adaptation methods with access to source data on both Office-Honne Home and VisDA. Note the extension of SHOT called SHOT++ [30] deploys extra self self-supervised training and semi- supervised learning, which are general improve the results (an evidence that the source model after these tricks gets huge improvement, e.g. 60.2% improves 66.6% Office-Homee not list it here for fair comparison.
Toy dataset. We carry out an experiment on the twinning moona dataset to ablate the influence of two terms in our objective Eq. 5 For the twinning moons dataset, the data from the source domain are represented by two inter-twinning moons, which contain 300 samples each. Data in the target domain are generated through rotating source data by 30". The domain shift here is instantiated as the rotation degree. First we train the model with 3 linear layers only on the source domain, and test the model all domains. As shown in the first image in Fig. | the source model performs badly on target data. Then we conduct several variants of our method to train the model. The visualization of the decision boundary Fig. indicates that both terms in Eq. are necessary, and decay of second term is shown to be important.
Table 6: Unsupervised hyperparameter selection of 3 with SND spond to better target model.
43
larger SND should corre-
VisDA
3
SNDT
Per-class
Office-31
Office-Home
0
8.1823
77.5
3
SND1
Avg
3
SNDT
Avg
1
8.2584
83.8
4.1366
88.0
3.7515
72.7
2
8.3214
86.7
0.25
4.3016
89.7
0.25
3.7402
72.6
3
8.3311
87.6
4.4494
89.9
0.5
3.7252
72.0
4
8.3540
88.0
4.4501
89.9
3.6923
70.6
5
8.3543
88.0
7
8.3530
88.1

---

Ratio of shared prediction
Ratio of correct shared prediction
09
68
63..83.5%
P-1: 82,8%
84
Bo2:
(123877.00 07%
66
9-3: 07.0%
35
8-5: 88 5%%
9-5: 58. 01.
10
2- 58.1%
B- 86.1%
23
2020
4000
6002
2200
10000
12000
14200
2000
4000
5000
5000
10000
12000
14000
Iteraiins
Iterations
Figure 2: (Left) Ratio of features which have 3 nearest neighbor features sharing the same predicted label. (Right) Ratio among above features which have nearest neighbor features sharing the same and correct predicted label.
Table 7: Runtime analysis on SHOT and our method. For SHOT, pseudo labels are computed at each epoch. 10% and 5% denote the percentage target features which are stored in the memory bank.
VisDA
Runtime Per-claas (%)
SHOT
618.82
82.9
Ours
520.13
88.0
Ours(10% for memory bank)
490.21
87.6
Ours(5% for memory bank)
482.77
87.5
Number of nearest neighbors (Nc.). For the number of nearest neighbors used for the first term in Eq. [5] we show in (Right) our method is robust to the choice of Ne., as the results imply that reasonable choice of Ne, (such as 3) works quite well all datasets, since only considering few neighbors (such as 1/2) may be too noisy if all of them are misclassified, while setting Nc. too larger may also potentially include samples of other categories. For larger dataset such as VisDA can choose a relatively larger Ne.,. Note the reason why we choose No, as in main experiments is
compare fairly with G-SFDA 666 and NRC 664
Decay factor 3. According the analysis in Sec. 3.2 the second term like a diversity term to avoid that all target features collapse to a limited set of categories. The role of the second term should be weakened during the training, but how to decay the second term is non-trivial. We directly adopt SND [43] which computes Soft Neighborhood Density for unsupervised hyperparameter selection of 3. The method is unsupervised and larger SND predicts better target models. The results of SND with different 8 are shown in Tab, 6 the results prove that SND works well to choose optimal B.
Runtime analysis. Instead of storing all features in the memory bank, we can only stores a limited number target features, by updating the memory bank at the end of each iteration by taking the n (batch size) embeddings from the current training iteration and concatenating them at the end of the memory bank, and discard the oldest n elements from the memory bank. We report the results with this type of memory bank of different buffer size in the Table 7 The results show that indeed this could be an efficient way to reduce computation on very large datasets.
Degree of clustering during training. We also plot how features are clustered with different decaying factors 3 on VisDA in Fig. 2 The left one shows the ratio of features which have 3-nearest neighbors all sharing the same prediction, which indicates the degree of clustering during training, and the right one shows the ratio among above features which have 3-nearest neighbor features sharing the same and correct predicted label. Those curves Fig. leff show that the target features are clustering, and those in Fig. 2 right indicate that clear category boundaries are emerging. The numbers in the legends denote the deployed 3 and the corresponding final accuracy. From the figures can draw the conclusion that with larger decay factor 3 on VisDA, features are quickly clustering

---

Table 8: Accuracy on Office-Home using ResNett-50 as backbone for Source-free open-set DA.
OS*, UNK and HOS mean average per-class accuracy across known classes, unknown accuracy and harmonic mean between known and unknown accuracy respectively.
Ar-
Ar
Ptr
Ar -7 Rw
Cl-+ Ar
CI
CI -I Rw
O8* UNK HOS
OS*
UNK
HOS
OS*
UNK HOS
OS* UNK HOS
OS* UNK
HOS
OS* UNK HOS
SHOT 28.0 39.5
81.8
26.3
39.8
87.5 32.1 47.0
66.8
46,2 54.6
77.5
27.2
40.2
80.0
25.9 39.1
AaD 50.7 66.4 $7.6
64.6
69.4
66.9
73.1 66.9 69.9
48.2
81.1 60.5
59.5
63.5
61.4
67.4
68.3 67.8
Pr Ar
Pr- C1
Pr -> Rw
Rw Ar
Rw
Rw
Avg.
OS* UNK HOS
OS* UNK HOS
OS*
UNK HOS
OS* UNK
HOS
OS* UNK
HOS
OS*
UNK
HOS UNK HOS 33,9
66.3
59.3
31.0
31.6
46.2
73.5
50.6
28,9
28.2
42.3
AaD 47.3 82.4
60.1
45.4
72.8 55.9
68.4
72.8
70.6
54.5 79.0
64.6
49.0
69.6
57.5
69.7
70.6
70.1
58.2 71,9 63.6
Table 9: Accuracy on Office-
using ResNet-50 as backbone for Source-free partial-set DA.
Partial-set DA Ar Ar-CCIA Ar -Pr Ar 1--Re
Re
CI +Ar Cl -Pr C1 -R-Re Py-AAr Pr- -CC Pr-rr--CC
PRe Re +Ar Re -CI Rc Pr Avg.
SHOT-IM
57.9
83.6
88.8
72.4
74.0
79.0
76.1
60.6
90.1
81.9
68.3
88.5
76.8
SHOT
64.8
85.2
92.7
76.3
77.6
88.8
79.7
64.3
89.5
80.6
66.4
85.8
79.3
AaD
67.0
83.5
93.1
80,5
76.0
87.6
78.1
65.6
90.2
83,5
64.3
87.3
79.7
and forming inter-class boundaries, since the ratio of features which share the same and correct prediction with neighbors are increasing faster. When decaying factor is too small, meaning training signal from the second term is strong, the clustering process is actually impeded. The curves in Fig, (left) signify that this ratio can also be used to choose / with higher performance unsupervisedly.
Source-free partial-set and open-set DA. We provide additional results under source-free partial- set and open-set DA (PDA and ODA) setting in Tab. and Tab. respectivel. where the open-set detection in ODA follows the same protocol to detect unseen categories as SHOT. On ODA, instead
of reporting average per-class accuracy OS =
|x,'
1xUNK where [Ca] is the number of Ic.+1
1c,1+1
known categories on source domain, we report results of HOS = 2x0s" xUNK which is harmonic +UNK
mean between known categories accuracy 05* and unknown accuracy UNK. As pointed out by [. OS is problematic since this metric can be quite high even when unknown class accuracy UNK is O, while unknown category detection is the key part in open-set DA. We reproduce SHOT under open-set DA and report results of OS*, UNK and HOS in Tab. 8 which shows our method gets much better balance between known and unknown accuracy.
Conclusion
We proposed to tackle source-free domain adaptation by encouraging similar features in feature space to have similar predictions while dispersing predictions of dissimilar features in feature space, to achieve simultaneously feature clustering and cluster assignment. We introduced an upper bound to our proposed objective, resulting in two simple terms. Further we showed that we can unify several popular domain adaptation, source-free domain adaptation and contrastive learning methods from the perspective of discriminability and diversity. The approach is simple but achieves state-of of-the-art performance on several benchmarks, and can be also adapted to source-free open-set and partial-set domain adaptation.
Acknowledgement
We acknowledge the support from Huawei Kirin Solution, and the project PID2019- PID2019-104174GBB 104174GB- (00/AE5/10.1003/5500 10000 1033 (MINECO, Spain), and the CERCA Programme of Generalitat de Catalunya.
References
[1] Silvia Bucci, Mohammad Reza Loghmani, and Tatiana Tommasi. On the effectiveness of image rotation for open set domain adaptation. In European Conference on Computer Vision, pages 422-438. Springer, 2020.
[2] Jianlong Chang, Lingfeng Wang, Gaofeng Meng. Shiming Xiang, and Chunhong Pan. Deep adaptive image clustering. In ICCV, pages 5879-5881 2017.
10

---

[3] Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs, discriminabiilit; Batch spectral penalization for adversarial domain adaptation. In ICML. pages 1081-1090, 2019.
[4] Safa Cicek and Stefano Soatto. Unsupervised domain adaptation via regularized conditional alignment. In ICCV, pages 1416-1425, 2019.
[5] Shuhao Cui, Shuhui Wang, Junbao Zhuo, Liang Li, Qingming Huang, and Qi Tian. Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations. CVPR, 2020.
[6] Shuhao Cui, Shuhui Wang. Junbao Zhuo, Liang Li. Qingming Huang, and Qi Tian. Fast batch nuclear-norm maximization and minimization for robust domain adaptation. arXiv preprint arXiv:2107.061 54, 2021.
Zhijie Deng, Yucen Luo, and Jun Zhu. Cluster alignment with teacher for unsupervised domain adaptation. In ICCV, pages 9944-9953, 2019.
[8] Debidatta Dwihedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanct, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. ICCV, 2021.
[9] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavio- lette, Mario Marchand, and Victor Lempitsky. Domain-adversarti training of neural networks. JMLR, 17(1):2096 200000000-000, 2016.
[10] Jacob Goldberger, Geoffrey E Hinton, Sam Roweis, and Russ R Salakhutdinov. Neighbourhood components analysis. NIPS, 17, 2004.
[11] Ryan Gomes, Andreas Krause, and Pietro Perona, Discriminative clustering by regularized information maximization. In NIPS, 2010.
[12] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kemel for unsupervised domain adaptation. In CVPR, pages 2066- 2073. IEEE, 2012.
[13] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR. pages 770-778, 2016.
[14] Weihua Hu, Takeru Miyato, Seiya Tokui, Eicchi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In ICML. pages 1558-1567, 2017.
[15] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu, Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. NewrlPS, 34, 2021.
[16] Zhizhong Huang. Jie Chen, Junping Zhang, and Hongming Shan. Exploring non non con-crntrett representation learning for deep clustering, arXiv preprint arXiv:2111.11821 2021.
[17] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint urXiv:1502.084 2015.
[18] Joao Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In ICCV, pages 9865-9874, 2019.
[19] Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. ECCV, 2020.
[20] Jogendra Nath Kundu, Naveen Venkat, and R Venkatesh Babu. Universal source-free domain adaptation. CVPR, 2020.
[21] Jogendra Nath Kundu, Naveen Venkat, Ambareesh Revanur, R Venkatesh Babu, et al. Towards inheritable models for open-set domain adaptation. In CVPR, pages 12376-12385, 2020.
[22] Chen- Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. In CVPR, pages 10285-10295, 2019.
[23] Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsupervised representations. In ICLR, 2021.
[24] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In CVPR, pages 9641-9650, 2020.
[25] Yunfan Li. Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive clustering. In AAAI, 2021.

---

[26] Jian Liang. Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. ICML, 2020.
[27] Jian Liang, Dapeng Hu, and Jiashi Feng. Domain adaptation with auxiliary target domain-oriented classifier. In CVPR, pages 16632-16642, 2021.
[28] Jian Liang, Dapeng Hu, Ran He, and Jiashi Feng. Distill and fine-tune: Effective adaptation from black-box source model. CVPR, 2022.
[29] Jian Liang, Dapeng Hu. Yunbo Wang, Ran He, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. arXiv preprint arXiv:2012.07299 2020.
[30] Jian Liang, Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[31] Hong Liu, Jianmin Wang, and Mingsheng Long. Cycle self-training for domain adaptation. In NeurlPS, 2021.
[32] Mingsheng Long. Yue Cao, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Transferable representation learning with deep adaptation networks. TPAMI, 41(12):307 10007711-38,,..001101111111 2018.
[33] Mingsheng Long, Yue Cao. Jianmin Wang, and Michael I Jordan. Learning transferable features with deep adaptation networks. ICML, 2015.
[34] Mingsheng Long, Zhangjie Cao. Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In NIPS, pages 1647-1657, 2018.
[35] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. In NIPS. pages
2016.
[36] Zhihe Lu, Yongxin Yang, Xiatian Zhu, Cong Liu. Yi-Zhe Song, and Tao Xiang. Stochastic classifiers for unsupervised domain adaptation. In CVPR, pages 9111-9120. 2020.
[37] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 10XKvv:887.0377777 2018
[38] Sinno Jialin Pan and Qiang Yang. A survey transfer learning. TKDE, 22(10):3345-3355 2009.
[39] Xingehao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924. 2017.
[40] Simone Romano, James Bailey, Vinh Nguyen, and Karin Verspoor. Standardized mutual information for clustering comparisons: step further in adjustment for chance. In ICML, pages 1143-1151, 2014.
[41] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models new domains. ECCV, pages 213-226. Springer, 2010.
[42] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. NeurlPS, 33, 2020.
[43] Kuniaki Saito, Donghyun Kim, Piotr Teterwak, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Tune the right way: Unsupervised validation of domain adaptation via soft neighhorhood density. ICCV, pages 9184-9193, 2021.
[44] Kuniaki Saito, Kohci Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR, pages 3723-3732, 2018
[45] Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint 1rXiv:1602.07888. 2016.
[46] Yuming Shen, Ziyi Shen, Menghan Wang, Jie Qin, Philip HS Torr, and Ling Shao. You never cluster alone. In NewrlPS, 2021.
[47] Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. ICLR, 2018
[48] Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adver- sarial networks. In ICLR, 2015.
12

---

[49] Baochen Sun, Jiashi Feng. and Kate Saenko. Return of frustratingly easy domain adaptation. In AAAI. 2016.
[50] Hui Tang, Ke Chen, and Kui Jia. Unsupervised domain adaptation via structurally regularized deep clustering. In CVPR, pages 8725-8735, 2020.
[51] Tsung Wei Tsai, Chongxuan Li, and Jun Zhu. Mice: Mixture of contrastive experts for unsupervised image clustering. In ICLR, 2021.
[52] Eric Tzeng, Judy Hoffiman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR. pages 7167-7176, 2017.
[53] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing domain invariance. arXiv preprint 4rXi::1412.3474, 2014.
[54] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, pages 5018 (11885027,000 2017.
[55] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. CVPR. 2022.
[56] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In ICML, pages 9929-9939. PMLR, 2020.
[57] Ximei Wang. Liang Li, Weirui Ye, Mingsheng Long, and Jianmin Wang. Transfcrable attention for domain adaptation. In AAAI, volume 33, pages 3345-5352) 2019.
[58] Jianlong Wu, Keyu Long, Fei Wang. Chen Qian, Cheng Li. Zhouchen Lin, and Hongbin Zha. Deep comprehensive correlation mining for image clustering. ICCV. pages $150-8159, 2019.
[59] Yuan Wu, Diana Inkpen, and Ahmed Fl-Roby. Dual mixup regularized learning for adversarial domain adaptation. ECCV, 2020.
[60] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised fcature learning via non- parammaeettic instance discrimination. In CVPR, pages 1733-3742, 2018.
[61] Haifeng Xia, Handong Zhao, and Zhengming Ding, Adaptive adversarial network for source-free domain adaptation. In ICCV, pages 1010-9019 2021.
[62] Renjun Xu, Pelen Liu, Liyan Wang, Chao Chen, and Jindong Wang. Reliable weighted optimal transport for unsupervised domain adaptation. In CVPR, pages 3399-4403 2020.
[63] Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In ICCV, Octoher 2019.
[64] Shiqi Yang. Joost van de Weijer, Luis Herranz, Shangling Jui, et al. Exploiting the intrinsic neighborhood structure for fource-free domain adaptation. NeurlPS, 34, 2021.
[65] Shiqi Yang, Yaxing Wang, Joost de Weijer, Luis Herranz, and Shangling Jui. Unsupervised domain adaptation without source data by casting bait. arXiv preprint arXiv:2010. 124220 2020.
[66] Shiqi Yang, Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In ICCV, pages 8978-8987, 2021.
[67] Yabin Zhang, Bin Deng, Kui Jia, and Lei Zhang. Label propagation with augmented anchors: A simple semi-supervised learning bascline for unsupervised domain adaptation. ECCV, pages 781-797, 2020.
[68] Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan. Domain-symmetric networks for adversarial domain adaptation. In CVPR, pages 5031-5040) 2019.
[69] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In ICML, pages 7404-7413, 2019.
[70] Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual emheddings. In ICCV, pages 6002-6012, 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "yPIxMQ82M6w", "reviewer": "Reviewer_FqZ2", "summary": "This paper proposed a clustering-based method for the Source-Free Unsupervised Domain Adaptation (SFDA) problem. Considering the semantic information embedded in the feature space of the source model, the authors proposed an AaD clustering objective, aiming to simultaneously encourage the model’s discriminability and diversity during the adaptation. Besides, this paper also compared and analogized their method with several previous SFDA methods and gave some insightful analysis. Finally, experimental results on several domain adaptation benchmarks under different SFDA settings verified the efficacy of the proposed method.", "questions": "1. For the description of Algorithm 1 in line 5, it is written, “Update model by minimizing Eq.5”. Does this mean only updating the feature extractor f as implemented in [1], or updating the feature extractor and the classifier g simultaneously?\n2. The choice and the analysis of $\\lambda$: according to the simplified upper-bound of the proposed AaD objective, theoretically, $\\lambda$ = $\\frac{N_{C_i}}{N_{B_i}}$. Although the authors have explained the treatment of $\\lambda$ and the introduction of decay exponent $\\beta$, I am still curious about the numerical connection between the truly utilized $\\frac{N_{C_i}}{N_{B_i}}$ (the theoretical value) and the evolving $\\lambda$.\n3. As we know, pseudo labeling is another widely leveraged technique for the SFDA problem, and it is usually combined with the hard or soft neighborhood clustering methods. I am curious whether the authors have considered combining pseudo labeling methods with the proposed AaD approach.\n\nOther comments:\n\nTypos:\nIn line 298, “Soruce-free” → “Source-free”.", "limitations": "The authors discussed the limitations in section 2.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. Overall, this paper is well-written and easy to follow.\n2. The proposed method is simple but effective, and the analysis between the proposed AaD objective and the previous SFDA methods in section 3.2 is insightful.\n3. The experiment part is rich, which covers a wide range of SFDA SOTA methods and different SFDA settings (e.g., close-set, open-set, and partial set). The experimental results can generally confirm the effectiveness of the proposed method.\n\nWeakness\n\n1. Novelty: Compared with the previous neighborhood clustering-based method and the contrastive learning method, the proposed approach seems to be an incremental work, and the novelty is relatively limited.\n2. The description of the algorithm and the training process is not completely clear. Please refer to the Questions section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Z_7CrVS2EHB", "reviewer": "Reviewer_G5ee", "summary": "The authors propose a simple SFDA method that aims to adapt the model to the target data in a self-supervised manner. The adaptation is based on contrastive learning; positive pairs are constructed by searching kNN of the target sample from memory bank, while all sample pairs within the mini-batch is considered as negative pairs. Differently from popular contrastive learning methods, the output space is used to compute loss in the proposed method, which strongly encourages the model to assign each target data into predefined class labels. The experimental results validate the advantage of the proposed method. \n", "questions": "Related to the concerns described at <quality>:\n- Do we really need P(B_i) to derive the loss function of the proposed method? What is the specific advange of considering P(B_i) here?\n- How is the sensitivity of the proposed method to batch size?\n\nMinor things:\n- Is N_{C_i} different from K?\n\n---\n\n<After reading authors' rebuttal>\n\nSince my concerns on <quality> have been resolved in the rebuttal, I updated my score from 4 to 6.\n", "limitations": "I think the proposed method does not perform well when batch size or the number of target data is too small.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I will describe strengths and weaknesses from four perspectives: originality, quality, clarity and significance.\n\n<originality>\n\nNot highly novel. Methodologically, the proposed method is similar with contrastive learning with InfoNCE loss, but it is different in how to compute similarity scores in the loss function. Specifically, while the positive and negative pairs are constructed at the feature space in a similar way with [R1], the similarity scores in the loss function are computed at the output space. The authors provides some theretical justification on the formulation of the proposed method, though I have a concern on its derivation, which will be described in <quality>.   \n[R1] \"Mean Shift for Self-Supervised Learning,\" ICCV 2021.\n\n<quality>\n\nMarginal. I have two concerns mainly on the derivation of the proposed method.\n- In line 144-145, the authors state \"we cannot get this upper-bound without P(B_i),\" but I respectfully disagree with it. Using exactly the same deformations with Eq. (3) and (4), we can also have the same results by starting from -log P(C_i) except for the difference of the coefficients of the second and third terms at the third line of Eq. (4), which will be ignored in Eq. (5). This is due to the approximation adopted at the second line of Eq. (4), in which the sum over the target data is replaced with that within a mini-batch. Since the derivation of the proposed loss is quite similar with that of InfoNCE loss if we ignore P(B_i), showing the necessity of P(B_i) should be important to validate the novelty as well as the advantage of the proposed method. \n- Due to the approximation mentioned above, it should be important to use a sufficiently large mini-batch as in contrastive learning with InfoNCE loss, which leads to a concern on sensitivity of the performance to batch size. However, the batch size seems to be fixed in all experiments.\n\n<clarity>\n\nGreat. This paper is well-written and also well-organized. Someone might think that Section 2 and Section 3.2 should be integrated, but I think it depends on preference.\n\n<significance>\n\nGood. This paper provides some significant results in terms of showing that such a simple method can perform on par or better than other existing methods in several benchmark datasets. It is also a good news to be able to effectively tune the hyperparameter beta in an unsupervised way.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MeZ7DlfC0uC", "reviewer": "Reviewer_KU52", "summary": "This paper approaches the source-free domain adaptation problem through the lens of clustering, which forces similar features to cluster and dissimilar features to disperse. A simple upper bound for the objective is introduced and optimized. On VisDA and Office-31, the proposed method achieves state-of-the-art performance.", "questions": "- The legends in Figure 2 are too small to read.", "limitations": "I couldn't find any discussion of the work's limitations or potential negative societal impact by the authors.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The paper is well written and easy to read. \n- Impressive performance improvement in VisDA over previous SOTA. \n\n### Weaknesses\n- The proposed method's concept and effect are very similar to previous contrastive learning methods, such as InfoNCE. InfoNCE or SimCLR also set positive/negative pairs and force the positive pairs to attract while the negative pairs disperse. The difference between SimCLR and the proposed method appears to be marginal; SimCLR applies the loss to the projection layer, whereas the proposed method applies the loss to the output layer. \n- Using the nearest neighbor as a positive pair in contrastive learning is not a new discovery for the deep learning community. [1] \n- The experimental results are not promising. The proposed method performs well on VisDA, but it only outperforms HCL by 0.1 percent on Office-31 and fails to achieve state-of-the-art performance on Office-Home. \n- Storing all target features in a memory bank and retrieving K-nearest neighbors from the memory bank for each batch appears to be a significant computational burden. Have the authors thought about this and computed the computational complexity of the proposed algorithm? Is this method capable of handling large target datasets? In comparison to existing methods, how much time or FLOPs does the proposed method take? \n\n### Overall \nThe proposed loss objective is straightforward, and the concept is simple to grasp. The theoretical novelty and significance of the contributions, however, appear to be insufficient for this venue.\n\n[1] Dwibedi, Debidatta, et al. \"With a little help from my friends: Nearest-neighbor contrastive learning of visual representations.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BFQNuOOaXA", "reviewer": "Reviewer_G5fg", "summary": "This paper proposed an efficient and effective approach for source-free domain adaptation, which encourages nearest neighbors in the feature space to be similar, while the rest of the samples within the minibatch are dissimilar.", "questions": "1. This paper claims the search of nearest neighbors within the mini-batch is efficient but has not been validated. Presenting the complexity analysis and the compute time would help to validate this claim.\n\n2. Some details of the model are not discussed in detail. For example, it is not clearly presented how to calculate the probability that the feature, $p_{ij}$, and how it has been involved in the back-propagation.\n\n3. What's the memory requirement for building up two memory banks that store all the target features?\n\n4. The experiment results on DA datasets show that the proposed method has higher average accuracy than the traditional DA approaches. How does the proposed method perform better than those algorithms without accessing the source samples and labels?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "A well-written paper that has clear math notations and solid derivations. And the results on both DA dataset and Source-free open-set DA datasets looks promising. However, although the intuition for the Attracting-and-Dispersing strategy is super clear but not much theoretical support is provided. Also, some details of the model are not discussed in detail and the claim on the efficiency needs to be validated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept