Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation
Authors: Shiqi Yang
Yaxing Wang
Joost van de weijer
Luis Herranz
SHANGLING JUI
Venue: NeurIPS 2021 Poster (2021)
Keywords: source-free domain adaptation
reciprocal nearest neighbors

Abstract: Domain adaptation (DA) aims to alleviate the domain shift between source domain and target domain. Most DA methods require access to the source data, but often that is not possible (e.g. due to data privacy or intellectual property). In this paper, we address the challenging source-free domain adaptation (SFDA) problem, where the source pretrained model is adapted to the target domain in the absence of source data. Our method is based on the observation that target data, which might no longer align with the source domain classifier, still forms clear clusters. We capture this intrinsic structure by defining local affinity of the target data, and encourage label consistency among data with high local affinity. We observe that higher affinity should be assigned to reciprocal neighbors, and propose a self regularization loss to decrease the negative impact of noisy neighbors. Furthermore, to aggregate information with more context, we consider expanded neighborhoods with small affinity values. In the experimental results we verify that the inherent structure of the target features is an important source of information for domain adaptation. We demonstrate that this local structure can be efficiently captured by considering the local neighbors, the reciprocal neighbors, and the expanded neighborhood. Finally, we achieve state-of-the-art performance on several 2D image and 3D point cloud recognition datasets. Code is available in https://github.com/Albert0147/SFDA_neighbors.

Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation
Shiqi Yang', Yaxing Wang Joost van Weijer', Luis Herranz', Shangling Jui Computer Vision Center, Universitat Autonoma de Barcelona, Barcelona, Spain
2 PCALab, Nanjing University of Science and Technology, China Huawei Kirin Solution, Shanghai, China es, .ui.ggl iianggiingg n hngghaawei.... com
(syang, yaxi ng, joost, Iherranz
Abstract
Domain adaptation (DA) aims alleviate the domain shift between source domain and target domain. Most DA methods require access to the source data, but often that is not possible (e.g. due data privacy or intellectual property). In this paper, we address the challenging source-free domain adaptation (SFDA) problem, where the source pretrained model is adapted to the target domain the absence of source data. Our method is based on the observation that target data, which might no longer align with the source domain classifier, still forms clear clusters. We capture this intrinsic structure by defining local affinity of the target data, and encourage label consistency among data with high local affinity, We observe that higher affinity should be assigned to reciprocal neighbors, and propose self regularization loss to decrease the negative impact of noisy neighbors. Furthermore, to aggregate information with more context, consider expanded neighborhoods with small affinity values. the experimental results we verify that the inherent structure of the target features is important source of information for domain adaptation. We demonstrate that this local structure can efficiently captured by considering the local neighbors, the reciprocal neighbors, and the expanded neighborhood. Finally, we achieve state-of of-the-art performance on several 2D image and 3D point cloud recognition datasets. Code is available in https: https:///ittann tht.b...tuuuuu//.////// com A hllm/Albeete www.leert0111. SSDAA wew.orrigghhss...
Introduction
Most deep learning methods rely on training on large amount of labeled data, while they cannot generalize well to a related yet different domain. One research direction to address this issue is Domain Adaptation (DA), which aims to transfer learned knowledge from a source to target domain. Most existing DA methods demand labeled source data during the adaptation period, however, it often not practical that source data are always accessible, such as when applied on data with privacy or property restrictions. Therefore, recently, there have emerged a few works [16] 17 20 21
tackling a new challenging DA scenario where instead of source data only the source pretrained model is available for adapting, i.e., source-free free domain adaptation (SFDA). Among these methods, USFDA [6 addresses universal DA [57] and [17] addresses open-set DA (3) In both universal and open-set DA the label set is different for source and target domains. SHOT [1] and 3C-GAN (0) are for closed-set DA where source and target domains have the same categories. 3C-GAN 20 is based target-style image generation with conditional GAN, and SHOT [2]] is based on mutual
information maximization and pseudo labeling. Finally, BAIT (56) extends MCD 35 to the SFDA
*Corresponding Author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

M
Had nH
(a)
Nearesi noightor
nANN
Target samples
misclesslfed
samples
RNN
Decision
Boundary
Pushing features to semantically similar neighbors
Before Adaptation
After Adaptation
(c)
Figure 1: (a) 1-SNE visualization of target features by source model. (b) Ratio of different type of nearest neighbor features of which: the predicted label is the same the feature, K is the number of nearest neighbors. The features in (a) and (b) are on task Ar-RR of Office-Home, (c) Illustration of method. In the left shows distinguish reciprocal and non-reciprocal neighbors. The adaptation achieved by pushed the features towards reciprocal neighbors heavily.
setting. However, these methods ignore the intrinsic neighborhood structure of the target data in feature space which can be very valuable to tackle SFDA.
In this paper, focus closed-set source-free domain adaptation. Our main observation that current DA methods do not exploit the intrinsic neighborhood structure of the target data. We use this term to refer to the fact that, even though the target data might have shifted the feature space (due to the covariance shift), target data of the same class is still expected to form a cluster in the embedding space. This can be implied to some degree from the t-SNE visualization of target features on the source model which suggests that significant cluster structure is preserved (see Fig. 1 (a)). This assumption is implicitly adopted by most DA methods, as instantiated by recent DA work A well-established way to assess the structure of points in nigh-dimensional spaces is by considering the nearest neighbors of points, which are expected to belong to the same class. However, this assumption is not true for all points; the blue curve in Figure 1(b) shows that around 75% of the nearest neighbors has the correct label. In this paper, we observe that this problem can be mitigated by considering reciprocal nearest neighbors (RNN); the reciprocal neighbors of point have the point as their neighbor. Reciprocal neighbors have been studied before in different contexts [4 60). The reason why reciprocal neighbors more trustworthy is illustrated in Fig, C). Fig. Tíb) shows the ratio of neighbors which have the correct prediction for different kinds nearest neighbors. The curves show that reciprocal neighbors indeed have more chances to predict the true label than non-reciprocal nearest neighbors (nRNN).
The above observation and analysis motivate us assign different weights to the supervision from nearest neighbors. Our method, called Neighborhood Reciprocity Clustering (NRC), achieves source- free domain adaptation by encouraging reciprocal neighbors to concord in their label prediction. In addition, we will also consider a weaker connection to the non-reciprocal neighbors. We define affinity values describe the degree of connectivity between each data point and its neighbors, which is also utilized to encourage caassconnistencyy between neighbors, and we propose to use self-regularizationn decrease the negative impact of potential noisy neighbors. Furthermore, inspired by recent graph based methods [
3, 61 which show that the higher order neighbors can provide relevant context, and also considering neighbors of neighbors more likely provide datapoints that are close on the data manifold (3). Thus, to aggregate wider local information, we further retrieve the expanded neighbors, i.e, neighbor of the nearest neighbors, for auxiliary supervision.
Our contributions can be summarized as follows, to achieve source-free domain adaptation: (i) we explicitly exploit the fact that same-class data forms cluster in the target embedding space, do

---

this by considering the predictions of neighbors and reciprocal neighbors, (ii) we further show that considering an extended neighborhood of data points further improves results (iii) the experiments results on three 21) image datasets and one 3D) point cloud dataset show that our method achieves state-of-the-ar performance compared with related methods.
2 Related Work
Domain Adaptation. Most DA methods tackle domain shift by aligning the feature distributions. Early DA methods such [23 41 45] adopt moment matching to align feature distributions. And in recent years, plenty of works have emerged that achieve alignment by adversarial training. DANN 7 formulates domain adaptation an adversarial two-player game. The adversarial training of CDAN [24] is conditioned on several sources of information. DIRT-T (40) performs domain adversarial training with an added term that penalizes violations of the cluster assumption. Addi- tionally, [8 26 35|
local or category- adopts prediction diversity between multiple learnable classifiers to achieve alignmen
feature alignment between source and target domains. AFN [52) shows that the erratic discrimination of target features stems from much smaller norms than those found in the source features. SRDC [22 proposes to directly uncover the intrinsic target discrimination via discriminative clustering to achieve adaptation. More related, [27 resorts to K-means clustering for open-set adaptation while considering global structure. Our method instead only focuses on nearest neighbors (local structure) for source-free adaptation.
Source-free Domain Adaptation. Source-presen, methods need supervision from the source domain during adaptation. Recently, there are several methods investigating source-free domain adaptation. USFDA [66] and FS [17 explore source-free universal DA (57] and open-set DA [36,, and they propose to synthesize extra training samples to make the decision boundary compact, thereby allowing to recognise the open classes. For closed-set DA setting. SHOT (2] proposes to fix the source classifier and match the target features to the fixed classifier by maximizing mutual information and a proposed pseudo label strategy which considers global structure. 3C-GAN (20 synthesizes labeled target-style training images based on the conditional GAN to provide supervision for adaptation. Finally, SFDA 22 is for segmentation based on synthesizing fake source samples.
Graph Clustering. Our method shares some similarities with graph clustering work such as [38 48 [54 (55) by utilizing neighborhood information. However, our methods are fundamentally different. Unlike those works which require labeled data to train the graph network for estimating the affinity, we instead adopt reciprocity to assign affinity.
3 Method
Notation. We denote the labeled source domain data with n, samples as D= [(ff, Th where the yf is the corresponding label of x;, and the unlabeled target domain data with n, samples as D. = {x/));; Both domains have the same C classes relosed-set setting). Under the SFDA setting DD is only available for model pretraining. Our method is based on neural network, which we split into two parts: a feature extractor f. and a classifier g. The feature output by the feature extractor is denoted as zxx) f (x), the output of network is denoted as p(x) = d(g(2)) E RC where ó the softmax function, for readability we will abandon the input and use 2, p in the following sections.
Overview. We assume that the source pretrained model has already been trained. As discusses in the introduction, the target features output by the source model form clusters. We exploit this intrinsic structure of the target data for SFDA by considering the neighborhood information, and the adaptation is achieved with the following objective:
Σ Σ B. E)Neghhh())))
Dsim (pe, Pj)
T4
Daio
where the Neigh(a::) means the nearest neighbors of T,, Dsim computes the similarity between predictions, and Das is constant measuring the semantic distance (dissimilarity) between data. The principle behind the objective to push the data towards their semantically close neighbors by encouraging similar predictions. In the next sections, will define Dsim and Dais-
3

---

3.1 Encouraging Class-Consisteccy with Neighborhood Affinity
To achieve adaptation without source data, we use the prediction of the nearest neighbor encourage prediction consistency. While the target features from the source model are not necessarily totally intrinsic discriminative, meaning some neighbors belong to different class and will provide the wrong supervision. To decrease the potentially negative impact of those neighbors, we propose to weigh the supervision from neighbors according to the connectivity (semantic similarity). We define affinity values to signify the connectivity between the neighbor and the feature, which corresponds to the 1 in Eq. indicating the semantic similarity. Das
To retrieve the nearest neighbors for batch training, similar 33 50
62
build two memory
banks: F stores all target features, and S stores corresponding prediction scores:
F= [2 22, and [pt, .2..... Pr.]]
We use the cosine similarity for nearest neighbors retrieving. The difference between ours and 33 50
lies in the fact that we utilize the memory bank to retrieve nearest neighbors while [3 50 adopts the memory bank to compute the instance discrimination loss. Before every mini-batch training, we simply update the old items in the memory banks corresponding to current mini-batcly Note that updating the memory bank is only done replace the old low-dimension vectors with new ones computed by the model, and does not require any additional computation.
We then use the prediction of the neighbors to supervise the training weighted by the affinity values, with the following objective adapted from Eq. [
LN
1 The Σ
Σ KEN
AuS, Ps
(3)
where we use the dot product to compute the similarity between predictions, corresponding to Dsim in Eq [ the l is the index of the k-th nearest neighbors of z, S₁ is the k-th item in memory bank S, A.k is the affinity value of k-th nearest neighbors of feature 2, Here the NVx is the index set of the K K-nearest neighbors of feature z,. Note that all neighbors are retrieved from the feature bank F. With the affinity value as weight, this objective pushes the features to their neighbors with strong connectivity and to lesser degree to those with weak connectivity.
To assign larger affinity values to semantic similar neighbors, we divide the nearest neighbors retrieved into two groups: reciprocal nearest neighbors (RNN) and non-reciprocal nearest neighbors (nRNN). The feature zj is regarded as the RNN of the feature z, if it meets the following condition:
E N AiE
(4)
Other neighbors which do not meet the above condition are nRNN. Note that the normal definition of reciprocal nearest neighbors (31 applies K M, while in this paper K and M can be different. We find that reciprocal neighbors have a higher potential to belong to the same cluster as the feature (Fig. [b)).. Thus, we assign high affinity value to the RNN features. Specifically for feature z,, the affinity value of its j-th K-nearest neighbor is defined as:
A1,5 = {
j E N A otherwise.
(5)
where r is a hyperparameter. If not specified is set 0.1.
To further reduce the potential impact of noisy neighbors in NK. which belong to the different class but still are RNN, propose simply yet effective way dubbed reguaraeulai,,aaioon that is, to not ignore the current prediction of ego feature:
1 The Σ>
Leels
ST.
where S; means the stored prediction in the memory bank, note this term is a constant vector and identical to the P: since we update the memory banks before the training, here the loss only mack-propagated for variable p;-
All indexes are in the same order for the dataset and memory banks.

---

Algorithm 1 Neighborhood Reciprocity Clustering for Source-free Domain Adaptation
Require: D, (only for source model training), Dr
I: Pre-train model on D,
2: Build feature bank FF and score bank S for Dr
3: while Adaptation do
Sample batch 7 from D;
5: Update J and S corresponding to current batch T
6: Retrieve nearest neighbors /V for each of T
7: Compute affinity value A
Rq
8: Retrieve expanded neighborhoods E for each of A
9: Compute loss and update the model
Eq
10: end while
To avoid the degenerated solution 8 39 where the model predicts all data as some specific classes (and does not predict other classes for any of the target data), we encourage the prediction to be balanced. We adopt the prediction diversity loss which is widely used in clustering [8] 9 13 and also in several domain adaptation works
21 39
Lais Σ (pol).)))) with P.
1 Σn! 14
pf and ((==I..,,C) = C
where the (0) is the score of the e-th class and Pe the empirical label distribution, it represents the predicted possibility of class and is uniform distribution.
3.2 Expanded Neighborhood Affinity
As mentioned in Sec. simple way to achieve the aggregation more information is by considering more nearest neighbors. However, drawback is that larger neighborhoods are expected to contain more datapoint from multiple classes, defying the purpose of class consistency. include more target features is by considering the M-nearest neighbor of each neighbor in N/x of z; Eq. i.e., the expanded neighbors. These target features are expected to closer on the target data manifold than the features that are included by considering larger number of nearest neighbors (43. The expanded neighbors of feature z; are defined as EM (zz) = NM(2,) Vjj (2:) note that Ear (z:) is still an index set and i (ego feature) & Em(z:). We directly assign small affinity value r to those expanded neighbors, since they are further than nearest neighbors and may contain noise. We utilize the prediction of those expanded neighborhoods for training:
better way to
LE --=--CE Σ
1hg Σ
TST
KEN mE E
where Ebr contain the M-nearest neighbors of neighbor k in Nx.
Although the affinity values of all expanded neighbors are the same, it does not necessarily mean that they have equal importance. Taking a closer look at the expanded neighbors (M,),,, some neighbors will show up more than once, for example 2m can the nearest neighbor of both 24 and zj where h.j E Vx(2;), and the nearest neighbors can also serve as expanded neighbor. It implies that those neighbors form compact cluster, and we posit that those duplicated expanded neighbors have potential to be semantically closer to the ego-feature 2;. Thus, we do not remove duplicated features in Ex(2) as those can lead to actually larger affinity value for those expanded neighbors. This is one advantage of utilizing expanded neighbors instead of more nearest neighbors, will verify the importance of maintaining the duplicated features the experimental section.
Final objective. Our method, called Neighborhood Reciprocity Clustering (NRC), is illustrated in Algorithm. |
The final objective for adaptation is:
C == Laiv + LN + LE + Leelf.
4 Experiments
Datasets. We use three 2D image benchmark datasets and a 3D point cloud recognition dataset. Office-31 [32] contains 3 domains (Amazon, Webcam, DSLR) with 31 classes and 4,652 images.
5

---

Table 1: Accuracies (%) on Office-31 for ResNet50- methods.
Method
SF
A-+D
A-W
D-W
W-+D
D->
W-+
Avg
MCD
X
88.6
98.5
100.0
69.5
86.5
CDAN 24
X
92.9
98.6
100.0
71.0
87.7
MDD 59
X
90.4
98.7
99.9
75.0
88.0
BNM 4
X
98.5
100.0
70.9
87.1
DMRL
X X
90.8
99.0
100.0
73.0
87.9
BDG 15
93.6
93.6
99.0
100.0
73.2
72.0
88.5
MCC
X
95.4
98.6
100.0
72.6
73.9
89.4
SRDC
X
95.7
100.0
76.7
90.8
RWOT [S]]
X
95.1
99.5
100.0
77.5
90.8
RSDA-MSTN 10
X
95.8
96.1
99.3
100.0
77.4
78.9
91.1
SHOT
90.1
74.7
88,6
3C-GAN
98.5
75.3
89.6
NRC
96.0
90.8
99.0
100.0
75.3
75.0
89.4
Table 2: Accuracies (%) Office-Home for ResNet50-based methods.
Method
AAr--CAAr- RR RWCC--ACC-PrCC- PrriCC---P-AArrrr AePP--AAPt-CIPr-----
Rtwwa Wewww-Praa
MCD
35
48.9
68.3
74.6
61.3
67.6
68.8
57.0
47.1
75.1
69.1
52.2
79.6
64.1
CDAN
24
50.7
70.6
76.0
57.6
70.0
70.0
57.4
50.9
77.3
70.9
56.7
81.6
65.8
SAFN
52.0
71.7
76.3
64.2
69.9
71.9
63.7
51.4
77.1
70.9
57.1
81.5
67.3
Symeets
47.7
72.9
64.2
74.2
64.2
79.5
$2.6
82.7
MDD 50
54.9
73.7
77.8
60.0
71.4
71.8
61.2
53.6
78.1
72.5
60.2
82.3
68.1
TADA 47
X
53.1
72.3
77.2
59.1
71.2
72.1
59.7
53.1
78.4
72.4
60.0
82.9
67.6
BNM
52.3
73.9
80.0
63.3
72.9
74.9
61.7
49.5
79.7
70.5
53.6
82.2
67.9
BDG
51.5
73.4
65.3
73.7
65.1
55.1
84.8
SRDC 42
52.3
76.3
81.0
69.5
76.2
78.0
68.7
53.8
81.7
76.3
57.1
85.0
RSDA-MSTN (T0
53.2
77.7
81.3
66.4
74.0
76.5
67.9
53.0
82.0
75.8
57.8
85.4
70.9
SHOT
21
57.1
78.1
81.5
68.0
78.2
78.1
67.4
54.9
82.2
73.3
58.8
84.3
71,8
NRC
57.7
80.3
82.0
68.1
79.8
78.6
65.3
56.4
83.0
71.0
58.6
85.6
72.2
Office-Home [46 contains 4 domains (Real, Clipart, Art, Product) with 65 classes and total of 15,500 images. VisDA (8 more challenging dataset, with 12-class synthetic-tor recognition tasks, its source domain contains of 152k synthetic images while the target domain has 55k real object images. PointDA-10 (30 is the first 3D point cloud benchmark specifically designed for domain adaptation, it has 3 domains with 10 classes, denoted as ModelNet- 10, ShapeNett- 10 and ScanNet-10, containing approximately 27.7k training and testing images together.
object
Evaluation. We compare with existing cource-present and source-free DA methods. All results are the average on three random runs. SF in the tables denotes source-free.
Model details. For fair comparison with related methods, we also adopt the backbone of ResNet-50 [11] for Office-Home and ResNet- 101 for VisDA, and PointNet [29] for PointDA- 10. Specifically, for 2D image datasets, we use the same network architecture as SHOT (2]) i.e., the final part of the network is: fully connected layer fully connected layer with weight normalization (37]. And for PointDA-10 (29], we use the code released by the authors for fair comparison with PointDAN 22), and only use the backbone without any of their proposed modules. To train the source model, we also adopt label smoothing as SHOT does. We adopt SGD with momentum 0.9 and batch size of 64 for all 2D datasets, and Adam for PointDA-10, The learning rate for Office-31 and Office-Home is set to le-3 for all layers, except for the last two newly added fe layers, where we apply le-2. Learning rates are set times smaller for VisDA. Learning rate for PointDA-10 is set to le-6. We train 30 epochs for Office-31 and Office- Home while epochs for VisDA, and 100 for PointDA-10. For the number of nearest neighbors (K) and expanded neighborhoods (M), we use 3,2 for Office-31, Office- Office-Home and PointDA- 10, since VisDA is much larger we set K, to 5. Experiments are conducted on a TITAN Xp.
Batch Normalization [12]
4.1 Results
2D image datasets. We first evaluate the target performance our method compared with existing
DA and SFDA methods three 2D image datasets. As shown in Table 3 the top part shows results for the nource-present methods with access source data during adaptation. The bottom shows results for the source-free DA methods. On Office-3L, our method gets similar results compared

---

Table Accuracies (%) on VisDA-C (Synthesis -+ Real) for ResNet1011 methods.
Method
SF
planc beycl
bus
car
horse knife meycl person plant sktbrd train truck Per-class
ADR 34
X 94.2
48.5
84.0
72.9 90.1
74.2
92,6
72.5
80.8
61.8
82.2
28.8
73.5
CDAN 24
X
85.2
66.9
83.0 50.8
84.2
74.9
88.1
74.5
83.4
76.0
81.9
38.0
73.9
CDAN+BSP
92.4
61.0
81.0
57.5
89.0
80.6
90.1
77.9
82.1
38.4
75.9
SAFN 52
93.6
61.3
84.1
70.6
94.1
79.0
91.8
79.6
89.9
55.6
89.0
24.4
76.1
SWD 19
X
90.8
82.5
81.7
70.5
91.7
69.5
86.3
77.5
87.4
63.6
29.2 85.6
76.4
MDD 59
74.6
DMRL.
75.5
MCC 1S
x 88.7
80.3
80.5
71.5
90.1
93.2
85.0
71.6
89.4
73.8
85.0
36.9
78.8
STAR 226
X
95.0
84.0
84.6
73.0
91.6
91.8
85.9
78.4
94.4
84.7
87.0
42.2
82.7
RWOT [51
95.1
80.3
83.7
90.0
92.4
68.0
92.5
82.2
87.9
78.4
90.4
68.2
84.0
3C-GAN 20
94.8
73.4
68.8
74.8
93.1
95.4
88.6
84.7
89.1
84.7
83.5
48.1
81.6
SHOT 21
94.3
88.5
80.1 57.3
93.1
94.9
80.7
80.3
91.5
89.1
86.3
58.2
82.9
NRC
96.8
91.3
82.4 62.4
96.2
95.9
86.1
80.6
94.8
94.1
90.4
59.7
85.9
Table 4: Accuracies (%) PointDA-10, 10.
The results except ours are from PointDAN 300
Model Shape Model SSaan Shape +Model Shape Scan Sean +Model Scan Shape Avg.
MMD 25
X
57.5
27.9
40.7
26.7
47.3
54.8
42.5
DANN
6
X
58.7
29.4
42.3
30.5
48.1
56.7
44,2
ADDA
44
61.0
30.5
40.4
29.3
48.9
51.1
43.5
MCD
B5
X
62.0
31.0
41.4
31.3
46.8
59.3
45.3
PointDAN
30
X
64.2
33.0
47.6
33,9
49.1
64.1
48.7
Source-only
43.1
17.3
40.0
15.0
33.9
47.1
32.7
NRC
64.8
25.8
59.8
26.9
70.1
68.1
52.6
with source-free method 3C-GAN and lower than source-present method RSDA-MSTN, And our method achieves state-of-the-art performance on Office-Home and VisDA, especially on VisDA our method surpasses the source-free free method SHOT and source source-present method RWOT by a wide margin (3% and 1.9% respectively). The reported results clearly demonstrate the efficiency of the proposed method for source-free domain adaptation. Interestingly, like already observed in the SHOT paper, source-free methods outperform methods that have access to source data during adaptation.
3D point cloud dataset. We also report the result for the PointDA--10. As shown in Table our method outperforms PointDA (30, which demands source data for adaptation and specifically tailored for point cloud data with extra attention modules, by a large margin (4%).
4.2 Analysis
Ablation study on neighbors N,, and affinity A. In the first two tables of Table 5 we conduct the ablation study and the 2-nd row from only training with the diversity loss Cdss- From the remaining rows, several conclusions can be drawn.
office-Home and VisDA. The 1-st contains results from the source model
First, the original supervision, which considers neighbors equally can lead to a decent performance (67.1 I Office-Home) Second. considering higher affinity values for reciprocal neighbors leads to large performance gain (69.1 on Office- Homme.. Last but not the least, the expanded neighborhoods
Table 5: Ablation study of different modules on Office- (left) and VisDA (middle), comparison between using expanded neighbors and larger nearest neighbors (right).
Lavo LN LE
Lg Avg
A
Laio EN
LE
Lg
A
Ace
59.5
44.6
&
Acc
62.1
47.8
VisDA (K=M=5)
85.9
67.1
74.6
VisDA w/o E (K=30)
84.0
69.1
81.5
OH (K=3.M=2)
72.2
65.2
61.2
OH w/o E (K=9)
69.5
72.2
85.9
69.1
82.0

---

Table Runtime analysis on SHOT and our method. For SHOT, pseudo labels are computed each
epoch. 20%, 10% and 5% denote the percentage of target features which are stored in the memory bank.
VisDA
Runtime (secpoch)Per-class (%)
SHOT
618.82
82.9
NRC
540,89
85.9
NRC(20% for memory bank)
507.15
85.3
NRC(10% for memory bank)
499.49
85.2
NRC(5% for memory bank)
499.28
85.1
Office- Hore
90.0
VisDA
Acc on VisDA with different
74
Wro self
w/o self
Acat%
70
72.2
05.0
81.5
82.7
84.8
70
80.0
75.0
50
80
E+E
70.0
AV
E
Figure 2: (Leff and middle) Ablation study of Lietf on Office-Home and VisDA respectively.
(Right) Performance with different VisDA.
can also be helpful, but only when combined with the affinity values A (72.2 on Office-Home). Using expanded neighborhoods without affinity obtains bad performance (65,2 on Office-Home We conjecture that those expanded neighborhoods, especially those neighbors of nRNN, may be noisy as discussed in Sec. 3.2 Removing the affinity means we treat all those neighbors equally, which not reasonable.
We also show that duplication in the expanded neighbors important in the last row of Table 5 where the Cg means we remove duplication in Eq. 8 The results show that the performance will degrade significantly when removing them, implying that the duplicated expanded neighbors are indeed more important than others.
Next ablate the importance of the expanded neighborhood in the right of Table5 We show that we increase the number of datapoints considered for class- cass-connsstency by simply considering a larger K, obtain significantly lower scores. We have chosen K so that the total number of points considered is equal to our method (i.e. 5+5#5=30 and 343*2=9). Considering neighbors of neighbors more likely to provide datapoints that close the data manifold [43), and therefore more likely to share the class label with the ego feature.
Runtime analysis. Instead of storing all feature vectors in the memory bank, follow the same memory bank setting as in SS which is for nearest neighbor retrieval. The method only stores a fixed number of target features, (batch size) embeddings from the current training iteration and concatenating them at the end of the memory bank, and discard the oldest elements from the memory bank. We report the results with this type of memory bank of different buffer size in the Table The results show that indeed this could be efficient way reduce computation very large datasets.
update the memory bank at the end of each iteration by taking the
Ablation study on self-regularization. In the left and middle Fig 2 we show the results with and without self-regularization Czels- The Ceels can improve the performance when adopting only nearest neighbors ' or all neighbors N E. The results imply that self-regularizztion can effectively reduce the negative impact of the potential noisy neighbors, especially on the Office-Hone dataset.
Sensitivity to hyperparameter. There are three hyperparameters in our method: K and M which are the number of nearest neighbors and expanded neighbors, which is the affinity value assigned to nRNN. We show the results with different r in the right of Fig. 2 Note we keep the affinity of expanded neighbors as 0.1. r means no affinity. = -1 means treating supervision of nRNN feature as totally wrong, which is not always the case and will lead quite lower result. 0 can also achieve good performance, signifying RNN can already work well. Results with

---

Acc (%6) with different K and M on VisDA
Target
Her Shared
Per Shared Correct
Perations
Figure 3: (Left) The three curves (on VisDA): target accuracy (Blue), ratio of features which have 5-nearest neighbors all sharing the same predicted label (dashed Red), and ratio of features which have 5-nearest neighbors all sharing the same and correct predicted label (dashed Black). (Right) Ablation study on choice of K and M on VisDA.
Predicted Label
90
76
RNN NN before
DRNN NE before
50
DRNN KN
Figure 4: (Left) Ratio of different type of nearest neighbor features which have the correct predicted label, before and after adaptation. (Right) Visualization target features after adaptation.
r= == 0.1/0. 1011/.15//0. 15.2.22 show that our method is not sensitive to the choice of a reasonable r. Note in DA, there is no validation set for hyperparameter tuning, we show the results varying the number of neighbors in the right of Tab. demonstrating the robustness to the choice of and M.
Training curve. We show the evolution of several statistics during adaptation on VisDA in the left of Tab. The blue curve is the target accuracy. The dashed red and black curves are the ratio of features which have 5-nearest neighbors all sharing the same (dashed Red), or the same and also correct (dashed Black) predicted label. The curves show that the target features are clustering during the training. Another interesting finding is that the curve 'Per Shared' correlates with the accuracy curve, which might therefore be used to determine training convergence.
Accuracy of supervision from neighbors. We also show the accuracy of supervision from neighbors on task Rr of Office-Homm in Fig. left). shows that after adaptation, the ratio of all types of neighbors having more correct predicted label, proving the effectiveness of the method.
t-SNE visualization. We show the t-SNE feature visualization task Ar-Rw of target features before (Fig. [aa) and after (Fig. rright) adaptation. After adaptation, the features are more compactly clustered.
5 Conclusions
We introduce source-free domain adaptation (SFDA) method by uncovering the intrinsic target data structure. We propose to achieve the adaptation by encouraging label consistency among local target features. We differentiate between nearest neighbors, reciprocal neighbors and expanded neighborhood. Experimental results verify the importance considering the local structure of the target features. Finally, experimental results on both 2D image and 3D point cloud datasets testify the efficacy our method.
19920019-041744BBBBB (MINECO, Spain) and Acknowledgement We acknowledge the support from Huawei Kirin Solution, and the project
(MICINN, Spain), Ramón y Cajal fellowship RYC2019-027020-1 and the CERCA Programme of Generalitat de Catalunya.

---

References
[1] Kristen M Altenburger and Johan Ugander. Monophily in social networks introduces similarity among friends-of-friends, Nature human behaviour, 2(4):284-290, 2018.
[2] Xinyang Chen. Sinan Wang, Mingsheng Long. and Jianmin Wang. Transferability vs, discriminability: Batch spectral penalization for adversarial domain adaptation. In International Conference on Machine Learning, pages 1081-1090, 2019.
[3] Alex Chin, Yatong Chen, Kristen M. Altenburger, and Johan Ugander. Decoupled smoothing on graphs. In The World Wide Web Conference, pages 263 272, 2019.
[4] Shuhao Cui, Shuhui Wang, Junhao Zhuo, Liang Li, Qingming Huang, and Qi Tian. Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations. CVPR, 2020.
[5] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbon contrastive learning of visual representations. ICCV, 2021.
[6] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by hackpropagation. arXiv preprint 100: 7495, 2014.
[7] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle. François Laviolette. Mario Marchand, and Victor Lempitsky. Domain-adversaare training of neural networks. The Journal of Machine Learning Research, 17(1):2096-200 2030, 2016.
[8] Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In Proceedings of the IEEE international conference on computer vision, pages 5736-5745, 2017.
[9] Ryan Gomes, Andreas Krause, and Pietro Perona. Discriminative clustering by regularized information maximization. 2010.
[10] Xiang Gu, Jian Sun, and Zongben Xu. Spherical space domain adaptation with robust pseudo-label loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9101-9110, 2020.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[12] Sergey loffc and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, arXiv preprint 1502
2015.
[13] Mohammed Jahi, Marco Pedersoli, Amar Mitiche, and Ismail Ben Ayed. Deep clustering: On the link between discriminative models and k-means. IEEE transactions on pattern analysis and machine intelligence, 2019.
[14] Herve Jegou, Hedi Harzallah, and Cordelia Schmid. A contextual dissimilarity measure for accurate and efficient image search. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8. IEEE, 2007.
[15] Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. ECCV, 2020.
[16] Jogendra Nath Kundu, Naveen Venkat, and R Venkatesh Babu. Universal source-free domain adaptation. CVPR, 2020.
[17] Jogendra Nath Kundu, Naveen Venkat, Ambareesh Revanur, R Venkatesh Babu, al. Towards inheritable models for open-set domain adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12376-12385 2020.
[18] Chen Yu Lee. Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[19] Chen- Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10285 10295, 2019.
[20] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9641-9650, 2020.
[21] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. ICML, 2020.
[22] Yuang Liu, Wei Zhang, and Jun Wang. Source-free free domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1215- 1224, 2021.
10

---

[23] Mingsheng Long. Yue Cao. Jianmin Wang, and Michael I Jordan. Learning transferable features with deep adaptation networks. ICML, 2015.
[24] Mingsheng Long. Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pages 1647-1657, 2018.
[25] Mingsheng Long. Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S Yu. Transfer feature learning with joint distribution adaptation. In Proceedings of the IEEE international conference on computer vision. pages 2200-2207, 2013.
[26] Zhihe Lu, Yongxin Yang, Xiatian Zhu, Cong Liu, Yi-Zhe Song, and Tao Xiang. Stochastic classifiers for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9111111111111 11-9120, 2020.
[27] Yingwei Pan, Ting Yao, Ychao Li, Chong-War Ngo, and Tao Mci. Exploring aategory-agnostic clusters for open-set domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13867-13875, 2020.
[28] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang. and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924, 2017.
[29] Charles Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. Proceedings of the IEEE conference on computer vision and pattern recogmition, pages 652-660, 2017.
[30] Can Qin, Haoxuan You, Lichen Wang. C-C Jay Kuo, and Yun Fu. Pointdan: A multi-scale 3d domain adaption network for point cloud representation. Advances in Neural Information Processing Systems, 32:7192-7203, 2019.
[31] Danfeng Qin. Stephan Gammeter, Lukas Bossard, Till Quack. and Luc Van Gool. Hello neighbor: Accurate object retrieval with k-reciprocal nearest neighbors. In CVPR 2011, pages 777-784. IEEE, 2011.
[32] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pages 213 213-226. Springer, 2010.
[33] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. Advances in Neural Information Processing Systems, 33. 2020.
[34] Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Adversarial dropout regularization. ICLR, 2018.
[35] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3723-3732, 2018
[36] Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adaptation by backpropagation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 153-168, 2018.
[37] Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint
2016.
[38] Saquib Sarfraz, Vivek Sharma, and Rainer Stiefelhagen. Efficient parameter-free clustering using first neigh- bor relations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8934 8943, 2019.
[39] Yuan Shi and Fei Sha. adaptation. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pages 1275-1282, 2012.
learning of discriminative clusters for unsupervised domain
[40] Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. ICLR, 2018
[41] Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Thirtieth AAAI Conference on Artificial Intelligence. 2016.
[42] Hui Tang, Ke Chen, and Kui Jia. Unsupervised domain adaptation via structurally regularized deep clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8725-8735, 2020.
[43] Joshua Tenenbaum, De Silva, and John C Langford. global geometric framework for nonlinear dimensionality reduction. science, 290(5500):2319-2555 233-2233, 2000.
[44] Eric Tzeng, Judy Hoffman, Kate Sacnko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7167-7176. 2017.
[45] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint 111 2014.
11

---

[46] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018-5027, 2017.
[47] Ximei Wang, Liang Li, Weirui Ye, Mingsheng Long, and Jianmin Wang. Transferable attention for domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5345-5352, 2019.
[48] Zhongdao Wang. Liang Zheng, Yali Li, and Shengjin Wang. Linkage based face clustering via graph convolution network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1117-1125, 2019.
[49] Yuan Wu, Diana Inkpen, and Ahmed El-Roby. Dual mixup regularized learning for adversarial domain adaptation. ECCV, 2020.
[50] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733-3742, 2018.
[51] Renjun Xu, Pelen Liu. Liyan Wang. Chao Chen, and Jindong Wang. Reliable weighted optimal transport for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 4394 4403, 2020.
[52] Ruijia Xu, Guanbin Li. Jihan Yang, and Liang Lin. Larger norm more transterable: An adaptive feature norm approach for unsupervised domain adaptation. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
[53] Guanglei Yang, Haifeng Xia, Mingli Ding, and Zhengming Ding. Bi-directional generation for unsupervised domain adaptation. In AAAI, pages 661.5-662,, 2020.
[54] Lei Yang, Dapeng Chen, Xiaohang Zhan, Rui Zhao, Chen Change Loy, and Dahua Lin. Learning cluster faces via confidence and connectivity estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13369-13378, 2020.
[55] Lei Yang. Xiaohang Zhan, Dapeng Chen, Junjie Yan, Chen Change Loy, and Dahua Lin. Learning to cluster faces an affinity graph. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2298-2306, 2019.
[56] Shiqi Yang. Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui. Unsupervised domain adaptation without souree data by casting bait, arXiv preprint arXiv:2010. 2010..227. 2020.
[57] Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan, Universal domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2720 2729, 2019.
[58] Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan. Domain-so Domain-symmetric networks for adversarial domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5031-5040, 2019.
[59] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pages 1404-7413. 2019.
[60] Zhun Zhong, Liang Zheng, Donglin Cao, and Shaozi Li. Re-ranking person re-identiffiction with k- reciprocal encoding. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition. pages 1318-1327. 2017.
[61] Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33, 2020.
[62] Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6002-6012, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
12

---

2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL.)? [Yes] We attach the code the supplemental material.
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they chosen)? [Yes] As in the model details in Sec
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)? [Yes] All main results are average over three running with random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "vLCOtruRYs", "reviewer": "Reviewer_R8cb", "summary": "This paper addresses the problem of Source Free Domain Adaptation for the classification task. SFDA is a challenging DA scenario where instead of source data only the source pretrained model is available for adaptation to unlabeled target. \n\nThis work focuses on closed-set domain adaptation task. The approach is motivated from an interesting observation that though targe samples do not align with the source classifier, these samples still form clusters with consistent class semantics. Following this, the authors aim to leverage this local neighborhood structure by encouraging label consistency among samples with high neighborhood affinity. The affinity resembles the degree of connectivity which is defined separately for general neighbors (nRNN), reciprocal neighbors (RNN), and extended neighbors (i.e., neighbor of neighbor).\n\nThe approach involves maintaining two memory banks, a) features and b) prediction scores for all the target samples. While neighbors are selected based on feature space similarity, the adaptation losses aim to maximize the affinity weighted prediction score similarity among the neighbors.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n+The authors introduce a source-free domain adaptation method by using the intrinsic data structures. They propose a method for adaptation by encouraging label consistency among local target features by differentiating between nearest neighbors, reciprocal neighbors, and expanded neighbors, and further exploiting that information.\n\n+The paper is well-written with clear justification and ablation analysis of the various learning components. The approach achieves significant improvement on VisDA-C (3%)  and marginal improvement (0.4%) over the prior-art [21] on the Office-Home dataset.\n\n\nWeaknesses:\n\n1) The minibatch update of $p_i$’s coupled with the self-regularization loss $L_{self}$ seems similar to an mommnetum update of $S=[p_1, p_2,..., p_{n_t}]$. While $p_i$ is computed based on the current model weights, $L_{self}$ does not allow drastic changes in $p_i$, thus acting as a regularizer. I am curious wether $L_{self}$ can be replaced by a simple momentum update of $S$. \n\n2) Following the previous, any comments on what would be the behavior if such momentum update is applied while updating $F$. As I see, all the affinity-based losses and self-regularization is applied on $p$ while $z$ is used only for neighborhood selection (i.e. for assigning affinities). What if we apply similar regularization on $z$.\n\n3) The proposed approach does not employ any pseudo-label-based loss as used in other prior arts. I am curious whether such a loss would further improve the adaptation performance. I observe that all the losses are defined on $p$, i.e., a softer form of pseudo-label.\n\n4) I am curious about the importance of the diversity loss, $L_{div}$. I would like to see the performance where all other losses are active but without the diversity loss, $L_{div}$ (in the Left and middle panel of Table 5). This would give us some idea regarding robustness of the proposed approach against imbalanced target.\n\n5) The authors claim that “source-free methods outperform methods that have access to source data during adaptation” (L215). They should have compared the same method with supervision from source data during adaptation. I would like them to show comparisons with other relevant non-SF prior-arts like A1. Also, can the performance of the proposed approach be improved further in the presence of source samples?\n6) The authors have set the affinity value for the expanded neighbors to an arbitrary value of 0.1 (L254). And, the authors stated that expanded neighbors should have less affinity than the nearest neighbors (L165), yet they are using the same affinity, 0.1 for both expanded neighbors and nRNNs (L143, L254). Please clarify.\n\nReferences:\n[A1] Na et al, FixBi: Bridging Domain Spaces for Unsupervised Domain Adaptation [CVPR’21]\n\n\n**** Post-rebuttal ****\n\nI thank the authors for the detailed rebuttal. The rebuttal addresses most of my concerns regarding the limitation which were not discussed before. Below I restate some of these:\na) In presence of source data the performance of the method degrades.\nb) Without diversity loss performance degrades significantly. This raises concerns about whether this method is at all suitable for open-set, partial, or universal DA settings.\nc) The significance of setting different affinities for the expanded neighbors and nRNNs must be clarified thoroughly.\n\nI hope that the authors would include a thorough discussion on the limitations of this approach in their revised draft. I'd like to raise the score to '6: Marginally above the acceptance threshold'.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No. The authors have not discussed any limitations of the approach. Certain limitations would be a) robustness to larger domain shifts, b) robustness to class-imbalanced target, etc. Further, the authors’ response of No to whether this approach could have potential negative societal impacts is a major oversight.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}, {"review_id": "imicABBW4Hf", "reviewer": "Reviewer_tPBC", "summary": "This work proposes a framework to perform Source-free Domain Adaptation by fully exploiting the local representation structure in the latent space. Specifically, it maintains two memory banks for logits and representations, respectively. Using the a sample in the mini-batch as query, it retrieve semantically similar neighborhoods from the bank, and constrains their logits to be similar. In addition, a self-regularization loss is applied to mitigate noisy neighbors, and a diversity loss is used to balance the sizes among different clusters. Experimental results on four benchmark datasets verify the superiority of the proposed method. Also, thorough ablation studies are conducted to analyze the effect of different model components. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. The proposed framework is technically sound, which can exploit the local representation structure from various granularities.\n2. The paper is well written, and it's easy to capture the gist of this work.\n3. Sufficient experiments are conducted to evaluate the whole model and various model components.\n\nWeakness and question:\n\n1. Technically, I'm convinced that the proposed method is able to promote the feature compactness within some feature cluster. However, except for the feature compactness, it is also important to constrain the inter-cluster feature separability so as to guarantee the good performance on target domain. How can the proposed model achieves this important property?\n2. In Eq. 1, authors use the distance towards query sample to depict semantics. Considering the representation bias of each specific instance, why not use the distance towards the cluster center associated to the query sample? This scheme may capture semantics more precisely.\n3. The objectives in Eq. 9 are composed of a diversity term, two neighborhood semantic similarity terms and a self-regularization term. There should be a trade-off among these three kinds of objectives, such that some trade-off parameters might further promote the performance of this model.\n\nJustification of rating:\n\nI am convinced by the technical merit of this paper, and think it as a well-suited framework for Source-free Domain Adaptation. Though there are some points needed to be further clarified, this work is decent enough as a whole.\n\n\n**Post Rebuttal**\n\nI have read the response from authors and other reviews. In general, I admire authors' efforts on exploring the local and global neighborhoods structures in the representation space of multiple domains. Most of my concerns are addressed in the response, and thus I keep my positive rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the proposed method are less discussed in the current version. Also, the potential social impact is lacked in the paper. Authors are encouraged to analyze how their method can potentially benefit and also hamper the related real-world applications.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "d5sa51SZxYO", "reviewer": "Reviewer_meMo", "summary": "This paper introduces a neighborhood affinity based consistency approach for source-free domain adaptation. While some prior adaptation methods use nearest neighbors to obtain pseudo labels for the unlabeled target samples, this work proposes the use of reciprocal nearest neighbors (RNN). The observation is that RNNs show a higher degree of label consistency than nearest neighbors. Accordingly, the adaptation is formulated as improving the label consistency with the Reciprocal Nearest Neighbors of a given target instance. The method yields excellent performance over 4 datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n========\n\na. The motivation for various nearest neighbor strategies is clear (L42-51). The proposed reciprocal nearest neighbor approach is intuitive and makes sense. The simplicity of the approach is a key highlight of this work. \n\nb. Good work on the analysis. The figures are helpful in understanding the effectiveness of the approach. Figure 4 (Left) presents another interesting observation where per-shared fraction (measured without using target labels) converges with target accuracy (more details in the next section). The ablation study is detailed and supports the claims made in the paper. Moreover, the insights presented in this work would be highly useful for research in Source-Free Domain Adaptation.\n\nc. The manuscript is mostly easy to follow (though some clarifications and typographical fixes are required). Overall, it is a good read.\n\nd. The inclusion of code is appreciated.\n\n\nScope of improvement\n=================\n\na. Scalability: The requirement of a memory bank and nearest neighbor computation seems like a large overhead (e.g. consider large scale datasets like ImageNet). While I understand that features $z_i$ and prediction scores $p_i$ are cheaper to memoize than the input data itself, it is imperative to investigate the effect of reducing the space and computational overhead. \n\n* A runtime analysis (training time, FLOPS / epoch etc.) is required to understand the scalability of the proposed framework. Could the authors provide a runtime comparison against the prior state-of-the-art methods? \n\n* Could the authors provide more details on how the nearest neighbors are computed? A typical nearest neighbor computation would entail $O(n_t * n_t)$  operations (quadratic) for each epoch (every target feature is compared with every other feature to obtain the list of nearest neighbors). In my opinion, this is a huge overhead considering that most adaptation algorithms require $O(n_t)$ operations (linear). \n\n* Consider the following approach: Suppose the target training dataset is split into, say, 10 mutually exclusive and exhaustive folds. Each fold pertains to 10% of the data (call this, one mini-epoch) and is trained upon sequentially, i.e., we reduce the memory bank to 10% of the data, and ensure that samples in this mini-epoch pertain to the corresponding fold. After one epoch on the entire dataset is completed (i.e. after 10 mini-epochs), the data is randomly shuffled to create the next 10 folds. In this manner, the model is trained on the entire dataset, however, would require 10 times less overhead for memoization and NN-computation. Furthermore, this could reveal that perhaps only a fraction of the dataset in-memory is sufficient and the performance gain saturates with an increase in samples. If this works, then one of a major drawbacks of the proposed framework could be solved. Could the authors verify this and comment on the efficacy of this approach?\n\nb. Writing:\n\n* L147-149: This part is confusing. My understanding is that $\\mathcal{S}\\_{i}^{T}$ is a constant vector (not “scalar”, L147) containing the prediction scores of $x\\_i^t$. The loss $\\mathcal{L}\\_{self}$ essentially computes a weighted sum (weights = $\\mathcal{S}\\_i$) of the class probabilities ($p\\_i$). The expression is written like so (and not $p\\_i^T \\cdot p\\_i$) because the loss $\\mathcal{L}\\_{self}$ is assumed to back-propagate only through the variable $p\\_i$ and not through the weights ($\\mathcal{S}\\_i$). If this is the case, please consider paraphrasing this part to clarify the same.\n\n* Some additional improvements are suggested in Minor Comments below.\n\nc. Affinity measure: In the current formulation, affinity is a hyperparameter ($A = 1$ or $r<=1$). Thus, affinity is assigning discrete weights to the similarity between pairs of samples in Eq. 1. Did the authors consider using continuous affinity values to obtain a “degree of connectivity” (L55), for e.g. using feature distance $D_{dis} = || z_i - z_j ||$)?\n\nd. The observation in Figure 4 (left) could potentially lead to a reliable convergence criterion. The usual practice in Unsupervised DA is to consider a labeled validation set for determining training convergence / selecting maximum epochs etc. A recent work [P1] presents an approach to determine training convergence without using target labels. In the proposed framework, the per-shared plot in Figure 4 (left) seems to correlate with the target accuracy. This could potentially be used as a convergence criterion. Could the authors provide more insights on this aspect?\n\n[P1] Venkat et al., \"Your Classifier can Secretly Suffice Multi-Source Domain Adaptation\", NeurIPS 2020.\n\n\nMinor Comments\n=============\n\n* A recent work by Wang et al., “Tent: Fully Test-Time Adaptation by Entropy Minimization”, ICLR 2021 also performs source-free adaptation using entropy minimization which results in a simple approach. It would be interesting to compare the runtime vs. performance trade-off between Tent and the proposed neighborhood approach. \n\n* Typos: L273 (consistence -> consistency), L217 (date -> data), L145 (simple -> simply), L115 (weight -> weigh), L103, 232 (discusses -> discussed), Fig. 1 caption (pushed -> pushing), L34 (vary -> very).\n\n* L90, L149, L270: Avoid 1-3 worded lines to save space (paraphrase the sentence).\n\n* L45-51: Here, it would be a good idea to introduce / define RNN and nRNN so that the figure 1(c) is comprehensible. Also, in Fig. 1(c) left, consider showing nearest neighbors connectivity on the left and the RNN/nRNN plot on the right (i.e. swap the two diagrams). This would enhance readability.\n\n\nI would be happy to increase the score if my concerns are addressed.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I could not find a discussion on potential limitations and societal impact. I would suggest the authors add a discussion on these aspects. As an example, the nearest neighbor calculation and memoization operations are compute-heavy, which might put organizations with access to large computational resources at a higher advantage than others.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "W97qac31Mzh", "reviewer": "Reviewer_4Suf", "summary": "The authors propose a source-free domain adaptation (SFDA) method by exploiting the intrinsic target neighborhood structure. Extensive experiments conducted on both 2D image and 3D point cloud datasets demonstrate the effectiveness of the proposed method to a certain extent.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.\tPlease give more explanations and discussions about the motivation of utilizing the memory bank to retrieve nearest neighbors.\n2.\tIs the hyperparameter r used in Eq.5 and Eq.9 the same? Why not choose different hyperparameters for these two parts? \n3.\tPlease provide comparative experiments with more source-free domain adaptation (SFDA) methods, such as [1] [2] [3].\n4.\tIt’s recommended to provide visualization of target features before adaptation in Fig.5 for better comparison.\n5.\tDoes the performance improve significantly at the expense of consuming large computational time cost? Please provide time complexity analysis of the proposed method.\n6.    The current manuscript should be carefully polished, e.g., some grammar errors, the layout of Figures and Tables.\n7.    The submitted code cannot re-implement the results in this paper, and there is a large performance gap. So I recommend to reject this paper.\n\n[1] Kurmi V K, Subramanian V K, Namboodiri V P. Domain Impression: A Source Data Free Domain Adaptation Method[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021: 615-625.\n[2] Tian J, Zhang J, Li W, et al. VDM-DA: Virtual Domain Modeling for Source Data-free Domain Adaptation[J]. arXiv preprint arXiv:2103.14357, 2021.\n[3] Yeh H W, Yang B, Yuen P C, et al. SoFA: Source-data-free Feature Alignment for Unsupervised Domain Adaptation[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021: 474-483.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "35h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a framework to perform Source-free Domain Adaptation by fully exploiting the local representation structure in the latent space. Specifically, it maintains two memory banks for logits and representations, respectively. Using the a sample in the mini-batch as query, it retrieve semantically similar neighborhoods from the bank, and constrains their logits to be similar. In addition, a self-regularization loss is applied to mitigate noisy neighbors, and a diversity loss is used to balance the sizes among different clusters. The reviewers found the paper to be technically sound and the approach interesting. There were some concerns about the technical novelty and the strength of the results, for which the paper can be argued from both a positive and negative perspective, since it builds on previous work (reciprocal NNs) and thus have some incremental flavor. Nevertheless, the positive reviewers highlighted the novelty of the method as a whole. After discussion it was concluded that the paper merits publication.  
Decision: Accept (Poster)