Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation
Authors: Shiqi Yang
Yaxing Wang
Joost van de weijer
Luis Herranz
SHANGLING JUI
Venue: NeurIPS 2021 Poster (2021)
Keywords: source-free domain adaptation
reciprocal nearest neighbors

Abstract: Domain adaptation (DA) aims to alleviate the domain shift between source domain and target domain. Most DA methods require access to the source data, but often that is not possible (e.g. due to data privacy or intellectual property). In this paper, we address the challenging source-free domain adaptation (SFDA) problem, where the source pretrained model is adapted to the target domain in the absence of source data. Our method is based on the observation that target data, which might no longer align with the source domain classifier, still forms clear clusters. We capture this intrinsic structure by defining local affinity of the target data, and encourage label consistency among data with high local affinity. We observe that higher affinity should be assigned to reciprocal neighbors, and propose a self regularization loss to decrease the negative impact of noisy neighbors. Furthermore, to aggregate information with more context, we consider expanded neighborhoods with small affinity values. In the experimental results we verify that the inherent structure of the target features is an important source of information for domain adaptation. We demonstrate that this local structure can be efficiently captured by considering the local neighbors, the reciprocal neighbors, and the expanded neighborhood. Finally, we achieve state-of-the-art performance on several 2D image and 3D point cloud recognition datasets. Code is available in https://github.com/Albert0147/SFDA_neighbors.

Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation
Shiqi Yang', Yaxing Wang Joost van Weijer', Luis Herranz', Shangling Jui Computer Vision Center, Universitat Autonoma de Barcelona, Barcelona, Spain
2 PCALab, Nanjing University of Science and Technology, China Huawei Kirin Solution, Shanghai, China es, .ui.ggl iianggiingg n hngghaawei.... com
(syang, yaxi ng, joost, Iherranz
Abstract
Domain adaptation (DA) aims alleviate the domain shift between source domain and target domain. Most DA methods require access to the source data, but often that is not possible (e.g. due data privacy or intellectual property). In this paper, we address the challenging source-free domain adaptation (SFDA) problem, where the source pretrained model is adapted to the target domain the absence of source data. Our method is based on the observation that target data, which might no longer align with the source domain classifier, still forms clear clusters. We capture this intrinsic structure by defining local affinity of the target data, and encourage label consistency among data with high local affinity, We observe that higher affinity should be assigned to reciprocal neighbors, and propose self regularization loss to decrease the negative impact of noisy neighbors. Furthermore, to aggregate information with more context, consider expanded neighborhoods with small affinity values. the experimental results we verify that the inherent structure of the target features is important source of information for domain adaptation. We demonstrate that this local structure can efficiently captured by considering the local neighbors, the reciprocal neighbors, and the expanded neighborhood. Finally, we achieve state-of of-the-art performance on several 2D image and 3D point cloud recognition datasets. Code is available in https: https:///ittann tht.b...tuuuuu//.////// com A hllm/Albeete www.leert0111. SSDAA wew.orrigghhss...
Introduction
Most deep learning methods rely on training on large amount of labeled data, while they cannot generalize well to a related yet different domain. One research direction to address this issue is Domain Adaptation (DA), which aims to transfer learned knowledge from a source to target domain. Most existing DA methods demand labeled source data during the adaptation period, however, it often not practical that source data are always accessible, such as when applied on data with privacy or property restrictions. Therefore, recently, there have emerged a few works [16] 17 20 21
tackling a new challenging DA scenario where instead of source data only the source pretrained model is available for adapting, i.e., source-free free domain adaptation (SFDA). Among these methods, USFDA [6 addresses universal DA [57] and [17] addresses open-set DA (3) In both universal and open-set DA the label set is different for source and target domains. SHOT [1] and 3C-GAN (0) are for closed-set DA where source and target domains have the same categories. 3C-GAN 20 is based target-style image generation with conditional GAN, and SHOT [2]] is based on mutual
information maximization and pseudo labeling. Finally, BAIT (56) extends MCD 35 to the SFDA
*Corresponding Author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

M
Had nH
(a)
Nearesi noightor
nANN
Target samples
misclesslfed
samples
RNN
Decision
Boundary
Pushing features to semantically similar neighbors
Before Adaptation
After Adaptation
(c)
Figure 1: (a) 1-SNE visualization of target features by source model. (b) Ratio of different type of nearest neighbor features of which: the predicted label is the same the feature, K is the number of nearest neighbors. The features in (a) and (b) are on task Ar-RR of Office-Home, (c) Illustration of method. In the left shows distinguish reciprocal and non-reciprocal neighbors. The adaptation achieved by pushed the features towards reciprocal neighbors heavily.
setting. However, these methods ignore the intrinsic neighborhood structure of the target data in feature space which can be very valuable to tackle SFDA.
In this paper, focus closed-set source-free domain adaptation. Our main observation that current DA methods do not exploit the intrinsic neighborhood structure of the target data. We use this term to refer to the fact that, even though the target data might have shifted the feature space (due to the covariance shift), target data of the same class is still expected to form a cluster in the embedding space. This can be implied to some degree from the t-SNE visualization of target features on the source model which suggests that significant cluster structure is preserved (see Fig. 1 (a)). This assumption is implicitly adopted by most DA methods, as instantiated by recent DA work A well-established way to assess the structure of points in nigh-dimensional spaces is by considering the nearest neighbors of points, which are expected to belong to the same class. However, this assumption is not true for all points; the blue curve in Figure 1(b) shows that around 75% of the nearest neighbors has the correct label. In this paper, we observe that this problem can be mitigated by considering reciprocal nearest neighbors (RNN); the reciprocal neighbors of point have the point as their neighbor. Reciprocal neighbors have been studied before in different contexts [4 60). The reason why reciprocal neighbors more trustworthy is illustrated in Fig, C). Fig. Tíb) shows the ratio of neighbors which have the correct prediction for different kinds nearest neighbors. The curves show that reciprocal neighbors indeed have more chances to predict the true label than non-reciprocal nearest neighbors (nRNN).
The above observation and analysis motivate us assign different weights to the supervision from nearest neighbors. Our method, called Neighborhood Reciprocity Clustering (NRC), achieves source- free domain adaptation by encouraging reciprocal neighbors to concord in their label prediction. In addition, we will also consider a weaker connection to the non-reciprocal neighbors. We define affinity values describe the degree of connectivity between each data point and its neighbors, which is also utilized to encourage caassconnistencyy between neighbors, and we propose to use self-regularizationn decrease the negative impact of potential noisy neighbors. Furthermore, inspired by recent graph based methods [
3, 61 which show that the higher order neighbors can provide relevant context, and also considering neighbors of neighbors more likely provide datapoints that are close on the data manifold (3). Thus, to aggregate wider local information, we further retrieve the expanded neighbors, i.e, neighbor of the nearest neighbors, for auxiliary supervision.
Our contributions can be summarized as follows, to achieve source-free domain adaptation: (i) we explicitly exploit the fact that same-class data forms cluster in the target embedding space, do

---

this by considering the predictions of neighbors and reciprocal neighbors, (ii) we further show that considering an extended neighborhood of data points further improves results (iii) the experiments results on three 21) image datasets and one 3D) point cloud dataset show that our method achieves state-of-the-ar performance compared with related methods.
2 Related Work
Domain Adaptation. Most DA methods tackle domain shift by aligning the feature distributions. Early DA methods such [23 41 45] adopt moment matching to align feature distributions. And in recent years, plenty of works have emerged that achieve alignment by adversarial training. DANN 7 formulates domain adaptation an adversarial two-player game. The adversarial training of CDAN [24] is conditioned on several sources of information. DIRT-T (40) performs domain adversarial training with an added term that penalizes violations of the cluster assumption. Addi- tionally, [8 26 35|
local or category- adopts prediction diversity between multiple learnable classifiers to achieve alignmen
feature alignment between source and target domains. AFN [52) shows that the erratic discrimination of target features stems from much smaller norms than those found in the source features. SRDC [22 proposes to directly uncover the intrinsic target discrimination via discriminative clustering to achieve adaptation. More related, [27 resorts to K-means clustering for open-set adaptation while considering global structure. Our method instead only focuses on nearest neighbors (local structure) for source-free adaptation.
Source-free Domain Adaptation. Source-presen, methods need supervision from the source domain during adaptation. Recently, there are several methods investigating source-free domain adaptation. USFDA [66] and FS [17 explore source-free universal DA (57] and open-set DA [36,, and they propose to synthesize extra training samples to make the decision boundary compact, thereby allowing to recognise the open classes. For closed-set DA setting. SHOT (2] proposes to fix the source classifier and match the target features to the fixed classifier by maximizing mutual information and a proposed pseudo label strategy which considers global structure. 3C-GAN (20 synthesizes labeled target-style training images based on the conditional GAN to provide supervision for adaptation. Finally, SFDA 22 is for segmentation based on synthesizing fake source samples.
Graph Clustering. Our method shares some similarities with graph clustering work such as [38 48 [54 (55) by utilizing neighborhood information. However, our methods are fundamentally different. Unlike those works which require labeled data to train the graph network for estimating the affinity, we instead adopt reciprocity to assign affinity.
3 Method
Notation. We denote the labeled source domain data with n, samples as D= [(ff, Th where the yf is the corresponding label of x;, and the unlabeled target domain data with n, samples as D. = {x/));; Both domains have the same C classes relosed-set setting). Under the SFDA setting DD is only available for model pretraining. Our method is based on neural network, which we split into two parts: a feature extractor f. and a classifier g. The feature output by the feature extractor is denoted as zxx) f (x), the output of network is denoted as p(x) = d(g(2)) E RC where ó the softmax function, for readability we will abandon the input and use 2, p in the following sections.
Overview. We assume that the source pretrained model has already been trained. As discusses in the introduction, the target features output by the source model form clusters. We exploit this intrinsic structure of the target data for SFDA by considering the neighborhood information, and the adaptation is achieved with the following objective:
Σ Σ B. E)Neghhh())))
Dsim (pe, Pj)
T4
Daio
where the Neigh(a::) means the nearest neighbors of T,, Dsim computes the similarity between predictions, and Das is constant measuring the semantic distance (dissimilarity) between data. The principle behind the objective to push the data towards their semantically close neighbors by encouraging similar predictions. In the next sections, will define Dsim and Dais-
3

---

3.1 Encouraging Class-Consisteccy with Neighborhood Affinity
To achieve adaptation without source data, we use the prediction of the nearest neighbor encourage prediction consistency. While the target features from the source model are not necessarily totally intrinsic discriminative, meaning some neighbors belong to different class and will provide the wrong supervision. To decrease the potentially negative impact of those neighbors, we propose to weigh the supervision from neighbors according to the connectivity (semantic similarity). We define affinity values to signify the connectivity between the neighbor and the feature, which corresponds to the 1 in Eq. indicating the semantic similarity. Das
To retrieve the nearest neighbors for batch training, similar 33 50
62
build two memory
banks: F stores all target features, and S stores corresponding prediction scores:
F= [2 22, and [pt, .2..... Pr.]]
We use the cosine similarity for nearest neighbors retrieving. The difference between ours and 33 50
lies in the fact that we utilize the memory bank to retrieve nearest neighbors while [3 50 adopts the memory bank to compute the instance discrimination loss. Before every mini-batch training, we simply update the old items in the memory banks corresponding to current mini-batcly Note that updating the memory bank is only done replace the old low-dimension vectors with new ones computed by the model, and does not require any additional computation.
We then use the prediction of the neighbors to supervise the training weighted by the affinity values, with the following objective adapted from Eq. [
LN
1 The Σ
Σ KEN
AuS, Ps
(3)
where we use the dot product to compute the similarity between predictions, corresponding to Dsim in Eq [ the l is the index of the k-th nearest neighbors of z, S₁ is the k-th item in memory bank S, A.k is the affinity value of k-th nearest neighbors of feature 2, Here the NVx is the index set of the K K-nearest neighbors of feature z,. Note that all neighbors are retrieved from the feature bank F. With the affinity value as weight, this objective pushes the features to their neighbors with strong connectivity and to lesser degree to those with weak connectivity.
To assign larger affinity values to semantic similar neighbors, we divide the nearest neighbors retrieved into two groups: reciprocal nearest neighbors (RNN) and non-reciprocal nearest neighbors (nRNN). The feature zj is regarded as the RNN of the feature z, if it meets the following condition:
E N AiE
(4)
Other neighbors which do not meet the above condition are nRNN. Note that the normal definition of reciprocal nearest neighbors (31 applies K M, while in this paper K and M can be different. We find that reciprocal neighbors have a higher potential to belong to the same cluster as the feature (Fig. [b)).. Thus, we assign high affinity value to the RNN features. Specifically for feature z,, the affinity value of its j-th K-nearest neighbor is defined as:
A1,5 = {
j E N A otherwise.
(5)
where r is a hyperparameter. If not specified is set 0.1.
To further reduce the potential impact of noisy neighbors in NK. which belong to the different class but still are RNN, propose simply yet effective way dubbed reguaraeulai,,aaioon that is, to not ignore the current prediction of ego feature:
1 The Σ>
Leels
ST.
where S; means the stored prediction in the memory bank, note this term is a constant vector and identical to the P: since we update the memory banks before the training, here the loss only mack-propagated for variable p;-
All indexes are in the same order for the dataset and memory banks.

---

Algorithm 1 Neighborhood Reciprocity Clustering for Source-free Domain Adaptation
Require: D, (only for source model training), Dr
I: Pre-train model on D,
2: Build feature bank FF and score bank S for Dr
3: while Adaptation do
Sample batch 7 from D;
5: Update J and S corresponding to current batch T
6: Retrieve nearest neighbors /V for each of T
7: Compute affinity value A
Rq
8: Retrieve expanded neighborhoods E for each of A
9: Compute loss and update the model
Eq
10: end while
To avoid the degenerated solution 8 39 where the model predicts all data as some specific classes (and does not predict other classes for any of the target data), we encourage the prediction to be balanced. We adopt the prediction diversity loss which is widely used in clustering [8] 9 13 and also in several domain adaptation works
21 39
Lais Σ (pol).)))) with P.
1 Σn! 14
pf and ((==I..,,C) = C
where the (0) is the score of the e-th class and Pe the empirical label distribution, it represents the predicted possibility of class and is uniform distribution.
3.2 Expanded Neighborhood Affinity
As mentioned in Sec. simple way to achieve the aggregation more information is by considering more nearest neighbors. However, drawback is that larger neighborhoods are expected to contain more datapoint from multiple classes, defying the purpose of class consistency. include more target features is by considering the M-nearest neighbor of each neighbor in N/x of z; Eq. i.e., the expanded neighbors. These target features are expected to closer on the target data manifold than the features that are included by considering larger number of nearest neighbors (43. The expanded neighbors of feature z; are defined as EM (zz) = NM(2,) Vjj (2:) note that Ear (z:) is still an index set and i (ego feature) & Em(z:). We directly assign small affinity value r to those expanded neighbors, since they are further than nearest neighbors and may contain noise. We utilize the prediction of those expanded neighborhoods for training:
better way to
LE --=--CE Σ
1hg Σ
TST
KEN mE E
where Ebr contain the M-nearest neighbors of neighbor k in Nx.
Although the affinity values of all expanded neighbors are the same, it does not necessarily mean that they have equal importance. Taking a closer look at the expanded neighbors (M,),,, some neighbors will show up more than once, for example 2m can the nearest neighbor of both 24 and zj where h.j E Vx(2;), and the nearest neighbors can also serve as expanded neighbor. It implies that those neighbors form compact cluster, and we posit that those duplicated expanded neighbors have potential to be semantically closer to the ego-feature 2;. Thus, we do not remove duplicated features in Ex(2) as those can lead to actually larger affinity value for those expanded neighbors. This is one advantage of utilizing expanded neighbors instead of more nearest neighbors, will verify the importance of maintaining the duplicated features the experimental section.
Final objective. Our method, called Neighborhood Reciprocity Clustering (NRC), is illustrated in Algorithm. |
The final objective for adaptation is:
C == Laiv + LN + LE + Leelf.
4 Experiments
Datasets. We use three 2D image benchmark datasets and a 3D point cloud recognition dataset. Office-31 [32] contains 3 domains (Amazon, Webcam, DSLR) with 31 classes and 4,652 images.
5

---

Table 1: Accuracies (%) on Office-31 for ResNet50- methods.
Method
SF
A-+D
A-W
D-W
W-+D
D->
W-+
Avg
MCD
X
88.6
98.5
100.0
69.5
86.5
CDAN 24
X
92.9
98.6
100.0
71.0
87.7
MDD 59
X
90.4
98.7
99.9
75.0
88.0
BNM 4
X
98.5
100.0
70.9
87.1
DMRL
X X
90.8
99.0
100.0
73.0
87.9
BDG 15
93.6
93.6
99.0
100.0
73.2
72.0
88.5
MCC
X
95.4
98.6
100.0
72.6
73.9
89.4
SRDC
X
95.7
100.0
76.7
90.8
RWOT [S]]
X
95.1
99.5
100.0
77.5
90.8
RSDA-MSTN 10
X
95.8
96.1
99.3
100.0
77.4
78.9
91.1
SHOT
90.1
74.7
88,6
3C-GAN
98.5
75.3
89.6
NRC
96.0
90.8
99.0
100.0
75.3
75.0
89.4
Table 2: Accuracies (%) Office-Home for ResNet50-based methods.
Method
AAr--CAAr- RR RWCC--ACC-PrCC- PrriCC---P-AArrrr AePP--AAPt-CIPr-----
Rtwwa Wewww-Praa
MCD
35
48.9
68.3
74.6
61.3
67.6
68.8
57.0
47.1
75.1
69.1
52.2
79.6
64.1
CDAN
24
50.7
70.6
76.0
57.6
70.0
70.0
57.4
50.9
77.3
70.9
56.7
81.6
65.8
SAFN
52.0
71.7
76.3
64.2
69.9
71.9
63.7
51.4
77.1
70.9
57.1
81.5
67.3
Symeets
47.7
72.9
64.2
74.2
64.2
79.5
$2.6
82.7
MDD 50
54.9
73.7
77.8
60.0
71.4
71.8
61.2
53.6
78.1
72.5
60.2
82.3
68.1
TADA 47
X
53.1
72.3
77.2
59.1
71.2
72.1
59.7
53.1
78.4
72.4
60.0
82.9
67.6
BNM
52.3
73.9
80.0
63.3
72.9
74.9
61.7
49.5
79.7
70.5
53.6
82.2
67.9
BDG
51.5
73.4
65.3
73.7
65.1
55.1
84.8
SRDC 42
52.3
76.3
81.0
69.5
76.2
78.0
68.7
53.8
81.7
76.3
57.1
85.0
RSDA-MSTN (T0
53.2
77.7
81.3
66.4
74.0
76.5
67.9
53.0
82.0
75.8
57.8
85.4
70.9
SHOT
21
57.1
78.1
81.5
68.0
78.2
78.1
67.4
54.9
82.2
73.3
58.8
84.3
71,8
NRC
57.7
80.3
82.0
68.1
79.8
78.6
65.3
56.4
83.0
71.0
58.6
85.6
72.2
Office-Home [46 contains 4 domains (Real, Clipart, Art, Product) with 65 classes and total of 15,500 images. VisDA (8 more challenging dataset, with 12-class synthetic-tor recognition tasks, its source domain contains of 152k synthetic images while the target domain has 55k real object images. PointDA-10 (30 is the first 3D point cloud benchmark specifically designed for domain adaptation, it has 3 domains with 10 classes, denoted as ModelNet- 10, ShapeNett- 10 and ScanNet-10, containing approximately 27.7k training and testing images together.
object
Evaluation. We compare with existing cource-present and source-free DA methods. All results are the average on three random runs. SF in the tables denotes source-free.
Model details. For fair comparison with related methods, we also adopt the backbone of ResNet-50 [11] for Office-Home and ResNet- 101 for VisDA, and PointNet [29] for PointDA- 10. Specifically, for 2D image datasets, we use the same network architecture as SHOT (2]) i.e., the final part of the network is: fully connected layer fully connected layer with weight normalization (37]. And for PointDA-10 (29], we use the code released by the authors for fair comparison with PointDAN 22), and only use the backbone without any of their proposed modules. To train the source model, we also adopt label smoothing as SHOT does. We adopt SGD with momentum 0.9 and batch size of 64 for all 2D datasets, and Adam for PointDA-10, The learning rate for Office-31 and Office-Home is set to le-3 for all layers, except for the last two newly added fe layers, where we apply le-2. Learning rates are set times smaller for VisDA. Learning rate for PointDA-10 is set to le-6. We train 30 epochs for Office-31 and Office- Home while epochs for VisDA, and 100 for PointDA-10. For the number of nearest neighbors (K) and expanded neighborhoods (M), we use 3,2 for Office-31, Office- Office-Home and PointDA- 10, since VisDA is much larger we set K, to 5. Experiments are conducted on a TITAN Xp.
Batch Normalization [12]
4.1 Results
2D image datasets. We first evaluate the target performance our method compared with existing
DA and SFDA methods three 2D image datasets. As shown in Table 3 the top part shows results for the nource-present methods with access source data during adaptation. The bottom shows results for the source-free DA methods. On Office-3L, our method gets similar results compared

---

Table Accuracies (%) on VisDA-C (Synthesis -+ Real) for ResNet1011 methods.
Method
SF
planc beycl
bus
car
horse knife meycl person plant sktbrd train truck Per-class
ADR 34
X 94.2
48.5
84.0
72.9 90.1
74.2
92,6
72.5
80.8
61.8
82.2
28.8
73.5
CDAN 24
X
85.2
66.9
83.0 50.8
84.2
74.9
88.1
74.5
83.4
76.0
81.9
38.0
73.9
CDAN+BSP
92.4
61.0
81.0
57.5
89.0
80.6
90.1
77.9
82.1
38.4
75.9
SAFN 52
93.6
61.3
84.1
70.6
94.1
79.0
91.8
79.6
89.9
55.6
89.0
24.4
76.1
SWD 19
X
90.8
82.5
81.7
70.5
91.7
69.5
86.3
77.5
87.4
63.6
29.2 85.6
76.4
MDD 59
74.6
DMRL.
75.5
MCC 1S
x 88.7
80.3
80.5
71.5
90.1
93.2
85.0
71.6
89.4
73.8
85.0
36.9
78.8
STAR 226
X
95.0
84.0
84.6
73.0
91.6
91.8
85.9
78.4
94.4
84.7
87.0
42.2
82.7
RWOT [51
95.1
80.3
83.7
90.0
92.4
68.0
92.5
82.2
87.9
78.4
90.4
68.2
84.0
3C-GAN 20
94.8
73.4
68.8
74.8
93.1
95.4
88.6
84.7
89.1
84.7
83.5
48.1
81.6
SHOT 21
94.3
88.5
80.1 57.3
93.1
94.9
80.7
80.3
91.5
89.1
86.3
58.2
82.9
NRC
96.8
91.3
82.4 62.4
96.2
95.9
86.1
80.6
94.8
94.1
90.4
59.7
85.9
Table 4: Accuracies (%) PointDA-10, 10.
The results except ours are from PointDAN 300
Model Shape Model SSaan Shape +Model Shape Scan Sean +Model Scan Shape Avg.
MMD 25
X
57.5
27.9
40.7
26.7
47.3
54.8
42.5
DANN
6
X
58.7
29.4
42.3
30.5
48.1
56.7
44,2
ADDA
44
61.0
30.5
40.4
29.3
48.9
51.1
43.5
MCD
B5
X
62.0
31.0
41.4
31.3
46.8
59.3
45.3
PointDAN
30
X
64.2
33.0
47.6
33,9
49.1
64.1
48.7
Source-only
43.1
17.3
40.0
15.0
33.9
47.1
32.7
NRC
64.8
25.8
59.8
26.9
70.1
68.1
52.6
with source-free method 3C-GAN and lower than source-present method RSDA-MSTN, And our method achieves state-of-the-art performance on Office-Home and VisDA, especially on VisDA our method surpasses the source-free free method SHOT and source source-present method RWOT by a wide margin (3% and 1.9% respectively). The reported results clearly demonstrate the efficiency of the proposed method for source-free domain adaptation. Interestingly, like already observed in the SHOT paper, source-free methods outperform methods that have access to source data during adaptation.
3D point cloud dataset. We also report the result for the PointDA--10. As shown in Table our method outperforms PointDA (30, which demands source data for adaptation and specifically tailored for point cloud data with extra attention modules, by a large margin (4%).
4.2 Analysis
Ablation study on neighbors N,, and affinity A. In the first two tables of Table 5 we conduct the ablation study and the 2-nd row from only training with the diversity loss Cdss- From the remaining rows, several conclusions can be drawn.
office-Home and VisDA. The 1-st contains results from the source model
First, the original supervision, which considers neighbors equally can lead to a decent performance (67.1 I Office-Home) Second. considering higher affinity values for reciprocal neighbors leads to large performance gain (69.1 on Office- Homme.. Last but not the least, the expanded neighborhoods
Table 5: Ablation study of different modules on Office- (left) and VisDA (middle), comparison between using expanded neighbors and larger nearest neighbors (right).
Lavo LN LE
Lg Avg
A
Laio EN
LE
Lg
A
Ace
59.5
44.6
&
Acc
62.1
47.8
VisDA (K=M=5)
85.9
67.1
74.6
VisDA w/o E (K=30)
84.0
69.1
81.5
OH (K=3.M=2)
72.2
65.2
61.2
OH w/o E (K=9)
69.5
72.2
85.9
69.1
82.0

---

Table Runtime analysis on SHOT and our method. For SHOT, pseudo labels are computed each
epoch. 20%, 10% and 5% denote the percentage of target features which are stored in the memory bank.
VisDA
Runtime (secpoch)Per-class (%)
SHOT
618.82
82.9
NRC
540,89
85.9
NRC(20% for memory bank)
507.15
85.3
NRC(10% for memory bank)
499.49
85.2
NRC(5% for memory bank)
499.28
85.1
Office- Hore
90.0
VisDA
Acc on VisDA with different
74
Wro self
w/o self
Acat%
70
72.2
05.0
81.5
82.7
84.8
70
80.0
75.0
50
80
E+E
70.0
AV
E
Figure 2: (Leff and middle) Ablation study of Lietf on Office-Home and VisDA respectively.
(Right) Performance with different VisDA.
can also be helpful, but only when combined with the affinity values A (72.2 on Office-Home). Using expanded neighborhoods without affinity obtains bad performance (65,2 on Office-Home We conjecture that those expanded neighborhoods, especially those neighbors of nRNN, may be noisy as discussed in Sec. 3.2 Removing the affinity means we treat all those neighbors equally, which not reasonable.
We also show that duplication in the expanded neighbors important in the last row of Table 5 where the Cg means we remove duplication in Eq. 8 The results show that the performance will degrade significantly when removing them, implying that the duplicated expanded neighbors are indeed more important than others.
Next ablate the importance of the expanded neighborhood in the right of Table5 We show that we increase the number of datapoints considered for class- cass-connsstency by simply considering a larger K, obtain significantly lower scores. We have chosen K so that the total number of points considered is equal to our method (i.e. 5+5#5=30 and 343*2=9). Considering neighbors of neighbors more likely to provide datapoints that close the data manifold [43), and therefore more likely to share the class label with the ego feature.
Runtime analysis. Instead of storing all feature vectors in the memory bank, follow the same memory bank setting as in SS which is for nearest neighbor retrieval. The method only stores a fixed number of target features, (batch size) embeddings from the current training iteration and concatenating them at the end of the memory bank, and discard the oldest elements from the memory bank. We report the results with this type of memory bank of different buffer size in the Table The results show that indeed this could be efficient way reduce computation very large datasets.
update the memory bank at the end of each iteration by taking the
Ablation study on self-regularization. In the left and middle Fig 2 we show the results with and without self-regularization Czels- The Ceels can improve the performance when adopting only nearest neighbors ' or all neighbors N E. The results imply that self-regularizztion can effectively reduce the negative impact of the potential noisy neighbors, especially on the Office-Hone dataset.
Sensitivity to hyperparameter. There are three hyperparameters in our method: K and M which are the number of nearest neighbors and expanded neighbors, which is the affinity value assigned to nRNN. We show the results with different r in the right of Fig. 2 Note we keep the affinity of expanded neighbors as 0.1. r means no affinity. = -1 means treating supervision of nRNN feature as totally wrong, which is not always the case and will lead quite lower result. 0 can also achieve good performance, signifying RNN can already work well. Results with

---

Acc (%6) with different K and M on VisDA
Target
Her Shared
Per Shared Correct
Perations
Figure 3: (Left) The three curves (on VisDA): target accuracy (Blue), ratio of features which have 5-nearest neighbors all sharing the same predicted label (dashed Red), and ratio of features which have 5-nearest neighbors all sharing the same and correct predicted label (dashed Black). (Right) Ablation study on choice of K and M on VisDA.
Predicted Label
90
76
RNN NN before
DRNN NE before
50
DRNN KN
Figure 4: (Left) Ratio of different type of nearest neighbor features which have the correct predicted label, before and after adaptation. (Right) Visualization target features after adaptation.
r= == 0.1/0. 1011/.15//0. 15.2.22 show that our method is not sensitive to the choice of a reasonable r. Note in DA, there is no validation set for hyperparameter tuning, we show the results varying the number of neighbors in the right of Tab. demonstrating the robustness to the choice of and M.
Training curve. We show the evolution of several statistics during adaptation on VisDA in the left of Tab. The blue curve is the target accuracy. The dashed red and black curves are the ratio of features which have 5-nearest neighbors all sharing the same (dashed Red), or the same and also correct (dashed Black) predicted label. The curves show that the target features are clustering during the training. Another interesting finding is that the curve 'Per Shared' correlates with the accuracy curve, which might therefore be used to determine training convergence.
Accuracy of supervision from neighbors. We also show the accuracy of supervision from neighbors on task Rr of Office-Homm in Fig. left). shows that after adaptation, the ratio of all types of neighbors having more correct predicted label, proving the effectiveness of the method.
t-SNE visualization. We show the t-SNE feature visualization task Ar-Rw of target features before (Fig. [aa) and after (Fig. rright) adaptation. After adaptation, the features are more compactly clustered.
5 Conclusions
We introduce source-free domain adaptation (SFDA) method by uncovering the intrinsic target data structure. We propose to achieve the adaptation by encouraging label consistency among local target features. We differentiate between nearest neighbors, reciprocal neighbors and expanded neighborhood. Experimental results verify the importance considering the local structure of the target features. Finally, experimental results on both 2D image and 3D point cloud datasets testify the efficacy our method.
19920019-041744BBBBB (MINECO, Spain) and Acknowledgement We acknowledge the support from Huawei Kirin Solution, and the project
(MICINN, Spain), Ramón y Cajal fellowship RYC2019-027020-1 and the CERCA Programme of Generalitat de Catalunya.

---

References
[1] Kristen M Altenburger and Johan Ugander. Monophily in social networks introduces similarity among friends-of-friends, Nature human behaviour, 2(4):284-290, 2018.
[2] Xinyang Chen. Sinan Wang, Mingsheng Long. and Jianmin Wang. Transferability vs, discriminability: Batch spectral penalization for adversarial domain adaptation. In International Conference on Machine Learning, pages 1081-1090, 2019.
[3] Alex Chin, Yatong Chen, Kristen M. Altenburger, and Johan Ugander. Decoupled smoothing on graphs. In The World Wide Web Conference, pages 263 272, 2019.
[4] Shuhao Cui, Shuhui Wang, Junhao Zhuo, Liang Li, Qingming Huang, and Qi Tian. Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations. CVPR, 2020.
[5] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbon contrastive learning of visual representations. ICCV, 2021.
[6] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by hackpropagation. arXiv preprint 100: 7495, 2014.
[7] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle. François Laviolette. Mario Marchand, and Victor Lempitsky. Domain-adversaare training of neural networks. The Journal of Machine Learning Research, 17(1):2096-200 2030, 2016.
[8] Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In Proceedings of the IEEE international conference on computer vision, pages 5736-5745, 2017.
[9] Ryan Gomes, Andreas Krause, and Pietro Perona. Discriminative clustering by regularized information maximization. 2010.
[10] Xiang Gu, Jian Sun, and Zongben Xu. Spherical space domain adaptation with robust pseudo-label loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9101-9110, 2020.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[12] Sergey loffc and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, arXiv preprint 1502
2015.
[13] Mohammed Jahi, Marco Pedersoli, Amar Mitiche, and Ismail Ben Ayed. Deep clustering: On the link between discriminative models and k-means. IEEE transactions on pattern analysis and machine intelligence, 2019.
[14] Herve Jegou, Hedi Harzallah, and Cordelia Schmid. A contextual dissimilarity measure for accurate and efficient image search. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8. IEEE, 2007.
[15] Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. ECCV, 2020.
[16] Jogendra Nath Kundu, Naveen Venkat, and R Venkatesh Babu. Universal source-free domain adaptation. CVPR, 2020.
[17] Jogendra Nath Kundu, Naveen Venkat, Ambareesh Revanur, R Venkatesh Babu, al. Towards inheritable models for open-set domain adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12376-12385 2020.
[18] Chen Yu Lee. Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[19] Chen- Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10285 10295, 2019.
[20] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9641-9650, 2020.
[21] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. ICML, 2020.
[22] Yuang Liu, Wei Zhang, and Jun Wang. Source-free free domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1215- 1224, 2021.
10

---

[23] Mingsheng Long. Yue Cao. Jianmin Wang, and Michael I Jordan. Learning transferable features with deep adaptation networks. ICML, 2015.
[24] Mingsheng Long. Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pages 1647-1657, 2018.
[25] Mingsheng Long. Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S Yu. Transfer feature learning with joint distribution adaptation. In Proceedings of the IEEE international conference on computer vision. pages 2200-2207, 2013.
[26] Zhihe Lu, Yongxin Yang, Xiatian Zhu, Cong Liu, Yi-Zhe Song, and Tao Xiang. Stochastic classifiers for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9111111111111 11-9120, 2020.
[27] Yingwei Pan, Ting Yao, Ychao Li, Chong-War Ngo, and Tao Mci. Exploring aategory-agnostic clusters for open-set domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13867-13875, 2020.
[28] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang. and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924, 2017.
[29] Charles Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. Proceedings of the IEEE conference on computer vision and pattern recogmition, pages 652-660, 2017.
[30] Can Qin, Haoxuan You, Lichen Wang. C-C Jay Kuo, and Yun Fu. Pointdan: A multi-scale 3d domain adaption network for point cloud representation. Advances in Neural Information Processing Systems, 32:7192-7203, 2019.
[31] Danfeng Qin. Stephan Gammeter, Lukas Bossard, Till Quack. and Luc Van Gool. Hello neighbor: Accurate object retrieval with k-reciprocal nearest neighbors. In CVPR 2011, pages 777-784. IEEE, 2011.
[32] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pages 213 213-226. Springer, 2010.
[33] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. Advances in Neural Information Processing Systems, 33. 2020.
[34] Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Adversarial dropout regularization. ICLR, 2018.
[35] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3723-3732, 2018
[36] Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adaptation by backpropagation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 153-168, 2018.
[37] Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint
2016.
[38] Saquib Sarfraz, Vivek Sharma, and Rainer Stiefelhagen. Efficient parameter-free clustering using first neigh- bor relations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8934 8943, 2019.
[39] Yuan Shi and Fei Sha. adaptation. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pages 1275-1282, 2012.
learning of discriminative clusters for unsupervised domain
[40] Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. ICLR, 2018
[41] Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Thirtieth AAAI Conference on Artificial Intelligence. 2016.
[42] Hui Tang, Ke Chen, and Kui Jia. Unsupervised domain adaptation via structurally regularized deep clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8725-8735, 2020.
[43] Joshua Tenenbaum, De Silva, and John C Langford. global geometric framework for nonlinear dimensionality reduction. science, 290(5500):2319-2555 233-2233, 2000.
[44] Eric Tzeng, Judy Hoffman, Kate Sacnko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7167-7176. 2017.
[45] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint 111 2014.
11

---

[46] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018-5027, 2017.
[47] Ximei Wang, Liang Li, Weirui Ye, Mingsheng Long, and Jianmin Wang. Transferable attention for domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5345-5352, 2019.
[48] Zhongdao Wang. Liang Zheng, Yali Li, and Shengjin Wang. Linkage based face clustering via graph convolution network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1117-1125, 2019.
[49] Yuan Wu, Diana Inkpen, and Ahmed El-Roby. Dual mixup regularized learning for adversarial domain adaptation. ECCV, 2020.
[50] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733-3742, 2018.
[51] Renjun Xu, Pelen Liu. Liyan Wang. Chao Chen, and Jindong Wang. Reliable weighted optimal transport for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 4394 4403, 2020.
[52] Ruijia Xu, Guanbin Li. Jihan Yang, and Liang Lin. Larger norm more transterable: An adaptive feature norm approach for unsupervised domain adaptation. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
[53] Guanglei Yang, Haifeng Xia, Mingli Ding, and Zhengming Ding. Bi-directional generation for unsupervised domain adaptation. In AAAI, pages 661.5-662,, 2020.
[54] Lei Yang, Dapeng Chen, Xiaohang Zhan, Rui Zhao, Chen Change Loy, and Dahua Lin. Learning cluster faces via confidence and connectivity estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13369-13378, 2020.
[55] Lei Yang. Xiaohang Zhan, Dapeng Chen, Junjie Yan, Chen Change Loy, and Dahua Lin. Learning to cluster faces an affinity graph. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2298-2306, 2019.
[56] Shiqi Yang. Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui. Unsupervised domain adaptation without souree data by casting bait, arXiv preprint arXiv:2010. 2010..227. 2020.
[57] Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan, Universal domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2720 2729, 2019.
[58] Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan. Domain-so Domain-symmetric networks for adversarial domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5031-5040, 2019.
[59] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pages 1404-7413. 2019.
[60] Zhun Zhong, Liang Zheng, Donglin Cao, and Shaozi Li. Re-ranking person re-identiffiction with k- reciprocal encoding. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition. pages 1318-1327. 2017.
[61] Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33, 2020.
[62] Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6002-6012, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
12

---

2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL.)? [Yes] We attach the code the supplemental material.
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they chosen)? [Yes] As in the model details in Sec
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)? [Yes] All main results are average over three running with random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "tSRw9Hfs3Sm", "reviewer": "Reviewer_4Ef6", "summary": "This paper proposes a method for source-free domain adaptation (SFDA). While methods for unsupervised domain adaptation access the labeled source domain data during adaptation, SFDA does not allow models to do so. The proposed method utilizes the neighborhood affinity of the target data. The experimental results show that the proposed method achieves state-of-the-art performance on image classification and point cloud classification.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n- In Section 5, the authors insist, \"To the best of our knowledge, existing works on DA do not explicitly exploit this source of information, and they could be combined with our method.\" However, the use of local structure in the target domain is pretty well-known. [a] is one of the examples having a similar motivation.\n\n[a] Pan et al., Exploring Category-Agnostic Clusters for Open-Set Domain Adaptation. CVPR, 2020\n\nQuality: \n- The papers below have a similar problem definition to that in this paper. \n\n[b] Li et al., Source-free domain adaptation for semantic segmentation. CVPR, 2021.\n\n[c] Kurmi et al., Domain Impression: A Source Data Free Domain Adaptation Method. WACV, 2021.\n\n[d] Yeh et al., SoFA: Source-data-free Feature Alignment for Unsupervised Domain Adaptation. WACV, 2021.\n\nClarity: \n- Figure (c) shows that the samples misclassified once can be pushed toward a cluster for each class. However, it is unclear why the proposed method can form completely clean clusters.\n\nSignificance:\n- In Section 5, the authors insist, \"To the best of our knowledge, existing works on DA do not explicitly exploit this source of information, and they could be combined with our method.\" The former is already addressed in the originality part. If the authors insist that the proposed method can be combined with their method, they could have shown some results.\n- The performance gain is sometimes very moderate. For example, results of Office-31 and Office-Home are very close to those by SHOT [21] and ModelAdaptation [20]. By the way, the authors of [20] have named their method not as ModelAdaptation but as Collaborative Class Conditional GAN (3C-GAN).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are no descriptions of limitations and societal impacts.\n", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "UelqsZH8Lfe", "reviewer": "Reviewer_ERRY", "summary": "The authors proposed a novel method for source-free domain adaptation. Based on empirical observation, they assume that the features of target data extracted by the pre-trained model form clear clusters. To explicitly capture this intrinsic structure in the feature space, the local affinity of the target data is defined, and the model is trained to predict consistent labels if there is high local affinity. The experimental results with several benchmark datasets show that the proposed method works well in the source-free setting. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "<Review summary>\n\nThis study is well-motivated and tackles an important problem, which is unsupervised domain adaptation without access to the source data. The proposed method is carefully designed to solve this problem and works well in several benchmark datasets, while there are some concerns on hyper-parameter setting and scalability. I vote for \"weak accept.\"\n\n\n<Details>\n\nStrength\n\n- This paper is well-written and easy-to-follow.\n\n- This study tackles an important and challenging problem, which is source-free UDA. The source-free UDA should be beneficial in many practical situations where domain adaptation is needed.\n\n- The experimental results show that the proposed method works well in several benchmark datasets. In addition, rigorous ablation studies validate the design of the proposed method.\n\n\n\nWeakness\n\n- Scalability to the number of target data. Since we need to store and continuously update the extracted features of all target data, it should spend a lot of memory, when the number of target data is large. In addition, computational cost can be also prohibitively large due to the nearest-neighbor search process in each iteration. Although small-scale target data can be expected in some situations where domain adaptation is required, this point should be discussed in the paper.\n\n- How to optimally set the number of nearest/expanded neighbors. Although Fig. 4 shows some robustness of the proposed method against suboptimal setting of K and M, this robustness may heavily depend on both the dimensionality of the features and the number of target data. Since it is difficult to tune hyper-parameters in the adaptation phase under the source-free setting, this point should be carefully analyzed/discussed.\n\n- It is not clear to see whether the final objective function is minimized if and only if all p_i are one-hot vectors with ground-truth labels. Showing loss curve with error curve may help.\n\n- Something wrong in Eq. (7). I think both \\bar{p}_c and q_c cannot be label distribution but scalar. Please check it.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no description on limitations. I strongly encourage the authors to describe some of them such as what I raised at <weakness> in Main Review.\n\nThere is no description on possible negative societal impact, but it is not so much problematic, because this study is on general machine learning techniques.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "eight hours."}, "extra_text": {}}, {"review_id": "KF5Y6TmFE9P", "reviewer": "Reviewer_eN4k", "summary": "This paper introduces a new method from the viewpoint of neighborhood regularization for source-free domain adaptation where only the source model instead of the raw source data is provided to the target domain. The proposed method distinguishes reciprocal nearest neighbors (RNN) and non-reciprocal nearest neighbors (nRNN) among the k-nn neighborhood. By exploiting the memory bank and the relationship with RNN, the proposed method obtains better results than previous source-free methods like SHOT and Model Adaptation on 2D image datasets and a 3D point-cloud dataset.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this paper is well-written and easy to read. The main contribution lies in the usage of reciprocal nearest neighbors (RNN) to preserve the intrinsic neighborhood structure of the target data during feature adaptation.\n\nPros:\n1. Different from previous methods using clustering or neighborhood consistency, the usage of RNN in this paper is interesting and effective for target feature adaptation;\n2.  Obtaining promising results on several cross-domain recognition benchmarks including Office-Home.\n\nCons:\n1. How to determine the parameters M and K, as shown in \"Model Details\", it seems that these two parameters are carefully tuned. \n2. During the proposed method, it is not clear how to calculate the affinity matrix A. Since we want to find the intrinsic neighbors, do we need to calculate the n_t*n_t distance matrix for each mini-batch? If so, the time cost would be quite large.\n3. In table 4, the authors are encouraged to run the results of SHOT and Model Adaptation for a complete comparison. Since on Office and Office-Home in Table 1-2, the advantages of the proposed method over SHOT are not significant.\n\n\nBesides, the results on a multi-class DomainNet (365 classes) would be more persuasive since the neighbors found may be noisy.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "1f3evwOrabB", "reviewer": "Reviewer_LSVJ", "summary": "This paper explores the effectiveness of reciprocal nearest neighbors for the challenging task of Source-Free domain adaptation where the target domain is used to train the model already pre-trained on the source domain. The paper exploits the clusterability of the target dataset in absence of the source domain and highlights the importance of a self regularization. The results demonstrate promising performance on 2D and 3D tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Nearest Neighbour (NN) based techniques have been used extensively in DA literature to structural properties of the target domain. The paper finds that using Reciprocal NN (RNN) instead of NN provides an improvement in the performance of the model\u0010. \n\nFurther, the paper identifies certain gaps with RNN and exploits extended neighborhoods for further improvements. The ideas described in this paper are novel and have been presented well. \n\nThe results are significant and demonstrate promising accuracy for 2D and 3D computer vision tasks.\n\nConcerns:\n1. The top K/M RNN points are shown to have a higher correct ratio percentage compared to the top K/M NN points. However, the performance of the source-free model really depends on the points that lie in low confidence regions (which are typically not present in the top K) because they contribute to the majority of the samples. So even if the extended neighbourhood is considered, it is not clear how RNN could perform better than NN since the most confident samples are classified correctly anyway.\n\n2. On Eq. 1 ($\\mathcal{L}_{\\mathcal{N}}$), each of the terms on the right-hand side ($A, \\mathcal{S}, p$) are always positive. So the dot product of these vectors will be limited to a positive regime. Wouldn't it be ideal to choose a negative $A$ value for the nRNN data point (consider it as a highly dissimilar point) that needs to be \"pushed away\" instead of $r=0.1$ (L143)? Also, in Fig. 3, a value of $r$ that lies in between -1 and 0 has not been shown. For e.g., a $r= -0.1$.\n\nSuggestions to authors:\n1. Training the model in a curriculum fashion could boost the performance of your method since the model has the flexibility to learn the signal from beyond the data points present at the top-K. [Ref 2]\n\nMinor typos:\n1. L136: \"non-reciprocal neighbors neighbors\" .. repeated words.\n2. L143: \"If not specified otherwise r is set to 0.1.\" .. remove otherwise here?\n\nConsider including the following references in your paper as they are closely related to your method.\n\n[Ref 1] Yuang Liu, Wei Zhang, Jun Wang, \"Source-Free Domain Adaptation for Semantic Segmentation', CVPR'21\n\n[Ref 2] Venkat N, Kundu JN, Singh DK, Revanur A, Babu RV, \"Your Classifier can Secretly Suffice Multi-Source Domain Adaptation\", NeurIPS'20\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not included any limitation or societal impact in the current draft of the paper.\nI encourage authors to consider how a source-trained model could inadvertently induce bias during source-free domain adaptation that might result in societal impact. For e.g., For a task that involves human facial images, choosing RNN to sample a set of points might automatically favor a certain group of people who are dominantly present in the source dataset.\n\nPlease consider including a broader impact statement as well, if you see fit.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Following the rebuttal and discussion period, this paper received borderline scores with three leaning towards acceptance and one recommending rejection. All reviewers agreed that this paper proposes a novel algorithm that leverages local structure in a way different from prior work (using nearest neighbor consistency instead of global cluster structure through K-means). Most reviewers found the empirical results to be compelling, though eN4k did have doubts about whether the results would scale (both in terms of performance and complexity) to larger datasets such as DomainNet. However, after considering all reviews, discussion, and rebuttal comments, the AC is inclined to agree that the algorithmic novelty is interesting and sufficiently empirically justified with the existing datasets to warrant publication. The authors should be sure to include the discussion around K-NN vs K-means structure and clarify their claims and the other minor points brought up in their final version. Further both scalability and how to set the hyperparams K, M were brought up by multiple reviewers, please add a discussion around these points. 
Decision: Accept (Poster)