Neighbor Class Consistency on Unsupervised Domain Adaptation
Authors: Chang Liu
Kai Li
Yun Fu
Venue:  (2021)
Keywords: Unsupervised domain adaptation
Consistency regularization
Neighbor

Abstract: Unsupervised domain adaptation (UDA) is to make predictions for unlabeled data in a target domain with labeled data from source domain available. Recent advances exploit entropy minimization and self-training to align the feature of two domains. However, as decision boundary is largely biased towards source data, class-wise pseudo labels generated by target predictions are usually very noisy, and trusting those noisy supervisions might potentially deteriorate the intrinsic target discriminative feature. Motivated by agglomerative clustering which assumes that features in the near neighborhood should be clustered together, we observe that target features from source pre-trained model are highly intrinsic discriminative and have a high probability of sharing the same label with their neighbors. Based on those observations, we propose a simple but effective method to impose Neighbor Class Consistency on target features to preserve and further strengthen the intrinsic discriminative nature of target data while regularizing the unified classifier less biased towards source data. We also introduce an entropy-based weighting scheme to help our framework more robust to the potential noisy neighbor supervision. We conduct ablation studies and extensive experiments on three UDA image classification benchmarks. Our method outperforms all existing UDA state-of-the-art. 

Under review as a conference paper at ICLR 2021
NEIGHBOR CLASS CONSISTENCY ON UNSUPERVISED DOMAIN ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Unsupervised domain adaptation (UDA) is to make a prediction for unlabeled data target domain with labeled data from the source domain available. Re- cent advances exploit entropy minimization and self-training to align the feature of two domains. However, as decision boundary largely biased towards source data, class-wise pscudo labels generated by target predictions usually very noisy, and trusting those noisy supervisions might potentially deteriorate the in- trinsic target discriminative feature. Motivated by agglomerative clustering which assumes that features in the near neighborhood should be clustered together, we observe that target features from source pre- treitraine model are highly intrinsic dis- criminative and have a high probability of sharing the same label with their neigh- bors. Based those observations,
propose simple but effective method to impose Neighbor Class Consistency on target features to preserve and further strengthen the intrinsic discriminative nature of target data while regularizing the unified classifier less biased towards source data. We also introduce entropy- based weighting scheme help our framework more robust to the potential noisy neighbor supervision. We conduct ablation studies and extensive experiments on three UDA image classification benchmarks. Our method outperforms all existing UDA state-of-the-art.
INTRODUCTION
Recent advances in deep neural network have dominated many computer vision tasks, such image recognition He et al, (2016), object detectior(Gishhik (2015), and semantic segmentation[_ongg et al.) (2015) However, collection and manual annotation need no trivial human effort, especially for vision tasks like semantic segmentation where dense annotations are required. Thanks to the growth of computer graphics field, it is now possible to leverage CNN to synthetic images with computer- generated annotations (Richter et al. (2016); Ros et al. (016)), so unlimited amount of data with free annotation is available for training network in scale. However, directly applying the model trained on synthetic source data to unlabeled target data leads to performance degradation and Unsupervised Domain Adaptation (UDA) aims tackle this domain shift problem.
widespread of UDA methods were proposed to align the domain-invariant representations by simuitaneously minimizing the source error and discrepancy(e.g. H-divergence [Ben-avvd et al. (2010); Hoffmma et al. (0016) H4. Seddiieergnneeeen--DDD
et al. (2010)) between two domain such the maximum mean discrepancy Tzeng et al. (2014), correlation distance(un al. (2016) and etc. Further, adversarial learning-basee based UDA Ganin & Lempitsky (2013); Tzeng et al. (2017) Radford
al. (2013); [Hoffman et al. (2018): Tsai al. (2018); Sankaranarayanan et al. (2018): Luo et al.
22019) methods aim to reduce this discrepancy between two domain by minimizing the adversarial loss. However, the major limitation of adversarial learning that it only aligns the global feature distribution of two domains without considering the class labels. As the result, small H.H distance does not guarantee the small error on ideal joint hypothesis on the features of two domains Liu et al. (2019).
To alleviate this issue, Entropy minimization (Grandvalet & Bengio 2005): Vu et al. (2019) and Self- Training (Lee (2013); Zou et al. (2018) the two dominant methods to enforce the clus- ter assumption such that network can learn a discriminative feature space by pushing the decision boundary away from densely-distributed area. However, decision boundary is largely biased to-

---

Under review as a conference paper at ICLR 2021
Anchor
Condaneey Lau
D
CNN
Update
Retrioved 1NN
Caniensry
Memory
(1)
(2)
Figure 1:
(1) Our Neighbor Class Consistency framework. (2) An overview of our approach:
(a) Self-Training methods ignore the intrinsic target structure while aligning the features of two domains based on biased classifier. They potentially deteriorate the intrinsic target clusters. (b) Our approach enforces the Neighbor Class Consistency on target features, therefore progressively strengthen the instrinsic discrimination of target features while regularizing the unified classifier less biased towards source data.
wards source data, trusting biased network predictions will push target features towards their nearest source class prototypes while deteriorating the intrinsic discriminative target structure as shown in Fig. I2.a).
Motivated by agglomerative clustering methods (Sarfraz et al. (2019)) which assume that features in the nearby region should be clustered together, investigate target features from source pre- trained model and observe that they are intrinsically discriminative and have very high possibility label with their neighbors shown in Fig. 2.b). To utilize this high-quality
of sharing the pairwise neighbor supervision, we propose a simple and effective approach to impose Neighbor Class Consistency between target samples and their neighbors. To alleviate propagated errors from false neighbor supervision, we introduce an Entropy-based weighting scheme to emphasize more on the reliable pairwise neighbor supervision. Additionally, we categorize Self Class Consistency as a special case of our method where the nearest neighbor of a sample is its celf-augmentation. Further, we explore feature representation learning based on the ranking relationship between self- augmentation and the first neighbor given an anchor. We enforce the features of anchors to be closer their self- auff-augmentation than their first neighbors.
In summary, our main contributions are shown as follows: (1) We revisit the source pre-trained model and observe the intrinsic discriminative nature of target features from source model. (2) Based on this observation, we propose Neighbor Class Consistency (NC) to utilize the high-quality pairwise neighbor pseudo supervision over noisy class-wise pseudo supervision from Self-Traine methods. (3) We introduce Entropy-based weighting scheme to help framework be more robust to unreliable neighbor supervision. (4) We categorize Self Class Consistency as special case of our framework and explore the first neighbor for feature representation learning. (5) We conduct extensive experiments on three UDA benchmarks datasets. NC outperforms all existing methods and achieves new UDA state-of-the-art performance. Notably, we achieve 86.2% on challenging VisDA17 dataset.
RELATED WORK
Discrepancy based domain adaptation Following the theoretical upper bound proposed in Ben- David et al. (2007), existing methods have explored to align the feature representations of the source and target images by minimizing the distribution discrepancy. For example, Maximum Mean Dis- crepancy (MMD) Tzeng et al. (2014) is proposed to match the mean and covariance and target distributions. Alternatively, adversarial domain adaptation methods Ganin & Lempitsky (2015); Tzeng et al. (2017): Radford et al. (2013); Hoffman et (2018): Tsai et al. (201)))))) narayanan et al. (2018)
source
(2015);(ootimmn
al. (201)) solve this domain discrepancy by training domain-
2

---

Under review as a conference paper at ICLR 2021
invariant feature generator which produces the features to fool discriminator that distinguishes the representations from source and target domains. However, since the domain discriminator aligns source and target features without considering the class labels, merely aligning the global marginal distribution of the features in the two domains fails to align the class-wise distribution.
Clustering based domain adaptation Entropy minimization (Grandvalet & Bengio 2005 Vu
et al. (2019)) and self-training (Zou et al. (2018); Gu et al.] (2000) are two streams of approaches realize the class-wise alignment across domains. However, minimizing the conditional entropy of target prediction based on source biased classifier will harm the intrinsic discriminative target structure. To this end, prototypical classifier and alignment Xie al. (2018); Pan et al. (2019) have been explored to mitigate the noisy supervision from source biased classifier. However, the prototypes of target samples are still noisy estimated cluster centers and therefore target samples away from prototypes may still risk of being wrongly classified. Similar effort has been made by Tang et al. (2020) on learning the latent discriminative target structure with respect to learnable clusters via deep clustering framework Ghasedi Dizaji et al. (2017).
Consistency Regularization In semi-supervised setting, mainstream methods apply various con- sistency regularization unlabeled data with different pre-defined positive counterpart. Among them, Tarvainen & Valpola (2017) impose consistency between predictions from student network and moving-aversen based teacher networks. Virtual Adversarial Training Miyato et al. (2018) tries to make network invariant to small adversarial permutations around the neighborhood of a sample while DTALLee al. (2019) enforces the target predictions from the networks with different choice of dropout mask to be consistent. In un/self-supervised setting, SimCLR (Chen et al. (2020) and MoCo (He et al. (2020)) are the two prevalent approaches conduct contrastive learning on the feature space among unlabeled data, its strong augmentation and other negative samples from ei- ther extremely large current batch or momentum updated memory banks. comparison to those un/self-supervised methods, our method exploreS neighbor samples for consistency regularization. In cross-domain person re-identification /Zhong et al. (2019) also utilize the neighbor information on target data. To emphasize the difference, our method focuses on regularizing the classifier to be less biased toward source domain by applying target neighbor class consistency while they focus on the feature representation learning by enforcing neighbor feature invariance.
METHOD
3.1 PROBLEM DEFINITION
In unsupervised domain adaptation(UDAA, source domain data is denoted D, = {(2 y where a; and yf denote the i-th training sample and its label, N₂ is the number of source images. Target domain data is denoted as D, = [ where N₂ is the number of target images. The objective of UDA is to train deep neural network G(-I0) which has access to the source data (z), v.) drawn from DD and target data 2 drawn from D, such that the model G(-I0) can generalize on target domain. Network () C F(- is comprised of feature extractor F(|(0) and classifier C(-10) where 0 denotes network parameters.
3.2 REVISIT SUPERVISED PRE-TRAINING FOR SOURCE DOMAIN
In general, UDA pre-trains a network G(-10) on source domain by standard cross entropy loss and then the network is transferred to inference the target data. The source supervised objective function in the form of,
N
-. 0 = N.
Σ Cce (C(F(x)]0)) vi).
(I)
However, the model trained on source data usually generalizes poorly on target data due to the domain shift between the joint distribution of two domains.
Self-Trainins (ST) and Entropy Minimization (Ent) methods are proposed to make the network be confident its target predictions following the cluster assumption and thus improve the discrim-
3

---

Under review as a conference paper at ICLR 2021
k-th Neighbor
k=1
k=3
k=5
Acc
96.4
92.3
86.4
Table I: Pairwise class consistency evaluation on target features extracted from Source model with their k-th Nearest neighbors on Office-31 A-W.
inativeness of network. Their objective functions are calculated as
(0)
Le (C(F(a)(0))
(2)
N.
Σ H (C(((())))) N.
1
(3)
where ÿf is the pseudo label for the target sample and H. is entropy function.
Nonetheless, the pseudo labels of target samples noisy the network is biased towards source data. Trusting noisy label or prediction has high risk of misleading the training and propagate errors.
Inspired by agglomerative clustering methods which claim that features should be clustered with their near neighborhood, we hypothesize that source domain pre-training can provide a feature space where target features intrinsically discriminative and have high probability of sharing the same labels with their neighbors.
To verify this, we apply k-Nearest Neighbor (kNN) search target features extracted from source model on Offce-31 A.W. We evaluate the pairwise label consistent accuracy between target fea- tures and their k-th neighbors. Table shows that target features from source model have very high accuracy of sharing the same label with their K-th neighbors. It demonstrates our hypothesis that target features from source model are locally discriminative.
This observation motivates us to leverage this relative "clean" pairwise pseudo supervision to help the network generalization on target domain over the noisy class-wise pseudo supervision.
3.3 NEIGHBORHOOD CLASS CONSISTENCY
1) Neighborhood Discovery. First, we extract target features z' F(x'(0) from source pre-train model and save them into a target feature memory bank V, = {2 Then we apply kNN on target features V, to explore the neighborhood Nx(2²) which is defined as follows:
Nx(2)) (2 2f) top-k in Wc),
where s a similarity metric and & is the number neighbors.
Note that we update the memory bank V, with the newest features in training and conduct kNN on Vt at every epoch.
2) Vanilla Neighborhood Class Consistency (VNC), Based motivation that target samples should share the same class label with their neighbors, we propose VNC to enforce the class assign- ment consistency between pairwise neighbors by mutual information (MT) maximization between their network predictions. Formally, we formulate the objective function of VNC in the form of:
Ny (0)
Σ MI (CCE)))) (EN((z))
(5)
N
is also worth noting that the memory bank serves as a look-up table to retrieve the target features and feed them into the classifier C for computing the Neighborhood Class Consistency loss. As the size of ie can be potentially large, using memory bank can implicitly enlarge the batch size by times without introducing extra computational cost and time.
4

---

Under review as a conference paper at ICLR 2021
3) Entropy-weighted Neighborhood Class Consistency (ENC). Table shows that the reliability of pairwise neighborhood supervision is decreased with the rise neighborhood size k. In other words, there is trade-off between large neighborhood diversity and reliability of neighbor supervi- sion.
A straightforward idea is treat each neighbor pairs differently in terms of loss weight such that hopefully positive neighbor pairs (Correct) will have higher loss weight than negative neighbor pairs (False). To achieve this, we propose entropy-based weighting scheme (EW) to assign different loss weight on different neighbor pairs given an anchor. As the entropy is a measurement of sample prediction certainty, the less entropy is. more confident the prediction is and thus more weight will assign on. Formally, we define the loss weight and objective of ENC as follows:
w0(2) H(C(2²10))
log M
Nr w w(z))
(
Σ JENN(#
MI (C(()
N₂
k
f=1
where M the number of class.
We claim that entropy-based weighting helps mitigate the problem of noisy pairwise supervision and it is more robust the neighhorhood size as will down-weight the neighbor pairs the selected neighbor samples have high entropy value.
3.4 SELF CLASS CONSISTENCY
In addition to neighborhood consistency, target samples should always share the same label with their self-auggentation and this pairwise supervision is 100% clean. Therefore, Self Class Con- sistency is a special case of NC where the nearest neighbor given an anchor is its augmentation. We adopt Cropping, grayscale and color distortion our data augmentation following Chen et al. (2020) and enforce the self-consistency in terms of class assignment on target samples and their augmentations as follows.
(0) N
MI (CCFFEE)))
(8)
where E is the data augmentation of i-th target sample.
3.5 FEATURE RANKING BETWEEN SELF-AUGMENT AUGMENTATION AND FIRST NEIGHBOR
As the first neighbors given target anchors might be negative (do not share the same label with an- chors) and the anchors' celf-augmentations might be heavily distorted, enforcing neighbor class sistency alone might not guarantee that the anchors are closer to their self-augmented samples than their first neighbors in feature space. We claim that introducing this feature ranking regularization benefits the feature representation learning by ensuring the positive samples (seff-augmemtationn) rank higher than the negative samples (the first neighbors) in feature space. When the first neighbors are positive, it can also enforce an inductive bias on model training to rank the positive samples with pre-defined variations (such cropping. grayscale and color distortion) higher than the positive samples with unknown variations (the first neighbors).
To impose this feature ranking regularization, we adopt triplet loss as objective function as shown below,
No max (0, [F([[((()))) F( + (G((())))))(((
(0) N
(9)
where NG (F(2((0))) is the feature of the first neighbor of target sample and is the margin.
5

---

Under review as a conference paper at ICLR 2021
3.6
TOTAL Loss
The objective of proposed Neighbor Class Consistency is in the form of,
Livc (0) = L (0) + (CEENc(0) + (0) + (ir(()))
(10)
As pre-training on source data only might not provide the optimal initial feature space to explore high-quality neighborhood, add target self-consistency loss with source supervised loss in pre- training stage and formulate it as:
(sp = (orc(() AC
(11)
4 EXPERIMENT
4.1 DATASETS
We conduct experiments on three widely used domain adaptation classification benchmarks: Office- 31 Saenko et al. 2010), VisDAI7 Peng et al. (2018) and ImageCLEF-DA
Office-31 is commonly used dataset for unsupervised domain adaptation. It includes 4652 images 31 classes from three domains: Amazon (A), Webcam (W) and DSLR (D). ImageCLEF-DA con- sists of 12 common classes shared by three public datasets (domains): Caltech-256 (C). ImageNet ILSVRC 2012 (I), and Pascal VOC 2012 (P). VisDA17 17 is large-scale dataset. It uses 152, 409 2D synthetic images from 12 classes as the source training set and 55, 400 real images from MS- COCO Lin al. (2014) the target set. 12 object categories are shared by these two domains.
4.2 IMPLEMENTATION DETAILS
We follow the standard protocol of UDA( Ganin et al. (2016); Zhang et al. (2019b); Xu et al. (2019); Zou et al. (2019) to use all labeled source samples and all unlabeled target samples as training data. The reported testing results the average accuracy
three random repeats with center-crop images. We adopt ResNet-50 (He et al. (2016) on Office-31 I and ImageCLEF-DA while ResNet101 on VisDA17 dataset, fine-tuned from the ImageNet (Deng et al. (2009) pre-trained model. We use Pytorch as implementation framework. We adopt Stochastic Gradient Descent (SGD) optimizer with learning rate of 10-3 weight decay 5 x 10-4, momentum 0.9 and batch size 32. We set A = 0.1 (Eq 10 for Office-31 and ImageCLEF-Da, while X 0.5 for VisDA17. We set k = 1 (Eq
7p to explore the most local neighbourhoods and triplet margin m 0.1 in (Eq 9.. The analysis on
the impact of is is at section 4.5.
4.3 ABLATION STUDY
In this section, we investigate the effectiveness of each components of NC in achieving the state- of-the art performance on Office-31. We name each components of NC as: (1) Source Model, which fine-tunes the base network on source labeled samples with [q (2) Vanilla Neighbor Class Consistency (VNC), which denotes the training with non-weighted Neighbor Class consistency
with Eq 5 (3) Entropy-Weighter Neighbor Class Consistency (ENC), which denotes the training
with Entropy-Weiin Weighted Neighbor Class consistency with Eq 7 (4) Self Class Consistency(SC),
which denotes the training with Self consistency with Eq 8 (5) Feature Ranking (FR), which
denotes the training with feature ranking relations between anchor, its self-augmentation and its first neighbor with Eq (6) supervseer Pretraining (SP), which fine-tunes the base network
with source labeled samples and SC with Eq I1 From the Tab 2 we observe that all proposed components play a contributing role to our final performance.
4.4 COMPARISONS WITH THE STATE OF THE ART
Results on Office-31 and CLFFDA based ResNet--0 are shown in Table Jnd Table 4 We can observe that NC-SP outperforms all the existing methods and more importantly boosts the

---

Under review as a conference paper at ICLR 2021
Method
A- W
D
D
A
D-A
-
Avg
Source Model
68.420.2
96.7.0.1
99.310.1
68.910.2
62.540.3
60.710.3
76.1
SC
10.2
98.3=0.0
100.0+0.00
92.3:0.1
75.0+0.2
75.3+0.2
88.8
VNC
93.8 +0.1
98.7=000
100.0:0.00
93.240.2
75.1+0.1
75.8+0.1
89.4
ENC
94.1=0.2
98.7-000
100.0:0.00
93.8
75.40.0
76.2 +0.1
89.7
ENC-SC
90.1
98.71000
100.0=0.00
95.2
75.3+0.1
76.7200.3
90.1
ENC-SC-SP
95.8 +001
98.7=000
100.0=0.00
90.4
76.9++0.1
77.0+-0.1
90.6
ENC-SC-FR (NC)
95.7+0022
98.7+000
100.0+0.0
95.2+0.1
77.5+0.1
76.9+0.0
90.6
ENC-SC-FR-SI (NC-SP)
96.4-0.1
98.7-0.0
100.010.0
95.810.33
77.610.2
77.4-0..
91.0
Table 2: Ablation studies using Office-31 based on ResNet-50. Please refer to Section 4.3 on what each component represents.
Method
A W
D
WD
A
D-
W
Avg
ResNet-50 He 2016
68.4
96.710.1
99.310.1
68.910.2
62.510.3
60.7
76.1
DANN Ganin al.
2016
82.0
96.9
99.1
79.710.4
68.210.4
67.410.5
82.2
ADDA zeng al. 2017
86.2.10.5
96.2.10.3
98.4.0.33
77.810.3
69.5100.4
68.910.5
82.9
ENT
(rrandvalet & Hengio
2005
89.0+0.1
99.0+0.11
100.0-0.0
86.3+0.3
67.5+0.2
63.0+0.1
84.1
JAN (Long et al. 201
85,420.3
97. 97.420.2
99.810.2
84.7.0.3
68.620.3
70.0=20.4
84.3
MRENT (Zou (Zou of al.
2019
88.0=0.4
98.6=#0.1
100.0=0.0
87.40088
72.7+002
71.0+0.4
86.4
SAFN+Enl Xu el 2019
90.1110.8
98.6100.
99,810.0
90.7.10.5
73.0.10.2
70.2.10.3
SymNets Zhang
2019a
90.8=0.1
98.820.3
100.0=0.0
93.910.5
74.610.6
72.510.5
88.4
MDD Zhang al.
(0199]
94.5
98.4:10.1
100.0-000
93.510.22
74.61.0.3
72.210011
88.9
CAN Kang al
2019
94.5=0.3
99.1=0.2
99.8=00.2
95.0+0.3
78.0+0.3
77.3+0.3
90.6
SRDC
Tang eral.
2020
95.7+0.2
99.2+0.11
100.0-0.0
95.8+0.2
76.7+0.3
77.1+0.1
90.8
NC-SP
96.4 0.1
98.7+0.0
100.0 0.0
95.8+0.3
77.6+0.2
77.4
91.0
Table 3: Experiment results on Office-31 classification using ResNet-50
performance substantially difficult transfer tasks such as A-> W, D- A and W-+ A. Notably, comparing to state-of-the-ar self- self-training (Zou et al. 2019) and entropy minimization methods (Xu et al. (2019)) based class-wise pseudo supervision, NC-SP outperforms them by 4.6% and 3.9% respectively, and it demonstrates that following the relatively clean neighborhood pseudo supervi- sion can alleviate the accumulation problem from self-training.
Results on V1 based on ResNet-101 are reported in Table 5 NC-SP achieves much better performance than all compared methods. Comparing DTA (Lee et al. (2019) which uses VAT (Miyato et al. (2018) as self-consitenncy and Adversarial Dropout as model-consisteney, we instead explore neighborhood consistency and demonstrate the effectiveness of NC-SP over other consis- tency regularization. Note: The results of existing methods in Table refer to their respective papers.
4.5 ANALYSIS
Feature visualization. We visualize the target feature embeddings of (a) source model, (b) entropy minimization, and (c) NC Office-31 W-+ via t-SNE (Maaten & Hinton (2008) in Fig 2 We can qualitatively observe that NC could learn more discriminative features than source model and preserves the intrinsic target features structure.
entropy minimization
Nearest neighbors visualization. We visualize the top 3 nearest neighbors given an anchor based on W in Fig 3J). We investigate both success
the target features from source model on Office-31 and failure cases to get extra insights into our method. For the first two row, features from source model could retrieve the correct neighbors for the mobile phone and backpack with certain level of appearance and pose variations. However, at the last row where the pose of that laptop sample is unusual, our method might fail in those cases. The proposed entropy weighting on neighbor class consistency is alleviate the misleading learning by false neighborhood supervision.
Impact of neighbourhood size kt Neighbourhood size is an important parameter as it controls the amount of pairwise neighborhood supervision. However, there is trade-off between increasing the neighbor diversity and increasing the risk of adding false neighborhood supervision. Empirically, the optimal & is associated with the size of target dataset and the number of class. We evaluate from (1, 3, 5) for both Vanilla NC and hatropy-weighher NC on Office-31 - W as shown in Fig
bb). We observe that the performance decreases when increasing the & for VNC while ENC is more

---

Under review as a conference paper at ICLR 2021
Method
I>>
P-11
1
C-1
C-P
P--C
Avg
ResNet-50 He 2016
74.8
83.9
91.5
78.0
65.5
91.2
80.7
DANN Gamm
2016
75.0
86.0
96.2
87.0
74.3
91.5
85.0
JAN
Long
2017))
76.8
88.0
94.7
89.5
74.2
91.7
85.8
CDAN Long
2018
76.7
90.6
97.0
90.5
74.5
93.5
87.1
SAFN+Fmt et 2019a
78.0
91.7
96.2
94.7
88.1
SymNets Zhang et 1
93.6
96.4
CAN Kang 2019
78.5
93.0
97.3
91.0
77.2
97.0
89.0
ALLP+CAN Zhangg al. (2020)
94.3
97.7
96.9
NC-SP
80.9
95.0
97.9
94.2
79.8
97.5
90.9
Table 4: Experiment results ImageCLEF- -A classification using ResNet-50
Method
Aero
Bike
Bus
Car
Herse
Knile
Mutoc
Person
Flant
Skalibboard
Train
Truck
Mican
Source
Salto
53.1
53.3
61.9
$1..6
17.3
79.7
31.2
81.0
26.3
13.3
8.3
52.4
MCD
Saiee
20150
87.0
60.9
83.7
61.0
88.9
79.5
84.7
76.9
88.6
40.3
83.0
25.8
71.9
ADR
Sallrre 21
20133
87.8
79.5
83.7
65.3
92.3
61.8
$8.9
73.2
87.8
60.0
85.5
32.3
74.8
SAEN
93.6
61.3
84.1
70.6
94,1
79.0
91.8
79.6
89.9
55.6
89.0
24.4
76.1
MRKLD+LREN
2014
19TA
Lay
2019
93.7
82.2
83.6
83.8
93.0
81.0
90.7
82.1
95.1
78.1
86.4
32.1
81.5
RWOT
[XB et
2020
95.1
80.3
83.7
90.0
92.4
68.0
92.5
82.2
87.9
78.4
90.4
68.2
$4.0
SC
97.1
88.5
90.0
65.2
96.7
92.9
90.1
81.3
91.6
89.5
89.0
38.8
86.2
Table 5: Experimental results on VisDA17 classification using ResNet-101.
robust to k. This is consistent with the intuition of Entropy-weighting which aims to alleviate the problem of false neighborhood supervision. For simplicity, we use k=1 through all the experiments while increasing & definitely has lots of potential to boost the performance further especially when applying Entropy weighting scheme.
How does entropy-based weighting looks like? In order to prove our hypothesis that negative neighbor pairs will have lower consistency loss weight, we compute the entropy weights for all the neighbor pairs (k=l) Office-31 A W and W A, and take average over entropy weights on positive pairs and negative pairs respectively. From Fig Fig c)), we can see that positive neighbor pairs have much higher consistency loss weight than negative pairs by 20% average. It demon- strates our hypothesis that Entropy-weighted NC could alleviate the false neighbor supervision via assigning less loss weight them.
Hyper-parameter sensitivity. We conduct the hyper-parameter sensitivity analysis on the in Eqn. 10 and the triplet margin in Eqn. on Office31 A W. As Fig. [ shows, the performance of VNC decreases with the increase of while the performance of NC is inversely proportional to the triplet margin m. To conclude, our method is robust the regional changes of hyper-parameters. The reason that large margin results in a performance drop might be that there exists portion of anchors whose first neighbors are positive and semantically more similar to the anchors than their self-augmentations. Setting the margin to very large value will emphasize the inductive bias too much on the pre-defined variations and thus harm our model to align those positive neighbors with unknown variations to the anchors. By setting small margin, our method could ensure the self- augmentations are more close their anchors when the first neighbors of anchors are negative. At the same time, can ensure that the self-auggeenations not too far away from the first neighbors when they are positive.
Sance
Soune
Soca
20-
-29
-20
(a)
Source only
Entropy Minimization
(c) Neighbor Consistency
Figure 2: The t-SNE visualization of target feature embedding (red) on Office-31 W - A.
8

---

Under review as a conference paper at ICLR 2021
Anchor
mobile plooe
FCE
Acunnee
back pwos
144
03
DBC
WA-4
Meggtortied
Figure 3: Analysis: (a) Top 3 nearest neighbors given anchor based source model on Office-31. (Green: positive sample; Red: negative sample.) (b) Impact of the neighbourhood size on Office- 31 A + W. (c) Entropy-basee based weighting for positive and negative neighbor pairs Office-31 A W, W - A.
93.8
93.5
93.4
Accuracy
03.7
Accuracy
93.0
94.5
92.8
94.0
93.5
92.2
0.1
0.25
0.5
0.75
0.1
0.5
Lambda
Triplet Margin
(b)
Figure 4:
sensitivity analysis: (a) Sensitivity of on Office-31 - W. (b) Sensi- tivity of m Office-31 A + W.
CONCLUSION
In this work, propose simple but effective appoarch, named Neighbor Class Consistency based on the observation that target features extracted from source pre-trained model are high intrinsically discriminative. We introduce entropy-ber -based weighting scheme improve the robustness of our framework to potential noisy neighbor supervision. We also incorporate self-supervision and impose metric learning based on feature ranking relationship. We conduct solid ablation studies to prove each proposed components contributing to the performance and extensive experiments on three UDA image classification benchmarks. Our method outperforms all existing UDA state-of-the-art.
REFERENCES
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances in Neural Information Processing Systems, pp. 137-144, 2007.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine Learning, 191--2): 2010.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255. IEFE, 2009.

---

Under review as a conference paper at ICLR 2021
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning, pp. 1180-1189. PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The Journal of Machine Learning Research, 17(1):096-2030) 2016.
Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In International Conference on Computer Vision, pp. 5736 5735,,745,00 2017.
Ross Girshick. Fast R-CNN. In IEEE International Conference on Computer Vision, pp. 1440-1448, 2015.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Ad- vances in Neural Information Processing Systems, pp. 529-536, 2005.
Xiang Gu, Jian Sun, and Zongben Xu. Spherical space domain adaptation with robust pseudo-label loss. In IEEE Conference on Computer Vision and Pattern Recognition, June 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Judy Hoffmman, Dequan Wang, Fisher Yu, and Trevor Darrell. Fens in the wild: Pixel-level adversar- ial and constraint- based adaptation. arXiv preprint 16/2.02644 2016.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International Conference on Machine Learning, pp. 1989-1998. PMLR, 2018.
Guoliang Kang, Lu Jiang, Yi Yang, and Alexander Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 4893-49922 2019.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. International Conference on Machine Learning Workshop, volume 3, 2013.
Seungmin Lee, Dongwan Kim, Namil Kim, and Seong-Gyun Jeong. Drop to adapt: Learning discriminative features for unsupervised domain adaptation. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pp. 91-100, 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO: Common objects context. In European Conference Computer Vision, 2014.
Hong Liu, Mingsheng Long, Jianmin Wang, Michael Jordan. Transferable adversarial train- ing: A general approach to adapting deep classifiers. In International Conference Machine Learming. pp. 4013 4022, 2019.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431-3440, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International Conference on Machine Learning, pp. 2208-2217 JMLR. org, 2017.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650, 2018.
10

---

Under review as a conference paper at ICLR 2021
Yawei Luo, Liang Zheng. Tao Guan, Junqing Yu, and Yi Yang. Taking closer look domain shift: Category-level adversaries for semantics consistent domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2507-2516, 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, (Nov))2579-2660 2008
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(8):1979-1993, 2018.
Yingwei Pan, Ting Yao, Yehao Li, Yu Wang, Chong- Wah Ngo, and Tao Mei. Transferrable proto- typical networks for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2239 2247,
Xingchao Peng, Ben Usman, Neela Kaushik, Dequan Wang, Judy Hoffman, Kate Saenko, Xavier Roynard, -ean-mmmanuel Deschaud, Francois Goulette, Tyler L. Hayes, al. Visda: A synthetic- to-real benchmark for visual domain adaptation. In IEEE Conference on Computer Vision and Putlern Recognition Workshops, 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06433 2015.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European Conference on Computer Vision, pp. 102-118. Springer, 2016.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: large collection of synthetic images for semantic segmentation of urban scenes. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3234-32433 2016.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European Conference on Computer Vision, 2010.
Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Adversarial dropout regular- ization. In International Conference on Learning Representations, 2018a.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier dis- crepancy for unsupervised domain adaptation. In IEEE Conference Computer Vision and Pattern Recognition, pp. 3723-3732, 2018b.
Swami Sankaranarayanan. Yogesh Balaji. Carlos D Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 8503-8512, 2018.
Saquib Sarfraz, Vivek Sharma, and Rainer Stiefelhagen. Efficient parameter-free clustering using first neighbor relations. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 1934-8943, 2019.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation, In Thirtieth AAAI Conference Artificial Intelligence, 2016.
Hui Tang, Ke Chen, and Kui Jia. Unsupervised domain adaptation via structurally regularized deep clustering. In IEEE Conference Computer Vision and Pattern Recognition, pp. 8725-8735, 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve sumi-supervieed deep learning results. Advances in Neural Information Processing Systems, pp. 1195-1204, 2017.
Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation. In JEEE Conference Computer Vision and Pattern Recognition, pp. 7472-7481, 2018.

---

Under review as a conference paper at ICLR 2021
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 7167-7176, 2017.
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adver- sarial entropy minimization for domain adaptation in semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2517-2526, 2019.
Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen. Learning semantic representations for unsupervised domain adaptation. In International Conference on Machine Learning, pp. 5423- 5432, 2018.
Renjun Xu, Pelen Liu, Liyan Wang, Chao Chen, and Jindong Wang. Reliable weighted optimal transport for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 4394-4403. 2020.
Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In IEEE International Conference on Computer Vision, pp. 1435, 2019.
Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin. Regularizing class-wise predictions via self-knowledge distillation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan. Domain-symmetric networks for adversarial domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 5031- 5040, 2019a.
Yabin Zhang, Bin Deng, Kui Jia, and Lei Zhang. Label propagation with augmented anchors: A simple cupisuupe learning baseline for unsupervised domain adaptation. In European Conference on Computer Vision, pp. 781-797. Springer, 2020.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pp. 7404-7413, 2019b.
Zhun Zhong, Liang Zheng, Zhiming Luo, Shaozi Li, and Yi Yang. Invariance matters: Exemplar memory for domain adaptive person re-identification. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 598-607, 088-677,40 2019.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Domain adaptation for semantic segmen- tation via
self self-training. arXiv preprint arXiv:1810.07911,, 2018
Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In IEEE International Conference on Computer Vision, pp. 5982-5991, 2019.
A APPENDIX
A.1 ADDITIONAL HYPER-PARAMMTER SENSITIVITY ANALYSIS
To select the best neighborhood size & and analyze the influence of & on the performance, we eval- uate from (1, 3, 5} for all the tasks Office31 and report the performance in Fig, 5 We observe that our method (NC) achieves the best performance when & = while the performance does not monotonically decrease with the increase of neighborhood size.
12

---

Under review as a conference paper at ICLR 2021
90.60
NC
90.55
90.50
Accuracy
90.45
90.40
90.35
90.30
Neighborhood size
Figure Hyper-parameter sensitivity of neighborhood size on Office-31 for all tasks.
A.2 COMPARISON TO THE RELATED WORK FROM OTHER DOMAIN
To differentiate method to the related work from other domains such self-supervised learning Chen et al. (2000), knowledge distillation (Yun et al.] (2020) and person re-identification tasks Zhong al. (2019)), we implement their idea in unsupervised domain adaptation setting Of- fice31 A + W in table Specifically, 1) for self-supervised learning, we utilize the self consistency SC (ours). Though our objective function is dif- ferent, we adopt the same data augmentation rule from from (Chen et al. (2000). 2) For knowledge distillation which
loss in Eqn. 8 only comparison and
supervised setting, we replace our objective function (Mutual information maximization) in our ENC to the KI. divergence objectives from (Yun et al. (2200) to compute the class consistency loss and term as CS-KD. 3) For person re-identification, we refer the official github repository of (Zhong et al, 2019)) which introduces the neighbors invariance in feature space and implement it in UDA setting. We term it as ECN.
As table 6 shows, our ENC outperforms other baselines by significant margin. It demonstrates that 1) the neighbor class consistency contributes more to the domain alignment than self consis- tency loss. 2) Our consistency loss based on mutual information maximization is better than KL divergence in terms of enforcing class prediction consistency. 3) Our neighbor class consistency loss more effective than enforcing neighbor feature invariance. We think the features of target neighbors are naturally close to each other in feature space but the source biased classifier fails to provide consistent class predictions between the target neighbor. Therefore, our point of view, providing regularization on the classifier via our neighbor class consistency loss is more important than enforcing neighbor feature invariance in UDA problem.
Method
A W
ECN (Zhong et al. (009
87.9
CS-KD Yun et al. 2020))
84.5
SC (ours)
91.8
ENC (ours)
94.1
Table 6: Comparison the related works Office-31 A- W where ECN explores neighbor feature invariance and KD utilizes the class-wise prediction regularization.
13

---

Under review as a conference paper at ICLR 2021
A.3 ADDING RESULTS FOR ENC ON IMAGECLEF-DA AND VISDA17
To show the effectiveness of our method with ENC only, we report the results on ImageCLEF- DA and VisDA17 in table and 8 We could observe that method with ENC only achieves the third best performance (exclude NC-SP) in Office-31, top performance in ImageCLEF-DA and the second best performance in VisDA17. Compared the self-training based methods (ENT, MRENT, SAFN+Ent), our ENC outperforms them by large margin without bells and whistles.
Method
1-P
P-11
|-C
C-+1
C-+P
P-+C
Avg
ResNet-30 (He et al.
2016
74.8
83.9
91.5
78.0
65.5
91.2
80.77
DANN Ganin et al. 2016
75.0
86.0
96.2
87.0
74.3
91.5
85.0
JAN
Long et al. (2077)
76.8
88.0
94.7
89.5
74.2
91.7
85.8
CDAN (ong el al. 2018
76.7
90.6
97.0
90.5
74.5
93.5
87.1
SAFN+Ent Xu et at 2019
78.0
91.7
96.2
91.1
77.0
94.7
88.
SymNets 7Zhag ef al. 2019%
80.2
93.6
97.0
93.4
78.7
96.4
89.9
LLP+CAN Zhung
22220
79.8
94.3
97.7
93.0
79.9
96,9
90.3
ENC
79.6
95.0
97.5
94.1
79.2
96.7
90.3
NC-SP
80.9
95.0
97.9
94.2
79.8
97.5
90.9
Table 7: Experiment results on ImageCLEF-DA classification using ResNet-50
Mechad
Bike
Bus
Horse
Knife
Mator
Plant
Skateboard
Train
Truck
Mean
DANN
Lann 8
201G
81.9
77.7
82.8
44.3
81.2
29.3
65.1
28.6
51.9
54.6
82.8
788
57.4
MCD
Sano ar
20100
87.0
60.9
83.7
64.0
$8.9
79.6
$4.7
76.9
88.6
40.3
83.0
25.8
71.9
ADR
Sahoo
20133
87.8
79.5
83.7
65.3
61.8
88.9
73.2
87.8
60.0
85.5
32.3
74.8
MRKLD 220
2019
88.0
79.2
61.0
60.0
87.5
86.3
78.8
85.6
86.6
73.9
68.8
78.1
DTA Lee ot
20191
92.7
82.2
85.6
83.8
93.0
$1.0
90.7
82.1
95.1
78.1
86.4
32.1
$1.5
RWOT (xu
95.1
80.3
83.7
90.0
92.1
68.0
92.5
62.2
87.9
78.1
90.4
68.2
81.0
91.0
NC-SP
97.1
88.5
90.0
65
96.7
92.9
90.1
81.5
94.6
89.5
89.0
58.8
86.2
Table 8: Experimental results on VisDA17 classification using ResNet- 101.
A.4 MUTUAL INFORMATION OBJECTIVE FUNCTION
We define the class prediction of target sample as P(z!) C(z 0) and its neighbor sample j as P(z{)= C(2)(0).
The objective of mutual information can be formulated as:
P(z),2))
MI (P(())), = P(.) log
(12)
((z))
where
N; (P(z)) + P(2)))()))
2()), N
(13)
14


Reviews:
[{"review_id": "dqzzzWSh6Nc", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the unsupervised domain adaption (UDA) problem. Particularly, the paper proposes to impose neighbor class consistency on target features to preserve intrinsic discriminative nature of target data and presents an entropy-based weighting scheme to improve robustness against the potential noisy neighbor supervision. The motivation of the paper is clear and the method is well presented. Extensive experiments show the effectiveness of the proposed method. However, the paper suffers some problems, such as\n1. The idea of neighborhood consistency in UDA is not very significant. It lies on the assumption that target features from source pre-trained model are highly intrinsic discriminative. However, the distribution discrepancy between source domain and target domain may be very large, such as unsupervised cross-dataset person re-identification. It is not sure whether such assumption/observation is still satisfied.\n2. From Table 3-5, NC-SP outperforms the state-of-the-art methods with only a small margin. From Table 3, SC, FR , SP make a great contribution to the performance. It is suggested to add some comparisons to Table 3-5 that only the proposed VNC or ENC is used without other tricks.\n3. It is suggested to evaluate the influence of hyper-parameter \\lambda in Eq. (10) and (11)\n4.  It is suggested to evaluate neighbor size K on more datasets, in order validate the association of K with target dataset and number of classes.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_-oX3u5U5b", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed neighbor class consistency regularization together with an entropy-based weighting factor to tackle the problem of unsupervised domain adaptation. Another self class consistency regularization was further introduced to help training. The difference between \"neighbor class\" and \"self class\" is the positive pair selection, where \"neighbor class\" uses k-nearest neighbors as positive pairs, and \"self class\" uses an augmented version of the anchor itself.\n\nThe paper is generally well-written and easy to follow. Sufficient experiments are conducted. However, I think the quality of the paper is not publishable now due to the following weaknesses.\n\nMajor weaknesses:\n\n1. The core contribution of this paper is the class consistency loss (Eq. (5)&(7)) between the anchor and its neighbors, however, similar ideas have been investigated in existing works, e.g. [A, B]. But unfortunately, this paper even did not include them in the related works.\n\n\n\n2. I think the work lacks novelty. The memory bank in Sec 3.3 (1) is similar to [C, D] and the data augmentation rules in Sec 3.4 are similar to [E]. The consistency loss is popular in knowledge distillation tasks. In my point of view, the paper heavily borrowed techniques from existing works.  The author should at least acknowledge the relations between the paper and these works.\n\n\n\n3. It's somewhat counter-intuitive that the paper required the similarity between the anchor and its augmented version to be closer than the similarity between the anchor and its nearest neighbor (Eq. (9)). A situation may exist: the augmentation is very heavy while the nearest neighbor shows very similar visual features with the anchor.\nThe author has shown the effectiveness of Eq. (9) in Table 2, namely \"FR\". However, it's not convincing enough, I recommend to evaluate \"ENC + SC + SP (NC-SP)\", i.e. removing \"FR\" from the whole model.\n\n\nMinor weaknesses:\n\n4. The paper claimed its state-of-the-art performances on various benchmarks. But it missed some previous methods, e.g. [F]. The paper achieved even inferior performance than [F] on the VisDA17 benchmark, i.e. 86.2 in this paper v.s. 87.2 in [F].\n\n5. As I mentioned in weakness-3, it's better to use \"minus\" rather than \"plus\" in ablation studies.\n\n6. More hyper-parameter analysis would help improve paper quality. For example, as described in Sec 4.2, the author adopted different $\\lambda$ values for different datasets. It's better to show how much does the value of $\\lambda$ impact the final performance.\n\n\n\n[A] Regularizing Class-wise Predictions via Self-knowledge Distillation. CVPR 2020.\n\n[B] Invariance Matters: Exemplar Memory for Domain Adaptive Person Re-identification. CVPR 2019.\n\n[C] Unsupervised Feature Learning via Non-Parametric Instance Discrimination. CVPR 2018.\n\n[D] Momentum Contrast for Unsupervised Visual Representation Learning. CVPR 2020.\n\n[E] A Simple Framework for Contrastive Learning of Visual Representations. ICML 2020.\n\n[F] Contrastive Adaptation Network for Unsupervised Domain Adaptation. CVPR 2019.\n\n\n=========================================================================================================\n\nUpdate:\n\nThanks for the authors' response. But unfortunately, the most significant concern has not been addressed well, which is regarding of the paper's novelty. I thought the contributions of this work are incremental, and the authors' rebuttal did not convince me well. Btw, the authors claimed that [A] used KL-divergence while their work used mutual information maximization, however, minimizing KL-divergence is actually one kind of mutual information maximization. What's more, the authors even did not clearly point out which kind of mutual information maximization they used in either the manuscript or the rebuttal. Thanks again for the authors' efforts, but I choose to maintain my original score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Yf3eF5g9Aas", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Based on the observation that the target features from source pre-trained model are highly intrinsic discriminative and have a high probability of sharing the same label with their neighbors, a simple but effective method to impose Neighbor Class Consistency on target features is proposed. The experimental results are promising.\n[1] Compared with VNC, the improvement of ENC is very limited. Whether the authors have repeated the experiments in several times. It is better to show the variance of the results in Table 2, which may be more suitable to demonstrate the effectiveness of ENC.\n[2] In Eq.(9), it forces that the anchor samples should be closed to the neighbor samples by comparing with the distance between the anchor sample and the augmented sample. What is the advantage of such a strategy compared with triplet loss. It is also better to show the comparison in the experiments.\n[3] For the neighbors, when k is equal to 1, the proposed method achieved the best performance. Hence, the neighbors with the minimum distance could not guarantee that they share the same labels. How does the proposed method estimate the confidence.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WvVLaG4FUIK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles Unsupervised Domain Adaptation. The authors focus on the intrinsic discriminative feature for target samples. The proposed method, Neighborhood Class Consistency among target samples and augmented ones, is proposed as a set of multiple losses to calculate the consistency from several aspects. The experimental results show that the proposed method achieves state-of-the-art performance using the same backbone network.\n\n**Pros**\n- The consistent approach making use of neighborhood structure in the target domain is well motivated and easy to understand.\n- The proposed method achieves SoTA performance on several benchmark datasets.\n\n**Cons**\n- There is a related paper by Gu et al. (2020), although it is not referred to in this paper. Gu et al. also focus on a pseudo labeling approach and discriminative distributions in the target domain. Additionally, the experimental results show that the performance is very close to that in this paper.  Although the detailed method is different, the authors should have compared them qualitatively and quantitatively.\n\nGu et al., Spherical Space Domain Adaptation with Robust Pseudo-label Loss. In CVPR, 2020.\n\n- Section 4.2 says that the authors use different values for the hyperparameter, such as $\\lambda$, for each dataset. How do they tune them? Since the method is for UDA, there cannot be a validation set in the target domain. Therefore, different values of $\\lambda$ seem unnatural. Alternatively, the authors should report the performance with different hyperparameter values not only for the number of neighbor samples in Fig. 3 (b) but also for $\\lambda$ and triplet margin $m$.　\n\n**Minor comments**\n- In the main text, there are some points the authors should insert a white space before a bracket and a citation.\n- In the last paragraph of Section 1, an abbreviation \"NC\" is used without an explanation. What does it stand for?\n- How often do we need to update the memory bank $\\mathbb{V}_t$ while a model is trained? Frequent updates of the memory would be time-consuming.\n-The format of References is messy. Some papers, such as CyCADA, should not be arXiv preprints but published ones. Some proceeding names are capitalized while the others are not. Some conference names are abbreviated while the others are not.\n-The blank Appendix section should be deleted.\n\n**Overall rating**\n\nAlthough there should be more descriptions about the existing method and experimental results, the reviewer is leaning toward acceptance. The rating can be upgraded if the authors could solve the cons above.\n\n**Additional comment after rebuttal**\n\nUnfortunately, the reviewer would like to downgrade my first score. The remaining concerns are about R2-A1 and R2-A2.\n\nIn R2-A1, the authors add some discussions about experimental results and explanations about Gu et al. (2020). The second discussion about generating pseudo labels would be the most considerable theoretical difference between the proposed method and Gu et al. (2020). The authors state that the proposed method addresses the problem by incorrect pseudo labeling of Gu et al. (2020); however, they fail to show quantitative nor qualitative discussions that support the statement. The third discussion seems just showing the proposed method achieves similar performance to that of Gu et al. (2020). As discussed in the fourth comment, the authors could add another result showing the proposed method is complementary to Gu et al. (2020).\n\nIn R2-A2, the authors honestly state that the hyperparameters are tuned according to the test data evaluation. However, it should be avoided to evaluate unsupervised domain adaptation methods since there are no labeled data in the target domain in a real setting. Moreover, Figure 4 (a) shows that the proposed method is sensitive to lambda on Office31 dataset. When lambda=0.5 on all dataset, the proposed method achieves SOTA on VisDA as shown in Table 5, but seems to fail on Office31.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method (via a novel objective) for feature alignment between source and target tasks in an unsupervised domain adaptation scenario.

Pro
- the proposed approach is sensible in many realistic scenarios of distribution shift
- the submission provides an extensive empirical evaluation establishing state of the art results on several benchmark tasks

Con
- there is no thorough discussion of the the underlying assumptions (when should we expect them to hold? for shich types of tasks? which types of shifts? can they generally be reliably tested? from which type of data? unlabeled?)
- one reviewer raised concerns over novelty, which should be more clearly addressed before publication
- two reviewers raised concerns over use of target data for hyper-parameter selection, which seem valid; these should be fixed or clearly explained (and implications of this be discussed) in subsequent versions of this work

I agree with concerns of the reviewers (the last two points), and would therefore not recommend this work for publication in the current state.
Decision: Reject