Neighbor Class Consistency on Unsupervised Domain Adaptation

1 INTRODUCTION . Recent advances in deep neural network have dominated many computer vision tasks , such as image recognition He et al . ( 2016 ) , object detectionGirshick ( 2015 ) , and semantic segmentationLong et al . ( 2015 ) . However , collection and manual annotation need no trivial human effort , especially for vision tasks like semantic segmentation where dense annotations are required . Thanks to the growth of computer graphics field , it is now possible to leverage CNN to synthetic images with computergenerated annotations ( Richter et al . ( 2016 ) ; Ros et al . ( 2016 ) ) , so unlimited amount of data with free annotation is available for training network in scale . However , directly applying the model trained on synthetic source data to unlabeled target data leads to performance degradation and Unsupervised Domain Adaptation ( UDA ) aims to tackle this domain shift problem . A widespread of UDA methods were proposed to align the domain-invariant representations by simultaneously minimizing the source error and discrepancy ( e.g . H-divergence Ben-David et al . ( 2010 ) ; Hoffman et al . ( 2016 ) H4H-divergenceBen-David et al . ( 2010 ) ) between two domain such as the maximum mean discrepancy Tzeng et al . ( 2014 ) , correlation distanceSun et al . ( 2016 ) and etc . Further , adversarial learning-based UDA Ganin & Lempitsky ( 2015 ) ; Tzeng et al . ( 2017 ) ; Radford et al . ( 2015 ) ; Hoffman et al . ( 2018 ) ; Tsai et al . ( 2018 ) ; Sankaranarayanan et al . ( 2018 ) ; Luo et al . ( 2019 ) methods aim to reduce this discrepancy between two domain by minimizing the adversarial loss . However , the major limitation of adversarial learning is that it only aligns the global feature distribution of two domains without considering the class labels . As the result , a small H4H distance does not guarantee the small error on ideal joint hypothesis on the features of two domains Liu et al . ( 2019 ) . To alleviate this issue , Entropy minimization ( Grandvalet & Bengio ( 2005 ) ; Vu et al . ( 2019 ) ) and Self-Training ( Lee ( 2013 ) ; Zou et al . ( 2018 ) ) are the two dominant methods to enforce the cluster assumption such that network can learn a discriminative feature space by pushing the decision boundary away from densely-distributed area . However , as decision boundary is largely biased to- wards source data , trusting biased network predictions will push target features towards their nearest source class prototypes while deteriorating the intrinsic discriminative target structure as shown in Fig . 1 ( 2.a ) . Motivated by agglomerative clustering methods ( Sarfraz et al . ( 2019 ) ) which assume that features in the nearby region should be clustered together , we investigate target features from source pretrained model and observe that they are intrinsically discriminative and have a very high possibility of sharing the same label with their neighbors as shown in Fig . 1 ( 2.b ) . To utilize this high-quality pairwise neighbor supervision , we propose a simple and effective approach to impose Neighbor Class Consistency between target samples and their neighbors . To alleviate propagated errors from false neighbor supervision , we introduce an Entropy-based weighting scheme to emphasize more on the reliable pairwise neighbor supervision . Additionally , we categorize Self Class Consistency as a special case of our method where the nearest neighbor of a sample is its self-augmentation . Further , we explore feature representation learning based on the ranking relationship between selfaugmentation and the first neighbor given an anchor . We enforce the features of anchors to be closer to their self-augmentation than their first neighbors . In summary , our main contributions are shown as follows : ( 1 ) We revisit the source pre-trained model and observe the intrinsic discriminative nature of target features from source model . ( 2 ) Based on this observation , we propose Neighbor Class Consistency ( NC ) to utilize the high-quality pairwise neighbor pseudo supervision over noisy class-wise pseudo supervision from Self-Training methods . ( 3 ) We introduce an Entropy-based weighting scheme to help our framework be more robust to unreliable neighbor supervision . ( 4 ) We categorize Self Class Consistency as a special case of our framework and explore the first neighbor for feature representation learning . ( 5 ) We conduct extensive experiments on three UDA benchmarks datasets . NC outperforms all existing methods and achieves a new UDA state-of-the-art performance . Notably , we achieve 86.2 % on challenging VisDA17 dataset . 2 RELATED WORK . Discrepancy based domain adaptation Following the theoretical upper bound proposed in BenDavid et al . ( 2007 ) , existing methods have explored to align the feature representations of the source and target images by minimizing the distribution discrepancy . For example , Maximum Mean Discrepancy ( MMD ) Tzeng et al . ( 2014 ) is proposed to match the mean and covariance of source and target distributions . Alternatively , adversarial domain adaptation methods Ganin & Lempitsky ( 2015 ) ; Tzeng et al . ( 2017 ) ; Radford et al . ( 2015 ) ; Hoffman et al . ( 2018 ) ; Tsai et al . ( 2018 ) ; Sankaranarayanan et al . ( 2018 ) ; Luo et al . ( 2019 ) solve this domain discrepancy by training a domain- invariant feature generator which produces the features to fool a discriminator that distinguishes the representations from source and target domains . However , since the domain discriminator aligns source and target features without considering the class labels , merely aligning the global marginal distribution of the features in the two domains fails to align the class-wise distribution . Clustering based domain adaptation Entropy minimization ( Grandvalet & Bengio ( 2005 ) ; Vu et al . ( 2019 ) ) . and self-training ( Zou et al . ( 2018 ) ; Gu et al . ( 2020 ) ) are two streams of approaches to realize the class-wise alignment across domains . However , minimizing the conditional entropy of target prediction based on source biased classifier will harm the intrinsic discriminative target structure . To this end , prototypical classifier and alignment Xie et al . ( 2018 ) ; Pan et al . ( 2019 ) have been explored to mitigate the noisy supervision from source biased classifier . However , the prototypes of target samples are still noisy estimated cluster centers and therefore target samples away from prototypes may still risk of being wrongly classified . Similar effort has been made by Tang et al . ( 2020 ) on learning the latent discriminative target structure with respect to learnable clusters via deep clustering framework Ghasedi Dizaji et al . ( 2017 ) . Consistency Regularization In semi-supervised setting , mainstream methods apply various consistency regularization on unlabeled data with different pre-defined positive counterpart . Among them , Tarvainen & Valpola ( 2017 ) impose consistency between predictions from student network and moving-average based teacher networks . Virtual Adversarial Training Miyato et al . ( 2018 ) tries to make network invariant to small adversarial permutations around the neighborhood of a sample while DTALee et al . ( 2019 ) enforces the target predictions from the networks with different choice of dropout mask to be consistent . In un/self-supervised setting , SimCLR ( Chen et al . ( 2020 ) ) and MoCo ( He et al . ( 2020 ) ) are the two prevalent approaches to conduct contrastive learning on the feature space among unlabeled data , its strong augmentation and other negative samples from either extremely large current batch or momentum updated memory banks . In comparison to those un/self-supervised methods , our method exploreS neighbor samples for consistency regularization . In cross-domain person re-identification , Zhong et al . ( 2019 ) also utilize the neighbor information on target data . To emphasize the difference , our method focuses on regularizing the classifier to be less biased toward source domain by applying target neighbor class consistency while they focus on the feature representation learning by enforcing neighbor feature invariance . 3 METHOD . 3.1 PROBLEM DEFINITION . In unsupervised domain adaptation ( UDA ) , source domain data is denoted as Ds = { ( xsi , ysi ) | Ns i=1 } where xsi and y s i denote the i-th training sample and its label , Ns is the number of source images . Target domain data is denoted as Dt = { xti| Nt i=1 } where Nt is the number of target images . The objective of UDA is to train a deep neural network G ( ·|θ ) which has access to the source data ( xsi , y s i ) drawn from Ds and target data xti drawn from Dt such that the modelG ( ·|θ ) can generalize on target domain . Network G ( ·|θ ) = C ◦ F ( ·|θ ) is comprised of a feature extractor F ( ·|θ ) and a classifier C ( ·|θ ) where θ denotes network parameters . 3.2 REVISIT SUPERVISED PRE-TRAINING FOR SOURCE DOMAIN . In general , UDA pre-trains a network G ( ·|θ ) on source domain by standard cross entropy loss and then the network is transferred to inference the target data . The source supervised objective function is in the form of , Lssrc ( θ ) = 1 Ns Ns∑ i=1 Lce ( C ( F ( xsi |θ ) ) , ysi ) . ( 1 ) However , the model trained on source data usually generalizes poorly on target data due to the domain shift between the joint distribution of two domains . Self-Training ( ST ) and Entropy Minimization ( Ent ) methods are proposed to make the network be confident on its target predictions following the cluster assumption and thus improve the discrim- inativeness of network . Their objective functions are calculated as LtST ( θ ) = 1 Nt Nt∑ i=1 Lce ( C ( F ( xti|θ ) ) , ỹti ) , ( 2 ) LtEnt ( θ ) = 1 Nt Nt∑ i=1 H ( C ( F ( xti|θ ) ) ) , ( 3 ) where ỹti is the pseudo label for the i target sample andH is entropy function . Nonetheless , the pseudo labels of target samples are noisy as the network is biased towards source data . Trusting noisy label or prediction has high risk of misleading the training and propagate errors . Inspired by agglomerative clustering methods which claim that features should be clustered with their near neighborhood , we hypothesize that source domain pre-training can provide a feature space where target features are intrinsically discriminative and have high probability of sharing the same labels with their neighbors . To verify this , we apply k-Nearest Neighbor ( kNN ) search on target features extracted from source model on Offce-31 A→W . We evaluate the pairwise label consistent accuracy between target features and their k-th neighbors . Table 1 shows that target features from source model have very high accuracy of sharing the same label with their K-th neighbors . It demonstrates our hypothesis that target features from source model are locally discriminative . This observation motivates us to leverage this relative ” clean ” pairwise pseudo supervision to help the network generalization on target domain over the noisy class-wise pseudo supervision . 3.3 NEIGHBORHOOD CLASS CONSISTENCY . 1 ) Neighborhood Discovery . First , we extract target features zt = F ( xt|θ ) from source pre-train model and save them into a target feature memory bank Vt = { zti | Nt i=1 } . Then we apply kNN on target features Vt to explore the neighborhood Nk ( zt ) which is defined as follows : Nk ( ztj ) = { zti |s ( zti , ztj ) is top-k in Vt } , ( 4 ) where s is a similarity metric and k is the number of neighbors . Note that we update the memory bank Vt with the newest features in training and conduct kNN on Vt at every epoch . 2 ) Vanilla Neighborhood Class Consistency ( VNC ) . Based on our motivation that target samples should share the same class label with their neighbors , we propose VNC to enforce the class assignment consistency between pairwise neighbors by mutual information ( MI ) maximization between their network predictions . Formally , we formulate the objective function of VNC in the form of : LtV NC ( θ ) = − 1 Nt Nt∑ i=1 1 k ∑ j∈Nk ( zti ) MI ( C ( zti |θ ) , C ( ztj |θ ) ) . ( 5 ) It is also worth noting that the memory bank serves as a look-up table to retrieve the target features and feed them into the classifier C for computing the Neighborhood Class Consistency loss . As the size of k can be potentially large , using memory bank can implicitly enlarge the batch size by k times without introducing extra computational cost and time . 3 ) Entropy-weighted Neighborhood Class Consistency ( ENC ) . Table 1 shows that the reliability of pairwise neighborhood supervision is decreased with the rise of neighborhood size k. In other words , there is a trade-off between large neighborhood diversity and reliability of neighbor supervision . A straightforward idea is to treat each neighbor pairs differently in terms of loss weight such that hopefully positive neighbor pairs ( Correct ) will have higher loss weight than negative neighbor pairs ( False ) . To achieve this , we propose a entropy-based weighting scheme ( EW ) to assign different loss weight on different neighbor pairs given an anchor . As the entropy is a measurement of sample prediction certainty , the less entropy is , more confident the prediction is and thus more weight we will assign on . Formally , we define the loss weight and objective of ENC as follows : w ( zt ) = 1− H ( C ( z t|θ ) ) logM , ( 6 ) LtENC ( θ ) = − 1 Nt Nt∑ i=1 w ( ztj ) k ∑ j∈Nk ( zti ) MI ( C ( zti |θ ) , C ( ztj |θ ) ) , ( 7 ) where M is the number of class . We claim that entropy-based weighting helps mitigate the problem of noisy pairwise supervision and it is more robust to the neighborhood size k as it will down-weight the neighbor pairs if the selected neighbor samples have high entropy value .

Summary: This paper proposed neighbor class consistency regularization together with an entropy-based weighting factor to tackle the problem of unsupervised domain adaptation. Another self class consistency regularization was further introduced to help training. The difference between "neighbor class" and "self class" is the positive pair selection, where "neighbor class" uses k-nearest neighbors as positive pairs, and "self class" uses an augmented version of the anchor itself.