CLDA: Contrastive Learning for Semi-Supervised Domain Adaptation

1 Introduction . Deep Convolutional networks [ 30 , 52 ] have shown impressive performance in various computer vision tasks , e.g. , image classification [ 19 , 22 ] and action recognition [ 48 , 23 , 57 , 32 ] . However , there is an inherent problem of generalizability with deep-learning models , i.e. , models trained on one dataset ( source domain ) does not perform well on another domain . This loss of generalization is due to the presence of domain shift [ 11 , 55 ] across the dataset . Recent works [ 46 , 29 ] have shown that the presence of few labeled data from the target domain can significantly boost the performance of the convolutional neural network ( CNN ) based models . This observation led to the formulation of Semi-Supervised Domain Adaption ( SSDA ) , which is a variant of Unsupervised Domain Adaptation where we have access to a few labeled samples from the target domain . Unsupervised domain adaptation methods [ 42 , 12 , 36 , 51 , 35 ] try to transfer knowledge from the label rich source domain to the unlabeled target domain . Many such existing domain adaptation approaches [ 42 , 12 , 51 ] align the features of the source distribution with the target distribution without considering the category of the samples . These class-agnostic methods fail to generate discriminative features when aligning global distributions . Recently , owing to the success of contrastive approaches [ 6 , 18 , 39 ] , in self-representation learning , some recent works [ 26 , 28 ] have turned to instance-based contrastive approaches to reduce discrepancies across domains . [ 46 ] reveals that the direct application of the well-known UDA approaches in Semi-Supervised Domain Adaptation yields sub-optimal performance . [ 29 ] has shown that supervision from labeled source and target samples can only ensure the partial cross-domain feature alignment . This creates 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . aligned and unaligned sub-distributions of the target domain , causing intra-domain discrepancy apart from inter-domain discrepancy in SSDA . In this work , we propose CLDA , a simple single-stage novel contrastive learning framework to address the aforementioned problem . Our framework contains two significant components to learn domain agnostic representation . First , Inter-Domain Contrastive Alignment reduces the discrepancy between centroids of the same class from the source and the target domain while increasing the distance between the class centroids of different classes from both source and target domain . This ensures clusters of the same class from both domains are near each other in latent space than the clusters of the other classes from both domains . Second , inspired by the success of self-representation learning in semi-supervised settings [ 17 , 6 , 49 ] , we propose to use Instance Contrastive Alignment to reduce the intra-domain discrepancy . In this , we first generate the augmented views of the unlabeled target images using image augmentation methods . Alignment of the features of the original and augmented images of the unlabeled samples from the target domain ensures that they are closer to each other in latent space . The alignment between two variants of the same image ensures that the classifier boundary lies in the low-density regions assuring that the feature representations of two variants of the unlabeled target images are similar , which helps to generate better clusters for the target domain . In summary , our key contributions are as follows . 1 ) We propose a novel , simple single-stage training framework for Semi-supervised Domain Adaptation . 2 ) We propose using alignment at class centroids and instance levels to reduce inter and intra domain discrepancies present in SSDA . 3 ) We evaluate the effectiveness of different augmentation approaches , for instance-based contrastive alignment in the SSDA setting . 4 ) We evaluate our approach over three well-known Domain Adaptation datasets ( DomainNet , Office-Home , and Office31 ) to gain insights . Our approach achieves the state of the art results across multiple datasets showing its effectiveness . We perform extensive ablation experiments highlighting the role of different components of our framework . 2 Related Works . 2.1 Unsupervised Domain Adaptation . Unsupervised Domain Adaptation ( UDA ) [ 14 ] is a well-studied problem , and most UDA algorithms reduce the domain gap by matching the features of the sources and target domain [ 16 , 4 , 24 , 36 , 51 , 27 ] . Feature-based alignment methods reduce the global divergence [ 16 , 51 ] between source and target distribution . Adversarial learning [ 12 , 5 , 34 , 35 , 42 , 41 ] based approaches have shown impressive performance in reducing the divergence between source and target domains . It involves training the model to generate features to deceive the domain classifier , invariantly making the generated features domain agnostic . Recently , Image translation methods [ 20 , 21 , 38 ] have been explored in UDA where an image from the target domain is translated to the source domain to be treated as an image from the source domain to overcome the divergence present across domains . Despite remarkable progress in UDA , [ 46 ] shows the UDA approaches do not perform well in the SSDA setting , which we consider in this work . 2.2 Semi-Supervised Learning . Semi-Supervised Learning ( SSL ) aims to leverage the vast amount of unlabeled data with limited labeled data to improve classifier performance . The main difference between SSL and SSDA is that SSL uses data sampled from the same distribution while SSDA deals with data sampled from two domains with inherent domain discrepancy . The current line of work in SSL [ 50 , 3 , 31 , 10 ] follows consistency-based approaches to reduce the intra-domain gap . Mean teacher [ 53 ] uses two copies of the same model ( student model and teacher model ) to ensure consistency across augmented views of the images . Weights of the teacher model are updated as the exponential moving average of the weights of the student model . Mix-Match [ 3 ] and ReMixMatch [ 2 ] use interpolation between labeled and unlabeled data to generate perturbed features . Recently introduced FixMatch [ 50 ] achieves impressive performance using the confident pseudo labels of the unlabeled samples and treating them as labels for the strongly perturbed samples . However , direct application of SSL in the SSDA setting yields sub-optimal performance as the presumption in the SSL is that distributions of labeled and unlabeled data are identical , which is not the case in SSDA . 2.3 Contrastive Learning . Contrastive Learning ( CL ) has shown impressive performance in self-representation learning [ 6 , 1 , 18 , 54 , 39 ] . Most contrastive learning methods align the representations of the positive pair ( similar images ) to be close to each other while making negative pairs apart . In semantic segmentation , [ 33 ] uses patch-wise contrastive learning to reduce the domain divergence by aligning the similar patches across domains . In domain adaptation , contrastive learning [ 28 , 26 ] has been applied for alignment at the instance level to learn domain agnostic representations . [ 26 , 28 ] use samples from the same class as positive pairs , and samples from different classes are counted as negative pairs . [ 26 ] modifies Maximum Mean Discrepancy ( MMD ) [ 16 ] loss to be used as a contrastive loss . In contrast to [ 28 , 26 ] , our work proposes to use contrastive learning in SSDA setting both at the class and instance level ( across perturbed samples of the same image ) to learn the semantic structure of the data better . 2.4 Semi-Supervised Domain Adaptation . Semi-Supervised Domain Adaptation ( SSDA ) aims to reduce the discrepancy between the source and target distribution in the presence of limited labeled target samples . [ 46 ] first proposed to align the source and target distributions using adversarial training . [ 29 ] shows the presence of intra domain discrepancy in the target distribution and introduces a framework to mitigate it . [ 25 ] uses consistency alongside multiple adversarial strategies on top of MME [ 46 ] . [ 9 ] introduced the meta-learning framework for Semi-Supervised Domain Adaptation . [ 58 ] breaks down the SSDA problem into two subproblems , namely , SSL in the target domain and UDA problem across the source and target domains , and learn the optimal weights of the network using co-training . [ 37 ] proposed to use pretraining of the feature extractor and consistency across perturbed samples as a simple yet effective strategy for SSDA . [ 44 ] introduces a framework for SSDA consisting of a shared feature extractor and two classifiers with opposite purposes , which are trained in an alternative fashion ; where one classifier tries to cluster the target samples while the other scatter the source samples , so that target features are well aligned with source domain features . Most of the above approaches are based on adversarial training , while our work proposes to use contrastive learning-based feature alignment at the class level and the instance level to reduce discrepancy across domains . 3 Methodology . In this section , we present our novel Semi-Supervised Domain Adaptation approach to learn domain agnostic representation . We will first introduce the background and notations used in our work and then describe our approach and its components in detail . 3.1 Problem Formulation . In Semi-Supervised Domain Adaptation , we have datasets sampled from two domains . The source dataset contains labeled images Ds = { ( xsi , ysi ) } Ns i=1 ⊂ Rd × Y sampled from some distribution PS ( X , Y ) . Besides that , we have two sets of data sampled from target domain distribution PT ( X , Y ) . We denote the labeled set of images sampled from the target domain as Dlt = { ( xlti , ylti ) } Nlt i=1 . The unlabeled set sampled from target domain Dt = { ( xti ) } Nt i=1 contains large number of images ( Nt Nlt ) without any corresponding labels associated with them . We also denote the labeled data from both domains as Dl = Ds ∪ Dlt . Labels ysi and ylti of the samples from source and labeled target set correspond to one of the categories of the dataset having K different classes/categories i.e Y = { 1 , 2 , ... K } . Our goal is to learn a task specific classifier using Ds , Dlt and Dt to accurately predict labels on test data from target domain . 3.2 Supervised Training . Labeled source and target samples are passed through the CNN-based feature extractor G ( . ) to obtain corresponding features , which are then passed through task-specific classifier F ( . ) to minimize the well-known cross-entropy loss on the labeled images from both source and target domains . Lsup = − K∑ k=1 ( yi ) k log ( F ( G ( ( xil ) ) k ( 1 ) 3.3 Inter-Domain Contrastive Alignment . Our method is based on the observation that the samples from the same category across domains must cluster in the latent space . However , this is observed only for the source domain due to the availability of the labels . Samples from the target domain do not align to form clusters due to the domain shift between the target and the source distributions . This discrepancy between the cluster of the same category across domains is reduced by aligning the centroids of each class of source and target domain . [ 6 , 17 ] have shown that having a separate projection space is beneficial for contrastive training . Instead of using a separate projection , we have used the outputs from the task-specific classifier as features to align the clusters across the domain . We represent the centroid of the images from the source domain belonging to class k as the mean of their features , which can be written as Csk = i=B∑ i=1 1 { ysi=k } F ( G ( x s i ) ) i=B∑ i=1 1 { ysi=k } ( 2 ) where B is the size of batch . We maintain a memory bank ( Cs = [ Cs1 , C s 2 , .... C s K ] ) to store the centroids of each class from source domain . We use exponential moving average to update these centroid values during the training Csk = ρ ( C s k ) step + ( 1− ρ ) ( Csk ) step−1 where ρ is a momentum term , and ( Csk ) step and ( C s k ) step−1 are the centroid values of class k at the current and previous step , respectively . We also need to cluster the unlabeled target samples for Inter-Domain Contrastive Alignment . The pseudo labels obtained from the task specific classifier as shown in Eq ( 3 ) is used as the class labels for the corresponding unlabeled target samples . ŷti = argmax ( ( F ( G ( x t i ) ) ) ( 3 ) Similar to the source domain , we also calculate the separate cluster centroid Ctk for each of the class k of the target samples present in the minibatch as per the Eq ( 2 ) where unlabeled target images replace the images from the source domain with their corresponding pseudo label . The model is then trained to maximize the similarity between the cluster representation of each class k from the source and the target domain . Csk and C t k form the positive pair while the remaining cluster centroids from both domains form the negative pairs . The remaining clusters from both domains are pushed apart in the latent space . This is achieved through employing a modified NT-Xent ( normalized temperature-scaled cross-entropy ) contrastive loss [ 6 , 39 , 49 , 33 ] for domain adaptation given by Lclu ( Cti , Csi ) = − log h ( Cti , C s i ) h ( Cti , C s i ) + K∑ r=1 q∈ { s , t } 1 { r 6=i } h ( Cti , C q r ) ( 4 ) where h ( u , v ) = exp ( u > v ||u||2||v||2 /τ ) measures the exponential of cosine similarity , 1 is an indicator function and τ is the temperature hyperparameter .

Summary: The paper is about semi-supervised domain adaptation.  The inputs are labeled source samples, as well as labeled and unlabeled target domain samples.  The task is to train a model that performs well on the target domain.  The authors propose a framework with two main components to learn a domain-agnostic representation.  - Inter-domain contrastive alignment: This module reduces the discrepancy between classes centroids of the same class from source and target domain, while increasing the distance between class centroids of different classes from both source and target domains. Class centroids are computed as an exponential moving average via equations (2) and (3).  A contrastive loss is used to pull and push the centroids. - Instance contrastive alignment: The goal of this module is to pull the classifier boundary into a low-density region, which is done by regularizing the classifier on the unlabeled target data.  Image augmentation is used to create perturbed images of unlabeled target data and a contrastive loss is employed to pull the representation of the same instance (perturbed and unperturbed) together while pushing it away from other samples.  Experiments on Office-Home, DomainNet and Office31 show promising results.