Pairwise Adversarial Training for Unsupervised Class-imbalanced Domain Adaptation

‘ 1 INTRODUCTION . Unsupervised domain adaptation ( UDA ) aims to achieve knowledge transfer from a labeled source domain to an unlabelled target domain . Recent years have witnessed the significant progress of UDA based on deep neural networks ( Pei et al. , 2018 ; Cui et al. , 2020 ; Hu et al. , 2020 ; Liang et al. , 2020 ; Na et al. , 2021 ) . Most of existing UDA methods assume that only covariate shift occurs in the source domain and target domain , while the label distributions in two domains are identical . However , this assumption may not hold in real-world applications . For instance , in wild-life pictures , the commonly seen animals such as rabbit and deer appear more frequently than the rare animal such as panda and crocodile . Public datasets such as DomainNet ( Peng et al. , 2019 ) and and MSCOCO ( Lin et al. , 2014 ) exhibit imbalanced class distribution . Figure 1 illustrates the imbalanced label distributions in the Real domain and Sketch domain from the DomainNet dataset . To address the issue of imbalanced label distributions in domain adaptation , some recent studies ( Wu et al. , 2019 ; Tan et al. , 2020 ; Jiang et al. , 2020 ) try to jointly model the conditional feature distribution shift and label distribution shift ( LDS ) . This problem is referred to as Classimbalanced Domain Adaptation ( CDA ) . Let x and y denote the samples and labels , respectively . p and q separately represent the probability distribution of source domain and target domain . The common assumptions in UDA involve the covariate shift ( i.e. , p ( x ) 6= q ( x ) ) and identical label distribution ( i.e. , p ( y ) = q ( y ) ) . In CDA , however , apart from the covariate shift , both the condi- tional feature shift and label shift exist , i.e. , p ( x|y ) 6= q ( x|y ) , p ( y ) 6= q ( y ) . CDA is a more challenging task than UDA . Recent studies ( Tan et al. , 2020 ) have demonstrated that the mainstream UDA methods will suffer significant performance drop , as the classifier will favor the majority classes . Only a few CDA approaches have been proposed by far . In Tan et al . ( 2020 ) ’ s work , the negative effect of label shift is reduced by exploiting the pseudo labelled target samples via self-training . Jiang et al . ( 2020 ) use an implicit sampling method based on pseudo labels to align the joint distribution between features and labels . However , one critical problem of these methods is that the pseudo labels are likely to suffer from ill-calibrated probabilities ( Guo et al. , 2017 ) , and thus the unreliable pseudo labels will cause error accumulation during the training process , which will largely degrade the model performance . Augmenting training data has been proven as an effective strategy to tackle the issue of biased label distributions in class-imbalance learning ( Chawla et al. , 2002 ; Chou et al. , 2020 ) . In addition to the traditional data augmentation techniques , adversarial training is also capable of generating semantically meaningful synthetic samples that help enhance the robustness of models . However , these approaches only consider a single domain , and they can not be directly applied to solve the CDA problem . In this paper , we propose a pairwise adversarial training ( PAT ) approach that augments training data for class-imbalanced domain adaptation . Unlike conventional adversarial training in which the adversarial samples are obtained from the ` p ball of the original data , we obtain the semantic adversarial samples from the interpolated line of the aligned pair-wise samples from source domain and target domain . Moreover , a class-imbalanced semantic centroid alignment strategy is designed to explicitly align the source and target domains in the feature space . The main contributions of this paper are three-fold . ( 1 ) We propose a novel pairwise adversarial training approach that generates adversarial samples from pairs of samples across the source and target domains , and further exploits these samples to augment training data . ( 2 ) We propose a new optimization algorithm to solve pairwise adversarial training problem . ( 3 ) We conduct extensive evaluations on benchmark datasets , and results show that our approach obtains competing performance compared with state-of-art CDA methods . 2 RELATED WORK . In this section , we briefly introduce three relevant research topics , including unsupervised domain adaptation , class-imbalanced domain adaptation and adversarial training . Unsupervised Domain Adaptation . In recent years , unsupervised domain adaption ( UDA ) has attracted increasing attention . Existing UDA methods could be roughly categorized into two groups , including the discrepancy-based methods and adversarial-based methods . The discrepancy-based methods usually align source and target feature distributions in the embedding space using various statistical distance metrics , such as Maximum Mean Discrepancy ( MMD ) ( Long et al. , 2016 ; 2017 ; Kang et al. , 2019 ) , Correlation Alignment ( CORAL ) ( Sun & Saenko , 2016 ) , and Wasserstein distance ( Lee & Raginsky , 2018 ; Shen et al. , 2018 ; Balaji et al. , 2019 ) . On the other hand , the adversarial-based methods focus on learning domain invariant features via domain adversarial training ( Ganin et al. , 2016 ; Shu et al. , 2018 ; Pei et al. , 2018 ; Saito et al. , 2018 ; Deng et al. , 2019 ; Yu et al. , 2019 ) . Recently , Zhang et al . ( 2019 ) proposed the margin disparity discrepancy ( MDD ) to measure the discrepancy of two domains with generalization bounds . This theory is tailored into an adversarial learning algorithm for domain adaptation . Unlike other adversarial learning based UDA methods that align two domains by confusing a domain discriminator , MDD aligns two domains by minimizing the maximum margin disparity discrepancy of an optimal classifier f and an auxiliary classifier f ′ . The optimization problem of MDD is formulated as : min f , ψ ε ( Ds ) + ηDγ ( Ds , Dt ) , ( 1 ) max f ′ Dγ ( Ds , Dt ) , ( 2 ) where ε is the classification loss on the source domain and Dγ measures the discrepancy of source domain and target domain . Specifically , ε ( Ds ) = E ( xs , ys ) ∼DsL ( f ( ψ ( x s ) ) , ys ) , ( 3 ) Ladv = Dγ ( Ds , Dt ) = Ext∼DtL′ ( f ′ ( ψ ( xt ) ) , f ( ψ ( xt ) ) ) − γExs∼DsL ( f ′ ( ψ ( xs ) ) ) , f ( ψ ( xs ) ) , ( 4 ) L is cross-entropy function , and L′ ( f ′ ( ψ ( xt ) ) , f ( ψ ( xt ) ) ) = log ( 1 − σy′ ( f ′ ( ψ ( xt ) ) ) . y′ is the pseudo label generated from an optimal classifier . MDD is the backbone of our method . Class-imbalanced Domain Adaptation . As a branch of domain adaptation , the class-imbalanced domain adaptation ( CDA ) aims to deal with data with biased class distribution . Tan et al . ( 2020 ) might be the first one to investigate the CDA problem , and they exploited the pseudo labelled target data to reduce the negative effect of label shift . Wu et al . ( 2019 ) proposed the asymmetrically-relaxed distances as replacement of the standard ones under biased label distribution . Jiang et al . ( 2020 ) adopted the implicit sampling strategy to ensure class alignment at the minibatch level . Prabhu et al . ( 2021 ) avoided the use of highly unreliable pseudo labels by assessing the reliability of target data with predictive consistency under random image transformations . Our method refrains from the exploitation of pseudo labeled target data directly in the training process , while reducing the effect of biased label shift by incorporating the semantic adversarial samples into the training process . Adversarial Training . Adversarial training ( AT ) ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) is an effective regularization method for enhancing the robustness and generalization ability of deep learning models . In particular , adversarial samples are incorporated in the model training process , which are intentionally designed to deceive the deep learning model by adding small perturbation on the original data . Furthermore , virtual adversarial training ( VAT ) has been proposed ( Miyato et al. , 2018 ) , which seeks the adversarial direction for regularization without using label information . Both AT and VAT have been employed to tackle the standard UDA problems Shu et al . ( 2018 ) . However , to the best of our knowledge , our work is the first attempt to address the class-imbalanced domain adaptation problem using adversarial training . 3 PROPOSED APPROACH . In this section , we first give the problem definition of CDA , and then present the details of the proposed pairwise adversarial training approach . Finally , we introduce how to integrate the pairwise adversarial training with MDD to address the CDA problem . 3.1 PROBLEM DEFINITION . In class-imbalanced domain adaptation , both the source and target domains suffer from label distribution shift . We are given a source domain Ds = { ( xsi , ysi ) } Ns i=1 with N s labelled samples and a target domain Dt = { xti } Nt i=1 with N t unlabelled samples . Each domain contains K classes , and the class label is denoted as ys ∈ { 0 , 1 , 2 , ... , K − 1 } . Let p and q denote the probability distributions of the source domain and target domain , respectively . We assume that both the covariate shift ( i.e. , p ( x ) 6= q ( x ) ) and label distribution shift ( i.e. , p ( y ) 6= q ( y ) and p ( x|y ) 6= q ( x|y ) ) exist in two domains . Our goal is to train a model that can learn domain invariant features , reduce the gap between source and target domains , and mitigate the label distribution shift . The model typically consists of a feature extractor ψ : X → Z and a classifier f : Z → Y that aims to minimize the target risk . 3.2 PAIRWISE ADVERSARIAL TRAINING ( PAT ) . We investigate how to mitigate the challenging issue of label distribution shift in CDA , as illustrated in Figure 1 . Previous studies ( Tan et al. , 2020 ) found that when the source domain is imbalanced , the model performance on target domains will be significantly dropped , especially when the target domain is also imbalanced . An intuitive solution is to augment the training data in two domains , such that the model training would not be dominated by the majority classes in either domain . However , this task is not trivial , considering the mixed effects of domain gap and imbalanced class distributions . Inspired by adversarial training , we aim to create adversarial samples to augment training data . In adversarial training , the adversarial samples will be exploited to enhance the robustness and generalization ability of model . The loss function of adversarial training is : Lce ( x+ δ∗ , y ; θ ) where δ∗ : = argmax ||δ||p≤ Lce ( x+ δ , y ; θ ) , ( 5 ) where x is the original sample , y is the ground-truth label of x , θ refers to model parameters , and δ is the perturbation added to x . The existing adversarial training methods could not be directly used to tackle the CDA problem for two reasons . First , existing methods simply generate adversarial samples within the neighborhood of the original samples , but they could not mitigate the gap between source and target domains . Second , existing methods treat majority classes and minority classes equally , so they are unable to address the class imbalance issue . In this paper , we propose pairwise adversarial training that generates adversarial samples from the linear interpolation of source and target samples and meanwhile reduces the domain discrepancy . In the following , we will introduce two key components of PAT , including the generation of interpolated adversarial samples and semantic centroid alignment .

Summary: This paper proposes a new method called Pairwise Adversarial Training (PAT) that augments training data for class-imbalanced domain adaptation (CDA). Different from vanilla unsupervised domain adaptation, the label distributions of different distributions are quite different in CDA. The proposed PAT approach mainly consists of two part, centroid alignment (CA) and interpolated adversarial samples (IAS). Experiments on several benchmarks verify the effectiveness of PAT for the CDA problem.