Pairwise Adversarial Training for Unsupervised Class-imbalanced Domain Adaptation

‘ 1 INTRODUCTION . Unsupervised domain adaptation ( UDA ) aims to achieve knowledge transfer from a labeled source domain to an unlabelled target domain . Recent years have witnessed the significant progress of UDA based on deep neural networks ( Pei et al. , 2018 ; Cui et al. , 2020 ; Hu et al. , 2020 ; Liang et al. , 2020 ; Na et al. , 2021 ) . Most of existing UDA methods assume that only covariate shift occurs in the source domain and target domain , while the label distributions in two domains are identical . However , this assumption may not hold in real-world applications . For instance , in wild-life pictures , the commonly seen animals such as rabbit and deer appear more frequently than the rare animal such as panda and crocodile . Public datasets such as DomainNet ( Peng et al. , 2019 ) and and MSCOCO ( Lin et al. , 2014 ) exhibit imbalanced class distribution . Figure 1 illustrates the imbalanced label distributions in the Real domain and Sketch domain from the DomainNet dataset . To address the issue of imbalanced label distributions in domain adaptation , some recent studies ( Wu et al. , 2019 ; Tan et al. , 2020 ; Jiang et al. , 2020 ) try to jointly model the conditional feature distribution shift and label distribution shift ( LDS ) . This problem is referred to as Classimbalanced Domain Adaptation ( CDA ) . Let x and y denote the samples and labels , respectively . p and q separately represent the probability distribution of source domain and target domain . The common assumptions in UDA involve the covariate shift ( i.e. , p ( x ) 6= q ( x ) ) and identical label distribution ( i.e. , p ( y ) = q ( y ) ) . In CDA , however , apart from the covariate shift , both the condi- tional feature shift and label shift exist , i.e. , p ( x|y ) 6= q ( x|y ) , p ( y ) 6= q ( y ) . CDA is a more challenging task than UDA . Recent studies ( Tan et al. , 2020 ) have demonstrated that the mainstream UDA methods will suffer significant performance drop , as the classifier will favor the majority classes . Only a few CDA approaches have been proposed by far . In Tan et al . ( 2020 ) ’ s work , the negative effect of label shift is reduced by exploiting the pseudo labelled target samples via self-training . Jiang et al . ( 2020 ) use an implicit sampling method based on pseudo labels to align the joint distribution between features and labels . However , one critical problem of these methods is that the pseudo labels are likely to suffer from ill-calibrated probabilities ( Guo et al. , 2017 ) , and thus the unreliable pseudo labels will cause error accumulation during the training process , which will largely degrade the model performance . Augmenting training data has been proven as an effective strategy to tackle the issue of biased label distributions in class-imbalance learning ( Chawla et al. , 2002 ; Chou et al. , 2020 ) . In addition to the traditional data augmentation techniques , adversarial training is also capable of generating semantically meaningful synthetic samples that help enhance the robustness of models . However , these approaches only consider a single domain , and they can not be directly applied to solve the CDA problem . In this paper , we propose a pairwise adversarial training ( PAT ) approach that augments training data for class-imbalanced domain adaptation . Unlike conventional adversarial training in which the adversarial samples are obtained from the ` p ball of the original data , we obtain the semantic adversarial samples from the interpolated line of the aligned pair-wise samples from source domain and target domain . Moreover , a class-imbalanced semantic centroid alignment strategy is designed to explicitly align the source and target domains in the feature space . The main contributions of this paper are three-fold . ( 1 ) We propose a novel pairwise adversarial training approach that generates adversarial samples from pairs of samples across the source and target domains , and further exploits these samples to augment training data . ( 2 ) We propose a new optimization algorithm to solve pairwise adversarial training problem . ( 3 ) We conduct extensive evaluations on benchmark datasets , and results show that our approach obtains competing performance compared with state-of-art CDA methods . 2 RELATED WORK . In this section , we briefly introduce three relevant research topics , including unsupervised domain adaptation , class-imbalanced domain adaptation and adversarial training . Unsupervised Domain Adaptation . In recent years , unsupervised domain adaption ( UDA ) has attracted increasing attention . Existing UDA methods could be roughly categorized into two groups , including the discrepancy-based methods and adversarial-based methods . The discrepancy-based methods usually align source and target feature distributions in the embedding space using various statistical distance metrics , such as Maximum Mean Discrepancy ( MMD ) ( Long et al. , 2016 ; 2017 ; Kang et al. , 2019 ) , Correlation Alignment ( CORAL ) ( Sun & Saenko , 2016 ) , and Wasserstein distance ( Lee & Raginsky , 2018 ; Shen et al. , 2018 ; Balaji et al. , 2019 ) . On the other hand , the adversarial-based methods focus on learning domain invariant features via domain adversarial training ( Ganin et al. , 2016 ; Shu et al. , 2018 ; Pei et al. , 2018 ; Saito et al. , 2018 ; Deng et al. , 2019 ; Yu et al. , 2019 ) . Recently , Zhang et al . ( 2019 ) proposed the margin disparity discrepancy ( MDD ) to measure the discrepancy of two domains with generalization bounds . This theory is tailored into an adversarial learning algorithm for domain adaptation . Unlike other adversarial learning based UDA methods that align two domains by confusing a domain discriminator , MDD aligns two domains by minimizing the maximum margin disparity discrepancy of an optimal classifier f and an auxiliary classifier f ′ . The optimization problem of MDD is formulated as : min f , ψ ε ( Ds ) + ηDγ ( Ds , Dt ) , ( 1 ) max f ′ Dγ ( Ds , Dt ) , ( 2 ) where ε is the classification loss on the source domain and Dγ measures the discrepancy of source domain and target domain . Specifically , ε ( Ds ) = E ( xs , ys ) ∼DsL ( f ( ψ ( x s ) ) , ys ) , ( 3 ) Ladv = Dγ ( Ds , Dt ) = Ext∼DtL′ ( f ′ ( ψ ( xt ) ) , f ( ψ ( xt ) ) ) − γExs∼DsL ( f ′ ( ψ ( xs ) ) ) , f ( ψ ( xs ) ) , ( 4 ) L is cross-entropy function , and L′ ( f ′ ( ψ ( xt ) ) , f ( ψ ( xt ) ) ) = log ( 1 − σy′ ( f ′ ( ψ ( xt ) ) ) . y′ is the pseudo label generated from an optimal classifier . MDD is the backbone of our method . Class-imbalanced Domain Adaptation . As a branch of domain adaptation , the class-imbalanced domain adaptation ( CDA ) aims to deal with data with biased class distribution . Tan et al . ( 2020 ) might be the first one to investigate the CDA problem , and they exploited the pseudo labelled target data to reduce the negative effect of label shift . Wu et al . ( 2019 ) proposed the asymmetrically-relaxed distances as replacement of the standard ones under biased label distribution . Jiang et al . ( 2020 ) adopted the implicit sampling strategy to ensure class alignment at the minibatch level . Prabhu et al . ( 2021 ) avoided the use of highly unreliable pseudo labels by assessing the reliability of target data with predictive consistency under random image transformations . Our method refrains from the exploitation of pseudo labeled target data directly in the training process , while reducing the effect of biased label shift by incorporating the semantic adversarial samples into the training process . Adversarial Training . Adversarial training ( AT ) ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) is an effective regularization method for enhancing the robustness and generalization ability of deep learning models . In particular , adversarial samples are incorporated in the model training process , which are intentionally designed to deceive the deep learning model by adding small perturbation on the original data . Furthermore , virtual adversarial training ( VAT ) has been proposed ( Miyato et al. , 2018 ) , which seeks the adversarial direction for regularization without using label information . Both AT and VAT have been employed to tackle the standard UDA problems Shu et al . ( 2018 ) . However , to the best of our knowledge , our work is the first attempt to address the class-imbalanced domain adaptation problem using adversarial training . 3 PROPOSED APPROACH . In this section , we first give the problem definition of CDA , and then present the details of the proposed pairwise adversarial training approach . Finally , we introduce how to integrate the pairwise adversarial training with MDD to address the CDA problem . 3.1 PROBLEM DEFINITION . In class-imbalanced domain adaptation , both the source and target domains suffer from label distribution shift . We are given a source domain Ds = { ( xsi , ysi ) } Ns i=1 with N s labelled samples and a target domain Dt = { xti } Nt i=1 with N t unlabelled samples . Each domain contains K classes , and the class label is denoted as ys ∈ { 0 , 1 , 2 , ... , K − 1 } . Let p and q denote the probability distributions of the source domain and target domain , respectively . We assume that both the covariate shift ( i.e. , p ( x ) 6= q ( x ) ) and label distribution shift ( i.e. , p ( y ) 6= q ( y ) and p ( x|y ) 6= q ( x|y ) ) exist in two domains . Our goal is to train a model that can learn domain invariant features , reduce the gap between source and target domains , and mitigate the label distribution shift . The model typically consists of a feature extractor ψ : X → Z and a classifier f : Z → Y that aims to minimize the target risk . 3.2 PAIRWISE ADVERSARIAL TRAINING ( PAT ) . We investigate how to mitigate the challenging issue of label distribution shift in CDA , as illustrated in Figure 1 . Previous studies ( Tan et al. , 2020 ) found that when the source domain is imbalanced , the model performance on target domains will be significantly dropped , especially when the target domain is also imbalanced . An intuitive solution is to augment the training data in two domains , such that the model training would not be dominated by the majority classes in either domain . However , this task is not trivial , considering the mixed effects of domain gap and imbalanced class distributions . Inspired by adversarial training , we aim to create adversarial samples to augment training data . In adversarial training , the adversarial samples will be exploited to enhance the robustness and generalization ability of model . The loss function of adversarial training is : Lce ( x+ δ∗ , y ; θ ) where δ∗ : = argmax ||δ||p≤ Lce ( x+ δ , y ; θ ) , ( 5 ) where x is the original sample , y is the ground-truth label of x , θ refers to model parameters , and δ is the perturbation added to x . The existing adversarial training methods could not be directly used to tackle the CDA problem for two reasons . First , existing methods simply generate adversarial samples within the neighborhood of the original samples , but they could not mitigate the gap between source and target domains . Second , existing methods treat majority classes and minority classes equally , so they are unable to address the class imbalance issue . In this paper , we propose pairwise adversarial training that generates adversarial samples from the linear interpolation of source and target samples and meanwhile reduces the domain discrepancy . In the following , we will introduce two key components of PAT , including the generation of interpolated adversarial samples and semantic centroid alignment .

Summary: This work proposes a method for solving the UDA problem with imbalanced class, which is a sub-problem of UDA. The challenge lies in how to handle the difficulties introduced by imbalanced classes. To this end, this work proposes a new data augmentation strategy, that is taking the interpolation of two samples from the same class but from different domains as the augmented samples. The traditional MMD loss and a class centroid distance based loss are also imposed for the model training. Experiments on multiple benchmark datasets are conducted.