Year: 2021

Abstract:
Unsupervised domain adaptation (UDA) has become an appealing approach for knowledge transfer from a labeled source domain to an unlabeled target domain. However, when the classes in source and target domains are imbalanced, most existing UDA methods experience significant performance drop, as the decision boundary usually favors the majority classes. Some recent class-imbalanced domain adaptation (CDA) methods aim to tackle the challenge of biased label distribution by exploiting pseudo-labeled target data during training process. However, these methods still suffer from the issues with unreliable pseudo labels and error accumulation during training. In this paper, we propose a pairwise adversarial training approach for class-imbalanced domain adaptation. Unlike conventional adversarial training in which the adversarial samples are obtained from the `p ball of the original data, we generate adversarial samples from the interpolated line of the aligned pairwise samples from source domain and target domain. Experimental results and ablation studies show that our method achieves considerable improvements on benchmarks compared with the state-of-art CDA methods.

text: ‘

heading: 1 INTRODUCTION
text: Unsupervised domain adaptation (UDA) aims to achieve knowledge transfer from a labeled source domain to an unlabelled target domain. Recent years have witnessed the significant progress of UDA based on deep neural networks (Pei et al., 2018; Cui et al., 2020; Hu et al., 2020; Liang et al., 2020; Na et al., 2021). Most of existing UDA methods assume that only covariate shift occurs in the source domain and target domain, while the label distributions in two domains are identical. However, this assumption may not hold in real-world applications. For instance, in wild-life pictures, the commonly seen animals such as rabbit and deer appear more frequently than the rare animal such as panda and crocodile. Public datasets such as DomainNet (Peng et al., 2019) and and MSCOCO (Lin et al., 2014) exhibit imbalanced class distribution. Figure 1 illustrates the imbalanced label distributions in the Real domain and Sketch domain from the DomainNet dataset.
To address the issue of imbalanced label distributions in domain adaptation, some recent studies (Wu et al., 2019; Tan et al., 2020; Jiang et al., 2020) try to jointly model the conditional feature distribution shift and label distribution shift (LDS). This problem is referred to as Classimbalanced Domain Adaptation (CDA). Let x and y denote the samples and labels, respectively. p and q separately represent the probability distribution of source domain and target domain. The common assumptions in UDA involve the covariate shift (i.e., p(x) 6= q(x)) and identical label distribution (i.e., p(y) = q(y)). In CDA, however, apart from the covariate shift, both the condi-
tional feature shift and label shift exist, i.e., p(x|y) 6= q(x|y), p(y) 6= q(y). CDA is a more challenging task than UDA. Recent studies (Tan et al., 2020) have demonstrated that the mainstream UDA methods will suffer significant performance drop, as the classifier will favor
the majority classes. Only a few CDA approaches have been proposed by far. In Tan et al. (2020)’s work, the negative effect of label shift is reduced by exploiting the pseudo labelled target samples via self-training. Jiang et al. (2020) use an implicit sampling method based on pseudo labels to align the joint distribution between features and labels. However, one critical problem of these methods is that the pseudo labels are likely to suffer from ill-calibrated probabilities (Guo et al., 2017), and thus the unreliable pseudo labels will cause error accumulation during the training process, which will largely degrade the model performance.
Augmenting training data has been proven as an effective strategy to tackle the issue of biased label distributions in class-imbalance learning (Chawla et al., 2002; Chou et al., 2020). In addition to the traditional data augmentation techniques, adversarial training is also capable of generating semantically meaningful synthetic samples that help enhance the robustness of models. However, these approaches only consider a single domain, and they cannot be directly applied to solve the CDA problem. In this paper, we propose a pairwise adversarial training (PAT) approach that augments training data for class-imbalanced domain adaptation. Unlike conventional adversarial training in which the adversarial samples are obtained from the `p ball of the original data, we obtain the semantic adversarial samples from the interpolated line of the aligned pair-wise samples from source domain and target domain. Moreover, a class-imbalanced semantic centroid alignment strategy is designed to explicitly align the source and target domains in the feature space.
The main contributions of this paper are three-fold. (1) We propose a novel pairwise adversarial training approach that generates adversarial samples from pairs of samples across the source and target domains, and further exploits these samples to augment training data. (2) We propose a new optimization algorithm to solve pairwise adversarial training problem. (3) We conduct extensive evaluations on benchmark datasets, and results show that our approach obtains competing performance compared with state-of-art CDA methods.

heading: 2 RELATED WORK
text: In this section, we briefly introduce three relevant research topics, including unsupervised domain adaptation, class-imbalanced domain adaptation and adversarial training.
Unsupervised Domain Adaptation. In recent years, unsupervised domain adaption (UDA) has attracted increasing attention. Existing UDA methods could be roughly categorized into two groups, including the discrepancy-based methods and adversarial-based methods. The discrepancy-based methods usually align source and target feature distributions in the embedding space using various statistical distance metrics, such as Maximum Mean Discrepancy (MMD) (Long et al., 2016; 2017; Kang et al., 2019), Correlation Alignment (CORAL) (Sun & Saenko, 2016), and Wasserstein distance (Lee & Raginsky, 2018; Shen et al., 2018; Balaji et al., 2019). On the other hand, the adversarial-based methods focus on learning domain invariant features via domain adversarial training (Ganin et al., 2016; Shu et al., 2018; Pei et al., 2018; Saito et al., 2018; Deng et al., 2019; Yu et al., 2019). Recently, Zhang et al. (2019) proposed the margin disparity discrepancy (MDD) to measure the discrepancy of two domains with generalization bounds. This theory is tailored into an adversarial learning algorithm for domain adaptation. Unlike other adversarial learning based UDA methods that align two domains by confusing a domain discriminator, MDD aligns two domains by minimizing the maximum margin disparity discrepancy of an optimal classifier f and an auxiliary classifier f ′. The optimization problem of MDD is formulated as:
min f,ψ
ε(Ds) + ηDγ(Ds,Dt), (1)
max f ′ Dγ(Ds,Dt), (2)
where ε is the classification loss on the source domain and Dγ measures the discrepancy of source domain and target domain. Specifically,
ε(Ds) = E(xs,ys)∼DsL(f(ψ(x s)), ys), (3)
Ladv = Dγ(Ds,Dt) = Ext∼DtL′(f ′(ψ(xt)), f(ψ(xt)))− γExs∼DsL(f ′(ψ(xs))), f(ψ(xs)), (4)
L is cross-entropy function, and L′(f ′(ψ(xt)), f(ψ(xt))) = log(1 − σy′(f ′(ψ(xt))). y′ is the pseudo label generated from an optimal classifier. MDD is the backbone of our method.
Class-imbalanced Domain Adaptation. As a branch of domain adaptation, the class-imbalanced domain adaptation (CDA) aims to deal with data with biased class distribution. Tan et al. (2020) might be the first one to investigate the CDA problem, and they exploited the pseudo labelled target data to reduce the negative effect of label shift. Wu et al. (2019) proposed the asymmetrically-relaxed distances as replacement of the standard ones under biased label distribution. Jiang et al. (2020) adopted the implicit sampling strategy to ensure class alignment at the minibatch level. Prabhu et al. (2021) avoided the use of highly unreliable pseudo labels by assessing the reliability of target data with predictive consistency under random image transformations. Our method refrains from the exploitation of pseudo labeled target data directly in the training process, while reducing the effect of biased label shift by incorporating the semantic adversarial samples into the training process.
Adversarial Training. Adversarial training (AT) (Szegedy et al., 2014; Goodfellow et al., 2015) is an effective regularization method for enhancing the robustness and generalization ability of deep learning models. In particular, adversarial samples are incorporated in the model training process, which are intentionally designed to deceive the deep learning model by adding small perturbation on the original data. Furthermore, virtual adversarial training (VAT) has been proposed (Miyato et al., 2018), which seeks the adversarial direction for regularization without using label information. Both AT and VAT have been employed to tackle the standard UDA problems Shu et al. (2018). However, to the best of our knowledge, our work is the first attempt to address the class-imbalanced domain adaptation problem using adversarial training.

heading: 3 PROPOSED APPROACH
text: In this section, we first give the problem definition of CDA, and then present the details of the proposed pairwise adversarial training approach. Finally, we introduce how to integrate the pairwise adversarial training with MDD to address the CDA problem.

heading: 3.1 PROBLEM DEFINITION
text: In class-imbalanced domain adaptation, both the source and target domains suffer from label distribution shift. We are given a source domain Ds = {(xsi , ysi )} Ns i=1 with N
s labelled samples and a target domain Dt = {xti} Nt i=1 with N
t unlabelled samples. Each domain contains K classes, and the class label is denoted as ys ∈ {0, 1, 2, ...,K − 1}. Let p and q denote the probability distributions of the source domain and target domain, respectively. We assume that both the covariate shift (i.e., p(x) 6= q(x)) and label distribution shift (i.e., p(y) 6= q(y) and p(x|y) 6= q(x|y)) exist in two domains. Our goal is to train a model that can learn domain invariant features, reduce the gap between source and target domains, and mitigate the label distribution shift. The model typically consists of a feature extractor ψ : X → Z and a classifier f : Z → Y that aims to minimize the target risk.

heading: 3.2 PAIRWISE ADVERSARIAL TRAINING (PAT)
text: We investigate how to mitigate the challenging issue of label distribution shift in CDA, as illustrated in Figure 1. Previous studies (Tan et al., 2020) found that when the source domain is imbalanced, the model performance on target domains will be significantly dropped, especially when the target domain is also imbalanced. An intuitive solution is to augment the training data in two domains, such that the model training would not be dominated by the majority classes in either domain. However, this task is not trivial, considering the mixed effects of domain gap and imbalanced class distributions. Inspired by adversarial training, we aim to create adversarial samples to augment training data. In adversarial training, the adversarial samples will be exploited to enhance the robustness and generalization ability of model. The loss function of adversarial training is:
Lce(x+ δ∗, y; θ) where δ∗ := argmax
||δ||p≤ Lce(x+ δ, y; θ), (5)
where x is the original sample, y is the ground-truth label of x, θ refers to model parameters, and δ is the perturbation added to x.
The existing adversarial training methods could not be directly used to tackle the CDA problem for two reasons. First, existing methods simply generate adversarial samples within the neighborhood
of the original samples, but they could not mitigate the gap between source and target domains. Second, existing methods treat majority classes and minority classes equally, so they are unable to address the class imbalance issue. In this paper, we propose pairwise adversarial training that generates adversarial samples from the linear interpolation of source and target samples and meanwhile reduces the domain discrepancy. In the following, we will introduce two key components of PAT, including the generation of interpolated adversarial samples and semantic centroid alignment.

heading: 3.2.1 INTERPOLATED ADVERSARIAL SAMPLE GENERATION
text: As shown in Figure 2, we will generate adversarial samples on the interpolated line from a source sample to a target sample of the same class. The interpolated adversarial samples (IAS) should have the same semantic meaning as their corresponding source/target sample pair, although they are not perfectly aligned with the original samples. We explicitly address the data imbalance issue in the source domain by exploiting interpolated adversarial samples and aligning the source and target domains. As a result, the generalization ability of the unbiased model will be improved and the data imbalance issue in the target domain could be implicitly addressed. For the kth class, the interpolated adversarial samples can be defined as:
X advk = {xadvk |xadvi = xsi +λ(xti−xsi ), λ ∈ [0, 1)C , ysi = ŷti = k}, (6)
where xi ∈ RC×H×W , λ is the coefficient measuring the contributing weights of the source sample and target sample. ŷti is the pseudo label of the target sample which is generated from the backbone domain adaptation model (e.g., the optimal classifier f in MDD). It is used for match of the corresponding source sample. In our work, though the pseudo labels might be incorrect, our PAT method will not suffer from the potential error accumulation issue. First, we choose the target samples with high confidence. Second, even if the pseudo label is incorrect, there is still a chance that the interpolated adversarial sample will be generated within the boundary as expected. Specifically, the misclassified target samples often appear near the decision boundary. So, even if the target sample is from a different class, the adversarial sample generated from the pair of source sample and target sample may not violate the decision boundary. Finally, the adversarial samples in our approach are generated dynamically, and the adverse effect of bad adversarial samples could be mitigated.
Then, the generation of interpolated adversarial samples could be achieved by solving the following optimization problem:
LIAS := D(x̂adv, y; θ) where x̂adv = argmax
xadv∈Xadv D′(xadv, y; θ). (7)
In Eq. (7), the outer minimization problem involves a standard cross-entropy loss function, i.e.,
D(x̂adv, y; θ) = LCE(x̂adv, y; θ) = − log(σy(f(ψ(x̂adv)))), (8) where ψ : X → Z denotes a feature extractor, f : Z → Y denotes a classifier, and σ is the softmax function. For the inner maximization problem, we use a modified cross-entropy function proposed by Goodfellow et al. (2014). The modified loss function can alleviate the problem of gradient exploding or vanishing when the entropy loss is maximized. The loss function of the inner maximization problem is written as:
D′(xadv, y; θ) = L′CE(xadv, y; θ) = log(1− σy(f(ψ(xadv))). (9)
Several optimization algorithms, such as the fast gradient sign method (Goodfellow et al., 2015) and projected gradient descent (Madry et al., 2018), have been commonly used for adversarial training. However, these algorithms aim to obtain the adversarial samples in the `∞ ball of the original samples, which cannot be directly applied to solving our problem. In our case, the interpolated adversarial samples are confined on the interpolated line of source data and target data. We propose a new optimization algorithm to solve the inner maximization optimization in Eq. (7). We initialize the interpolated adversarial samples with random λ and update them by back propagation in each iteration. The main procedures of our algorithm are summarized in Algorithm 1.
Algorithm 1 Solving the maximization problem in PAT Input: Source samples from a mini-batch {(xsi , ysi )} Bs i=1, target samples and their pseudo labels
{(xti, ŷti)} Nt i=1, probability threshold of each class {Pk}Kk=1
Output: Interpolated adversarial samples {x̂advi } Nadv i=1
for each source sample xsi in the mini-batch do if rand() > Pk(k = ysi ) then
Choose one target sample xti with pseudo label equals to k Initialize xadv with random λ using Eq. (6) repeat
Calculates the gradient of λ with loss function in Eq. (7). gλ = α∇λD′(xadv, y; θ) Update λ with the gradient, λ← λ+ gλ Clip the λ between 0 and 1, λ← λ · clip(0, 1) Update adversarial sample xadv with new λ using Eq. (6).
until the optimization converges end if
end for
Generating interpolated samples has been explored in literature, such as mix-up (Chou et al., 2020) and its variants. However, mix-up based methods and our method have significant differences. First, mix-up based methods focus on the single-domain classification or standard domain-adaptation problems, while our method focuses on the data imbalance problem in domain adaptation. Second, the ideas for generating samples and labels in mix-up and our method are different. Mix-up creates virtual samples and their labels from two randomly chosen samples. While in our method, an adversarial sample is generated using a pair of samples from two domains with the same label. Third, mix-up needs to manually define a hyper-parameter to control the strength of interpolation. However, in our method, the parameter for controlling the strength of interpolation is adaptively updated along with adversarial training.

heading: 3.2.2 CLASS-IMBALANCED SEMANTIC CENTROID ALIGNMENT
text: Without careful control of the generation mechanism, the interpolated adversarial samples may not alleviate the issue of imbalanced class distribution. Moreover, although the interpolated adversarial samples bridge the source and target domains to some extent, the discrepancy between source and target domains is not explicitly reduced. To address these issues, we propose to explicitly align the source and target domains with imbalanced class distributions using two strategies.
First, we propose a strategy to guide the generation of interpolated adversarial samples. For training samples in each mini-batch from the source domain, they should not have the equal opportunity to generate interpolated adversarial samples. Since the decision boundary usually favors the majority classes, the probability of generating adversarial samples for minority classes should be larger than that for majority classes. For the k-th class, we set a probability threshold Pk as follows:
Pk = nk
nmax + , (10)
where nk is the number of the samples from the kth class. nmax = maxk{nk}Kk=1, and is the bias. For a specific class, if a random number r ∈ [0, 1) is larger than the corresponding threshold, the adversarial samples will be generated, as shown in Algorithm 1. We also adopt class-balanced sampling on the source data to alleviate the biased occurrence of the majority classes. Specifically, each class will be selected with an equal chance, in order to reduce the model prediction bias towards the majority classes.
Second, we incorporate the moving average centroid alignment (Xie et al., 2018) to align the conditional feature distributions of source and target domains by explicitly matching the centroids of two domains. As illustrated in Figure 3, without centroid alignment, the adversarial samples may be generated from a pair of samples in which one of the samples is misaligned to other classes, thus making the embedding of adversarial samples fall out of the decision boundary. With centroid alignment, we can eliminate the occurrence of such out-of-bound adversarial samples, and the interpolated adversarial samples could provide meaningful support for the minority class in the target domain. The loss function of moving average centroid alignment is defined as LCA = ∑K k=1 dist(Csk, Ctk), where Csk and Ctk denote the centroids of the kth class in the source domain and target domain, respectively. dist() can be implemented by the Euclidean distance or cosine distance.

heading: 3.3 PAT FOR CLASS IMBALANCED DOMAIN ADAPTATION
text: The proposed PAT approach could be integrated with many existing domain adaptation frameworks to enhance their performance on class-imbalanced data. In this paper, we adopt MDD (Zhang et al., 2019) as an example backbone model and showcase how to integrated PAT with it.
The MDD framework consists of a feature extractor G, an optimal classifier F , and an auxiliary classifier F ′. The loss function of MDD is introduced in Eq. (1) and Eq. (2). Adversarial samples designed to maximize the modified cross-entropy functionL′CE are generated from the pair of source data and target data. Finally, the overall loss function of our framework is:
L = LMDD + αLIAS + βLCA, (11)
where α and β are two trade-off parameters.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Office-31 is a widely used benchmark image dataset for domain adaptation (Saenko et al., 2010). It contains 31 classes in three domains: Amazon (A), Dslr (D) and Webcam (W). The standard Office-31 doesn’t exhibit obvious label distribution shift (LDS), so a new imbalanced Office-31 is created by sampling from standard ones as suggested by Tan et al. (2020). The distribution conforms to Paredo distribution (Reed, 2001) and follows the Reversely-unbalanced Source and Unbalanced Target (RS-UT) protocol. Both the source domain and target domain have shifted label distribu-
tions, and the label distribution of source domain is a reversed version of that of target domain. Office-Home is a large benchmark dataset containing 65 classes of objects commonly found in office and home scenarios (Venkateswara et al., 2017). It has four domains: Real-World (Rw), Clipart
(Cl), Product (Pr) and Art (Ar). In our experiments, we use the existing imbalanced Office-Home with RS-UT distributions generated in Tan et al. (2020) to train and test our approach. Since there are very limited samples in the art (Ar) domain, we only conduct domain adaptation tasks on the other three domains. DomainNet is a large-scale benchmark dataset for domain adaptation (Peng et al., 2019). Since there are mislabeled samples in some classes and domains, we follow Tan et al. (2020) and adopt only 40 common classes from four domains: Real (R), Clipart (C), Painting (P), and Sketch (S). Different from Office-31 and Office-Home, the selected samples in DomainNet already exhibit obvious label distribution shift in the source domain and target domain. So there is no need to sample this dataset again. Figure 5 illustrates the label distributions of imbalanced Office31 and imbalanced Office-Home datasets.
We use PyTorch to implement our approach. We train our model with the mini-batch SGD, a Nesterov momentum of 0.9, and a weight decay of 0.0005. The learning rate of classifiers is 10 times larger than that of feature extractor, and all the learning rates are adjusted by every iteration. In order to obtain the interpolated adversarial sample from a pair of source sample and target sample from the same class, we utilize a memory pool to store the pseudo labels of all the target data. The pseudo labels are updated in every iteration. Note that we utilize a class-balanced sampler on the source data, which can be referred to as N-way (number of classes per batch) and K-shot (number of examples per class). The coefficient α is set to 0.5 and β is set to 0.05 for all the experiments. Following existing work on CDA, we adopt the per-class mean accuracy as our evaluation metric. All the experiments are implemented on Nvidia RTX A5000 platform.

heading: 4.2 COMPARISON WITH STATE-OF-ART CDA METHODS
text: We compare our approach with several state-of-art methods for class imbalanced domain adaptation. The baseline methods can be divided into two categories. The first category of methods are specifically designed to solve the CDA problem, including Sentry (Prabhu et al., 2021), MDD+implicit Jiang et al. (2020), COAL (Tan et al., 2020), and F-DANN (Wu et al., 2019). The second category of methods aim to solve the standard unsupervised domain adaptation problem, including the InstaPBM (Li et al., 2020) and BSP (Chen et al., 2019).
Table 1 shows the per-class average accuracy and overall average accuracy of our approach and baselines on the imbalanced Office-Home dataset. The results of the baseline are cited from Prabhu et al. (2021). Our proposed model achieves the best results in 4 out of 6 tasks. The accuracies of our approach in these 4 tasks are more than 3% higher than the second-best results. In addition, our model obtains the second-best results in the Rw→CL and Pr→Cl tasks. Overall, our method achieves the highest average accuracy (i.e., 66.07%) among all the compared methods. We also investigate the class-specific accuracy of MDD (Zhang et al., 2019) and our method on minority classes, in order to understand how well our method addresses the data imbalance issue. Table 3 shows that our PAT method can significantly boost the performance on the minority classes from the imbalanced Office-Home dataset.
Table 2 shows the per-class average accuracies of our approach and baselines on the DomainNet dataset. Our approach achieves the best results in 4 tasks and second-best results in 8 tasks. In R→S, P→S and S→P, our model can achieve 76.08%, 75.46%, which are 1.65% and 1.48% higher than the second-best results, respectively. Though in some tasks our results are lower than Sentry (Prabhu et al., 2021), the overall average accuracy of our approach is comparable to that of Sentry.
We manually sample the standard Office-31 dataset and construct the imbalanced Office-31 dataset, in which the label distribution conforms to the Paredo distribution (Reed, 2001). We compare our model with current state-of-art methods that focus on the CDA problem. Among all the 6 tasks, our model achieves best results in 4 tasks. In these tasks, our results are 3.82%, 2.41%, 3.28% and 3.77% higher than the second best results. In the D→W and W→D tasks, the performance of our approach are only 0.12% and 0.32% less than the best results.

heading: 4.3 ABLATION STUDIES
text: We further investigate the performance of our approach from several aspects. First, we evaluate the contribution of each component in the loss function. Second, we evaluate the sensitivity of the model to the change of the hyper-parameters. Third, we evaluate the performance of our method on the standard unsupervised domain adaptation problem. We have also evaluated: (1) the effect of the number of the iteration in the inner maximization of PAT; and (2) the integration of PAT and other domain adaptation methods including CDAN, CDANE and Sentry, and reported the results in the Appendix due to space limit.
The proposed pairwise adversarial training approach includes two major components, i.e., interpolated adversarial samples (IAS) and centroid alignment (CA). We choose three tasks from each of the three datasets separately. Table 5 shows the performance of our backbone model MDD, Ours w/o IAS, Ours w/o CA, and our full model on the imbalanced Office-Home dataset. Detailed results on the other two datasets are provided in the Appendix. Ours w/o IAS and Ours w/o CA consistently achieve better performance than the baseline model MDD, which validates the effectiveness of both methods in dealing with the biased label distribution shift. Our full model further improves the classification accuracy, demonstrating the complementary roles of pairwise adversarial samples and centroid alignment in our approach.
Furthermore, we evaluate the sensitivity of our model to the change of two hyperparameters α and β . In particular, we first set β to 0.05 and choose α from [0, 2.5]. Then, α is set to 0.5, and β is chosen from [0, 0.25]. Figure 6 shows the per-class average accuracy of our model when varying the hyper-parameter values. It shows that our model is not very sensitive to the settings of hyper-parameters in a relatively wide range.
Finally, we evaluate our method in the standard UDA setting, by comparing the results against the baseline MDD (Zhang et al., 2019) on the standard Office-Home dataset. The results from Table 6 shows that the classification accuracy in all the four tasks are improved after incorporating our interpolation method, which proves the effectiveness of our method in unsupervised domain adaptation problem.

heading: 5 CONCLUSION
text: In this paper, we propose a pairwise adversarial training approach to tackle the class-imbalanced unsupervised domain adaptation (CDA) problem. Our approach generates interpolated adversarial samples across source and target domains. In order to alleviate the biased label distribution shift issue, we use the interpolated adversarial samples to augment the training data (especially the minority classes) and meanwhile adopt the centroid alignment strategy to explicitly align source and target domains. Experimental results on three CDA benchmark datasets show that, our approach yields competing performance compared with state-of-art CDA methods.

heading: A APPENDIX
text: We evaluate the effect of the number of iterations of the inner maximization in pairwise adversarial training. We choose Rw→Pr from Office-Home, R→C from DomainNet and A→W from Office-31 as our evaluation tasks. The results are illustrated in Figure 7. It shows that the performance of our approach will generally increase with more iterations. At some points the performance experiences the slight fluctuation. Though performance reaches the maximum value at iteration of 12, it is relatively time-consuming in the training process.
Table 7 and Table 8 show the per-class average accuracy of MDD, Ours w/o IAS, Ours w/o CA, and our full model on three tasks from the DomainNet dataset and Office-31 dataset, respectively. The results demonstrate the effectiveness of two major components (i.e., interpolated adversarial samples and centroid alignment) in our approach.
Furthermore, as our PAT method is an independent module, which could be combined with other domain alignment techniques such as CDAN and Sentry. We integrate PAT with CDAN and CDANE, and report the experimental results on the RS-UT Office-Home dataset in Table 9. Results show that our PAT module could significantly improve the performance: 54.07%→60.38% for CDAN, and
55.70%→62.61% for CDANE. The results in Table 10 further validate that our PAT method could boost the performance of the state-of-the-art CDA method Sentry.

Review summaries:
This paper proposes a pairwise adversarial training approach for class-imbalanced domain adaptation. Specifically, the adversarial samples are generated from the interpolated line of the aligned pairwise source domain samples and target domain samples. The generated adversarial data can augment the training data and help enhance the robustness of models.

This work proposes a method for solving the UDA problem with imbalanced class, which is a sub-problem of UDA. The challenge lies in how to handle the difficulties introduced by imbalanced classes. To this end, this work proposes a new data augmentation strategy, that is taking the interpolation of two samples from the same class but from different domains as the augmented samples. The traditional MMD loss and a class centroid distance based loss are also imposed for the model training. Experiments on multiple benchmark datasets are conducted. 

This paper proposes a new method called Pairwise Adversarial Training (PAT) that augments training data for class-imbalanced domain adaptation (CDA). Different from vanilla unsupervised domain adaptation, the label distributions of different distributions are quite different in CDA. The proposed PAT approach mainly consists of two part, centroid alignment (CA) and interpolated adversarial samples (IAS). Experiments on several benchmarks verify the effectiveness of PAT for the CDA problem.