Pairwise Adversarial Training for Unsupervised Class-imbalanced Domain Adaptation
Authors: Weili Shi
Ronghang Zhu
Sheng Li
Venue: ICLR 2022 Submitted (2022)
Keywords: domain adaptation adversarial training imbalanced class distribution

Abstract: Unsupervised domain adaptation (UDA) has become an appealing approach for knowledge transfer from a labeled source domain to an unlabeled target domain. However, when the classes in source and target domains are imbalanced, most existing UDA methods experience significant performance drop, as the decision boundary usually favors the majority classes. Some recent class-imbalanced domain adaptation (CDA) methods aim to tackle the challenge of biased label distribution by exploiting pseudo-labeled target data during training process. However, these methods may be challenged with the problem of unreliable pseudo labels and error accumulation during training. In this paper, we propose a pairwise adversarial training approach to augment training data for unsupervised class-imbalanced domain adaptation. Unlike conventional adversarial training in which the adversarial samples are obtained from the $\ell_p$ ball of the original data, we obtain the semantic adversarial samples from the interpolated line of the aligned pair-wise samples from source domain and target domain. Experimental results and ablation study show that our method can achieve considerable improvements on the CDA benchmarks compared with the state-of-art methods focusing on the same problem.


Under review as a conference paper at ICLR 2022
PAIRWISE ADVERSARIAL TRAINING FOR UNSUPER- VISED CLASS-IBBALANCED DOMAIN ADAPTATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Unsupervised domain adaptation (UDA) has become an appealing approach for knowledge transfer from a labeled source domain to an unlabeled target domain. However, when the classes in source and target domains are imbalanced, most existing UDA methods experience sigmificant performance drop, as the decision boundary usually favors the majority classes. Some recent class-imbalaneed do- main adaptation (CDA) methods aim to tackle the challenge of biased label distri- bution by exploiting ppeudo-labeled target data during training process. However, these methods still suffer from the issues with unreliable pseudo labels and error accumulation during training. In this paper, we propose pairwise adversarial training approach for class-imbalanced domain adaptation. Unlike conventional adversarial training in which the adversarial samples are obtained from the lp ball of the original data, we generate ad adversarial samples from the interpolated line of the aligned pairwise samples from source domain and target domain. Exper- imental results and ablation studies show that our method achieves considerable improvements on benchmarks compared with the state-of-art CDA methods.
INTRODUCTION
Unsupervised domain adaptation (UDA) aims to achieve knowledge transfer from a labeled source domain to an unlabelled target domain. Recent years have witnessed the significant progress of UDA based on deep neural networks (Pei et al. 2018 Cui et al. 2020 Hu et al. 2020 Liang al.) 2020 Na et al.] (021). Most of existing UDA methods assume that only covariate shift occurs in the source domain and target domain, while the label distributions in two domains are identical. However, this assumption may not hold in real-world applications. For instance, in wild-life pictures, the commonly seen animals such as rabbit and deer appear more frequently than the rare animal such as panda and crocodile. Public datasets such DomainNet (Peng al. 2019) and and MSCOCO (Lin et al. 2014) exhibit imbalanced class distribution. Figure illustrates the imbalanced label distributions in the Real domain and Sketch domain from the DomainNet dataset.
To address the issue of imbalanced label distributions in domain adaptation, some recent studies (Wu et al. 2019 Tan et al. 22200] Jiang et al. 2020) try to jointly model the conditional feature distribution shift and label distri- bution shift (LDS). This problem is referred to as Class- imbalanced Domain Adaptation (CDA). Let x and y de- note the samples and labels, respectively. p and sepa- rately represent the probability distribution of source do- main and target domain. The common assumptions in UDA involve the covariate shift (i.c., p(z) * q(x)) and identical label distribution (i.e., p(y) = q(v)). In CDA, however, apart from the covariate shift, both the condi- tional feature shift and label shift exist, i.e., p(xly) # q(<l)), p(v) # q(v).
Denaiitte Label diteownn Bool to Seach
Gutch
400
Figure 1: Illustration of label distribu- tion shift in DomainNet dataset.
CDA is a more challenging task than UDA. Recent studies (Tan et al. 2020) have demonstrated that the mainstream UDA methods will suffer significant performance drop, as the classifier will favor

---

Under review as a conference paper at ICLR 2022
the majority classes. Only few CDA approaches have been proposed by far. In Tan et al. (2020) work, the negative effect of label shift is reduced by exploiting the pseudo labelled target samples via self-training Diang et al. 2020 use an implicit sampling method based pseudo labels to align the joint distribution between features and labels. However, one critical problem of these methods is that the pseudo labels are likely to suffer from ill-calibrated probabilities (Guo al., (017). and thus the unreliable pseudo labels will cause error accumulation during the training process, which will largely degrade the model performance.
Augmenting training data has been proven as an effective strategy to tackle the issue of biased label distributions in insalance learning (Chawla et al. 2002] Chou et al. 2020). In addition the traditional data augmentation techniques, adversarial training is also capable of generating seman- tically meaningful synthetic samples that help enhance the robustness of models. However, these approaches only consider single domain, and they cannot be directly applied to solve the CDA problem. In this paper, we propose a pairwise adversarial training (PAT) approach that augments training data for class-imbalanced domain adaptation. Unlike conventional adversarial training in which the adversarial samples are obtained from the lp ball of the original data, we obtain the se- mantic adversarial samples from the interpolated line of the aligned pair-win pair-wise samples from source domain and target domain. Moreover, a haass-imallanced semantic centroid alignment strategy is designed to explicitly align the source and target domains in the feature space.
The main contributions of this paper are three-fold. (1) We propose a novel pairwise adversarial training approach that generates adversarial samples from pairs of samples across the source and target domains, and further exploits these samples to augment training data. (2) We propose new optimization algorithm to solve pairwise adversarial training problem. (3) We conduct extensive evaluations on benchmark datasets, and results show that our approach obtains competing perfor- mance compared with sfate-of-art CDA methods.
RELATED WORK
In this section, briefly introduce three relevant research topics, including unsupervised domain adaptation, class-imbalanced domain adaptation and adversarial training.
Unsupervised Domain Adaptation. In recent years, unsupervised domain adaption (UDA) has at- tracted increasing attention. Existing UDA methods could be roughly categorized into two groups, including the discrepancy-baag bbaed methods and adversarial-basee methods. The discrepancy-basee methods usually align source and target feature distributions in the embedding space using vari- ous statistical distance metrics, such Maximum Mean Discrepancy (MMD) (Long et al. 2016 2017 Kang et al.] 2019). Correlation Alignment (CORAL) (Sun & Saenko 2016). and Wasserstein distance (Lee Raginsky 2018 Shen et al.) 2018 Balaji et al.] 2019). On the other hand, the adversarial-based methods focus on learning domain invariant features via domain adversarial train- ing (Ganin et al. 2016; Shu et al., 2018 Pei et al., 2018 Saito et al. 2018 Deng et al.) 2019 Yu eet al. 2019). Recently, Zhang al. (2019) proposed the margin disparity discrepancy (MDD) to measure the discrepancy of two domains with generalization bounds. This theory is tailored into an adversarial learning algorithm for domain adaptation. Unlike other adversarial learning based UDA methods that align two domains by confusing a domain discriminator, MDD aligns two domains by minimizing the maximum margin disparity discrepancy of an optimal classifier and auxiliary classifier f". The optimization problem of MDD is formulated as:
min e (D.) + (D., Dr), f.
(1)
max (D. D,), 8
(2)
where E is the classification loss on the source domain and D., measures the discrepancy of source domain and target domain. Specifically,
<(D.) D. E. (( f(((())) - (Brr
(3)
= DD
ortp::/06......a..../...../////- f(b("))
C is cross-entropy function, and ('(f((vxx)) )) f()(=))) = (f (+ of is the
pseudo label generated from optimal classifier. MDD is the backbone of our method.
2

---

Under review as a conference paper at ICLR 2022
Class-ibbalanced Domain Adaptation. As branch of domain adaptation, the domain adaptation (CDA) aims to deal with data with biased class distribution. TTan et al. (2020)
might be the first one to investigate the CDA problem, and they exploited the pseudo labelled target data to reduce the negative effect of label shift. [W et al. | (2019) proposed the asymmetrically-rees rellead
distances replacement of the standard ones under biased label distribution. Jiang al. (2020) adopted the implicit sampling strategy to ensure class alignment at the minibatch level. PPrabhu et al. (202I) avoided the use of highly unreliable pseudo labels by assessing the reliability of target data with predictive consistency under random image transformations. Our method refrains from the exploitation of pseudo labeled target data directly in the training process, while reducing the effect of biased label shift by incorporating the semantic adversarial samples into the training process.
Adversarial Training. Adversarial training (AT) (Szegedy et al., 2014; Goodfellow et al.) 2015) an effective regularization method for enhancing the robustness and generalization ability of deep learning models. In particular, adversarial samples are incorporated in the model training process, which are intentionally designed to deceive the deep learning model by adding small perturbation on the original data. Furthermore, virtual adversarial training (VAT) has been proposed (Miyato et al. 2018), which seeks the adversarial direction for regularization without using label information. Both AT and VAT have been employed to tackle the standard UDA problems (Shu et al. (2018). However, the best of our knowledge, our work is the first attempt to address the class- dass-imballaceed domain adaptation problem using adversarial training.
3 PROPOSED APPROACH
In this section, we first give the problem definition of CDA, and then present the details of the proposed pairwise adversarial training approach. Finally, we introduce how to integrate the pairwise adversarial training with MDD to address the CDA problem.
3.1 PROBLEM DEFINITION
In plass-imbalancee domain adaptation, both the source and target domains suffer from label distri- bution shift. We are given a source domain D, = {= v ) with N labelled samples and target domain D, J with No unlabelled samples. Each domain contains K classes, and the class label is denoted as y E {0, ..., K - 1). Let and denote the probability distributions of the source domain and target domain, respectively. We assume that both the covariate shift (i.e., p(x) * q(x)) and label distribution shift (i.e., p(y) # q(B) and p(zly) * q(x))) y)) exist in two do- mains. Our goal is to train model that can learn domain invariant features, reduce the gap between source and target domains, and mitigate the label distribution shift. The model typically consists of a feature extractor v X + and a classifier f + y that aims to minimize the target risk.
3.2 PAIRWISE ADVERSARIAL TRAINING (PAT)
We investigate how to mitigate the challenging issue of label distribution shift in CDA, illustrated in Figure Previous studies (Tan et al. 2020) found that when the source domain is imbalanced, the model pertormance target domains will be significantly dropped, especially when the target do- main is also imbalanced. An intuitive solution is to augment the training data in two domains, such that the model training would not be dominated by the majority classes either domain. However, this task is not trivial, considering the mixed effects of domain gap and imbalanced class distribu- tions. Inspired by adversarial training, data. In adversarial training, the adversarial samples will be exploited to enhance the robustness and generalization ability of model. The loss function of adversarial training is:
aim create adversarial samples to augment training
Loe + 6" 1/.
where
8 = arg max Lee (x + 6.y:0) 1|
(5)
where x is the original sample, y is the ground-trui label of x, refers to model parameters, and is the perturbation added to
The existing adversarial training methods could not be directly used to tackle the CDA problem for two reasons. First, existing methods simply generate adversarial samples within the neighborhood
3

---

Under review as a conference paper at ICLR 2022
of the original samples, but they could not mitigate the gap between source and target domains. Second, existing methods treat majority classes and minority classes equally, so they are unable to address the class imbalance issue. In this paper, propose pairwise adversarial training that gen- erates adversarial samples from the linear interpolation of source and target samples and meanwhile reduces the domain discrepancy. In the following, we will introduce two key components of PAT, including the generation of interpolated adversarial samples and semantic centroid alignment.
3.2.1 INTERPOLATED ADVERSARIAL SAMPLE GENERATION
As shown in Figure 2 we will generate adversarial samples on the interpolated line from source sample to a target sample of the same class. The interpolated adversarial samples (IAS) should have the same semantic meaning their corresponding source/tareet sample pair, although they are not perfectly aligned with the original sam- ples. We explicitly address the data imbalance issue in the source domain by exploiting interpolated adversarial samples and aligning the source and target domains. As a result, the generalization ability of the unbiased model will be improved and the data imbalance issue in the target domain could be implicitly addressed. For the kth class, the interpolated adversarial samples can be defined as:
Toge
loogle
Figure 2: Illustration of an interpolated ad adversarial sam- ple, which lies on the inter- polated line of the source and target samples.
Ngde
{agde (oqdu x; Aa2 21), E [0, 1) 10 - k}.
(6)
where x, E RCCHHW À is the coefficient measuring the contributing weights of the source sample and target sample. if is the pseudo label of the target sample which is generated from the backbone domain adaptation model (e.g., the optimal classifier f in MDD). It is used for match of the corre- sponding source sample. In our work, though the pseudo labels might be incorrect, our PAT method will not suffer from the potential error accumulation issue. First, we choose the target samples with high confidence. Second, even if the pseudo label is incorrect, there is still a chance that the in- terpolated adversarial sample will be generated within the boundary as expected. Specifically, the misclassified target samples often appear near the decision boundary. So, even if the target sample is from a different class, the adversarial sample generated from the pair of source sample and target sample may not violate the decision boundary. Finally, the adversarial samples our approach are generated dynamically, and the adverse effect of bad adversarial samples could be mitigated.
Then, the generation of interpolated adversarial samples could be achieved by solving the following optimization problem:
LiAS := D(āadu 0)
where gadv =
arg max D'(2.dee 0) zadv exad
In Eq. (7p. the outer minimization problem involves standard cross-entropy loss function, i.c.,
(8) where 3) : X + denotes a feature extractor, f Z + y denotes a classifier, and a is the softmax function. For the inner maximization problem, we use a modified cross-entropy function proposed by GGoodfellw et al. (2014). The modified loss function can alleviate the problem of gradient exploding or vanishing when the entropy loss is maximized. The loss function of the inner maximization problem is written as:
D(aadv y; 0) = LcE (aadv y; 0) = log(a,([(((aadd))
a'd y: 0) C'CE(xad, 0) = log(1 o (f (f
Several optimization algorithms, such as the fast gradient sign method (Goodfellow et al. 2015) and projected gradient descent (Madry et al. 2018), have been commonly used for adversarial train- ing. However, these algorithms aim to obtain the adversarial samples in the loo ball of the original samples, which cannot be directly applied to solving our problem. In our case, the interpolated ad- versarial samples are confined on the interpolated line of source data and target data. We propose a new optimization algorithm to solve the inner maximization optimization in Eq. (7). We initialize the interpolated adversarial samples with random À and update them by back propagation in each iteration. The main procedures of our algorithm are summarized in A Algorithm
4

---

Under review as a conference paper at ICLR 2022
Algorithm Solving the maximization problem in PAT
Input: Source samples from mini-batch {(0,, wf) j.n target samples and their pseudo labels ((rt, (f) probability threshold of each class {Px)
Output: Interpolated adversarial samples {=idde) Nadv for each source sample xa in the mini-batch do if rand() > P& = yf) then
Ji=1
Choose one target sample z with pscudo label equals to At
Initialize zddv with random À using Eq. (6)
repeat
Calculates the gradient of À with loss function in Eq. 77. 99 -a D'vaD' y/; 0)
Update with the gradient, À + x 9A
Clip the À between 0 and 1, À f À clip(0. 1)
Update adversarial sample add with new À using Eq. (6 until the optimization converges
end if' end for
Generating interpolated samples has been explored in literature, such as mix-up (Chou et al. 2020) and its variants. However, mix-up based methods and our method have significant differences. First, mix-up based methods focus the single-domain classification standard Nomain-adaptattinn problems, while our method focuses on the data imbalance problem in domain adaptation. Sec- ond, the ideas for generating samples and labels in mix-up and method are different. Mix-up creates virtual samples and their labels from two randomly chosen samples. While our method, an adversarial sample is generated using a pair of samples from two domains with the same label. Third, mix-up needs to manually define a hyper-parameter to control the strength of interpolation. However, in our method, the parameter for controlling the strength of interpolation is adaptively updated along with adversarial training.
3.2.2
SEMANTIC CENTROID ALIGNMENT
Without careful control of the gener- ation mechanism, the interpolated ad- versarial samples may not alleviate the issue of imbalanced class distribution. Moreover, although the interpolated adversarial samples bridge the source and target domains to some extent, the discrepancy between source and target domains is not explicitly reduced. To address these issues, we propose to ex- plicitly align the source and target do- mains with imbalanced class distribu- tions using strategies.
Joon bargle
Jowne homale
Advessdd lang's
adorneral loodl
Figure 3: Illustration of generation of adversarial samples (a) without centroid alignment and (b) with centroid align- ment. In (a), adversarial samples have a larger chance to violate the decision boundary.
First, we propose a strategy to guide the generation of interpolated adversarial samples. For training samples in each mini-batch from the source domain, they should not have the equal opportunity to generate interpolated adversarial samples. Since the decision boundary usually favors the majority classes, the probability of generating adversarial samples for minority classes should be larger than that for majority classes. For the k-th class, we set a probability threshold Pk as follows:
77
Ps
(10)
Umaz +
where n& is the number the samples from the kth class. Rmaz = max (ru) 1K and is the bias. For specific class, if random number r E [0, 1) is larger than the corresponding threshold, the adversarial samples will be generated, as shown in Algorithm [ We also adopt class- balanced sampling the source data to alleviate the biased occurrence of the majority classes. Specifically, each class will be selected with an equal chance, in order to reduce the model prediction bias towards the majority classes.
5

---

Under review as a conference paper at ICLR 2022
Cis
Liss
Lea
source
adversorial
Lea
samples
lurgel cemplexs
GRL
?
maxe
Ladv
Figure 4: Illustration of our framework based on PAT and MDD. It includes feature extractor G, an optimal classifier F, and an auxiliary classifier F'. The dashed line represents the data flow in pairwise adversarial training. The margin disparity discrepancy of two domains is diminished by aligning the two one-hot labels. We also explicitly align the pair of class conditioned samples by minimizing the distance of the centroids denoted LCA-
Second, we incorporate the moving average centroid alignment (Xie et al., 2018) to align the condi- tional feature distributions of source and target domains by explicitly matching the centroids two domains. As illustrated in Figure 3 without centroid alignment, the adversarial samples may be gen- erated from a pair of samples in which one of the samples is misaligned to other classes, thus making the embedding of adversarial samples fall out of the decision boundary. With centroid alignment, we can eliminate the occurrence of such out-of-bound adversarial samples, and the interpolated ad- versarial samples could provide meaningful support for the minority class in the target domain. The loss function of moving average centroid alignment is defined as Lc = Σ dist(C Ct), where Cg and Cj2 denote the centroids of the kthh class in the source domain and target domain, respectively.
dist() can be implemented by the Euclidean distance or cosine distance.
3.3 PAT FOR CLASS IMBALANCED DOMAIN ADAPTATION
The proposed PAT approach could be integrated with many existing domain adaptation frameworks to enhance their performance
haaslimmalanceed data. In this paper, adopt MDD (Zhang al. 2019) as an example backbone model and showcase how to integrated PAT with it.
The MDD framework consists of feature extractor G, optimal classifier F, and auxiliary classifier F". The loss function of MDD is introduced in Eq. ( and Eq. (2). Adversarial samples designed to maximize the modified cross-entropy function L' are generated from the pair of source data and target data. Finally, the overall loss function of our framework is:
LODD + aLrAS + BLCA.
(11)
where a and B3 are two trade-off parameters.
4 EXPERIMENTS
Office1 ESUl Aeaass Mecce
4.1
EXPERIMENTAL SETTINGS
Office-31 is a widely used benchmark image dataset for do- main adaptation (Saenko al. 2010). It contains 31 classes in three domains: Amazon (A), Dslr (D) and Webcam (W). The standard Office-31 doesn't exhibit obvious label distri- bution shift (LDS), so a new imbalanced Office-31 is cre- ated by sampling from standard ones as suggested by Tan et al.) (2020). The distribution conforms to Paredo distri- bution (Reed, (2001) and follows the Reversely-unbalanced Source and Unbalanced Target (RS-UT) protocol. Both the source domain and target domain have shifted label distribu- tions, and the label distribution of source domain is reversed version of that of target domain. Office- Pffice-Homiin is a large benchmark dataset containing 65 classes of objects commonly found of- fice and home scenarios (Venkateswara et al. 2017). It has four domains: Real-World (Rw), Clipart
Figure 5: Biased label distribution shift on Tmazonn+Webere from imbalanced Office-31.

---

Under review as a conference paper at ICLR 2022
Table 1: Per-class average accuracy Office-Home dataset with RS UT label shift. Bold and underscore denote the best and second-best performing methods, respectively.
Method
Rw -P
Rw-+CI
Pr-
Pr-+CI
CI->Rw
CI-Pr
AVG
Source
70.74
44.24
67.33
38.68
53.51
51.85
54.39
BSP (Chn et al.
2019
72.80
23.82
66.19
20.05
32.59
30.36
40.97
PADA
CCa ct al. 2018
60.77
32.28
57.09
26.76
40.71
38.34
42.66
BBSE
Lipton al.
2018
61.10
33.27
62.66
31.15
39.70
38.08
44.33
MCD
Long
2018
66.03
33.17
62.95
29.99
44.47
39.01
45,94
DAN Long al.
2015
69.35
40.84
66.93
34.66
53.55
52.09
52.90
E-DANN Wu ct 2019
58.56
40.57
67.32
37.33
55.84
53.67
53.88
JAN Long et ul. 2017
67.20
43.60
68.87
39.21
57.98
48.57
54.24
DANN (Ganin & Lempitsky
2015
71.62
46.51
68.40
38.07
$8,83
58.05
56.91
MDD Zhang et al.. 2019
71.21
44.78
69.31
42.56
52.10
$2.70
55.44
COAL Tan el al. 2020
73.65
42.58
73.26
40.61
59.22
57.33
58.40
InstaPDM Li ul.
2020
75.56
42.93
70.30
39.32
61,87
63.40
58.90
MDD+implicil Jaang el
2020
76.08
50.04
74.21
45.38
61.15
63.15
61.67
SENTRY (Prabhu et al. 2021
76.12
56.80
73.60
54.75
65.94
64.29
65.25
Ours
79.88
54.68
77.32
50.21
67.29
67.04
66.07
Data of the baseline methods are cited from Prabhu et
2021
Table 2: Per-class average accuracy on DomainNet dataset. Bold and underscore denote the best and second- bocodd-best performing methods respectively.
3C
18
source
65.75
68.84
59.15
77.71
60.60
57.87
84.45
62.35
65.07
77.10
63.00
59.72
66.80
BBSE Lipton
42.36
PADA L30 et al
2015
65.91
67.13
58.43
74.69
53.09
52.86
79.84
59.33
57.87
76.52
66.97
61.08
64.48
81.74
DAN
Leng mh.
2013
64.36
70.65
58.44
79.44
56.78
60.05
84.36
61.62
62.21
79.60
65.01
62.04
67.07
E-DANN
2019
66.15
71,80
61.53
60.06
61.22
84.46
66.81
62.84
81
66.50
69.52
TAN
ala
2019
71.10
68.90
67.10
83.15
63.30
64.66
83.95
65.35
67.06
82:22
70.64
68.09
72.05
JAN
2017
65.57
73.58
67.61
85.02
64.96
67.17
S7.06
67.92
66.10
84.54
72.77
67.51
72.48
ETN 30 213
2019
69.22
72.14
63.63
86
65.33
63.34
85.04
68.78
84.93
72.17
68.99
73.99
BSP Iic of mi
2019
67.29
73.47
69.31
86.50
67.52
70.90
86.83
70.33
68.75
84.34
72.40
71.47
74.09
Grnin
Lempitsky
73.56
70.58
85.73
75.16
COAL
Lan ala
2020
73.58
75.37
70,50
89.63
69.98
71.29
89.81
68.01
70.49
87.97
73.21
70.53
75.89
MDD
75.54
74.30
70.02
88.17
70.50
70.30
87.94
72.03
72.29
88.85
76.12
71.21
76.44
InstaPBM
Li et
2020
80.10
75.87
70.84
89.67
70.21
72.76
89.60
74.41
72.19
87. 00.00
79.66
71.75
77.84
Sentry Prabbu al
2022
83,89
76.72
74.43
90.61
76.02
79.47
90.27
82,91
75.60
90.41
82.40
73.98
81.39
80.15
72.25
26.25
76.32
89.61
80.71
80.23
Dota of the beseline methods are clted Eron/Paahaaa Prahiu et al.
2021
(CI), Product (Pr) and Art (Ar). In our experiments, we use the existing imbalanced Office-Home with RS-UT distributions generated in Tan et al. (2020) to train and test our approach. Since there very limited samples in the art (Ar) domain, only conduct domain adaptation tasks the other three domains. DomainNet a large-scale benchmark dataset for domain adaptation (Peng et al. 2019). Since there are mislabeled samples in some classes and domains, follow Tan al. (2020) and adopt only 40 common classes from four domains: Real (R), Clipart (C), Painting (P), and Sketch (S). Different from Office-31 and Office-Home, the selected samples in DomainNet al- ready exhibit obvious label distribution shift in the source domain and target domain. So there is no need sample this dataset again. Figure 5 Billustrates the label distributions of imbalanced Office31 and imbalanced Office-Home datasets.
We use PyTorch implement our approach. We train our model with the mini-batch SGD, Nes- terov momentum of 0.9, and weight decay of 0,0005. The learning rate of classifiers is 10 times larger than that of feature extractor, and all the learning rates are adjusted by every iteration. In order to obtain the interpolated adversarial sample from a pair of source sample and target sample from the same class, we utilize a memory pool to store the pseudo labels of the target data. The pseudo labels are updated in every iteration. Note that we utilize class-balanced sampler on the source data, which can be referred to as N-way (number of classes per batch) and K-shot (number of examples per class). The coefficient is set to 0.5 and 3 is set to 0.05 for all the experiments. Following existing work on CDA, we adopt the per-class mean accuracy as our evaluation metric. All the experiments implemented Nvidia RTX A5000 platform.

---

Under review as a conference paper at ICLR 2022
Table 3: Accuracy of MDD and our full model on minority classes from imbalanced Ofmece-Home (Rw+>rr). Bold denotes the best performing method.
Method
Batteries
Bed
Bike
Bottle
Calculator
Chair
Clipboards
AVG
MDD (baseline)
51.61
95.52
19.79
85.91
71.42
93.05
62.22
68.50
Ours
59.67
100.0
29.17
88.73
73.47
100.0
64.44
73.64
Table 4: Per-class average accuracy on Office-31 with RS-UTT label shift. Bold and underscore denote the best and second-best performing methods respectively.
Method
A-W
D-*
W-+
A-D
D-
W-A
AVG
source
71.77
90.86
93.06
72.25
59.03
58.34
74.21
F-DANN (Wu et al. 2019
69.83
93.56
93.95
76.45
58.57
58.11
75.07
COAL (Tan al. 2020)
81.18
91.12
95.46
81.67
66.08
66.60
80.35
MDD+implicit (Jiane et al. 2020
85.79
96.20
97.40
84.25
68.11
66.63
83.06
Sentry (Prabhu el al. 2021
81.77
90.95
93.50
83.91
62.72
64.00
79.48
Ours
89.61
96.08
97.08
86.66
71.93
70.40
85.29
4.2
COMPARISON WITH OFAT CDA METHODS
We compare approach with several state- of-art methods for class imbalanced domain adaptation. The baseline methods can be divided into two categories. The first category of methods are specif- ically designed solve the CDA problem, including Sentry (Prabhu al. (021). MDD++implicit Jiang et al. (2020), COAL (Tan et al., 2020), and F-DANN (Wu et al. 2019). The second cate-
gory of methods aim solve the standard unsupervised domain adaptation problem, including the
InstaPBM (Li et al. 2020) and BSP (Chen et al. 2019).
Table shows the per-class average accuracy and overall average accuracy of our approach and baselines on the imbalanced Office-Home dataset. The results of the baseline are cited from Prabhu al. (202T). Our proposed model achieves the best results in 4 out of 6 tasks. The accuracies of our approach in these 4 tasks are more than 3% higher than the second-best results. In addition, our model obtains the second-best results in the Rw-CC and Pr-XCI tasks. Overall, our method achieves the highest average accuracy (i.c., 66.07%) among all the compared methods. We also investigate the class-specific accuracy of MDD (Zhagg et al., 2019) and our method on minority classes, in order to understand how well our method addresses the data imbalance issue. Table B shows that our PAT method can significantly boost the performance on the minority classes from the imbalanced Office- dataset.
Table 2 shows the per-class average accuracies of our approach and baselines on the DomainNet dataset. Our approach achieves the best results in 4 tasks and second-best results in tasks. In R-S, P-+ and S-P, our model can achieve 76.08%, 75. 35.46%, which are 1.65% and 1.48% higher than the second-bes best results, respectively. Though in some tasks our results are lower than Sentry (Prabhu et al. 2021). the overall average accuracy of our approach is comparable to that of Sentry.
We manually sample the standard Office-31 dataset and construct the imbalanced Office-31 dataset, in which the label distribution conforms to the Paredo distribution (Reed, 2001). We compare our model with current state-of-art methods that focus on the CDA problem. Among all the tasks, our model achieves best results in 4 tasks. In these tasks, our results are 3.82%, 2.41%, 3.28% and 3.77% higher than the second best results. In the D-W and W->D tasks, the performance of our approach are only 0.12% and 0.32% less than the best results.
4.3 ABLATION STUDIES
We further investigate the performance of our approach from several aspects. First, we evaluate the contribution of each component in the loss function. Second, we evaluate the sensitivity of the model to the change of the hyper-parameters. Third, we evaluate the performance of our method on the standard unsupervised domain adaptation problem. We have also evaluated: (1) the effect of the number of the iteration in the inner maximization of PAT; and (2) the integration of PAT and other domain adaptation methods including CDAN, CDANE and Sentry, and reported the results in the Appendix due to space limit.
8

---

Under review as a conference paper at ICLR 2022
Table 5: Per-class average accuracy of MDD, Ours w/o IAS, Ours w/o CA, and full model
three tasks from imbalanced Office-Home dataset. Bold denotes the best performing method.
Method
Rw-+P
Rw Rw-CC
Pr->Rw
Pr-+C
CI-Rww
CI-+Pr
AVG
MDD (baseline) f
75.96
47.38
71.56
42.73
57.46
58.76
58.98
Ours w/o IAS
78.38
51.85
75.72
48.31
67.16
65.80
64.26
Ours w/o CA
77.37
53.02
76.12
47.08
64.99
64.73
63.61
Ours
79.88
54.68
77.32
50.21
67.29
67.04
66.07
We adopt class-balanced source sampling on all these methods.
Table 6: Accuracy of MDD and our model on four tasks from standard Office-Home. Bold denotes the best performing method.
Method
Rw-Pr
Ar-+Rw
Cl-Prr
Pr-CI
AVG
MDD (baseline)
82.3
77.8
71.4
53.6
71.3
Ours
84.7
80.2
75.1
54.3
73.6
Data of the haseline methods are cited from Zhhang et al.
2019)
The proposed pairwise adversarial training approach includes two major components, i.c., interpo- lated adversarial samples (IAS) and centroid alignment (CA). We choose three tasks from each of the three datasets separately. Table [ shows the performance of our backbone model MDD, Ours w/o IAS, Ours w/o CA, and our full model the imbalanced Office-Homm dataset. Detailed results on the other two datasets are provided in the Appendix. Ours w/o IAS and Ours w/o CA consis- tently achieve better performance than the baseline model MDD, which validates the effectiveness of both methods dealing with the biased label distribution shift. Our full model further improves the classification accuracy, demonstrating the complementary roles of pairwise adversarial samples and centroid alignment in our approach.
Furthermore, evaluate the sensitiv- ity of our model to the change two hyperparameters a and | In partic- ular, we first set / to 0.05 and choose from [0, 2.5]. Then, a and 3 is chosen from [0, 0.25]. Fig- ure 6 shows the per-class average ac- curacy of our model when varying the hyper-parameter values. It shows that model is not very sensitive to the settings of hhper--pramne parametrsss in a rela- tively wide range.
0.5,
2
0.0
0.36
3.10
0.20
0.0
(a)
(b)
Figure 6: Average accuracy of our model with: (a) varying a when 3 = 0.05, and (b) varying when a 0.5 on
Rw-Pr in imbalanced Office-Home.
Finally, we evaluate our method in the standard UDA setting, by comparing the results against the baseline MDD (Zhang et al.] (019) the standard Office-Home dataset. The results from Table/]]show that the classification accuracy all the four tasks are improved after incorporating interpolation method, which proves the effectiveness of our method in unsupervised domain adaptation problem.
5 CONCLUSION
In this paper, we propose a pairwise adversarial training approach to tackle the class-imbalanced unsupervised domain adaptation (CDA) problem. Our approach generates interpolated adversarial samples across source and target domains. In order to alleviate the biased label distribution shift issue, we use the interpolated adversarial samples to augment the training data (especially the mi- nority classes) and meanwhile adopt the centroid alignment strategy explicitly align source and target domains. Experimental results on three CDA benchmark datasets show that, our approach yields competing performance compared with state-of-art CDA methods.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Yogesh Balaji, Rama Chellappa, and Soheil Feizi. Normalized wasserstein for mixture distributions with applications adversarial learning and domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6500-6508, 2019.
Zhangjie Cao, Lijia Ma, Mingsheng Long, and Jianmin Wang. Partial adversarial domain adaptation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 135-150, 2018.
Zhangjic Cao, Kaichao You, Mingsheng Long, Jianmin Wang. and Qiang Yang. Learning to transfer examples for partial domain adaptation. In CVPR, pp. 2985-2994, Computer Vision Foundation
/ /IEEE, 2019.
Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res., 16:321-357, 2002.
Xinyang Chen, Sinan Wang. Mingsheng Long, and Jianmin Wang. Transferability discrim- inability: Batch spectral penalization for adversarial domain adaptation. In ICML, volume 97 of Proceedings of Machine Learning Research, pp. 1081-1096 PMLR, 2019.
Hsin-Ping Chou, Shih-Chieh Chang, Jia-Yu Yu Pan, Wei Wei, and Da-Cheng Juan. Remix: Rebalanced mixup. In ECCV Workshops (6), volume 12540 of Lecture Notes Computer Science, pp. 91. Springer, 2020.
Shuhao Cui, Shuhui Wang, Junbao Zhuo, Chi Su, Qingming Huang. and Qi Tian. Gradually van- ishing bridge for adversarial domain adaptation. In CVPR, pp. 12452-12466 Computer Vision Foundation IEEE, 2020.
Zhijie Deng, Yucen Luo, and Jun Zhu. Cluster alignment with a teacher for unsupervised domain adaptation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 9944-9953, 2019.
Yaroslav Ganin and Victor S. Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1180-1189, JMLR.org. 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarr training of neural net- works. The journal of machine learning research, 17(1):2096-25 2016.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pp. 2672-2680, 2014.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR (Poster), 2015.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In ICML, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 2017.
Lanqing Hu, Meina Kan, Shiguang Shan, and Xilin Chen. Unsupervised domain adaptation with hierarchical gradient synchronization. In CVPR, pp. 4042-4051. Computer Vision Foundation / IEEE, 2020.
Xiang Jiang, Qicheng Lao, Stan Matwin, and Mohammad Havaei. Implicit class-conditioned do- main alignment for unsupervised domain adaptation. In International Conference Machine Learning, pp. 4816-4822 PMLR, 2020.
Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4893 4902, 2019.
10

---

Under review as a conference paper at ICLR 2022
Jacho Lee and Maxim Raginsky. Minimax statistical learning with wasserstein distances. Advances in Neural Information Processing Systems, 2018:2687-2666, 2018
Bo Li, Yezhen Wang, Tong Che, Shanghang Zhang, Sicheng Zhao, Pengfei Xu, Wei Zhou, Yoshua Bengio, and Kurt Keutzer. Rethinking distributional matching based domain adaptation. CoRR. abs/2006. 1300006.13322. 13352, 2020.
Jian Liang, Dapeng Hu, and Jiashi Feng. we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 6028-6039 PMLR, 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV (5), volume 8693 of Lecture Notes in Computer Science, pp. 740-755. Springer, 2014.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pp. 3122-3130. PMLR, 2018.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable fea- tures with deep adaptation networks. In ICML, volume 37 of JMLR Workshop and Conference Proceedings, pp. 97-105. JMLR.org, 2015.
Mingsheng Long. Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. Advances in Neural Information Processing Systems, 29:136- 144, 2016.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International conference on machine learning, pp. 2208-2217. PMLR, 2017.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Conditional adversarial domain adaptation. In NeurlPS, pp. 1647-1657, 2018.
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR (Poster). OpenReview.net, 2018.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979-199; 2018.
Jaemin Na, Heechul Jung, Hyung Jin Chang, and Wonjun Hwang. Fixbi: Bridging domain spaces for unsupervised domain adaptation. In CVPR, pp. 1094-1103. Computer Vision Foundation / IEEE, 2021.
Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Multi-adversarial domain adap- tation. In Thirty-second AAAI conference artificial intelligence, 2018.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. ICCV, pp. 1406-1415. IEEE, 2019.
Viraj Prabhu, Shivam Khare, Deeksha Karthik, and Judy Hoffman. Selective entropy optimization via committee consistency for unsupervised domain adaptation. In International Conference in Computer Vision (ICCV), 2021.
William Reed. The pareto, zipf and other power laws. Economics letters, 74(1):15-19, 2001.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV (4), volume 6314 of Lecture Notes in Computer Science, pp. 213-226. Springer, 2010.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3723-3732, 2018.

---

Under review as a conference paper at ICLR 2022
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Thirty- -eccnd AAAI Conference on Artificial Intelligence, 2018.
Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. In International Conference on Learning Representations. 2018.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pp. 443 450. Springer, 2016.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference Learning Representations, ICLR 2014, 2014.
Shuhan Tan, Xingchao Peng, and Kate Saenko. Class-imbalanced domain adaptation: an empirical odyssey. In European Conference on Computer Vision, pp. 585-602. Springer, 2020.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, pp. 5385-5394. IEEE Computer Society, 2017.
Yifan Wu, Ezra Winston, Divyansh Kaushik, and Zachary Lipton. Domain adaptation with asymmettrically-rrlaxed distribution alignment. In International Conference on Machine Learning, pp. 6872-6881. PMLR, 2019.
Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen. Learning semantic representations for unsupervised domain adaptation. In ICML, volume 80 of Proceedings of Machine Learning Research, pp. 5419-5428. PMLR, 2018.
Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Universal domain adaptation. In CVPR, pp. 2720-2729. Computer Vision Foundation IEEE, 2019.
Chaohui Yu, Jindong Wang, Yiqiang Chen, and Meiyu Huang. Transfer learning with dynamic adversarial adaptation network. In 2019 IEEE International Conference Data Mining (ICDM), pp. 778-786. IEEE, 2019.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael I. Jordan. Bridging theory and algorithm for domain adaptation. In ICML, volume 97 of Proceedings of Machine Learning Research, pp. 7404 7413. PMLR, 2019.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
We evaluate the effect of the number of iterations of the inner maximization in pairwise adversarial training. We choose Rw-+Pr from Office-Home, R-CC from DomainNet and A-W from Office-31 as our evaluation tasks. The results are illustrated in Figure It shows that the performance of our approach will generally increase with more iterations. At some points the performance experiences the slight fluctuation. Though performance reaches the maximum value at iteration of 12, it is relatively comemconu in the training process.
100.0
Rw->Pr
Pen-class average accuracy
95.0
R-SC
A-SW
90.0
85.0
80.0
75.0
70.0
10
12
number of Iterations
Figure 7: Per-class average accuracy of our model with different number of iterations.
Table 7 and Table S show the per-class average accuracy of MDD, Ours w/o IAS, Ours w/o CA, and our full model on three tasks from the DomainNet dataset and Office-31 dataset, respectively. The results demonstrate the effectiveness of two major components (i.e., interpolated adversarial samples and centroid alignment) in our approach.
Table 7: Per-class average accuracy of MDD, Ours w/o IAS, Ours w/o CA, and our full model on three tasks from DomainNet dataset. Bold denotes the best performing method.
Method
R R-C
C-+P
C-R
AVG
MDD (baseline)
75.30
70.38
87.94
77.87
Ours w/o IAS
77.03
69.43
88.60
78.35
Ours w/o CA
77.65
71.25
88.49
79.13
Ours
80.15
72.25
89.87
80.76
We adopt class-balanced source sampling on all these methods.
Table 8: Per-class average accuracy of MDD, Ours w/o IAS, Ours w/o CA, and full model imbalanced Office-31 dataset. Bold denotes the best performing method.
Method
A-+W
WDD
W-A
AVG
MDD (baseline)
85.86
96.12
65.20
82.39
Ours w/o IAS
87.41
96.44
68.42
84.09
Ours w/o CA t
90.58
96.55
69,99
85.71
Ours f
89.61
97.08
70.04
85.58
We adopt class-balanced source sampling on all these meth- ods.
Furthermore, as our PAT method is an independent module, which could be combined with other do- main alignment techniques such as CDAN and Sentry. We integrate PAT with CDAN and CDANE, and report the experimental results on the RS-UT Office-Home dataset in Table 9 Results show that PAT module could significantly improve the performance: 54.07% +6..8% for CDAN, and
13

---

Under review as a conference paper at ICLR 2022
55.70%-662.61% % for CDANE. The results in Table 10 further validate that our PAT method could boost the performance of the state-of-the-ae CDA method Sentry.
Table 9: Per-class average accuracy CDAE, CDANE and our methods on imbalanced Office-
Home dataset with RS- label shift. Bold denotes the best performing method.
Model
Rw-Pr
Pr-CI
Cl-+rr
AVG
CADN
70.78
38.46
52.99
54.07
CDAN+PAT (ours)
77.44
42.16
61.54
60.38
CDANE
72.16
40.39
54.57
55,70
CDANE+PAT (ours)
78.59
44.20
65.04
62.61
Table 10: Per-class average accuracy of Sentry and our method on DomainNet dataset. Bold denotes the best performing method.
Model
R-C
P-R
S-R
AVG
Sentry
83.89
90.27
90.41
88.19
Sentry PAT (ours)
86.42
91.33
91.37
89.71
14


Reviews:
[{"review_id": "i4IEV1G-_SO", "reviewer": "Reviewer_p4yb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. The interpolation based data augmentation method is interesting. \n2. The overall approach is simple and easy to implement. \n\nWeakness:\n1. Novelty is limited. The interpolation is similar to mix-up and follow-up works. Relation with mix-up should be discussed here.\n2. The proposed method does not outperform the baseline Sentry. \n3. Some statements are based on intuition but not well supported. For instance, the proposed method is claimed to help the model generalize better on the target domain for minority classes but the corresponding experiments are lacking. ", "summary_of_the_paper": "This work proposes a method for solving the UDA problem with imbalanced class, which is a sub-problem of UDA. The challenge lies in how to handle the difficulties introduced by imbalanced classes. To this end, this work proposes a new data augmentation strategy, that is taking the interpolation of two samples from the same class but from different domains as the augmented samples. The traditional MMD loss and a class centroid distance based loss are also imposed for the model training. Experiments on multiple benchmark datasets are conducted. ", "summary_of_the_review": "Overall, this work introduces a neat idea, that is taking the interpolation of two anchor samples to augment the minority class. But overall the paper is lacking in depth and the experiment results are not very convincing. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "hkDejGmYRol", "reviewer": "Reviewer_orbW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n- Since the target domain is unlabeled, the interpolation between a source sample and a target sample of the same class is not reliable, which may face the same error accumulation issue as the pseudo-labeling methods. If the pseudo-label is not correct, the augmented data may not only cause the error accumulation but also damage the domain alignment.\n\n- How does the interpolated adversarial sample generation method deal with the data imbalance issue in the target domain? It seems that the generation process that generates data merely resorts to the guidance of the class probability of the source domain data. \n\n- The interpolation method looks quite general, will this method also work well on the UDA problem?\n\n- In experiments, more discussions are required for explaining why the proposed method performs worse than Sentry (Prabhu et al., 2021).\n", "summary_of_the_paper": "This paper proposes a pairwise adversarial training approach for class-imbalanced domain adaptation. Specifically, the adversarial samples are generated from the interpolated line of the aligned pairwise source domain samples and target domain samples. The generated adversarial data can augment the training data and help enhance the robustness of models.", "summary_of_the_review": "In general, the proposed method is somewhat novel. However, the method is not very well justified and some parts of the method are unclear. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RtA_NlDGoLB", "reviewer": "Reviewer_sUHK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper solve the class-imbalanced problem from two aspects:\n1. pairwise adversarial training and generate the samples on the interpolated line from a source sample to a target sample of the same class. And samples from the minority class will have larger chance.\n2. Align the conditional feature distributions of source and target domains by explicitly matching the centroids of two domains.\n\nHowever, my major concerns are as follows:\n1. During the data generating process, the pseudo-labels of the target data are still not accurate. Won't there still exist error accumulation?\n2. The improvements of the experimental results are not obvious.", "summary_of_the_paper": "This paper proposes an adversarial data augmented method to solve the class-imbalanced problem in domain adaptation. ", "summary_of_the_review": "The novelty is limited and some statements should be explained further so that the advantage of the proposed method could be more obvious.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BWA43wuh5h", "reviewer": "Reviewer_Cnqr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. the studied problem called class-imbalanced domain adaptation is interesting and practical\n2. the proposed method especially the IAS part is simple and easy to understand\n3. the results on CDA are impressive \n\nWeakness:\n1. the part called semantic centroid alignment (CA) within the proposed PAT is commonly used in the domain adaptation field (Xie et al., 2018), and the moving average strategy and the balancing strategy are also not new\n\n2. the authors aim to address the CDA problem, however, it sounds that PAT could work well for vaninna UDA problems, the authors are suggested to perform comparison with UDA methods on these datasets. For example, the subsets of the VISDA-2017 dataset are not label-balanced, PAT could be compared with recent UDA methods [a-b] directly w.r.t. the per-class accuracy.\n\n[a]. Liang, Jian, Dapeng Hu, and Jiashi Feng. \"Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.\" International Conference on Machine Learning. PMLR, 2020.\n\n[b]. Na, Jaemin, et al. \"FixBi: Bridging Domain Spaces for Unsupervised Domain Adaptation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n3. besides, it seems that PAT is independent on MDD (adopted in this paper), could the authors further show its combination with else domain alignment techniques like CDAN? And the results in Table 4 only show 3 out of 6 tasks on the OfficeHome dataset, and the contribution of IAS seems not significant (I think IAS is main contribution of this paper).\n\n4. minor concerns: how to determine the parameters like \\alpha and \\beta within PAT, results in Fig.7 show PAT is somewhat sensitive to these parameters. ", "summary_of_the_paper": "This paper proposes a new method called Pairwise Adversarial Training (PAT) that augments training data for class-imbalanced domain adaptation (CDA). Different from vanilla unsupervised domain adaptation, the label distributions of different distributions are quite different in CDA. The proposed PAT approach mainly consists of two part, centroid alignment (CA) and interpolated adversarial samples (IAS). Experiments on several benchmarks verify the effectiveness of PAT for the CDA problem. ", "summary_of_the_review": "Class-imbalanced domain adaptation is an interesting and important problem in the UDA field. The strategy of interpolated adversarial samples (IAS) is new and interesting, however, the overall novelty of the proposed method sounds not much high for a top-tier conference. In addition, the experiments do not fully verify the effectiveness of PAT especially for IAS, thus I vote for \"weak reject.\"", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to address the imbalanced class problem in unsupervised domain adaptation. The challenge lies in how to handle the difficulties introduced by imbalanced classes. To this end, this work proposes a new data augmentation strategy by taking the interpolation of two samples from the same class but from different domains as the augmented samples. The experiments demonstrate promising performance on the class-imbalanced domain adaptation datasets.

However, there are several concerns raised by the reviewers. 1) The interpolation between a source and target sample of the same class can potentially be unreliable as the pseudo label methods. 2) Some statements are based on intuition but not well supported by either theoretical analysis or experimental evaluations. 3) The proposed method is inferior to baseline methods on some datasets, it would be helpful to have further analysis of the advantages and limitations of the proposed method. 

Overall, the paper provides some new and interesting ideas. However, given the above concerns, the novelty and significance of the paper will degenerate. More discussions on the principles behind the proposed method and more experimental studies are needed. Addressing the concerns needs a significant amount of work. Although we think the paper is not ready for ICLR in this round, we believe that the paper would be a strong one if the concerns can be well addressed.
Decision: Reject