Title: DOMAIN ADAPTATION WITH FEATURE AND LABEL DISTRIBUTION CO-ALIGNMENT

Abstract: Unsupervised knowledge transfer has a great potential to improve the generalizability of deep models to novel domains. Yet the current literature assumes that the label distribution is domain-invariant and only aligns the feature distributions or vice versa. In this paper, we explore the task of Generalized Domain Adaptation: How to transfer knowledge across different domains in the presence of both shift in the feature and label distribution? We propose a feature and label distribution CO-ALignment (COAL) model to tackle this problem. Our model leverages prototype-based conditional alignment and label distribution estimation to alleviate the negative effect of the the types of domain shift respectively. We demonstrate experimentally that when both types of shift exist in the data, COAL leads to state-of-the-art performance on several cross-domain benchmarks.

1. INTRODUCTION:

The success of deep learning models is highly dependent on the assumption that the training and testing data are i.i.d and sampled from the same distribution. In reality, they are typically collected from different but related domains, leading to a phenomenon known as domain shift (Quionero-Candela et al., 2009) . To bridge the domain gap, Unsupervised Domain Adaptation (UDA) transfers the knowledge learned from a labeled source domain to an unlabeled target domain by statistical distribution alignment (Long et al., 2015; Tzeng et al., 2014a) or adversarial alignment (Tzeng et al., 2017; Ganin & Lempitsky, 2015a; Saito et al., 2018) . Though recent UDA work has made great progress, it has mostly failed to address the case of Shift in Label Distribution, i.e., a changing prior over the labels. Denote the input data as x and output labels as y, and let the source and target domain be characterized by probability distributions p and q, respectively. The majority of methods assume that the conditional label distribution is invariant (p(y|x) = q(y|x)), and only the Shift in Feature Distribution (SFD) (p(x) = q(x)) needs to be tackled, neglecting any potential Shift in Label Distribution (SLD) (p(y) = q(y)). However, recent theoretical work (Zhao et al., 2019a) has demonstrated that tackling SLD is crucial to solving the domain adaptation problem. SLD also occurs in real applications; for example, an autonomous driving system should be able to handle changing frequencies of pedestrians and cars when adapting from a rural to a downtown area.

In this paper, we propose Generalized Domain Adaptation (GDA), a more challenging but practical domain adaptation setting where the conditional Shift in Feature Distribution and Shift in Label Distribution are required to be minimized simultaneously. Specifically, in addition to the Covariate Shift assumption (p(x) = q(x), p(y|x) = q(y|x)), we further assume p(x|y) = q(x|y) and p(y) = q(y). The main challenges of GDA are: (1) SLD exists between the source and target domain, which hampers the effectiveness of mainstream domain adaptation methods that only minimize SFD, (2) aligning the conditional feature distributions (p(x|y), q(x|y)) is difficult in the presence of SLD, and (3) when the data in one or both of the domains are unequally distributed across different categories, it is difficult to train an unbiased classifier. An overview of GDA is shown in Figure 1 .

Mainstream unsupervised domain adaptation aligns the marginal feature distributions of two domains by methods that include minimizing the Maximum Mean Discrepancy (Long et al., 2015; Tzeng et al., 2014a) , aligning high-order moments (Zellinger et al., 2017; Peng et al., 2019a) , or adversarial training (Tzeng et al., 2017; Ganin & Lempitsky, 2015a) . These methods have achieved state-of-the-art performance on several domain adaptation benchmarks (Saenko et al., 2010; Venkateswara et al., 2017; Peng et al., 2017) which have significant SFD but limited SLD. In this way, we can better align the feature distribution of each category. Due to label shift, previous methods that learn marginal domain-invariant features will incorrectly align samples from different categories, leading to negative transfer.

These models are limited when applied to the GDA task as they only align the feature distribution, ignoring the real issue of SLD which exists in many real-world applications. Another line of works (Lipton et al., 2018a; Azizzadenesheli et al., 2019a) assume that only SLD exists (p(y) = q(y)) between two domains and the conditional feature distribution is invariant (p(x|y) = q(x|y)). These methods have achieved good performance when the data in both domains are sampled from the same data distributions but under different label distributions. However, these models cannot handle the GDA task as the label distribution is not well aligned. Taking steps towards minimizing the SFD and SLD simultaneously, several works (Zhang et al., 2013; Gong et al., 2016; Wu et al., 2019) provide a theoretical analysis with additional constraints on distributions p and q, such as a linearity assumption between p(x|y) and q(x|y) (Zhang et al., 2013) , which does not necessarily hold in many real applications. In addition, no practical algorithm which can solve real-world cross-domain problems has been proposed by these papers.

We postulate that it is essential to align the conditional feature distributions as well as the label distributions to tackle the GDA task. In this work, we address GDA with SFD and SLD CO-ALignment (COAL). Specifically, our approach diminishes the negative effect of SFD and SLD with prototypebased conditional distribution alignment and label distribution estimation, respectively. First, to reduce the SFD in the context of SLD, it is essential to align the conditional rather than marginal feature distributions, to avoid the negative transfer effects which are caused by matching marginal feature distributions, according to a theoretical proof from Zhao et al. (2019a) (illustrated in Figure 2) . To this end, we propose a prototype-based method to align the conditional feature distributions of the two domains. The source prototypes are computed by learning a similarity-based classifier and the target prototypes are estimated by a minimax entropy algorithm (Saito et al., 2019b) . Second, we align the label distributions in the context of SFD by estimating the target label distribution. We incorporate feature distribution and label distribution alignment into an end-to-end deep learning framework, as illustrated in Figure 1 . Comprehensive experiments on standard crossdomain recognition benchmarks demonstrate that COAL achieves significant improvements over the state-of-the-art methods on the task of GDA.

The main contributions of this paper are highlighted as follows: (1) to the best of our knowledge, we provide the first practical solution and set of benchmarks for joint SFD and SLD in deep learning;

(2) we develop an end-to-end CO-ALignment (COAL) framework which leverages conditional prototype-based alignment to solve the problem caused by SFD and label distribution estimation to solve the problem caused by the SLD; and (3) we deliver extensive experiments and analysis to show the effectiveness of COAL, and importantly, we empirically demonstrate that state-of-the-art methods fail to align conditional feature distribution in the presence of label distribution.

2. RELATED WORK:

Unsupervised Domain Adaptation Domain adaptation aims to transfer the knowledge learned from one or more source domains to a target domain. Recently, many unsupervised domain adaptation methods have been proposed. These methods can be taxonomically divided into three categories (Wang & Deng, 2018) . The first category is the discrepancy-based approach, which leverages different measures to align the marginal feature distributions between source and target domains. Figure 2 : Overview of the proposed COAL model. Our model is trained iteratively between two steps. In step A, we forward the target samples through our model to generate the pseudo labels and mask. In step B, we train our models by self-training with the pseudo labels and mask to align the label distributions, and prototype-based conditional alignment with the minimax entropy.

Commonly used measures include Maximum Mean Discrepancy (MMD) (Long et al., 2017; Tzeng et al., 2014b ), H-divergence (Ben-David et al., 2010) , Kullback-Lerbler (KL) divergence (Zhuang et al., 2015) , and Wasserstein distance (Lee & Raginsky, 2017; Shen et al., 2017) . The second category is the adversarial-based approach (Tzeng et al., 2017; Liu & Tuzel, 2016; Peng et al., 2019b) which uses a domain discriminator to encourage domain confusion via an adversarial objective. The third category is the reconstruction-based approach. Data are reconstructed in the new domain by an encoder-decoder (Bousmalis et al., 2016; Ghifary et al., 2016) or a GAN discriminator, such as dual-GAN (Yi et al., 2017) , cycle-GAN (Zhu et al., 2017) , disco-GAN (Kim et al., 2017) , and CyCADA (Hoffman et al., 2018) . However, these methods mainly consider aligning the marginal distributions to decrease feature shift, neglecting label shift. To the best of our knowledge, we are the first the propose an end-to-end deep model to tackle both feature and label shift between the source and target domains.

Learning with Label Shift Despite its wide applicability, learning under label shift remains underexplored. Existing works tackle this challenge by importance reweighting or target distribution estimation. Specifically, Zhang et al. (2013) exploit importance reweighting to enhance knowledge transfer under label shift. Recently, Lipton et al. (2018b) introduce a test distribution estimator to detect and correct for label shift. Based on it, Azizzadenesheli et al. (2019b) propose a regularized algorithm to better estimate the target label distribution. These methods assume that the source and target domains share the same generative distributions and only differ in the marginal label distribution. In this work, we explore transfer learning between domains under domain and label shifts. In this direction, Zhao et al. (2019b) introduce a theoretical analysis to show that only learning domain-invariant features is not sufficient to solve domain adaptation task since the label priors are not aligned. In related work, Wu et al. (2019) propose asymmetrically-relaxed distribution alignment to overcomes the limitations of standard domain adaptation algorithms which aims to extract domain-invariant representations. Panareda Busto & Gall (2017) propose open set domain adaptation where the categories in the training domain and testing domain are not fully overlapped. Cao et al. (2018b) introduce partial domain adaptation where the categories in the target domain are a subset of those in the source domain.

Domain adaptation with self-training Self-training methods utilize pseudo-labels to compensate for the lack of categorical information in the target domain. The intuition is to assign pseudo-labels to unlabeled samples based on the predictions of one or more classifiers. Long et al. (2013) jointly align the marginal and conditional distributions using an MMD loss with pseudo-label refinement. Saito et al. (2017) leverage an asymmetric tri-training strategy to assign pseudo-labels to the unlabeled target domain. Xie et al. (2018) propose to assign pseudo-labels to all target samples and use them to achieve semantic alignment across domains. Zhang et al. (2018) progressively enlarge the training data with pseudo-labeled target samples assigned by the classifier from the previous training epoch and retrain the model with the enlarged training set. Recently, Chen et al. (2019a) propose to progressively label the target samples and align the prototypes of source domain and target domain to achieve domain alignment. However, to the best of our knowledge, self-training has not been applied for DA with label shift.

3. CO-ALIGNMENT OF FEATURE AND LABEL DISTRIBUTION:

In domain adaptation with shift in the feature and label distribution, we are given a source domain D S = {(x s i , y s i ) Ns i=1 } with N s labeled examples, and a target domain D T = {(x t i ) Nt i=1 } with N t unlabeled examples. We assume that p(x|y) = q(x|y) and p(y) = q(y). We aim to construct a end-to-end deep neural network which is able to transfer the knowledge learned from D S to D T , and train a classifier y = θ(x) which can minimize task risk

T (θ) = Pr (x,y)∼q [θ(x) = y].

Previous works either focus on aligning the marginal feature distributions (Long et al., 2015; Tzeng et al., 2017) or aligning the label distributions (Lipton et al., 2018b) . These approaches are not able to fully tackle the domain adaptation with shift in the feature and label distribution as they only align one of the two marginal distributions. In this work, we tackle GDA with prototype-based conditional alignment and label distribution estimation.

3.1. PROTOTYPE-BASED CONDITIONAL ALIGNMENT OF FEATURE DISTRIBUTION:

The mainstream idea in covariate-shift oriented methods is to learn domain-invariant features by aligning the marginal feature distributions, which was proved to be inferior in the presence of label shift (Zhao et al., 2019a) . Instead, we propose to align the conditional feature distributions. To this end, we leverage a similarity-based classifier to estimate p(x|y), and a minimax entropy algorithm to align it with q(x|y). We achieve conditional feature distribution alignment by aligning the source and target prototypes in an adversarial process.

Similarity-based Classifier:

The architecture of our COAL model contains a feature extractor F and a similarity-based classifier C. Similarity-or prototype-based classifiers perform well in fewshot learning settings (Chen et al., 2019b) , which motivates us to adopt them since in label-shift settings some categories can have low frequencies. Specifically, C is composed of a weight matrix W ∈ R d×c and a temperature parameter T , where d is the dimension of feature generated by F , and c is the total number of classes. Denote W as [w 1 , w 2 , ..., w c ], this matrix can be seen as c d-dimension vectors, one for each category. For each input feature F (x), we compute its similarity with the i th weight vector as s i = F (x)wi T F (x) . Then, we compute the probability of the sample being labeled as class i by h i (x) = σ( F (x)wi T F (x) ), normalizing over all the classes. Finally, we can compute the prototype-based classification loss for D S with standard cross-entropy loss:

EQUATION

The intuition behind this loss is that the higher the confidence of sample x being classified as class i, the closer the embedding of x is to w i . Hence, when optimizing Equation 1, we are reducing the intra-class variation by pushing the embedding of each sample x closer to its corresponding weight vector in W. In this way, w i can be seen as a representative data point (prototype) for p(x|y = i).

Prototype-based Conditional Alignment by Minimax Entropy Due to the lack of categorical information in the target domain, it is infeasible to utilize Equation 1 to obtain target prototypes. Instead, we propose to tackle this problem by 1) moving each source prototype to be closer to its nearby target samples, and 2) clustering target samples around this moved prototype. Inspired by Saito et al. (2019a) , we propose to achieve these two objectives jointly by entropy minimax learning. Specifically, for each sample x t ∈ D T fed into the network, we can compute the entropy of the classifier's output by

EQUATION

Larger H indicates the target samples are similar to all the weight vectors (prototypes) of C. We achieve conditional feature distributions alignment by aligning the source and target prototypes in an adversarial process: (1) we train C to maximize H, aiming to move the prototypes from the source samples towards the neighboring target samples;

(2) we train F to minimize H, aiming to make the embedding of target samples closer to their nearby prototypes. By training with these two objectives as a min-max game between C and F , we can align source and target prototypes. Practically, we add a gradient-reverse layer (Ganin & Lempitsky, 2015a) between C and F to flip the sign of gradient.

Algorithm 1: Conditional Feature Alignment with Decision Boundary Refinement Input:

F , C, D S , D T , N iter , N epoch , N b , k = k 0 Output: F , Ĉ

train F and C with D S using Equation 1for i = 0, ..., N epoch do

Step A:

compute DT = {(x t i , ŷt i , m i ) Nt i=1 } with k as in Section 3.2

Step B:

for j = 0, ..., N iter do

randomly sample mini-batch {(x s i , y s i ) N b i=1 } ∈ D S , {(x t i , ŷt i , m i ) N b i=1 } ∈ DT train F and C with Equation 4 k = min(k + k step , k max )

3.2. LABEL DISTRIBUTION ESTIMATION WITH SELF-TRAINING:

As the source label distribution p(y) is different from that of the target q(y), it is not guaranteed that the classifier C which has low risk on D S will have low error on the target domain. Intuitively, if the classifier is trained with imbalanced source data, the decision boundary will be dominated by the most frequent categories in the training data, leading to a "biased" classifier. When the classifier is applied to target domain with a different label distribution, its accuracy will degrade as it is highly biased. To tackle this problem, we employ the self-training method to estimate the target label distribution and refine the decision boundary. In addition, we leverage balanced sampling of the source data to further diminish the effect of label shift.

Self-training:

In order to refine the decision boundary, we propose to estimate the target label distribution with self-training (Zou et al., 2018) . We will assign pseudo labels ŷ to all the target samples according to the output the classifier C. As we have aligned the conditional feature distributions (p(x|y) and q(x|y)), we assume that the pseudo label distribution q(ŷ) is the approximation of the real label distribution q(y) for the target domain.

For each category, we select top-k percent of the target samples with the highest confidence scores belonging to that category. We utilize the highest probability in h(x) as the classifier's confidence on sample x. Specifically, for each pseudo-labeled sample (x, ŷ), we set its selection mask m = 1 if h(x) is among the top-k percent of all the target samples with the same pseduo-label, otherwise m = 0. Denote the pseudo-labeled target set as DT = {(x t i , ŷt i , m i ) Nt i=1 }, we leverage the data and pseudo labels from DT to train the classifier C, aiming to refine the decision boundary with target label distribution. The total loss function for classification is:

L ST = L SC + E (x,ŷ,m)∈ DT L ce (h(x), ŷ) • m (3)

where ŷ indicates the pseudo labels and m indicates selection masks. In our approach, we choose the top-k percent of the highest confidence target samples within each category, instead of universally. This is crucial to estimate the real target label distribution, otherwise, the easy-to-transfer category(ies) will dominate DT , leading to inaccurate estimation of the target label distribution.

Balanced Sampling of Source Data In addition to estimating the target label distribution, we propose a balanced sampling process to enhance label shift alignment. When coping with label shift, the label distribution of the source domain could be highly unbalanced. A classifier trained on unbalanced categories will make highly-biased predictions for the samples from the target domain (He & Garcia, 2009) . This effect also hinders the self-training process discussed above, as the label distribution estimation will also be biased. To tackle these problems, we apply a balanced mini-batch sampler to generate training data from the source domain and ensure that each source mini-batch contains roughly the same number of samples for each category.

3.3. TRAINING PROCESS:

In this section, we combine the above ideas into an end-to-end training process. Given input samples from source domain D S and target domain D T , we first initialize our network F and C with only labeled data D S . Then, we iterate between the assignment of pseudo labels and adaptive learning until convergence or reaching the maximum number of iterations. An overview of this process can be seen in Algorithm 1. The overall training objective is as follows:

Ĉ = arg min C L ST -αH F = arg min F L ST + αH (4)

3.4. THEORETICAL INSIGHTS:

Conditional Feature Alignment According to Zhao et al. (2019b) , the target error in domain adaptation is bounded by three terms: 1) source error, 2) the discrepancy between the marginal distributions and 3) the distance between the source and target optimal labeling functions. Denote h ∈ H as the hypothesis, S (•) and T (•) as the expected error of a labeling function on source and target domain, and f S and f T as the optimal labeling functions in the source and target domain. Then, we have:

EQUATION

where d Ĥ denote the discrepancy of the marginal distributions (Zhao et al., 2019b) . The bound demonstrates that the optimal labeling functions f S and f T need to generalize well in both domains, such that the term min{ S (f T ), T (f S )} can be bounded. Conventional domain adaptation approaches which only align marginal feature distribution cannot guarantee that min{ S (f T ), T (f S )} is minimized. This motivates us to align the conditional feature distribution, i.e. p(x|y) and q(x|y). Our model COALestimates and aligns the source and target prototypes to align p(x|y) and q(x|y).

Decision Boundary Refinement Theorem 4.3 in Zhao et al. (2019b) indicates that the target error T (h) can not be minimized if we only align the feature distributions and neglect the shift in label distribution. Denote d JS as the Jensen-Shannon(JS) distance between two distributions, Zhao et al. (2019b) propose:

EQUATION

This theorem demonstrates that when the divergence between label distributions d JS (p(y), q(y)) is significant, minimizing the divergence between marginal distributions d JS (p(x), q(x)) and the source task error S (h) will enlarge the target task error T (h). Motivated by this, we propose to align the empirical label distributions with self-training algorithm.

4. EXPERIMENTS:

We evaluate our CO-ALignment (COAL) model with three popular benchmarks, i.e., Digits, Office-Home (Venkateswara et al., 2017) and DomainNet (Peng et al., 2019a) . Sample images of these datasets can be found in Figure 3a . As Digits and Office-Home dataset contains limited SLD, we artificially create considerable SLD by random sub-sampling the data for each category.

4.1. SETUP:

Sub-sampling Protocol. To create significant SLD between source and target domains, we subsample the current datasets with Reversely-unbalanced Source and Unbalanced Target (RS-UT) protocol. In this setting, both the source and target domains have unbalanced label distribution, while the label distribution of the source domain is a reversed version of that of the target domain.

An illustration of this setting can be found in Figure 3 (a). We refer our reader to supplementary material for more details! Digits We select four digits datasets: MNIST (LeCun et al., 1998) , USPS (Hull, 1994) , SVHN (Netzer et al., 2011) and Synthetic Digits (SYN) (Ganin & Lempitsky, 2015b) . In this work, we investigate four domain adaptation tasks: MNIST → USPS, USPS → MNIST, SVHN → MNIST, and SYN → MNIST.

Office-Home (Venkateswara et al., 2017 ) is a dataset collected in office and home environment with 65 object classes and four domains: Real World (Rw), Clipart (Cl), Product (Pr), Art (Ar). Since the "Art" domain is too small to sample an imbalanced subset, we focus on the remaining domains and explore all the six adaptation scenarios.

DomainNet (Peng et al., 2019a ) is a large-scale testbed for domain adaptation, which contains six domains with about 0.6 million images distributed among 345 classes. Since some domains and classes contains many outliers, we select 40 commonly-seen classes from four domains: Real (R), Clipart (C), Painting (P), Sketch (S). Different from two datasets above, the SLD in DomainNet is significant enough, as illustrated in Figure 3 . Moreover, the images are directly collected from image search engine and they are in consistence with the natural label distribution in the real world.

Baselines We compare covarite and SLD CO-ALignment (COAL) with state-of-the-art domain adaptation methods that mainly aligns feature distribution: Deep Adaptation Network (DAN) (Long et al., 2015) , Joint Adaptation Network (JAN) (Long et al., 2017) , Domain Adversarial Neural Network (DANN) (Ganin & Lempitsky, 2015a) , Batch Spectral Penalization (BSP) (Chen et al., 2019c) , f-divergence Domain Adversarial Neural Network (F-DANN) (Wu et al., 2019) , Maximum Classifier Discrepancy (MCD) (Saito et al., 2018) . We also compare our model with methods that aligns the marginal label distribution: Partial Adversarial Domain Adaptation (PADA) (Cao et al., 2018a) , Black Box Shift Estimation (BBSE) (Lipton et al., 2018b) .

Implementation Details. We implement all our experiments in Pytorch1 platform. For the Digits dataset, we adopt the network architecture proposed by Saito et al. (2018) . We adopt SGD with the momentum of 0.9 and learning rate of 0.01 for the linear classifier and 0.001 for all other layers. The batch size is set as 32 for samples from each domain. For the other two datasets, we utilize ResNet- Table 3 : Per-class mean accuracy on DomainNet dataset with natural label shifts. Our method achieve 74.96% average accuracy across the 12 experiments. We observe that our model has significant performance boost on several tasks, such as R→C, and C→P. Note that DomainNet contains about 0.6 million images, it is non-trivial to have even one percent performance boost. 50 (He et al., 2015) as our backbone network, and replace the last fully-connected layer with a randomly initialized N-way classifier layer (for N categories). We also use SGD with momentum of 0.9 while setting the learning rate to be 0.001 for linear layers and 0.0001 for all the other layers. The batch size is set as 16 for each domain. We set the α = 1 in Equation 4 and set k 0 = 5, k step = 5, k max = 50 as the parameters for self-training selection policy. For all the compared methods, we decide their hyper-parameters on the validation set of Painting → Clipart task in DomainNet.

Evaluation metric. When the target domain is highly unbalanced, conventional overall average accuracy that treats every class uniformly is not an appropriate performance metric (He & Garcia, 2008) . Therefore, we follow Dong et al. (2019) to use the per-class mean accuracy in our main results. Formally, we denote S i = n (i,i) ni as the accuracy for class i, where n (i,j) represents the number of class i samples labeled as class j, and n i = c j=1 n (i,j) represents the number of samples in class i. Then, the per-class mean accuracy is computed as S = 1 c c i=1 S i .

4.2. RESULTS:

We first show the experimental results of our model and the compared baselines on Digits dataset in Table 1 . The covariate shift between the source and target domains is related small on Digits dataset. For fair comparison, we leverage the same backbones for all the models. From the results, we can make the following observations. (1) Our model achieves 84.33% average accuracy across four experimental setting, outperforming the best-performing baselines by 8.4%! (2) Within the eight selected baselines, only DAN and DANN have marginal improvement (less than 2%) over the source-only baseline. The other six compared methods encounter the negative transfer (Pan & Yang, 2010) issue and perform worse than the source-only baselines. These results demonstrate that aligning only the marginal feature distributions or only the label distributions can not fully tackle GDA task. In contrast, our model co-aligns the conditional feature distributions and label distributions and outperforms the baselines by a large margin.

Next, we show the experimental results on more challenging real-image datasets, i.e. Office-Home and DomainNet, in Table 2 and Table 3 , respectively. In Office-Home experiments, our model achieves 58.40% average accuracy across the six GDA tasks, outperforming other compared baselines. Our model has 5.59% improvement from the source-only result. We also notice that the BSP, PADA, BBSE and MCD models perform worse than the source-only baselines, which is in consistent with the results on Digits dataset. The empirical results on Office-Home dataset demonstrate the effectiveness of co-alignment of conditional covariate shift and SLD.

In DomainNet experiments, our model get 74.96% average accuracy across the 12 experiments, outperforming the compared baselines. Note we have carefully tuned the hyper-parameters for the compared domain adaptation methods. Without the hyper-parameters tuning, i.e. directly applying the model released by the authors, these models perform worse than the source-only baselines. In addition, our model improves the performance of source-only models by 12.4%.

4.3. ANALYSIS:

Ablation Study. Our COAL method has mainly two objectives: 1) alignment of conditional feature distribution and 2) alignment of label distribution. To show the importance of these two objectives in GDA, we show the performance of our method without each of these objectives respectively on multiple tasks. The results in Table 4 showed the importance of both objectives. For example, for task USPS → MNIST, if we remove the conditional feature distribution alignment objective, the accuracy of our model will drop by 2.55%. Similarly, if we remove the label distribution alignment objective, the accuracy will drop by 2.9%. These results demonstrate that both the alignment of conditional feature distribution and label distribution are important to GDA task.

Feature Visualization. In this section, we plot the learned features with t-SNE (van der Maaten & Hinton, 2008) in Figure 4 . We investigate the Real to Clipart task in DomainNet experiment with ResNet-50 backbones. From (a)-(d), we can observe that our method has better ability to cluster each sample towards its class centroid. This confirms the effectiveness of the entropy minimization in Equation 4. From (e)-(h), we observe that our method can better align source and target features in each category, while other methods either leave the feature distributions unaligned, or incorrectly aligned samples in different categories. These results further show the importance of prototypebased conditional feature alignment for generalized domain adaptation task.

Effect of Source Balanced Sampler. Source balanced samplers described in Section 3.2 can help us tackle the bias-classifier problem caused by the imbalanced data distribution of source domain. A significant performance boost can be observed after applying the balanced sampler for our COAL model, as well as the compared baselines. In this section, we specifically show the effect of using source balanced samplers. We show in Table 5 the performance of several methods with and without source balanced samplers on 5 adaptation tasks from multiple datasets. We observe that for 22 of the total 25 tasks (5 models on 5 adaptation tasks), using source balanced samplers Different Degrees of SLD. In Section 4, we only investigated the performance of each method under a certain kind of SLD in each dataset, i.e., RS-UT. In this section, we investigate the effect of different degrees of SLD on the performance of domain adaptation methods. Specifically, we create 4 interval degrees of SLD between the BS-BT (Blanced Source and Blanced Target) and RS-UT setting. To this end, we compute the proportions of each category by linear interpolation between its proportions in BS-BT and RS-UT. We denote BS-BT and RS-UT as 0% and 100% degree of SLD respectively, and create datasets with SLDs of 20%, 40%, 60% and 80% on the linear interpolation. For fair comparison, all the datasets have the same total amount of samples. With these datasets, we evaluated the performances of different methods on the USPS → MNIST task of Digits. The results in Figure 5 show that the performances of previous domain adaptation methods will be significantly affected by SLD. For example, the accuracy of MCD drastically drops from 91.45% to 77.18%. In contrast, the performance our method is much more stable, which ranges between 93.42% and 88.12%. It shows that our method are robust to different degrees of SLD.

5. CONCLUSION:

In this paper, we first propose a generalized domain adaptation (GDA) learning schema and demonstrate the importance of GDA in practical scenarios. Then we deliver theoretically analysis to demonstrate that aligning conditional feature distributions (p(x|y), q(x|y)) and aligning the label distributions (p(y),q(y)) are essential for GDA. Towards tackling GDA task, we have proposed conditional feature distribution and label distribution CO-ALignment (COAL) approach, which leverages prototype-based conditional alignment and self-training method. Empirically, we demonstrate that previous UDA models which learn domain-invariant features by aligning the marginal distributions fail to tackle GDA task, due to the presence of shift in the label distribution. An extensive empirical evaluation on GDA benchmarks demonstrate the efficacy of the proposed model against several state-of-the-art domain adaptation algorithms.

A ADDITIONAL RESULTS:

We have shown in the paper the performance of different domain adaptation methods when conditional covariant shift and label shift exist simultaneously. To analysis the influence of label shift in unsupervised domain adaptation, in this appendix we also include the results on datasets without label shift. To this end, we also created datasets with the Balanced Source Balanced Target (BS-BT) setting. Specifically, under BS-BT every class has the same number of samples. We use this setting to sub-sample the BS-BT datasets from Digits and Office-Home. For fair comparison with the results in Section 4, we constrain that the total amount of samples of each domain is the same in BS-BT and RS-UT. We show the results in We can observe that conventional methods that focus on covariate shift can achieve decent improvement over the Source Only baseline in BS-BT setting. For example, MCD is largely improving the performance when there is no label shift between the source and target domain. However, as we observed in Table 1 of our paper, MCD is actually having negative transfer effect when applied to the more general GDA setting. This comparison demonstrates the importance of considering both covariate and label shift in our generalized setting. Please note that the results in this section are produced by models with hyper-parameters tuned on unbalanced DomainNet.

B CREATION DETAILS FOR LABEL SHIFT:

In order to create unbalanced label distribution in each dataset, inspired by Liu et al. (2019) , we follow the Paredo distribution set different proportions for each category. By using this distribution, we can create long-tailed label distribution, which is frequently seen in real applications and benchmarks (Van Horn et al., 2018; Peng et al., 2019a) .

The shape of Paredo distribution (Reed, 2001) is controlled by parameter α. Because different datasets have different amount of samples, to avoid making some classes in the unbalanced dataset to have too few samples, we use a different parameter for each dataset. Specifically, we set α = 1 for Digits dataset, and α = 100 for Office-Home dataset.

We further assign each computed proportion to each category by following the descending order of the original class index. Specifically, in the target domain of RS-UT, we assign the k th largest propotion to class k -1, with class index starting from 0. For Digits, we set the index of each class directly as the digit it represents. For Office-Home, we set the class index in alphabetical ascending order.

C IMPLEMENTATION DETAILS FOR COMPARED METHODS:

In our experiments, we use the authors' official implementations for DAN (Long et al., 2015) , and JAN (Long et al., 2017) , PADA (Cao et al., 2018b) , MCD (Saito et al., 2018) , and BSP (Chen et al., 2019c) . For DANN (Ganin & Lempitsky, 2015a) , BBSE (Lipton et al., 2018b) and FDANN (Wu et al., 2019) , we implement their method by ourselves.

We tune the hyper-parameters of each method on Painting → Clipart task in DomainNet. Specifically, for PADA, BSP, DAN, JAN and DANN we tune the weight α of the marginal feature alignment loss. We empirically find that these method achieve better performance when we set α to be 5 10 times lower than default. Intuitively, it means that we can achieve better performance under generalized domain shift setting if we relax the strength of marginal feature alignment. For MCD we tune the number of feature generator updating times n.

D DETAILED INFORMATION FOR DATASETS:

We provide detailed information for datasets in Table 3 

:

https://pytorch.org/

Figures and Tables:

Figure1: We propose the GDA setting, where we consider SFD and SLD simultaneously. To tackle this problem, we propose to use self-training to estimate and align the target label distribution, and use a prototypebased method for conditional alignment. In this way, we can better align the feature distribution of each category. Due to label shift, previous methods that learn marginal domain-invariant features will incorrectly align samples from different categories, leading to negative transfer.

Sample images of the datasets we use in our experiments.

Figure 3: (a): Image examples from Digits, Office-Home (Venkateswara et al., 2017), and Domain-Net Peng et al. (2019a). (b): illustrations of Reversely-unbalanced Source (RS) and Unbalanced Target (UT) distribution in MNIST→USPS task. (c): Natural SLD of DomainNet.

Figure 4: Feature visualization: t-SNE plot for source features, DAN features, DANN features, and COAL features on GDA task Real → Clipart. Figure (a)-(d): features from each class. Different colors denote different categories. Figure (e)-(h): features from each domain. Blue and red points represents features from the source domain and target domain, respectively.



Table content: <table><tr><td/><td/><td/><td>Step A</td><td/><td/><td/><td/><td>Step B</td><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">𝑳 𝑪𝑬 (𝑷, 𝒚 𝑺 , ෝ 𝒚 𝑻 , 𝒎)</td></tr><tr><td>𝒙 𝑻</td><td>𝑭</td><td>𝑪</td><td>Softmax</td><td>𝒂𝒓𝒈𝒎𝒂𝒙 top-𝒌 ?</td><td>𝒎 ෝ 𝒚 𝑻</td><td>𝒙 𝑺 , 𝒙 𝑻</td><td>𝑭</td><td>𝑪</td><td>Softmax</td><td>𝑷</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>Gradient Reverse</td><td/><td>-𝑯(𝑷)</td></tr><tr><td>𝑭</td><td/><td/><td>𝑪</td><td/><td/><td/><td/><td/><td colspan="2">= Gradient Path</td></tr></table>



Table content: <table><tr><td>Methods</td><td colspan="5">USPS→MNIST MNIST→USPS SVHN→MNIST SYN →MNIST AVG</td></tr><tr><td>Source Only</td><td>75.31±0.09</td><td>87.92±0.74</td><td>50.25±0.81</td><td>85.74±0.49</td><td>74.81</td></tr><tr><td>F-DANN</td><td>72.59±1.61</td><td>81.62±2.38</td><td>45.65±2.93</td><td>82.07±1.65</td><td>70.48</td></tr><tr><td>JAN</td><td>75.75±0.75</td><td>78.82±0.93</td><td>53.21±3.94</td><td>75.64±1.42</td><td>70.86</td></tr><tr><td>BBSE</td><td>75.01±3.68</td><td>78.84±10.73</td><td>49.01±2.02</td><td>85.69±0.71</td><td>72.14</td></tr><tr><td>BSP</td><td>71.99±1.52</td><td>89.74±0.77</td><td>50.61±1.67</td><td>77.30±1.20</td><td>72.41</td></tr><tr><td>PADA</td><td>73.66±0.15</td><td>78.59±0.23</td><td>54.13±1.61</td><td>85.06±0.60</td><td>72.86</td></tr><tr><td>MCD</td><td>77.18±5.65</td><td>85.34±4.07</td><td>53.52±4.23</td><td>76.37±3.48</td><td>73.10</td></tr><tr><td>DAN</td><td>79.12±1.34</td><td>87.15±1.71</td><td>53.63±1.80</td><td>80.89±2.00</td><td>75.20</td></tr><tr><td>DANN</td><td>77.28±2.13</td><td>91.88±0.74</td><td>57.16±1.83</td><td>77.60±1.29</td><td>75.98</td></tr><tr><td>COAL (Ours)</td><td>88.12±0.37</td><td>93.04±1.67</td><td>65.67±1.29</td><td>90.60±0.44</td><td>84.33</td></tr></table>

Per-class mean accuracy on Office-Home dataset with RS-UT label shifts. Our model achieve 58.40% average accuracy across six GDA tasks, outperforming other baselines. .56 72.63 86.47 65.73 70.58 86.94 73.19 70.15 85.73 75.16 70.04 74.46 COAL (Ours) 73.85 74.82 70.50 85.68 70.86 71.29 87.71 69.90 67.28 83.82 73.21 70.53 74.96

Table content: <table><tr><td>Methods</td><td>Rw→Pr</td><td>Rw→Cl</td><td>Pr→Rw</td><td>Pr→Cl</td><td>Cl→Rw</td><td>Cl→Pr</td><td>AVG</td></tr><tr><td>Source Only</td><td>69.77</td><td>38.35</td><td>67.31</td><td>35.84</td><td>53.31</td><td>52.27</td><td>52.81</td></tr><tr><td>BSP</td><td>72.80</td><td>23.82</td><td>66.19</td><td>20.05</td><td>32.59</td><td>30.36</td><td>40.97</td></tr><tr><td>PADA</td><td>60.77</td><td>32.28</td><td>57.09</td><td>26.76</td><td>40.71</td><td>38.34</td><td>42.66</td></tr><tr><td>BBSE</td><td>61.10</td><td>33.27</td><td>62.66</td><td>31.15</td><td>39.70</td><td>38.08</td><td>44.33</td></tr><tr><td>MCD</td><td>66.03</td><td>33.17</td><td>62.95</td><td>29.99</td><td>44.47</td><td>39.01</td><td>45.94</td></tr><tr><td>DAN</td><td>69.35</td><td>40.84</td><td>66.93</td><td>34.66</td><td>53.55</td><td>52.09</td><td>52.90</td></tr><tr><td>F-DANN</td><td>68.56</td><td>40.57</td><td>67.32</td><td>37.33</td><td>55.84</td><td>53.67</td><td>53.88</td></tr><tr><td>JAN</td><td>67.20</td><td>43.60</td><td>68.87</td><td>39.21</td><td>57.98</td><td>48.57</td><td>54.24</td></tr><tr><td>DANN</td><td>71.62</td><td>46.51</td><td>68.40</td><td>38.07</td><td>58.83</td><td>58.05</td><td>56.91</td></tr><tr><td>COAL (Ours)</td><td>73.65</td><td>42.58</td><td>73.26</td><td>40.61</td><td>59.22</td><td>57.33</td><td>58.40</td></tr><tr><td>Methods</td><td colspan="7">R→C R→P R→S C→R C→P C→S P→R P→C P→S S→R S→C S→P AVG</td></tr><tr><td>Source Only</td><td colspan="7">58.84 67.89 53.08 76.70 53.55 53.06 84.39 55.55 60.19 74.62 54.60 57.78 62.52</td></tr><tr><td>BBSE</td><td colspan="7">55.38 63.62 47.44 64.58 42.18 42.36 81.55 49.04 54.10 68.54 48.19 46.07 55.25</td></tr><tr><td>PADA</td><td colspan="7">65.91 67.13 58.43 74.69 53.09 52.86 79.84 59.33 57.87 76.52 66.97 61.08 64.48</td></tr><tr><td>DAN</td><td colspan="7">64.36 70.65 58.44 79.44 56.78 60.05 84.56 61.62 62.21 79.69 65.01 62.04 67.07</td></tr><tr><td>MCD</td><td colspan="7">61.97 69.33 79.78 79.78 56.61 53.66 83.38 58.31 60.98 81.74 56.27 66.78 67.38</td></tr><tr><td>FDANN</td><td colspan="7">66.15 71.80 61.53 81.85 60.06 61.22 84.46 66.81 62.84 81.38 69.62 66.50 69.52</td></tr><tr><td>JAN</td><td colspan="7">65.57 73.58 67.61 85.02 64.96 67.17 87.06 67.92 66.10 84.54 72.77 67.51 72.48</td></tr><tr><td>BSP</td><td colspan="7">67.29 73.47 69.31 86.50 67.52 70.90 86.83 70.33 68.75 84.34 72.40 71.47 74.09</td></tr><tr><td>DANN</td><td>63.37 73</td><td/><td/><td/><td/><td/><td/></tr></table>

Ablation study of different objectives in our method. We randomly select 8 sets of experiments to perform the ablation study. The results demonstrate that the prototype-based conditional alignment and self-training are both critical for GDA.

Table content: <table><tr><td>Methods</td><td/><td colspan="10">U→M M→U Syn→M Cl→Rw Pr→Rw R→C P→C P→R</td><td>AVG</td></tr><tr><td>w/o self-training</td><td/><td colspan="10">85.22 85.94 55.17 58.38 69.39 71.92 62.86 77.45 70.79</td></tr><tr><td colspan="12">w/o conditional alignment 85.57 92.28 63.34 59.41 72.11 71.34 68.50 87.14 74.96</td></tr><tr><td colspan="2">COAL (full model)</td><td colspan="10">88.12 93.04 65.67 59.81 73.46 73.65 69.90 87.71 76.42</td></tr><tr><td>Methods</td><td colspan="2">USPS→MNIST w/o with</td><td colspan="2">SVHN→MNIST w/o with</td><td>w/o</td><td colspan="2">Pr→Cl</td><td>with</td><td colspan="2">Cl→Rw w/o with</td><td>w/o</td><td>R→S</td><td>with</td></tr><tr><td>SourceOnly</td><td>71.35</td><td>74.98</td><td>50.35</td><td>50.25</td><td colspan="2">34.99</td><td colspan="2">35.84</td><td>50.64</td><td>53.31</td><td>50.16</td><td>53.08</td></tr><tr><td>DAN</td><td>64.81</td><td>79.37</td><td>22.05</td><td>48.19</td><td colspan="2">32.93</td><td colspan="2">34.66</td><td>45.18</td><td>53.55</td><td>64.78</td><td>58.44</td></tr><tr><td>DANN</td><td>42.77</td><td>77.48</td><td>27.60</td><td>56.35</td><td colspan="2">35.17</td><td colspan="2">38.07</td><td>47.19</td><td>58.05</td><td>68.92</td><td>72.63</td></tr><tr><td>MCD</td><td>20.15</td><td>79.59</td><td>44.83</td><td>51.91</td><td colspan="2">33.06</td><td colspan="2">29.99</td><td>49.57</td><td>44.47</td><td>58.50</td><td>79.78</td></tr><tr><td>COAL</td><td>87.50</td><td>88.12</td><td>60.12</td><td>65.67</td><td colspan="2">34.03</td><td colspan="2">40.61</td><td>57.67</td><td>59.22</td><td>59.23</td><td>70.50</td></tr></table>

The performance of five models w. or w/o. source balanced sampler. We observe a significant performance boost when the source balanced sampler is applied, both for our model and the compared baselines, demonstrating the effectiveness of source balanced sampler to GDA task.will improve the domain adaptation performance. Furthermore, in USPS → MNIST and SVHN → MNIST, source balance samplers can prevent the baseline methods from negative transfer. These results show the effectiveness of having a source balanced sampler when tackling GDA task.

Table content: <table><tr><td/><td>90</td><td/></tr><tr><td>Accuracy</td><td>80 85</td><td>DAN DANN MCD Ours</td></tr><tr><td/><td/><td>0% 20% 40% 60% 80% 100% Label Shift</td></tr><tr><td colspan="3">Figure 5: Performance on USPS→MNIST</td></tr><tr><td colspan="3">task with different degrees of SLD. 0% and</td></tr><tr><td colspan="3">100% denote the BS-BT and RS-UT settings</td></tr><tr><td colspan="3">respectively. Others are the linear interpo-</td></tr><tr><td colspan="3">lations of BS-BT and RS-UT. The results</td></tr><tr><td colspan="3">demonstrate that our model are more robust</td></tr><tr><td colspan="3">to different degrees of SLD.</td></tr></table>

Table 1, and 2. Class-balanced accuracy on Digit dataset with BS-BT label shifts

Table content: <table><tr><td>Methods</td><td colspan="7">USPS→MNIST MNIST→USPS SVHN→MNIST SynD→SVHN AVG</td></tr><tr><td>Source Only</td><td colspan="2">71.49±0.08</td><td>84.73±1.34</td><td colspan="2">56.95±1.52</td><td>86.33±0.56</td><td>76.00</td></tr><tr><td>FDANN</td><td colspan="2">72.13±1.88</td><td>87.30±0.99</td><td colspan="2">34.16±1.64</td><td>80.77±1.45</td><td>67.41</td></tr><tr><td>BBSE</td><td colspan="2">79.42±3.04</td><td>74.07±3.56</td><td colspan="2">53.73±3.19</td><td>87.14±0.88</td><td>71.65</td></tr><tr><td>PADA</td><td colspan="2">73.07±0.13</td><td>81.39±0.11</td><td colspan="2">56.82±0.97</td><td>85.88±0.35</td><td>74.70</td></tr><tr><td>JAN</td><td colspan="2">87.69±0.51</td><td>93.93±0.34</td><td colspan="2">56.00±4.30</td><td>82.65±3.02</td><td>77.53</td></tr><tr><td>DAN</td><td colspan="2">83.09±2.10</td><td>86.16±0.49</td><td colspan="2">63.42±2.08</td><td>86.48±0.97</td><td>78.69</td></tr><tr><td>BSP</td><td colspan="2">83.33±1.77</td><td>89.21±1.44</td><td colspan="2">63.31±2.43</td><td>93.24±0.53</td><td>81.92</td></tr><tr><td>DANN</td><td colspan="2">83.92±1.19</td><td>90.29±0.94</td><td colspan="2">68.96±3.73</td><td>93.43±0.36</td><td>84.23</td></tr><tr><td>MCD</td><td colspan="2">97.53±0.04</td><td>96.86±0.60</td><td colspan="2">88.47±1.96</td><td>96.82±0.32</td><td>94.92</td></tr><tr><td>COAL (Ours)</td><td colspan="2">92.87±0.60</td><td>95.40±0.59</td><td colspan="2">82.01±2.36</td><td>96.39±0.26</td><td>91.27</td></tr><tr><td>Methods</td><td>Rw→Pr</td><td>Rw→Cl</td><td>Pr→Rw</td><td>Pr→Cl</td><td>Cl→Rw</td><td>Cl→Pr</td><td>AVG</td></tr><tr><td>Source Only</td><td>72.91</td><td>39.21</td><td>69.84</td><td>38.16</td><td>56.06</td><td>54.08</td><td>55.04</td></tr><tr><td>PADA</td><td>64.44</td><td>33.89</td><td>59.21</td><td>30.35</td><td>41.92</td><td>41.19</td><td>45.17</td></tr><tr><td>BBSE</td><td>64.25</td><td>34.85</td><td>63.56</td><td>34.19</td><td>40.57</td><td>38.57</td><td>46.00</td></tr><tr><td>BSP</td><td>72.85</td><td>50.29</td><td>69.35</td><td>25.20</td><td>34.31</td><td>31.92</td><td>47.32</td></tr><tr><td>MCD</td><td>68.76</td><td>37.14</td><td>66.78</td><td>36.31</td><td>51.77</td><td>48.50</td><td>51.54</td></tr><tr><td>DAN</td><td>71.67</td><td>38.35</td><td>67.42</td><td>36.07</td><td>55.48</td><td>53.25</td><td>53.71</td></tr><tr><td>FDANN</td><td>71.20</td><td>39.11</td><td>67.30</td><td>33.86</td><td>60.85</td><td>53.64</td><td>54.33</td></tr><tr><td>JAN</td><td>66.69</td><td>42.28</td><td>69.61</td><td>38.65</td><td>59.42</td><td>57.19</td><td>55.64</td></tr><tr><td>DANN</td><td>74.70</td><td>47.10</td><td>70.07</td><td>39.35</td><td>60.56</td><td>54.88</td><td>57.78</td></tr><tr><td>COAL</td><td>77.44</td><td>47.42</td><td>71.23</td><td>42.87</td><td>63.46</td><td>61.24</td><td>60.61</td></tr></table>

Class-balanced accuracy on Office-Home dataset with BS-BT label shifts

Table content: <table/>

. Detailed information for datasets

Table content: <table><tr><td>Digits</td></tr></table>

Peer review (iclr2020):
The paper deals with covariate and label shift in common evaluated on some standard benchmark data. The paper is scientific sound and appears to be in good shape, just a few comments: - how is your label shift different to concept drift in supervised learning? --> if this is basically the same I would expect that you can use,mention methods from there - the plots with t-SNE are obviously colorful but do not provide a lot of information - they should be removed - I do not see a benefit - it is very common that all these methods (like yours) are provided on some kind of image data ... is there a particular reason / limitation? - I would like to see additional experiments on similar text data -> reuters - self-training is a concept from semi-supervised learning and not particular well supported in the community - how do you make sure that the result remain valid - how do you make sure that the adaptation of the labels, the covariate shift and the classifier training are not in facting cheating the result to an optimum within the optimized cost function?