Title: OPEN-SET DOMAIN ADAPTATION WITH CATEGORY-AGNOSTIC CLUSTERS Anonymous

Abstract: Unsupervised domain adaptation has received significant attention in recent years. Most of existing works tackle the closed-set scenario, assuming that the source and target domains share the exactly same categories. In practice, nevertheless, a target domain often contains samples of classes unseen in source domain (i.e., unknown class). The extension of domain adaptation from closed-set to such open-set situation is not trivial since the target samples in unknown class are not expected to align with the source. In this paper, we address this problem by augmenting the state-of-the-art domain adaptation technique, Self-Ensembling, with categoryagnostic clusters in target domain. Specifically, we present Self-Ensembling with Category-agnostic Clusters (SE-CC) -a novel architecture that steers domain adaptation with the additional guidance of category-agnostic clusters that are specific to target domain. These clustering information provides domain-specific visual cues, facilitating the generalization of Self-Ensembling for both closed-set and open-set scenarios. Technically, clustering is firstly performed over all the unlabeled target samples to obtain the category-agnostic clusters, which reveal the underlying data space structure peculiar to target domain. A clustering branch is capitalized on to ensure that the learnt representation preserves such underlying structure by matching the estimated assignment distribution over clusters to the inherent cluster distribution for each target sample. Furthermore, SE-CC enhances the learnt representation with mutual information maximization. Extensive experiments are conducted on Office and VisDA datasets for both open-set and closedset domain adaptation, and superior results are reported when comparing to the state-of-the-art approaches.

1. INTRODUCTION:

Convolutional Neural Networks (CNNs) have driven vision technologies to reach new state-ofthe-arts. The achievements, nevertheless, are on the assumption that large quantities of annotated data are accessible for model training. The assumption becomes impractical when cost-expensive and labor-intensive manual labeling is required. An alternative is to recycle off-the-shelf learnt knowledge/models in source domain for new domain(s). Unfortunately, the performance often drops significantly on a new domain, a phenomenon known as "domain shift." One feasible way to alleviate this problem is to capitalize on unsupervised domain adaptation, which leverages labeled source samples and unlabeled target samples to generalize a target model. One of the most critical limitations is that most existing models simply align data distributions between source and target domains. As a consequence, these models are only applicable in closed-set scenario (Figure 1(a) ) under the unrealistic assumption that both domains should share exactly the same set of categories. This adversely hinders the generalization of these models in open-set scenario to distinguish target samples of unknown class (unseen in source domain) from the target samples of known classes (seen in source domain).

The difficulty of open-set domain adaptation mainly originates from two aspects: 1) how to distinguish the unknown target samples from known ones while classifying the known target samples correctly? 2) how to learn a hybrid network for both closed-set and open-set domain adaptation? One straightforward way (Figure 1(b) ) to alleviate the first issue is by employing an additional binary classifier for assigning known/unknown label to each target sample Panareda Busto & Gall (2017) . All the unknown target samples are further taken as outlier and will be discarded during the adaptation from source to target. As the unknown target samples are holistically grouped as one generic class, the inherent data structure is not fully exploited. In the case when the distribution of these target samples is diverse or the semantic labels between known and unknown classes are ambiguous, the performance of binary classification is suboptimal. Instead, we novelly perform clustering over all unlabeled target samples to explicitly model the diverse semantics of both known and unknown classes in target domain, as depicted in Figure 1(c) . All target samples are firstly decomposed into clusters, and the learnt clusters, though category-agnostic, convey the discriminative knowledge of unknown and known classes specific to target domain. As such, by further steering domain adaptation with category-agnostic clusters, the learnt representations are expected to be domain-invariant for known classes, and discriminative for unknown and known classes in target domain. To address the second issue, we remould Self-Ensembling French et al. (2018) with an additional clustering branch to estimate the assignment distribution over all clusters for each target sample, which in turn refines the learnt representations to preserve inherent structure of target domain.

To this end, we present a new Self-Ensembling with Category-agnostic Clusters (SE-CC), as shown in Figure 2 . Specifically, clustering is firstly implemented to decompose all the target samples into a set of category-agnostic clusters. The underlying structure of each target sample is thus formulated as its inherent cluster distribution over all clusters, which is initially obtained by utilizing a softmax over the cosine similarities between this sample and each cluster centroid. With this, an additional clustering branch is integrated into student model of Self-Ensembling to predict the cluster assignment distribution of each target sample. For each target sample, the KL-divergence is exploited to model the mismatch between its estimated cluster assignment distribution and the inherent cluster distribution. By minimizing the KL-divergence, the learnt feature is enforced to preserve the underlying data structure in target domain. Moreover, we uniquely maximize the mutual information among the input intermediate feature map, the output classification distribution and cluster assignment distribution of target sample in student to further enhance the learnt feature representation. The whole SE-CC framework is jointly optimized.

2. RELATED WORK:

Unsupervised Domain Adaptation. One common solution for unsupervised domain adaptation in closed-set scenario is to learn transferrable feature in CNNs by minimizing domain discrepancy through Maximum Mean Discrepancy (MMD) Gretton et al. (2012) . Tzeng et al. (2014) is one of early works that integrates MMD into CNNs to learn domain invariant representation. Long et al. (2016) additionally incorporates a residual transfer module into the MMD-based adaptation of classifiers. Inspired by Goodfellow et al. (2014) , another direction of unsupervised domain adaptation is to encourage domain confusion across different domains via a domain discriminator, which is devised to predict the domain (source/target) of each input sample. In particular, a domain confusion loss Tzeng et al. (2015) in domain discriminator is devised to enforce the learnt representation to be domain invariant. Ganin & Lempitsky (2015) formulates domain confusion as a task of binary classification and utilizes a gradient reversal algorithm to optimize domain discriminator. Saito et al. (2018b) utilizes adversarial training to learn feature representations that could separate the target samples of unknown class from the known target samples. Furthermore, Baktashmotlagh et al. (2019) factorizes the source and target data into the shared and private subspace. The shared subspace models the target and source samples from known classes, while the target samples from unknown class are modeled with a private subspace, tailored to the target domain. t and x T t , before injected into student and teacher models separately. Conditional entropy is applied to x S t in student pathway and self-ensembling loss is adopted to align the classification predictions between teacher and student.

Open:

To further exploit the underlying data structure of target domain, we perform clustering to decompose the whole unlabeled target samples into a set of category-agnostic clusters (top right), which will be incorporated into Self-Ensembling to facilitate both closed-set and open-set scenarios. Specifically, an additional clustering branch is integrated into student to infer the assignment distribution over all clusters for each target sample x S t . By aligning the estimated cluster assignment distribution to the inherent cluster distribution learnt from original clusters via minimizing their KL-divergence, the feature representation is enforced to preserve the underlying data structure in target domain. Furthermore, the feature representation of student is enhanced by maximizing the mutual information among its feature map, classification and cluster assignment distributions (bottom right). The maximization is conducted at both global and local levels as detailed in Appendix A.

Summary. In summary, similar in spirit as previous methods Baktashmotlagh et al. (2019) ; Panareda Busto & Gall (2017), SE-CC utilizes unlabeled target samples for learning task-specific classifiers in the open-set scenario. Different from these approaches, SE-CC leverages categoryagnostic clusters for representation learning. The learnt feature is driven to preserve the target data structure during domain adaption. The structure preservation enables effective alignment of sample distributions within known and unknown classes, and discrimination of samples between known and unknown classes. As a by-product, the preservation, which is represented as a cluster probability distribution, is exploited to further enhance representation learning. This is achieved through maximizing the mutual information among input feature, its cluster and class probability distributions. To the best of our knowledge, there is no study yet to fully explore the advantages of category-agnostic clusters for open-set domain adaptation.

3. APPROACH: SELF-ENSEMBLING WITH CATEGORY-AGNOSTIC CLUSTERS:

In this paper, we remold Self-Ensembling to suit both closed-set and open-set scenarios by integrating category-agnostic clusters into domain adaptation procedure. An overview of our Self-Ensembling with Category-agnostic Clusters (SE-CC) model is depicted in Figure 2 .

3.1. NOTATION:

In open-set domain adaptation, we are given the labeled samples X s = {(x s , y s )} in source domain and the unlabeled samples X t = {x t } in target domain belonging to N classes, where y s is the class label of sample x s . The set of N classes is denoted as C, which consists of N -1 known classes shared between two domains and an additional unknown class that aggregates all samples of unlabeled classes. The goal of open-set domain adaptation is to learn the domain-invariant representations and classifiers for recognizing the N -1 known classes in target domain and meanwhile distinguishing the unknown target samples from known ones.

3.2. SELF-ENSEMBLING IN CLOSED-SET ADAPTATION:

We first briefly recall the method of Self-Ensembling French et al. (2018) . Self-Ensembling mainly builds upon the Mean Teacher Tarvainen & Valpola (2017) for semi-supervised learning, which consists of a student model and a teacher model with the same network architecture. The main idea behind Self-Ensembling is to encourage consistent classification predictions between teacher and student under small perturbations of the input image. In other words, despite of different augmentations imposed on a target sample, both teacher and student models should predict similar classification probability distribution over all classes. Specifically, given two perturbed target samples x S t and x T t augmented from an unlabeled sample x t , the self-ensembling loss penalizes the difference between the classification predictions of student and teacher:

EQUATION

where P S cls (x S t ) ∈ R N and P T cls (x T t ) ∈ R N denote the predicted classification distribution over N classes via the classification branch in student and teacher. During training, the student is trained using gradient descent, while the weights of the teacher are directly updated as the exponential moving average of the student weights. Inspired by Shu et al. (2018) , we additionally adopt the unsupervised conditional entropy loss to train the classification branch in student, aiming to drive the decision boundaries of the classifier far away from high-density regions in target domain.

Accordingly, the overall training loss of our Self-Ensembling is composed of supervised cross entropy loss (L CSE ) on source data, unsupervised self-ensembling loss (L SE ) and conditional entropy loss (L CDE ) of unlabeled target data, balanced with two tradeoff parameters (λ 1 and λ 2 ):

L = (xs,ys)∈S L CSE (xs, ys) + x t ∈T (λ1L SE (xt) + λ2L CDE (xt)).

(2)

3.3. SE-CC FOR OPEN-SET ADAPTATION:

Open-set is more difficult than closed-set domain adaptation because it is required to classify not only inliers but also outliers into N -1 known and one unknown classes. The most typical way is by learning a binary classifier to recognize each target sample as known/unkown class. Nevertheless, such recipe oversimplifies the problem by assuming that all unknown samples belong to one class, while leaving the inherent data distribution among them unexploited. The robustness of this approach is questionable when the unknown samples span across multiple unknown classes and may not be properly grouped as one generic class. To alleviate this issue, we perform clustering to explicitly model the diverse semantics in target domain as the distilled category-agnostic clusters, which are further integrated into Self-Ensembling to guide domain adaptation. Specifically, we design an additional clustering branch in student of Self-Ensembling to align its estimated cluster assignment distribution with the inherent cluster distribution among category-agnostic clusters.

Hence, the learnt feature representations are enforced to be domain-invariant for known classes and meanwhile more discriminative for unknown and known classes in target domain.

Category-agnostic Clusters. Clustering is an essential data analysis technique for grouping unlabeled data in unsupervised machine learning Jain et al. (1999) . Here we utilize k-means MacQueen et al. (1967) , the most popular clustering method, to decompose all unlabeled target samples X t into a set of K clusters {C k } K k=1 , where C k represents the set of target samples from the k-th cluster. Accordingly, the obtained clusters {C k } K k=1 , though category-agnostic, is still able to reveal the underlying structure tailored to target domain, where the target samples with similar semantics stay closer with local discrimination. In our implementations, we directly represent each target sample x t as the output feature (x t ) of CNNs pre-trained on ImageNet Russakovsky et al. (2015) for clustering. We also tried to refresh the clusters according to learnt features periodically (e.g., every 5 training epoches), but that did not make a major difference.

We encode the underlying structure of each target sample x t as the joint relations between this sample and all category-agnostic clusters, i.e., the inherent cluster distribution over all clusters. Specifically, for each target sample x t , we measure its inherent cluster distribution Pclu (x t ) ∈ R K through a softmax over the cosine similarities between this sample and each cluster centroid. The k-th element represents the cosine similarity between x t and the centroid µ k of k-th cluster:

EQUATION

where cos (•) is cosine similarity function and ρ is the temperature parameter of softmax for scaling. The centroid of each cluster µ k is defined as the average of all samples belonging to that cluster.

Clustering Branch. An additional branch in student, named as clustering branch, is especially designed to predict the distribution over all category-agnostic clusters for cluster assignment of each target sample x S t . Concretely, we denote the feature of target sample x S t along student pathway as x S t ∈ R M . Hence, depending on the input feature x S t , clustering branch infers its cluster assignment distribution P clu (x S t ) ∈ R K over all K clusters via a modified softmax layer Liu et al. (2017) :

EQUATION

where P k clu (x S t ) is the k-th element in P clu representing the probability of assigning target sample x S t into the k-th cluster. W k is the k-th row of the parameter matrix W ∈ R K×M in the modified softmax layer, which denotes the cluster assignment parameter matrix for the k-th cluster.

KL-divergence Loss. The clustering branch is trained with the supervision from the inherent cluster distribution of each target sample. To measure the mismatch between the estimated cluster assignment distribution and the inherent cluster distribution, a KL-divergence loss is defined as

EQUATION

By minimizing the KL-divergence loss, the learnt representation is enforced to preserve the underlying data structure of target domain, pursuing to be more discriminative for both unknown and known classes. Moreover, we incorporate the inter-cluster relationship into the KL-divergence loss as a constraint to preserve the inherent relations among the cluster assignment parameter matrices.

The spirit behind follows the philosophy that the cluster assignment parameter matrices of two semantically similar clusters should be similar. Hence, the KL-divergence loss with the constraint of inter-cluster relationships is formulated as

EQUATION

The KL-divergence loss in Eq.( 6) is further relaxed as:

EQUATION

3.4. MUTUAL INFORMATION MAXIMIZATION IN STUDENT:

Given the input feature of a target sample, the student in our SE-CC produces both classification and cluster assignment distributions via the two parallel branches in a multi-task paradigm. To further strengthen the learnt target feature in an unsupervised manner, we leverage Mutual Information Maximization (MIM) Hjelm et al. (2019) in student to maximize the mutual information among the input feature and the two output distributions. The rationale behind follows the philosophy that the global/local mutual information between input feature and output high-level features can be used to tune the feature's suitability for downstream tasks. As a result, we design a MIM module in student to simultaneously estimate and maximize the local and global mutual information among input feature map, the output classification distribution, and cluster assignment distribution.

Global Mutual Information. Technically, let x S t ∈ R H×H×D0 be the output feature map of the last convolutional layer in student model for the input target sample x S t (H: the size of height and width; D 0 : the number of channels). We encode this feature map into a global feature vector G(x S t ) ∈ R D1 via a convolutional layer (kernel size: 3 × 3; stride size: 1; filter number: D 1 ) plus an average pooling layer. Next, we concatenate the global feature vector G(x S t ) with the conditioning classification distribution P S cls (x S t ) and cluster assignment distribution P clu (x S t ). The concatenated feature will be fed into the global Mutual information discriminator for discriminating whether the input global feature vector is aligned with the given classification and cluster assignment distributions. Here the global Mutual information discriminator is implemented with three stacked fully-connected network plus nonlinear activation. The final output score of global Mutual information discriminator is V g ([G(x S t ), P S cls (x S t ), P clu (x S t )]), which represents the probability of discriminating the real input feature with matched classification and cluster assignment distributions. As such, the global Mutual Information is estimated via Jensen-Shannon MI estimator Nowozin et al. (2016) :

EQUATION

where ϕ (•) is softplus function and G(x S t ) denotes the global feature of a different target image xS t . Local Mutual Information. In addition, we exploit the local Mutual Information among the local input feature at every spatial location, and the output classification and cluster assignment distributions. In particular, we spatially replicate the two distributions P S cls (x S t ) and P clu (x S t ) to construct H × H × N and H × H × K feature maps respectively, and then concatenate them with the input feature map x S t along the channel dimension. The concatenated feature map L(x S t , P S cls (x S t ), P clu (x S t )) ∈ R H×H×(D0+N +K) will be fed into the local Mutual information discriminator for discriminating whether each input local feature is matched with the given classification and cluster assignment distributions. The local Mutual information discriminator is constructed with three stacked convolutional layer (kernel size: 1 × 1) plus nonlinear activation. Hence the final output score map of local Mutual information discriminator is V l (L(x S t , P S cls (x S t ),

P clu (x S t ))) ∈ R H×H . The i-th element V i l (L(x S t , P S cls (x S t ), P clu (x S t )

)) in score map denotes the probability of discriminating the real input local feature at the i-th spatial location with matched classification and cluster assignment distributions. As such, the local Mutual Information is estimated as:

EQUATION

Accordingly, the final objective for MIM module is measured as the combination of local and global Mutual Information estimations, balanced with tradeoff parameter α:

EQUATION

)

Appendix A conceptually depicts the process of both local and global mutual information estimation.

3.5. TRAINING:

The overall training objective of our SE-CC integrates the cross entropy loss on source data, unsupervised self-ensembling loss, conditional entropy loss, KL-divergence loss of clustering branch in Eq.( 7 VisDA is a large-scale dataset for the challenging synthetic-real image transfer, consisting of 280k images from three domains. The synthetic images generated from 3D CAD models are taken as the training domain. The validation domain contains real images from COCO Lin et al. (2014) and the testing domain includes video frames in YTBB Real et al. (2017) . Given the fact that the ground truth of testing set are not publicly available, the synthetic images in training domain are taken as source and the COCO images in validation domain are taken as target for evaluation. In particular, for open-set adaptation, we follow the open-set setting in Peng et al. (2018) and take the 12 classes as the known classes for source & target domains, the 33 background classes as the unknown classes in source, and the other 69 COCO categories as the unknown classes in target. The known-to-unknown ratio of samples in target domain is strictly set as 1:10. Three metrics, i.e., Knwn, Mean, and Overall, are adopted for evaluation. Here Knwn denotes the accuracy averaged over all known classes, Mean is the accuracy averaged over all known & unknown classes, and Overall is the accuracy over all target samples. For closed-set adaptation, we report the accuracy of all the 12 classes for adaptation, as in the closed-set setting of Peng et al. (2018) . We utilize ResNet152 as the backbone of CNNs for clustering and adaptation in both closed-set and open-set scenarios.

4.1. PERFORMANCE COMPARISON:

Open-Set Domain Adaptation on Office. The performances of different models on Office for open-set adaptation are shown in Table 1 . It is worth noting that AODA adopts a different open-set setting where unknown source samples are absent. For fair comparison with AODA, we additionally include a variant of our SE-CC (dubbed as SE-CC ♦ ) which learns classifier without unknown source samples. Specifically, the classifier in SE-CC ♦ is naturally able to recognize only the N-1 known classes and the target samples will be recognized as unknown if the predicted probability is lower than the threshold for any class as performed in open set SVM Jain et al. ( 2014).

Overall, the results across two metrics consistently indicate that our SE-CC obtains better performances against other state-of-the-art closed-set adaptation models (RTN and RevGrad) and open-set adaptation methods (AODA, ATI-λ, and FRODA) on most transfer directions. Please also note that our SE-CC improves the classification accuracy evidently on the harder transfers, e.g., D → A and W → A, where the two domains are substantially different. The results generally highlight the key advantage of exploiting underlying target data structure implicit in category-agnostic clusters for open-set domain adaptation. Such design makes the learnt feature representation to be domaininvariant for known classes while discriminative enough to segregate target samples from known and unknown classes. Specifically, by aligning the data distributions between source and target domains, RTN and RevGrad exhibit better performance than Source-only that trains classifier only on source data while leaving unlabeled target data unexploited. By rejecting unknown target samples as outliers and aligning data distributions only for inliers, the open-set adaptation techniques (AODA, ATI-λ, and FRODA) outperform RTN and RevGrad. This confirms the effectiveness of excluding unknown target samples from the known target samples during domain adaptation in open-set scenario. Nevertheless, AODA, ATI-λ, and FRODA are still inferior to our SE-CC which steers the domain adaptation by injecting the distribution of category-agnostic clusters as a constraint for feature learning and alignment. Closed-Set Domain Adaptation on Office and VisDA. To further verify the generality of our proposed SE-CC, we additionally conduct experiments for domain adaptation in closed-set scenario. Tables 3 and 4 show the performance comparisons on Office and VisDA datasets for closed-set domain adaptation. Similar to the observations for open-set domain adaptation task on these two datasets, our SE-CC achieves better performances than other state-of-the-art closed-set adaptation techniques. The results basically demonstrate the advantage of exploiting the underlying data structure in target domain via category-agnostic clusters, for domain adaptation, even on closed-set scenario without any diverse and ambiguous unknown samples. 

5. CONCLUSION:

We have presented Self-Ensembling with Category-agnostic Clusters (SE-CC), which exploits the category-agnostic clusters in target domain for domain adaptation in both open-set and closed-set scenarios. Particularly, we study the problem from the viewpoint of how to separate unknown target samples from known ones and how to learn a hybrid network that nicely integrates category-agnostic clusters into Self-Ensembling. We initially perform clustering to decompose all target samples into a set of category-agnostic clusters. Next, an additional clustering branch is integrated into student model to align the estimated cluster assignment distribution to the inherent cluster distribution implicit in category-agnostic clusters. That enforces the learnt feature to preserve the underlying data structure in target domain. Moreover, the mutual information among the input feature, the outputs of classification and clustering branches is exploited to further enhance the learnt feature. Experiments conducted on Office and VisDA for both open-set and closed-set adaptation tasks verify our proposal. Performance improvements are observed when comparing to state-of-the-art techniques.

A GLOBAL/LOCAL MUTUAL INFORMATION MAXIMIZATION:

In this section, we illustrate the detailed frameworks for global and local mutual information estimation in Figure 3 11) and α in Eq.( 10) on two datasets for open-set and closed-set adaptation tasks. In particular, the number of clusters (K) is determined using Gap statistics method. λ 1 = 10 is fixed based on French et al. (2018) for all the experiments, and the other four parameters are tuned as in Hjelm et al. (2019) ; Shu et al. (2018) . We restrict the hyper-parameter search for each transfer in range of λ 2 = {10 -2 , 10 -1 , 1}, λ 3 = {10 -2 , 10 -1 , 1}, λ 4 = {10 -4 , 10 -3 , 10 -2 }, and α = {1, 5, 10}.

C EXPERIMENTAL ANALYSIS:

Evaluation of Clustering Branch. To study how the design of loss function in clustering branch affects the performance, we compare the use of KL-divergence in our proposed SE-CC with L 1 and L 2 distance. The results in Table 7 (a) verify that KL-divergence is a better measure of mismatch between the classification and cluster assignment distributions than L 1 and L 2 distance, which yield inferior performance.

Evaluation of Mutual Information Maximization. Next, we evaluate different variants of MIM module in our SE-CC by estimating mutual information between input feature and different outputs, as shown in Table 7 (b). CLS, CLU and CLS+CLU estimates the local and global mutual information between input feature and the output of classification branch, the output of clustering branch, and the combined output of two branches, respectively. Compared to our SE-CC without MIM module (Knwn: 69.3%, Mean: 69.3%, and Overall: 69.1%), CLS and CLU slightly improves the performances by additionally exploiting the mutual information between input feature and the output of each branch. Furthermore, CLS+CLU obtains a larger performance boost, when combining the outputs from both branches for mutual information estimation. The results demonstrate the merit of exploiting the mutual information among the input feature and the combined outputs of two downstream tasks (i.e., classification and cluster assignment) in our MIM module. 

:



Figures and Tables:

Figure 1: A comparison between (a) closed-set domain adaptation, (b) existing methods for openset domain adaptation, and (c) our open-set domain adaptation with category-agnostic clusters.

-Set Domain Adaptation. The task of open-set domain adaptation goes beyond the traditional domain adaptation to tackle a realistic open-set scenario, in which the target domain includes numerous samples from completely new and unknown classes not present in source domain. Panareda Busto & Gall (2017) is one of the early attempts to tackle the realistic open-set scenario. Busto et al. additionally exploit the assignments of target samples as know/unknown classes when learning the mapping of known classes from source to target domain. Later on,

Figure 2: An overview of our SE-CC. Each labeled source image is fed into student model to train the classifier with cross entropy. Each unlabeled target image xt is transformed into two perturbed samples, i.e., x St and x T t , before injected into student and teacher models separately. Conditional entropy is applied to x S t in student pathway and self-ensembling loss is adopted to align the classification predictions between teacher and student.To further exploit the underlying data structure of target domain, we perform clustering to decompose the whole unlabeled target samples into a set of category-agnostic clusters (top right), which will be incorporated into Self-Ensembling to facilitate both closed-set and open-set scenarios. Specifically, an additional clustering branch is integrated into student to infer the assignment distribution over all clusters for each target sample x S t . By aligning the estimated cluster assignment distribution to the inherent cluster distribution learnt from original clusters via minimizing their KL-divergence, the feature representation is enforced to preserve the underlying data structure in target domain. Furthermore, the feature representation of student is enhanced by maximizing the mutual information among its feature map, classification and cluster assignment distributions (bottom right). The maximization is conducted at both global and local levels as detailed in Appendix A.

), and the local & global Mutual Information estimation in Eq.(10) on target data: xt) + λ2LCDE(xt)) + λ3LKL -λ4LMIM , (11) where λ 3 and λ 4 are tradeoff parameters. 4 EXPERIMENTS We empirically verify the merit of our SE-CC by conducting experiments on Office Saenko et al. (2010) and VisDA Peng et al. (2018) datasets for both open-set and closed-set domain adaptation. Office is the standard benchmark for domain adaptation, which contains 4,110 images from 31 categories. They are collected from three domains: Amazon (A), DSLR (D), and Webcam (W). Six directions of transfer among them are evaluated for both open-set and closed-set adaptation. For open-set adaptation, as in Panareda Busto & Gall (2017), we firstly take 10 classes as the known classes shared between source and target domains. In alphabetical order, the classes with labels 11-20 are taken as the unknown classes in source, and the ones with labels 21-31 are unknown classes in target. Two metrics OS and OS*, are adopted for evaluation (OS: the accuracy on all known & unknown target samples; OS*: the accuracy on the target samples of the 10 known classes). We adopt AlexNetKrizhevsky et al. (2012) pre-trained on ImageNetRussakovsky et al. (2015) as the basic CNNs architecture for clustering and adaptation. For closed-set adaptation, we followLong et al. (2017) and report accuracy on target domain over all 31 classes. The basic architecture of CNNs for clustering and adaptation is ResNet50He et al. (2016) pre-trained on ImageNet.

Figure 3: Framework of global mutual information estimation in our SE-CC.

Figure 4: Framework of local mutual information estimation in our SE-CC.

Figure 5: The t-SNE visualization of features learnt by (a) Source-only, (b) SE, and (c) SE-CC on VisDA dataset for open-set domain adaptation.

Performance comparison with the state of arts on Office for open-set domain adaptation. ♦ indicates a different open-set setting without unknown source examples.

Table content: <table><tr><td>Method</td><td colspan="14">A → D OS OS* OS OS* OS OS* OS OS* OS OS* OS OS* OS OS* A → W D → A D → W W → A W → D Avg</td></tr><tr><td>Source-only</td><td colspan="14">67.1 67.0 64.6 63.8 61.9 60.7 90.6 92.3 60.2 59.7 96.7 98.7 73.5 73.7</td></tr><tr><td>RTN Long et al. (2016)</td><td colspan="14">76.6 74.7 73.0 70.8 57.2 53.8 89.0 88.1 62.4 60.2 98.8 98.3 76.2 74.3</td></tr><tr><td colspan="15">RevGrad Ganin &amp; Lempitsky (2015) 78.3 77.3 75.9 73.8 57.6 54.1 89.8 88.9 64.0 61.8 98.7 98.0 77.4 75.7</td></tr><tr><td>AODA ♦ Saito et al. (2018b)</td><td colspan="14">76.6 76.4 74.9 74.3 62.5 62.3 94.4 94.6 81.4 81.2 96.8 96.9 81.1 80.9</td></tr><tr><td colspan="15">ATI-λ Panareda Busto &amp; Gall (2017) 79.8 79.2 77.6 76.5 71.3 70.0 93.5 93.2 76.7 76.5 98.3 99.2 82.9 82.4</td></tr><tr><td colspan="2">FRODA Baktashmotlagh et al. (2019) 88.0</td><td>-</td><td>78.7</td><td>-</td><td>76.5</td><td>-</td><td>98.0</td><td>-</td><td>73.7</td><td>-</td><td>94.6</td><td>-</td><td>84.9</td><td>-</td></tr><tr><td>SE-CC ♦</td><td colspan="14">80.6 84.0 82.4 84.2 83.2 90.3 92.9 96.6 82.7 85.9 96.8 99.1 86.4 90.0</td></tr><tr><td>SE-CC</td><td colspan="14">85.3 84.5 85.1 84.3 87.9 89.5 97.7 97.8 86.8 87.5 99.4 99.6 90.4 90.5</td></tr></table>

Performance comparison with the state of arts on VisDA for open-set domain adaptation (Known-to-Unknown Ratio = 1:10). ♦ indicates a different open-set setting without unknown source examples. † indicates the results are referred from the official leaderboard VisDA (2018) (visda SE and visda AODA).

Table content: <table><tr><td>Method</td><td colspan="2">aero bike bus car horse knife mbike person plant skbrd train truck unk Knwn Mean Overall</td></tr><tr><td>Source-only</td><td colspan="2">53.8 54.2 50.3 48.7 72.7 5.3 82.0 27.0 49.6 43.4 78.0 5.1 44.2 46.9 47.3 44.8</td></tr><tr><td colspan="3">RevGrad Ganin &amp; Lempitsky (2015) 33.0 57.3 44.1 33.9 72.1 46.9 82.2 26.8 36.8 50.4 89.4 9.8 47.8 48.6 48.5 47.8</td></tr><tr><td>RTN Long et al. (2016)</td><td colspan="2">49.2 72.6 66.5 39.5 80.8 18.8 73.8 56.8 47.4 45.2 74.0 4.5 48.7 52.4 52.1 49.0</td></tr><tr><td>SE  † French et al. (2018)</td><td colspan="2">94.2 74.1 86.1 68.1 91.0 26.1 95.2 46.0 85.0 40.4 79.2 11.0 51.0 66.4 65.2 52.7</td></tr><tr><td>AODA ♦  † Saito et al. (2018b)</td><td>80.2 63.1 59.1 63.1 83.2 12.1 89.1</td><td>5.0 61.0 14.0 79.2 0.0 69.0 50.8 52.2 67.6</td></tr><tr><td colspan="3">ATI-λ Panareda Busto &amp; Gall (2017) 85.7 74.9 60.3 49.9 80.0 19.3 88.8 40.8 54.0 59.2 66.4 18.2 59.5 58.1 58.2 59.3</td></tr><tr><td>SE-CC ♦</td><td colspan="2">82.1 80.7 59.7 50.0 80.6 36.7 83.1 56.2 56.6 21.9 57.7 4.0 70.6 55.8 56.9 69.2</td></tr><tr><td>SE-CC</td><td colspan="2">94.2 79.0 83.4 70.7 91.0 43.5 89.3 73.3 69.4 58.8 79.4 12.8 71.6 70.4 70.5 71.6</td></tr></table>

Performance comparison with the state of arts on Office dataset for closed-set domain adaptation.

Table content: <table><tr><td>Method</td><td colspan="6">A → D A → W D → A D → W W → A W → D Avg</td></tr><tr><td>RTN Long et al. (2016)</td><td>77.5</td><td>84.5</td><td>66.2</td><td>96.8</td><td>64.8</td><td>99.4 81.6</td></tr><tr><td colspan="2">RevGrad Ganin &amp; Lempitsky (2015) 79.7</td><td>82.0</td><td>68.2</td><td>96.9</td><td>67.4</td><td>99.1 82.2</td></tr><tr><td>JAN Long et al. (2017)</td><td>85.1</td><td>86.0</td><td>69.2</td><td>96.7</td><td>70.7</td><td>99.7 84.6</td></tr><tr><td>SimNet Pinheiro (2018)</td><td>85.3</td><td>88.6</td><td>73.4</td><td>98.2</td><td>71.8</td><td>99.7 86.2</td></tr><tr><td colspan="2">GTA Sankaranarayanan et al. (2018) 87.7</td><td>89.5</td><td>72.8</td><td>97.9</td><td>71.4</td><td>99.8 86.5</td></tr><tr><td>iCAN Zhang et al. (2018)</td><td>90.1</td><td>92.5</td><td>72.1</td><td>98.8</td><td>69.9</td><td>100 87.2</td></tr><tr><td>SE-CC</td><td>91.6</td><td>90.8</td><td>74.2</td><td>99.0</td><td>73.2</td><td>100 88.1</td></tr></table>

Performance comparison with the state of arts on VisDA dataset for closed-set domain adaptation.Open-Set Domain Adaptation on VisDA. The performance comparison on VisDA for open-set adaptation is summarized in Table2. Our SE-CC performs consistently better than other methods across all the three metrics. In particular, the Mean accuracy averaged over 12 known classes plus one unknown class of our SE-CC can achieve 70.5%, making the absolute improvement over the best closed-set adaptation method (SE) and open-set adaptation approach (ATI-λ) by 5.3% and 12.3%, respectively. Similar to the observations on Office for open-set adaptation, the open-set adaptation approaches (AODA and ATI-λ) exhibit better performance than RTN and RevGrad, by additionally separating unknown target samples from known target samples for open-set adaptation. Note that although the closed-set technique SE achieves higher Mean per-category accuracy than the open-set techniques (AODA and ATI-λ), the Overall accuracy over all target samples of SE are still worse than open-set techniques. This is because SE aligns unknown samples across different domains and thus fails to recognize unknown target samples. Furthermore, by integrating category-agnostic clusters into SE and steering domain adaptation to preserve the underlying target data structure of both known and unknown classes, SE-CC boosts the performances in terms of all metrics.

Table content: <table><tr><td>Method</td><td colspan="5">aero bike bus car horse knife mbike person plant skbrd train truck Mean</td></tr><tr><td>Source-only</td><td>67.1 51.4 50.8 64.5 83.4 13.0</td><td>89.9</td><td>34.4</td><td>78.8 47.0 88.1 2.0</td><td>55.9</td></tr><tr><td colspan="2">RevGrad Ganin &amp; Lempitsky (2015) 81.9 77.7 82.8 44.3 81.2 29.5</td><td>65.1</td><td>28.6</td><td>51.9 54.6 82.8 7.8</td><td>57.4</td></tr><tr><td>RTN Long et al. (2016)</td><td>89.1 56.4 72.4 69.7 77.9 49.5</td><td>87.7</td><td>13.0</td><td>88.1 77.4 86.7 7.2</td><td>64.6</td></tr><tr><td>MCD Saito et al. (2018a)</td><td>87.0 60.9 83.7 64.0 88.9 79.6</td><td>84.7</td><td>76.9</td><td colspan="2">88.6 40.3 83.0 25.8 71.9</td></tr><tr><td>SimNet Pinheiro (2018)</td><td>94.3 82.3 73.5 47.2 87.9 49.2</td><td>75.1</td><td>79.7</td><td colspan="2">85.3 68.5 81.1 50.3 72.9</td></tr><tr><td>SE French et al. (2018)</td><td>96.2 87.8 84.4 66.5 96.1 96.1</td><td>90.5</td><td>81.5</td><td colspan="2">95.3 91.5 87.5 51.6 85.4</td></tr><tr><td>SE-CC</td><td>96.9 86.5 83.0 79.9 96.3 97.4</td><td>90.5</td><td>83.4</td><td colspan="2">94.2 94.1 88.2 56.0 87.2</td></tr></table>

Performance contribution of each design Table5details the performance improvements on VisDA by considering different designs and their contributions for open-set domain adaptation in our SE-CC. CE is a general way to enhance classifier for target domain irrespective of any domain adaptation architectures. In our case, CE improves the Mean accuracy from 65.2% to 66.3%, which demonstrates that CE is an effective choice. KL and MIM are two specific designs in our SE-CC and the performance gain of each is 3.0% and 1.2% in Mean metric. In other words, our SE-CC leads to a large performance boost of 4.2% in total in terms of Mean metric. The results verify the idea of exploiting underlying target data structure and mutual information maximization for open-set adaptation.

Table content: <table><tr><td colspan="2">(i.e., Conditional Entropy (CE), KL-divergence</td></tr><tr><td colspan="2">Loss (KL), and Mutual Information Maximization</td></tr><tr><td colspan="2">(MIM)) in SE-CC on VisDA for open-set transfer.</td></tr><tr><td colspan="2">Method CE KL MIM Knwn Mean Overall</td></tr><tr><td>SE</td><td>66.4 65.2 52.7</td></tr><tr><td>+CE</td><td>67.3 66.3 55.8</td></tr><tr><td>+KL</td><td>69.3 69.3 69.1</td></tr><tr><td>SE-CC</td><td>70.4 70.5 71.6</td></tr></table>

Hyperparameters for each task.

Table content: <table><tr><td>Task</td><td>K</td><td>λ 1</td><td>λ 2</td><td>λ 3</td><td>λ 4</td><td>α</td></tr><tr><td>Open-set domain adaptation (Office)</td><td colspan="3">25 10 0.1</td><td>1</td><td>0.001</td><td>1</td></tr><tr><td>Open-set domain adaptation (VisDA)</td><td colspan="3">500 10 0.1</td><td>1</td><td>0.01</td><td>5</td></tr><tr><td>Closed-set domain adaptation (Office)</td><td colspan="6">35 10 0.01 0.1 0.0001 1</td></tr><tr><td colspan="4">Closed-set domain adaptation (VisDA) 500 10 0.01</td><td>1</td><td>0.01</td><td>5</td></tr></table>



Table content: <table/>

Evaluation of (a) clustering branch with different loss functions (i.e., L1: L1 distance, L2: L2 distance, and KL: KL-divergence) to measure the mismatch between two distributions and (b) mutual information estimated over input feature and different outputs (i.e., CLS: output of classification branch, CLU: output of clustering branch, and CLS+CLU: combined output of classification and clustering branches) on VisDA dataset for open-set domain adaptation.

Table content: <table/>

Peer review (iclr2020):
Comments: The authors proposed a method based on self-ensembling and clustering for open-set domain adaptation. The proposed method steers category-agnostic clusters to learn the underlying structure of test samples, which is used as a constraint to learn better representation. Experiments are compared with different baseline methods and comprehensive analysis is carried out. Pros: 1. The proposed method incorporates the clustering method into model training to get better representation to keep the original data structure in the target domain. 2. Sufficient experiments are compared with different baseline methods on different datasets. The performance of the proposed method is good in most settings. Especially, on the close-set domain adaptation task, the method is also good compared with baseline methods. Besides, the authors also analyze the influence of different factors on the experimental results. For example, different distance metrics in the clustering branch and different branches in the MIM part. Cons: 1. The whole framework is complicated and seems a combined method with different existing components. The necessity and importance of different components are not clearly claimed. How to determine the choice of hyperparameters in the unsupervised domain adaptation setting? The performance would be very sensitive and unreliable. There is no sensitivity analysis for performance. 2. For the concept of agnostic clustering, the author may have an incorrect understanding. In the Probably Approximately Correct (or PAC) learning model, the assumption is that the data distribution over labeled examples is correctly classified by some fixed but unknown concept in some concept classes, e.g., by a linear separator. In the agnostic setting, however, the assumption is weakened to the hope that most of the data is correctly classified by some fixed but unknown concept in some concept space, and the goal is to compete with the best concept in the class by an efficient algorithm. I suggest two papers involved with agnostic clustering for the authors: Balcan, M. F., Röglin, H., & Teng, S. H. (2009, October). Agnostic clustering. In International Conference on Algorithmic Learning Theory (pp. 384-398). Springer, Berlin, Heidelberg. Kothari, P.K. and Steinhardt, J., 2017. Better agnostic clustering via relaxed tensor norms. arXiv preprint arXiv:1711.07465. 3. Technically, I got lost on Eq. (3), which also is a very important step in the proposed SE-CC framework. How do the authors perform clustering? Why do the authors use such an update policy for centres? If it follows the k-means model, the centre update policy needs to be reproduced after you use a different metric function. However, I found the authors still used the average value update policy. It is very strange and unclear. 4. For the main technique equation (5), it is very sudden that the authors use KL to do minimization. If the number of is small, the L_KL may lead to a small value. There must be enough samples for the following optimization. However, the sample numbers of x_T^S and x_t is unclear. 5. The writing may be improved and some parts are not clearly explained. For example, 1) in the *Summary* part of section 2, *The structure preservation enables effective alignment of sample distributions within known and unknown classes*; 2) It would be better, if the A -> D, A -> W and others are explained in the caption of Table 1. Questions: 1. In the introduction part, the authors claimed the binary classifier cannot get good performance because of ambiguous semantic labels between known and unknown classes. Why is the clustering method able to tackle the samples with ambiguous semantic labels? Do the authors have any proof? 2. In figure 2, 1) Is the MI maximization discriminator is necessary? The authors claimed that this constraint can enhance the learned feature representation. What does *enhance* mean? 2) How can we understand the results (real or fake) of Mutual Information Maximization after the discriminator? 3) How can you choose the input image hat_{x}_t^S? Why do you need the MI constraint between only one image and the input image, not between all the other images(excluding the input image only) and the input image? 3. In MI optimization, a global and local strategy is applied. It is unclear why the authors use these steps and what are their convergence conditions and results? 4. The authors claimed that *Hence, the learned feature representations are enforced to be domain-invariant for known classes and meanwhile more discriminative for unknown and known classes in target domain*. How can you guarantee this? I can not see why the representations are more discriminative for unknown and known classes in the target domain. 5. In table 2, 1) why is the performance of different classes so diverse? Some are around 90%, some are around 10%. 2) SE is better than the proposed method in many classes. Could you explain more?