Title: UNSUPERVISED DOMAIN ADAPTATION FOR DISTANCE METRIC LEARNING

Abstract: Unsupervised domain adaptation is a promising avenue to enhance the performance of deep neural networks on a target domain, using labels only from a source domain. However, the two predominant methods, domain discrepancy reduction learning and semi-supervised learning, are not readily applicable when source and target domains do not share a common label space. This paper addresses the above scenario by learning a representation space that retains discriminative power on both the (labeled) source and (unlabeled) target domains while keeping representations for the two domains well-separated. Inspired by a theoretical analysis, we first reformulate the disjoint classification task, where the source and target domains correspond to non-overlapping class labels, to a verification one. To handle both within and cross domain verifications, we propose a Feature Transfer Network (FTN) to separate the target feature space from the original source space while aligned with a transformed source space. Moreover, we present a non-parametric multi-class entropy minimization loss to further boost the discriminative power of FTNs on the target domain. In experiments, we first illustrate how FTN works in a controlled setting of adapting from MNIST-M to MNIST with disjoint digit classes between the two domains and then demonstrate the effectiveness of FTNs through state-of-the-art performances on a cross-ethnicity face recognition problem.

1. INTRODUCTION:

Despite strong performances on facial analysis using deep neural networks (Taigman et al., 2014; Sun et al., 2014; Schroff et al., 2015; Parkhi et al., 2015) , learning a model that generalizes across variations in attributes like ethnicity, gender or age remains a challenge. For example, it is reported by Buolamwini & Gebru (2018) that commercial engines tend to make mistakes at detecting gender for images of darker-skinned females. Such biases have enormous social consequences, such as conscious or unconscious discrimination in law enforcement, surveillance or security (WIRED, 2018a;b; NYTimes, 2018; GIZMODO, 2018) . A typical solution is to collect and annotate more data along the underrepresented dimension, but such efforts are laborious and time consuming. This paper proposes a novel deep unsupervised domain adaptation approach to overcome such biases in face verification and identification.

Deep domain adaptation (Long et al., 2013; 2015; 2016; Tzeng et al., 2015; Ganin et al., 2016; Sohn et al., 2017; Haeusser et al., 2017; Luo et al., 2017) allows porting a deep neural network to a target domain without extensive labeling efforts. Currently, there are two predominant approaches to deep domain adaptation. The first approach, domain divergence reduction learning, is motivated by the works of (Ben- David et al., 2007; 2010) . It aims to reduce the source-target domain divergence using domain adversarial training (Ganin et al., 2016; Sohn et al., 2017; Tran et al., 2018) or maximum mean discrepancy minimization (Tzeng et al., 2015; Long et al., 2015; 2016) , while leveraging supervised loss from labeled source examples to maintain feature space discriminative power. Since the theoretical basis of this approach (Ben-David et al., 2007) assumes a common task between domains, it is usually applied to a classification problem where the source and target domains share the same label space and task definition. The second approach considers domain adaptation as a semi-supervised learning problem and applies techniques such as entropy minimization (Grandvalet & Bengio, 2005) or self-ensembling (Laine & Aila, 2017; Tarvainen & Valpola, 2017; French et al., 2018) on target examples to encourage decisive and consistent predictions. However, neither of those are applicable if the label spaces of source and target domains do not align. As a motivating example, consider a cross-ethnicity generalization of face recognition problem, where the source ethnicity (e.g., Caucasian) contains labeled examples and the target ethnicity (e.g., African-American) contains only unlabeled examples. When it is cast as a classification problem, the tasks of the two domains are different due to disjoint label spaces. Moreover, examples from different ethnicity domains almost certainly belong to different identity classes. To satisfy such additional label constraints, representations of examples from different domains should ideally be distant from each other in the embedding space, which conflicts with the requirements of domain divergence reduction learning as well as entropy minimization on target examples with source domain class labels.

In this work, we aim at learning a shared representation space between a source and target domain with disjoint label spaces that not only remains discriminative over both domains but also keep representations of examples from different domains well-separated, when provided with additional label constraints. Firstly, to overcome the limitation of domain adversarial neural network (DANN) (Ganin et al., 2016) , we propose to convert disjoint classification tasks (i.e., the source and target domains correspond to non-overlapping class labels) into a unified binary verification task. We term adaptation across such source and target domains as cross-domain distance metric adaptation (CD2MA). We demonstrate a generalization of the theory of domain adaptation (Ben-David et al., 2007) to our setup, which bounds the empirical risk for within-domain verification of two examples drawn from the unlabeled target domain. While the theory does not guarantee verification between examples from different domains, we propose approaches that also address such cross-domain verification tasks.

To this end, we introduce a Feature Transfer Network (FTN) that separates the target features from the source features while simultaneously aligning them with an auxiliary domain of transformed source features. Specifically, we learn a shared feature extractor that maps examples from different domains to representations far apart. Simultaneously, we learn a feature transfer module that transforms the source representation space to another space used to align with the target representation space through a domain adversarial loss. By forging this alignment, the discriminative power from the augmented source representation space would ideally be transferred to the target representation space. The verification setup also allows us to introduce a novel entropy minimization loss in the form of N -pair metric loss (Sohn, 2016) , termed multi-class entropy minimization (MCEM), to further leverage unlabeled target examples whose label structure is not known. MCEM samples pairs of examples from a discovered label structure within the target domain using an offline hierarchical clustering algorithm such as HDBSCAN (Campello et al., 2013) , computes the N -pair metric loss among these examples (Sohn, 2016) , and backpropagates the resulting error derivatives.

In experiments, we first perform on a controlled setting by adapting between disjoint sets of digit classes. Specifically, we adapt from 0-4 of MNIST-M (Ganin et al., 2016) dataset to 5-9 of MNIST dataset and demonstrate the effectiveness of FTN in learning to align and separate domains. Then, we assess the impact of our proposed unsupervised CD2MA method on a challenging cross-ethnicity face recognition task, whose source domain contains face images of Caucasian identities and the target domain of non-Caucasian identities, such as African-American or East-Asian. This is an important problem since existing face recognition datasets show significant label biases towards Caucasian ethnicity, leading to sub-optimal recognition performance for other ethnicities. The proposed method demonstrates significant improvement in face verification and identification compared to a source-only baseline model and a standard DANN. Our proposed method also closely matches the performance upper bounds obtained by training with fully labeled source and target domains.

2. RELATED WORK:

Research efforts in deep domain adaptation have explored a proper metric to measure the variational distance between two domains and subsequently regularize neural networks to minimize this distance. For example, maximum mean discrepancy (Long et al., 2013; 2016; Tzeng et al., 2014; Fernando et al., 2015; Tzeng et al., 2015; Sun & Saenko, 2016) estimates the domain difference based on kernels. As another example, domain adversarial neural networks (Ganin et al., 2016; Bousmalis et al., 2016; 2017; Sohn et al., 2017; Luo et al., 2017; Tran et al., 2018) , measuring the distance using a trainable and flexible discriminator often parameterized by an MLP, have been successfully adopted for several computer vision applications, such as semantic segmentation (Hoffman et al., 2016; Tsai et al., 2018; Zhang et al., 2018) and object detection (Chen et al., 2018) . Most of those works assume a common classification task between two domains, whereas we tackle a cross-domain distance metric adaptation problem where label spaces of source and target domains are different. Moreover, our problem setting, an adaptation from labeled source to unlabeled target with disjoint label spaces, contains flavors from both domain adaptation (DA) and transfer learning (TL), following the nomenclature of (Pan et al., 2010). The difference in input distribution between source and target domains and the lack of labels in the target domain are similar to that of DA or transductive TL (Pan et al., 2010), while the difference in label distribution and task definitions between two domains is akin to inductive TL (Pan et al., 2010; Daumé III, 2007) . In our work, we formalize this problem in domain adaptation framework using verification as a common task. This is a key contribution that allows theoretical analysis on the generalization bound as presented in Section 3 and Appendix A, while allowing novel applications like cross-ethnicity face recognition.

In terms of task objective, (Hu et In terms of network design, residual transfer network (Long et al., 2016) , which learns two classifiers differ by a residual function for the source and the target domain, is closely related. However, it only tackles the scenario where source and target domains share a common label space for classification.

3. REVISITING THE THEORY OF DOMAIN ADAPTATION FOR VERIFICATION:

Under the domain adaptation assumption, Ben-David et al. (2007) show that the empirical risk on the target domain X T is bounded by the empirical risk on the source domain X S and the variational distance between the two domains, provided that the source and the target domains share the classifiers. Therefore, this bound is not applicable to our CD2MA setup where the label spaces of two domains are often different. To generalize those theoretical results to our setting, we reformulate the verification task as a binary classification task shared across two domains. This new binary classification task takes a pair of images as an input and predicts the label of 1 if the pair of images shares the same identity and 0 otherwise. Furthermore, if we now define the new source domain to be pairs of source images and the new target domain to be pairs of target images, then Theorem 1 and 2 from (Ben-David et al., 2007) can be directly carried over to bound the new target domain binary classification error in the same manner. That is, the empirical with-in target domain verification loss is bounded by with-in source domain verification loss and the variational distance between X S × X S and X T × X T . 1 Note that inputs to the binary classifier are pairs of images from the same domain. Thus, this setup only addresses adaptation of within-domain verification to unlabeled target domains.

There are two implications from the theoretical insights on domain adaptation using verification as a shared classification task. Firstly, domain adversarial training, reducing the discrepancy between the source and the target product spaces, coupled with supervised source domain binary classification loss (i.e., verification loss using source domain labels) can yield target representations with high discriminative power when performing within-domain verification. Note that in practice we approximately reduce the product space discrepancy by generic adversarial learning as done in (Ganin et al., 2016; Sohn et al., 2017) . Secondly, there is no guarantee that the aligned source and target feature spaces possess any discriminative power for cross-domain verification task. Thus, additional actions in the form of a feature transfer module and domain separation objective are required to address this issue. These two consequences together motivate the design of our proposed framework, which is introduced in the next section.

4. FEATURE TRANSFER NET: LEARNING TO ALIGN AND SEPARATE DOMAINS:

In this section, we first define the CD2MA problem setup and motivate our proposed feature transfer network (FTN). Then we elaborate on the training objectives that help our model achieve its desired properties. Lastly, we provide practical considerations to implement our proposed algorithm.

4.1. PROBLEM STATEMENT AND ALGORITHM OVERVIEW:

Recall the description of CD2MA, given source and target domain data distributions X S and X T , our goal is to verify whether two random samples x, x drawn from either of the two distributions (and we do not know which distribution x or x come from a priori) belong to the same class.

There are 3 scenarios of constructing a pair: x, x ∈ X S , x, x ∈ X T , or x ∈ X S , x ∈ X T . We refer the task of the first two cases as within-domain verification and the last as cross-domain verification.

x s

x t 0/1 0/1 D 1 D 2 Tx (g) f s f t g(f s ) x s x t L adv L sep L vrf L vrf Gen (f)

Figure 1 : Training of Feature Transfer Network (FTN) for verification, composed of feature generation module (Gen; f ), feature transfer module (Tx; g), and two domain discriminators D1 and D2. Verification objective Lvrf's are applied to source (fs) pairs and transformed source (g(fs))) pairs. Our FTN applies domain adversarial objective Ladv for domain alignment between transformed source and target domains by D1 and applies Lsep to distinguish source domain from both target and transformed source domains by D2.

If x, x ∈ X S (or X T ), we need a source (or target) domain classifier 2 . For the source domain, we are provided with adequate labeled training examples to learn a competent classifier. For the target domain, we are only given unlabeled examples. However, with our extension of Theorem 1 and 2 from (Ben-David et al., 2007), discriminative power of the classifier can be transferred to the target domain by adapting the representation spaces of X T × X T and X S × X S , that is, we can utilize the same competent classifier from the source domain to verify target domain pairs if two domains are well-aligned. For the third scenario where x ∈ X S but x ∈ X T , we assume that the two examples cannot be of the same class, which is true for problems such as cross-ethnicity face verification.

Our proposed framework, Feature Transfer Network (FTN), is designed to solve all these verification scenarios in an unified framework. FTN is composed of multiple modules as illustrated in Figure 1 . First, a feature generation module f : X → Z denoted as "Gen" in Figure 1 ideally maps X S and X T to distinguishable representation spaces, that is, f (X S ) and f (X T ) are far apart. To achieve this, we introduce a domain separation objective. 3 Next, the feature transfer module g : Z → Z denoted as "Tx" in Figure 1 transforms f (X S ) to g(f (X S )) for it to be aligned with f (X T ). To achieve this, we introduce a domain adversarial objective. Finally, we apply verification losses on f (X S ) and g(f (X S )) using classifiers h f , h g : Z × Z → {0, 1}. During testing, we compare the metric distance between f (x) and f (x ). Overall, we achieve the following desired capabilities:

• If x, x are from different domains, f (x) and f (x ) will be far away due to the functionality of the feature generation module. • If x, x ∈ X S , then f (x) and f (x ) will be close if they belong to the same class and far away otherwise, due to the discriminative power acquired from optimizing h f . • If x, x ∈ X T , then f (x) and f (x ) will be close if they belong to the same class and far otherwise, due to the discriminative power acquired by optimizing h g with domain adversarial training.

4.2. TRAINING OBJECTIVES:

We first define individual learning objectives of the proposed Feature Transfer Network and then present overall training objectives of FTN. For ease of exposition, all objectives are to be maximized.

Verification Objective. For a pair of source examples, we evaluate the verification losses at two representations spaces f (X S ) and g(f (X S )) using classifiers h f and h g as follows:

EQUATION

EQUATION

where

g i = g(f (x i )), f i = f (x i

) and y 12 = 1 if x 1 and x 2 are from the same class and 0 otherwise. While classifiers h f , h g can be parameterized by neural networks, we aim to learn a generator f and g whose embeddings can be directly used as a distance metric. Therefore, we use non-parameteric classifiers

h f = σ(f 1 f 2 ), h g = σ(g 1 g 2 ) where σ(a) = 1 1+exp(-a) .

Domain Adversarial Objective. Let D 1 : Z → (0, 1) be a domain discriminator. As mentioned earlier, D 1 is trained to discriminate distributions f (X T ) and g(f (X S )) and then produces gradient for them to be indistinguishable. The learning objectives are written as follows:

EQUATION

Note that when feature transform module is an identity mapping, i.e., g(f (x)) = f (x), Equation (3) defines the training objective of standard DANN.

Domain Separation:

Objective. The goal of this objective is to distinguish between source and target at representation spaces of generation module. To this end, we formulate the objective using another domain discriminator D 2 : Z → (0, 1):

EQUATION

Note that, in L sep , the source space f (X S ) is not only pushed apart from the target space f (X T ) but also from the augmented source space g(f (X S )) to ensure that g learns meaningful transformation of source domain representation beyond identity transformation.

Training FTN. Now we are ready to present the overall training objectives L f and L g :

EQUATION

with λ 1 for domain adversarial objective and λ 2 for domain separation objective. We use L D1 in Equation ( 3) for D 1 and

L D2 = L sep for D 2 .

We alternate updating between D 1 and (f, g, D 2 ).

4.3. PRACTICAL CONSIDERATIONS:

Preventing Mode Collapse via Feature Reconstruction Loss. The mode collapsing phenomenon with generative adversarial networks (GANs) (Goodfellow et al., 2014) has received much attention (Salimans et al., 2016) . In the context of domain adaptation, we also find it critical to treat the domain adversarial objective with care to avoid similar optimization instability.

In this work, we prevent the mode collapse issue for domain adversarial learning with an additional regularization method similar to (Sohn et al., 2017) . Assuming the representation of the source domain is already close to optimal, we regularize the features of source examples to be similar to those from the reference network f ref : X → Z, which is pretrained on labeled source data and fixed during the training of f . Furthermore, we add a similar but less emphasized (λ 4 < λ 3 ) regularization to target examples, simultaneously avoiding collapsing and allowing more room for target features to diverge from the original representations. Finally, the feature reconstruction loss is written as follows:

L recon = -λ 3 E x∈X S f (x) -f ref (x) 2 2 + λ 4 E x∈X T f (x) -f ref (x) 2 2 (6)

We empirically find that without the feature reconstruction loss, the training would become unstable, reach an early local optimum and lead to suboptimal performance (see Section 6 and Appendix C). Thus, we always include the feature reconstruction loss to train DANN or FTN models unless stated otherwise.

Replacing Verification Loss with N -pair Loss. Our theoretical analysis in Section 3 (and Appendix A) suggests to use a verification loss that compares similarity between a pair of images. In practice, however, the pairwise verification loss is too weak to learn a good deep distance metric. Following (Sohn, 2016) , we propose to replace the verification loss with an N -pair loss, defined as follows:

EQUATION

where x n and x + n are from the same class and x n and x + k , n = k, are from different classes. Replacing L vrf into L N , the training objective of FTN with N -pair loss is written as follows: 

L f = 1 2 L N (g) + L N (f ) + λ 1 L adv + λ 2 L sep + L recon , L g = L N (g) + λ 2 E X S log(1-D 2 (g)) (8)

5. ENTROPY MINIMIZATION VIA HIERARCHICAL CLUSTERING:

Entropy minimization (Grandvalet & Bengio, 2005 ) is a popular training objective in unsupervised domain adaptation: unlabeled data is trained to minimize entropy of a class prediction distribution so as to form features that convey confident decision rules. However, it is less straightforward how to apply entropy minimization when label spaces for source and target are disjoint. Motivated from Section 3, we extend entropy minimization for distance metric adaptation using verification as a common task for both domains:

EQUATION

where

p ij p ij (f ) = σ(f (x i ) f (x j ))

. This formulation encourages a more confident prediction for verifying two unlabeled images, whether or not coming from the same class.

However, recall that for the source domain, we use N -pair loss instead of pair-wise verification loss for better representation learning. Therefore, we would like to similarly incorporate the concept of Npair loss on the target domain by forging a multi-class entropy minimization (MCEM) objective. This 

EQUATION

where x n and x + n are from the same cluster and x n and x + k , n = k are from different clusters. The objective can be combined with L f in Equation ( 8) to optimize f .

6. EXPERIMENTS:

In this section, we first experiment on digit datasets as a proof of concept and compare our proposed FTN to DANN. Then, we tackle the problem of cross-ethnicity generalization in the context of face recognition to demonstrate the effectiveness of FTN. In all experiments, we use N -pair loss as defined in Equation ( 8) to update f and g for better convergence and improved performance. We also use the same learning objectives for DANN while fixing g to the identity mapping and λ 2 = 0.

6.1. PROOF OF CONCEPT: MNIST-M (0-4) TO MNIST (5-9):

To provide insights on the functionality of FTN, we conduct an experiment adapting the digits 0-4 from MNIST-M (Ganin et al., 2016) to 5-9 from MNIST. In other words, the two domains in our setting not only differ in foreground and background patterns but also contain non-overlapping digit classes, contrasting the usual adaptation setup with a shared label space. Our goal is to learn a feature space that separates the digit classes not only within each domain, but also across the two.

We construct a feature generator f composed of a CNN encoder followed by two fully-connected (FC) layers and a feature transfer module g composed of MLP with residual connections. Outputs of f and g are then fed to discriminators D 1 and D 2 parameterized by MLPs to induce domain adversarial and domain separation losses respectively. We provide more architecture details in Appendix B.1.

We visualize t-SNE plots of generator features in Figure 2 . Without an adaptation (Figure 2 (a)), features of digits from the target domain are heavily mixed with those from the source domain as , 2015) . Note that these datasets also exhibit significant ethnicity bias. 5To highlight the effectiveness of the proposed adaptation approach, we construct individual test set for CAU, AA, EA, each of which contains 10 face images from 200 identities. We refer to our testing set as the Cross-Ethnicity Faces (CEF) dataset. We apply two evaluation metrics on CEF dataset, verification accuracy and identification accuracy. For verification, following the standard protocol (Huang et al., 2007), we construct 10 splits, each containing 900 positive and 900 negative pairs, and compute the accuracy on each split using the threshold found from the other 9 splits. For identification, a pair composed of the reference and the query images from the same identity is considered correct if there is no image from different identity that has higher similarity to the reference image than the query image. We evaluate identification accuracy per ethnicity (200-way) as well as across all ethnicities (600-way).

Results. The results on CEF are summarized in Table 1 . Cross domain identification accuracy is reported in Table 2 , where we use AA and EA as negative classes when evaluating accuracy on CAU and vice versa, as a measure to indicate domain discrepancy. Among adaptation models, DANN without feature reconstruction loss (DANN\L recon ) shows unstable training and easily degenerate, which leads to only marginal improvement upon Sup C . Similar trend is observed while training FTN. Therefore, to ensure training stability, we impose L recon as a regularization term for all adaptation models. More analysis on the effectiveness of L recon is provided in Appendix C.

When testing on AA and EA with model trained on only the labeled source CAU domain (Sup C ), we observe significant performance drops in Table 1 . Meanwhile, in Table 2 , cross domain identification accuracy is much higher than within domain identification accuracy, i.e., 96.14% of AA vs. CAU is much higher than 71.92% of AA identification in Table 1 , indicating 1) significant discrepancy between the feature spaces of the source and target domains and 2) lack of discriminative power for within domain verification task on target ethnicity.

Comparing to Sup C , both DANN and FTN show moderate improvement when testing on AA and EA from CEF (Table 1 ), demonstrating the effectiveness of domain adversarial learning in transferring within domain verification capability from labeled source domain to unlabeled target domain. Despite the improvement, DANN suffers a notable drawback from adversarial objective which attempts to align identities from different domains, resulting a poor cross domain identification accuracy as shown in Table 2 . In contrast, the proposed FTN achieves much higher cross domain identification accuracy, demonstrating both within and cross domain discriminative power.

Additionally, in combination with the multi-class entropy minimization (FTN+MCEM), we further boost the verification and identification accuracy over FTN on AA and EA as well as approach the accuracy of Sup C,A,E , the performance upper bound. This indicates that the HDBSCAN-based hierarchical clustering provides high quality pseudo-class labels for MCEM to be effective. Indeed, the clustering algorithm achieves F-score as high as 96.31% and 96.34% on AA and EA. We provide more in-depth analysis on the clustering strategy in Appendix D.

Finally, Table 3 reports the performance of face recognition models on standard verification and recognition benchmarks. We observe similar improvements with our proposed distance metric adaptation when only using labeled CAU, i.e., source domain, as training data. Once the task becomes more challenging thus demands more discriminative power, the advantage of our method becomes more evident, such as in the case of open-set recognition and verification at low FAR.

7. CONCLUSION:

We address the challenge of unsupervised domain adaptation when the source and the target domains have disjoint label spaces by formulating the classification problem into a verification task. We Our feature generator module is composed of 6 convolution layers and 3 max-pooling layers followed by 2 fully-connected layers. We use ReLU (Nair & Hinton, 2010) after convolution layers. The output dimension of the feature generator module is 128 and is normalized to have L2-norm of 2.

The full description of the generator module is in Table S1 .

The feature transfer module maps 128 dimensional vector into the same dimensional vector using two fully-connected layers (128 -256 -256 -128) and residual connection as in Figure 1(a) . Discriminator architectures are similar to that in Figure 1 (b) but with fully-connected layers whose output dimensions are 128 instead of 256.

We use Adam stochastic optimizer with learning rate of 0.0003, λ 1 = 0.3 and λ 2 = 0.03 to train FTN.

B.2 CROSS ETHNICITY FACE VERIFICATION AND RECOGNITION:

Our experimental protocols, such as data preprocessing and network architecture, closely follow those of (Sohn et al., 2017) . We preprocess face images by detecting (Yang et al., 2016) , aligning (Yu et al., 2016) , and cropping to provide face images of size 110 × 110. The data is prepared for network training by random cropping into 100 × 100 with horizontal flip with a 50% chance and converting into gray-scale.

Our feature generation module contains 38 layers of convolution with several residual blocks and max pooling layers. We use ReLU (Nair & Hinton, 2010) for most of the layers in combination with maxout nonlinearities (Goodfellow et al., 2013) . We add 7 × 7 average pooling layer on top of the last convolution layer. The output of the feature generation module is 320 dimensional vector and is normalized to have L2-norm of size 12. The full description of the model is in Table S2 .

The feature transfer module maps 320 dimensional output vector from feature generation module into the same dimensional vector using two fully-connected layers and residual connection. The architecture of feature transfer module is described in Figure 1 (a). Discriminators have similar network architecture besides different numbers of neurons and omitted residual connection.

All models, including supervised CNNs (Sup C , Sup C,A,E ), are trained with 4096-pair loss. For Sup C and Sup C,A,E , we use Adam stochastic optimizer (Kingma & Ba, 2015) with the learning rate of 0.0003 for the first 12K updates and 0.0001 and 0.00003 for the next two subsequent 3K updates.

Our feature generation module is initialized with the Sup C model, which is also used as a reference network for feature reconstruction loss as described in Section 4.3. Other modules of our model, such as feature generation module and discriminators, are initialized randomly. All modules are then updated with the learning rate of 0.00003. Hyperparameters of different models are summarized in Table S3 . As we see in Figure S2 , the performance of all models on the target ethnicities start to improve in the beginning of training from those of the pretrained reference network. Soon after, however, the accuracy starts to drop when values of either λ 3 or λ 4 are set to 0. Note that even in that situation the performance on the CAU set still remains high, which implies the failure of discriminative 

D PERFORMANCE OF UNSUPERVISED HIERARCHICAL CLUSTERING:

In this section, we provide analysis on the performance of our clustering strategy by measuring the clustering accuracy. Specifically, we measure the verification precision and recall as follows:

Precision = x1,x2∈X T 1{y 1 = y 2 , ŷ1 = ŷ2 } x1,x2∈X T 1{ŷ 1 = ŷ2 } , Recall = x1,x2∈X T 1{y 1 = y 2 , ŷ1 = ŷ2 } x1,x2∈X T 1{y 1 = y 2 } (S1)

where y i is the ground-truth class label of an example x i , and ŷi is an index of an assigned cluster. Precision computes the proportion of positive pairs among pairs assigned to the same cluster, i.e., purity of returned clusters, and recall computes the proportion of positive pairs assigned to the same cluster. Ideally, we expect high precision and high recall, i.e., high F-score, to ensure examples with the same class labels are assigned to the same cluster. Note that we only use clusters of size 5 or larger as new target classes and discard examples assigned to a cluster whose size is less than 5.

Here, in addition to our proposed clustering strategy, we also evaluate the clustering performance that clusters target examples by finding a nearest classes or examples from the source domain, which are shown to be effective for zero-shot learning (Vinyals et al., 2016) or semi-supervised domain adaptation with disjoint source and target classes (Luo et al., 2017) . In this case, we call two examples from the target domain are assigned to the same cluster if the nearest source examples are the same. We also measure the clustering performance by matching the nearest source classes.

The summary result is provided in Table S4 . Firstly, we observe extremely low precision when using source domain examples or clusters as a proxy to relate target examples. We believe that this idea of "clustering by finding the nearest source classes" works under a cross-category similarity assumption between disjoint classes of source and target domains. In other words, it assumes that there exists a certain source class closer to examples from certain target class, so that those examples from the same target class can be clustered around that source class, even though those matching source and target classes are indeed different (e.g., 3/5, 2/8, 4/9, and 0/6 in Section 6.1). Unfortunately, such an assumption does not hold for our problem, maybe due to the huge number of identity classes (60K) in the source domain.

On the other hand, using hierarchical clustering on target features achieves significantly higher precision and recall. Especially, when using embedding vectors of Sup C , we achieve 100% precision, which means that all clusters are pure even though some ground-truth classes might be separated into multiple clusters. We observe slightly lower precision using FTN features but much higher recall, achieving higher F-score overall. Further, the number of examples returned with FTN feature (253K and 195K for AA and EA, respectively) is higher than with Sup C feature (217K and 165K).

Repeating the process using feature of FTN+MCEM model improves the F-score while returning more target examples that are with cluster assignment (276K and 214K). This not only shows the 

:

Mathematical details formalizing our theoretical analysis are in the Appendix A.

Here we use the term "classifier" to denote a prediction module for the verification of a pair.

The term "domain separation" indicates that the representation space can be separated with respect to domain definitions (such as, source or target). This is unrelated to Domain Separation Network(Bousmalis et al., 2016), where it denotes the separation of the representation space into shared and private subspaces.

We ask AMT to organize the ethnicity of face images into five categories, Caucasian, African-American, East-Asian, South-Asian and Latino. Sample annotated images are shown in Appendix E.

We find that LFW dataset is composed of 84.1% of CAU, 9.4% of AA, and

6.5% of EA. IJB-A dataset is less biased, but still with a dominating 71.6% CAU versus 8.2% AA and 10.6% EA.

Rigorously, Y (x1, x2) is a Bernoulli random variable with outcome space containing 0 and 1.

Figures and Tables:

Figure 2: t-SNE visualizations of source (0-4 from MNIST-M) and target (5-9 from MNIST) representations by different learning methods: (a) deep neural network without adaptation, (b) domain adversarial neural network (DANN) and (c) our feature transfer network (FTN). While domain adversarial learning results in significant confusion of digits classes between source and target domains (e.g., 3/5, 2/8, 4/9, or 0/6 in (b)), the proposed FTN transfers discriminative power to target domain while successfully separating them from the source domain.

demands N pair examples to be sampled from the target domain. As the target domain is unlabeled, we ought to first discover a plausible label structure, which is done off-line via HDBSCAN (Campello et al., 2013; McInnes et al., 2017), a fast and scalable density-based hierarchical clustering algorithm. The returned clusters provide pseudo-labels to individual examples of the target domain, allowing us to sample N pair examples to evaluate the following MCEM objective:

NETWORK ARCHITECTURE AND TRAINING DETAILS B.1 TOY EXPERIMENTS: MNIST-M (0 -4) TO MNIST (5 -9) Following (Haeusser et al., 2017), we preprocess the data by subtracting a channel-wise pixel mean and dividing by channel-wise standard deviation of pixel values. For MNIST examples, we also apply color-intensity inversion. All images are resized into 32×32 with 3 channels.

Figure S2: Performance curves of identification accuracy per ethnicity subset on the CEF datasets. The accuracy of DANNs with different values of λ3 for λ4 are visualized.

Verification and identification accuracy on the Cross Ethnicity Faces (CEF) dataset. For supervised models, we report results trained on labeled CAU (Sup C ) or on labeled CAU, AA, EA domains (Sup C,A,E ); for adaptation, we evaluate DANN and FTN, without and with multi-class entropy minimization (MCEM).Such imbalance across ethnicity in labeled training data can result in significant drop in identification performance on data-scarce minorities: the second row of Table1shows a model trained on Caucasian dominated dataset performs poorly on the other ethnicities. As expected, if the training data is composed of only Caucasian identities as source domain, the performance over the target domains consisting of the other ethnicities further deteriorates (see row 1 of Table1). Provided the available labeled source domain contains only Caucasian identities, we subsequently demonstrate that our method can effectively leverage unlabeled data from the non-Caucasian target ethnicity to substantially improve their face verification performances.Experimental Setup. We perform an adaptation from CAU to a mixture of AA and EA. Our experiments use the MS-1M dataset. We first remove identities that both appear in the training and testing sets. The resulting training set consists of 4.04M images from 60K CAU identities, 398K images from 7K AA identities, and 308K images from 4.6K EA identities. For domain adaptation experiments, we use labeled CAU images and unlabeled AA, EA images for training. For supervised experiments to obtain performance lower and upper bound, we use labeled CAU images to train Sup C and labeled CAU, AA, EA images to train Sup C,A,E . scheme result in strongly competitive face recognition performance, comparing to other state-of-the-art methods such as FaceNet(Schroff et al., 2015) on YouTube Faces (Wolf et al., 2011) (97.32% (ours) vs 95.12%) and Neural Aggregation Network (Yang et al., 2017) on IJB-A (see row 2 of Table 3). The complete network architecture and training details are provided in Appendix B.2.Evaluation. We report the performance of the baseline and our proposed models on two standard face recognition benchmarks LFW (Huang et al., 2007) and IJB-A (Klare et al.

Table content: <table><tr><td/><td/><td>CAU</td><td>AA</td><td>EA</td><td>ALL</td><td>CAU</td><td>AA</td><td>EA</td><td>ALL</td></tr><tr><td/><td>Sup C</td><td colspan="7">98.39 92.24 93.41 95.58 90.07 69.64 76.37 77.97</td></tr><tr><td/><td>Sup C,A,E</td><td colspan="7">98.43 97.16 97.05 98.15 90.16 84.02 84.38 85.75</td></tr><tr><td/><td>DANN\Lrecon</td><td colspan="7">98.36 94.54 95.02 96.84 90.01 73.05 74.94 77.99</td></tr><tr><td/><td>DANN</td><td colspan="7">98.36 95.37 96.36 97.34 90.34 74.88 79.39 79.83</td></tr><tr><td/><td>FTN</td><td colspan="7">98.36 95.62 96.64 97.68 90.54 75.35 80.69 81.28</td></tr><tr><td/><td colspan="8">DANN+MCEM 98.39 96.36 97.34 97.88 90.77 80.30 83.07 82.69</td></tr><tr><td/><td>FTN+MCEM</td><td colspan="7">98.37 96.76 97.40 98.08 90.95 80.75 83.71 84.16</td></tr><tr><td>Model Sup C</td><td colspan="5">CAU vs. AA, EA AA vs. CAU EA vs. CAU 91.67 95.42 94.87</td><td colspan="3">Table 2: Cross domain identification accuracy on CEF, with CAU evaluated against AA + EA combined, AA against</td></tr><tr><td>DANN</td><td>89.91</td><td/><td>84.78</td><td/><td>91.47</td><td colspan="3">CAU and EA against CAU.</td></tr><tr><td>FTN</td><td>92.29</td><td/><td>88.09</td><td/><td>92.07</td><td/><td/></tr></table>

Face verification and recognition performance on LFW and IJB-A. From left to right, verification (VRF), closed-set (CLS) and open-set recognition at FAR = 0.01 and 0.001 (Best-Rowden et al., 2014) on LFW, and verification at different FAR and identification (id.) at rank-k on IJB-A are reported.propose a Feature Transfer Network, allowing simultaneous optimization of domain adversarial loss and domain separation loss, as well as a variant of N -pair metric loss for entropy minimization on the target domain where the ground-truth label structure is unknown, to further improve the adaptation quality. Our proposed framework excels at both within-domain and cross-domain verification tasks. As an application, we demonstrate cross-ethnicity face verification that overcomes label biases in training data, achieving high accuracy even for unlabeled ethnicity domains, which we believe is a result with vital social significance.

Table content: <table><tr><td>Model</td><td/><td colspan="2">LFW</td><td/><td colspan="3">IJB-A (verification)</td><td colspan="2">IJB-A (id.)</td></tr><tr><td/><td>VRF</td><td>CLS</td><td>0.01</td><td>0.001</td><td>0.01</td><td>0.001</td><td colspan="3">0.0001 rank-1 rank-5</td></tr><tr><td>Sup C</td><td>99.57</td><td>98.95</td><td>86.07</td><td>66.61</td><td>92.67</td><td>76.65</td><td>50.32</td><td>94.31</td><td>97.25</td></tr><tr><td>Sup C,A,E</td><td colspan="2">99.72 98.79</td><td>96.81</td><td colspan="3">91.11 95.57 87.45</td><td>76.45</td><td>94.73</td><td>97.19</td></tr><tr><td>DANN\Lrecon</td><td>99.43</td><td>98.98</td><td>96.81</td><td>91.44</td><td>94.23</td><td>86.87</td><td>73.80</td><td>94.27</td><td>97.03</td></tr><tr><td>DANN</td><td>99.63</td><td>98.95</td><td>97.15</td><td>93.46</td><td>95.54</td><td>88.64</td><td>77.13</td><td colspan="2">94.59 97.31</td></tr><tr><td>FTN</td><td>99.63</td><td>99.11</td><td>97.15</td><td>92.95</td><td>95.07</td><td>88.45</td><td>77.70</td><td>94.48</td><td>97.19</td></tr><tr><td colspan="2">DANN+MCEM 99.63</td><td colspan="5">99.08 97.65 94.97 95.28 88.78</td><td>77.30</td><td>94.75</td><td>97.30</td></tr><tr><td>FTN+MCEM</td><td colspan="3">99.65 99.14 96.98</td><td>93.46</td><td>94.63</td><td>88.28</td><td colspan="3">77.98 94.79 97.00</td></tr></table>

Network architecture for digit experiments.

Table content: <table/>

Network architecture for face experiments.

Table content: <table><tr><td>operation</td><td/><td>kernel</td><td/><td>output size</td></tr><tr><td>Conv1-1 + ReLU</td><td/><td>3×3</td><td/><td>100×100×32</td></tr><tr><td>Conv1-2 + Maxout (2)</td><td/><td>3×3</td><td/><td>100×100×64</td></tr><tr><td>max pooling</td><td/><td>2×2</td><td/><td>50×50×64</td></tr><tr><td>ResBlock + ReLU ×2</td><td colspan="3">3×3, 64 -64 -64</td><td>50×50×64</td></tr><tr><td>Conv2 + Maxout (2)</td><td/><td>3×3</td><td/><td>50×50×128</td></tr><tr><td>max pooling</td><td/><td>2×2</td><td/><td>25×25×128</td></tr><tr><td>ResBlock + ReLU ×4</td><td colspan="3">3×3, 128 -96 -128</td><td>25×25×128</td></tr><tr><td>Conv3 + Maxout (2)</td><td/><td>3×3</td><td/><td>25×25×192</td></tr><tr><td>max pooling</td><td/><td>2×2</td><td/><td>13×13×192</td></tr><tr><td>ResBlock + ReLU ×8</td><td colspan="3">3×3, 192 -128 -192</td><td>13×13×192</td></tr><tr><td>Conv4 + Maxout (2)</td><td/><td>3×3</td><td/><td>13×13×256</td></tr><tr><td>max pooling</td><td/><td>2×2</td><td/><td>7×7×256</td></tr><tr><td>ResBlock + ReLU ×2</td><td colspan="3">3×3, 256 -160 -256</td><td>7×7×256</td></tr><tr><td>Conv5 + Maxout (2)</td><td/><td>3×3</td><td/><td>7×7×320</td></tr><tr><td>avg pooling</td><td/><td>7×7</td><td/><td>1×1×320</td></tr><tr><td>Normalize and Scale (12)</td><td/><td>-</td><td/><td>320</td></tr><tr><td/><td>λ1</td><td>λ2</td><td>λ3</td><td>λ4</td></tr><tr><td>DANN</td><td>0.1</td><td>-</td><td>0.1</td><td>0.01</td></tr><tr><td>FTN</td><td colspan="3">0.03 0.1 0.03</td><td>0.01</td></tr><tr><td colspan="5">FTN+MCEM 0.03 0.1 0.03 0.003</td></tr></table>

Optimal hyperparameter settings of different adaptation models.C IMPACT OF FEATURE RECONSTRUCTION LOSS ON DOMAIN ADVERSARIAL TRAININGWe demonstrate the effectiveness of feature reconstruction loss in stabilizing the domain adversarial training in DANN framework. We train four different DANN models with different configurations of λ 3 and λ 4 . We visualize in FigureS2the performance curves of identification accuracy evaluated on the AA, EA, and CAU ethnicities of CEF dataset. Note that we stop training early on when the performance start to degrade significantly. Therefore, x-axis, the number of training epoch, of different curves are different. y-axis represents the identification accuracy.

Table content: <table/>

Verification precision and recall of clustering methods, such as projection to source example or source class center, or hierarchical clustering using embeddings of Sup C (HDBSCAN) or our proposed FTN model. Furthermore, we repeat the clustering using the FTN with multi-class entropy minimization model (FTN+MCEM) and report the clustering accuracy.improved discriminative quality of features by FTNs, but also suggests a potential tool for automatic labeling of unlabeled data by iterative training of FTN model and hierarchical clustering.

Table content: <table><tr><td/><td colspan="2">source example</td><td colspan="2">source center</td><td colspan="2">HDBSCAN</td><td>FTN</td><td/><td colspan="2">FTN+MCEM</td></tr><tr><td/><td>AA</td><td>EA</td><td>AA</td><td>EA</td><td>AA</td><td>EA</td><td>AA</td><td>EA</td><td>AA</td><td>EA</td></tr><tr><td>Precision</td><td>0.11</td><td>0.12</td><td>0.30</td><td>0.08</td><td>100</td><td>100</td><td colspan="4">95.36 96.23 95.79 96.10</td></tr><tr><td>Recall</td><td>25.64</td><td>31.11</td><td colspan="8">22.54 48.60 88.66 81.74 97.29 96.46 97.81 96.89</td></tr><tr><td>F-score</td><td>0.22</td><td>0.25</td><td>0.58</td><td>0.16</td><td colspan="6">93.99 89.95 96.31 96.34 96.79 96.49</td></tr></table>

Peer review (iclr2019):
 The authors studied an interesting problem of unsupervised domain adaptation when the source and the target domains have disjoin labels spaces. The paper proposed a novel feature transfer network, that optimizes domain adversarial loss and domain separation loss. Strengths: 1) The proposed approach on Feature Transfer Network was novel and interesting. 2) The paper was very well written with a good analysis of various choices. 3) Extensive empirical analysis on multi-class settings with a traditional MNIST dataset and a real-world face recognition dataset. Weakness: 1) Practical considerations addressing feature reconstruction loss needs more explanation. Comments: The technical contribution of the paper was sound and novel. The paper considered existing work and in a good way generalizes and extends into disjoint label spaces. It was easy to read and follow, most parts of the paper including the Appendix make it a good contribution. However, the reviewer has the following suggestions" 1. Under the practical considerations for preventing the mode collapse via feature reconstruction, how is the reference network trained? In the Equation(6) for feature reconstruction, the f_ref term maps the source and target domain examples to new feature space. What do you mean by references network trained on the label data? Please clarify. 2. Under the practical considerations for replacing the verification loss, it is said that "Our theoretical analysis suggests to use a verification the loss that compares the similarity between a pair of images" - Can you please cite the references to make it easier for the reader to follow.