Title: No title found

Abstract: There is a strong incentive to develop versatile learning techniques that can transfer the knowledge of class-separability from a labeled source domain to an unlabeled target domain in the presence of a domain-shift. Existing domain adaptation (DA) approaches are not equipped for practical DA scenarios as a result of their reliance on the knowledge of source-target label-set relationship (e.g. Closed-set, Open-set or Partial DA). Furthermore, almost all the prior unsupervised DA works require coexistence of source and target samples even during deployment, making them unsuitable for incremental, real-time adaptation. Devoid of such highly impractical assumptions, we propose a novel two-stage learning process. Initially, in the procurement-stage, the objective is to equip the model for future sourcefree deployment, assuming no prior knowledge of the upcoming category-gap and domain-shift. To achieve this, we enhance the model's ability to reject out-of-source distribution samples by leveraging the available source data, in a novel generative classifier framework. Subsequently, in the deployment-stage, the objective is to design a unified adaptation algorithm capable of operating across a wide range of category-gaps, with no access to the previously seen source samples. To achieve this, in contrast to the usage of complex adversarial training regimes, we define a simple yet effective source-free adaptation objective by utilizing a novel instancelevel weighing mechanism, named as Source Similarity Metric (SSM). A thorough evaluation shows the practical usability of the proposed learning framework with superior DA performance even over state-of-the-art source-dependent approaches.

1. INTRODUCTION:

Deep learning models have proven to be highly successful over a wide variety of tasks (Krizhevsky et al., 2012; Ren et al., 2015) . However, a majority of these remain heavily dependent on access to a huge amount of labeled samples to achieve a reliable level of generalization. A recognition model trained on a certain distribution of labeled samples (source domain) often fails to generalize (Chen et al., 2017) when deployed in a new environment (target domain) in the presence a discrepancy in the input distribution (Shimodaira, 2000) . Domain adaptation (DA) algorithms seek to minimize this discrepancy either by learning a domain invariant feature representation (Long et al., 2015; Kumar et al., 2018; Ganin et al., 2016; Tzeng et al., 2015) , or by learning independent domain transformations (Long et al., 2016) to a common latent representation through adversarial distribution matching (Tzeng et al., 2017; Nath Kundu et al., 2018) , in the absence of target label information.

Most of the existing approaches (Zhang et al., 2018c; Tzeng et al., 2017 ) assume a common label-set shared between the source and target domains (i.e. C s = C t ), which is often regarded as Closed-Set DA (see Fig. 1 ). Though this assumption helps to analyze various insights of DA algorithms, such an assumption rarely holds true in real-world scenarios. Recently researchers have independently explored two broad adaptation settings by partly relaxing the above assumption. In the first kind, Partial DA (Zhang et al., 2018b; Cao et al., 2018a; b) , the target label space is considered as a subset of the source label space (i.e. C t ⊂ C s ). This setting is more suited for large-scale universal source datasets, which will almost always subsume the label-set of a wide range of target domains. However, the availability of such a universal source is highly questionable for a wide range of input domains and tasks. In the second kind, regarded as Open-set DA (Baktashmotlagh et al., 2019; Ge et al., 2017) , the target label space is considered as a superset of the source label space (i.e. C t ⊃ C s ). The major challenge in this setting is attributed to detection of target samples from the unobserved categories in a fully-unsupervised scenario. Apart from the above two extremes, certain works define a partly mixed scenario by allowing "private" label-set for both source and target domains (i.e. C s \ C t = ∅ and C t \ C s = ∅) but with extra supervision such as few-shot labeled data (Luo et al., 2017) or access to the knowledge of common categories (Panareda Busto & Gall, 2017). Most of the prior approaches consider each scenario in isolation and propose independent solutions. Thus, they require access to the knowledge of label-set relationship (or category-gap) to carefully choose a DA algorithm, which would be suitable for the problem in hand. Furthermore, all the prior unsupervised DA works require coexistence of source and target samples even during deployment, hence not source-free. This is highly impractical, as labeled source data may not be accessible after deployment due to several reasons such as, privacy concerns, restricted access to proprietary data, accidental loss of source data or other computational limitations in real-time deployment scenarios.

Acknowledging the aforementioned shortcomings, we propose one of the most convenient DA frameworks which is ingeniously equipped to address source-free DA for all kinds of label-set relationships, without any prior knowledge of the associated category-gap (i.e. universal-DA). We not only focus on identifying the key complications associated with the challenging problem setting, but also devise insightful ideas to tackle such complications by adopting learning techniques much different from the available DA literature. This leads us to realize a holistic solution which achieves superior DA performance even over prior source-dependent approaches.

2. RELATED WORK:

We briefly review the available domain adaptation methods under the three major divisions according to the assumption on label-set relationship. a) Closed-set DA. 

3. PROPOSED APPROACH:

The problem setting for source-free domain adaptation is broadly divided into a two stage process.

a) Procurement stage. In this stage, we are given full access to the labeled samples of source domain, D s = {(x s , y s ) : x s ∼ p, y s ∈ C s }, where p is the distribution of source samples and C s denotes the label-set of the source domain. Here, the objective is to equip the model for the second stage, i.e. the Deployment stage, in the presence of a discrepancy in the distribution of input target samples. To achieve this we rely on an artificially generated negative dataset,

D n = {(x n , y n ) : x n ∼ p n , y n ∈ C n }

, where p n is the distribution of negative source samples such that C n ∩ C s = ∅. b) Deployment stage. After obtaining a trained model from the Procurement stage, the model will have its first encounter with the unlabeled target domain samples from the deployed environment. We denote the unlabeled target data by D t = {x t : x t ∼ q}, where q is the distribution of target samples. Note that, access to the source dataset D s from the previous stage is fully restricted during adaptation in the Deployment stage. Suppose that, C t is the "unknown" label-set of the target domain. We define the common label space between the source and target domain as C = C s ∩ C t . The private label-set for the source and the target domains is represented as C s = C s \ C t and C t = C t \ C s respectively.

3.1. LEARNING IN THE PROCUREMENT STAGE:

3.1.1 Challenges. The available DA techniques heavily rely on the adversarial discriminative (Tzeng et al., 2017; Saito et al., 2018a) strategy. Thus, they require access to the source samples to reliably characterize the source domain distribution. Moreover, these approaches are not equipped to operate in a source-free setting. Though a generative model can be used as a memory-network (Sankaranarayanan et al., 2018; Bousmalis et al., 2017) to realize source-free adaptation, such a solution is not scalable for large-scale source datasets (e.g. ImageNet (Russakovsky et al., 2015) ), as it introduces unnecessary extra parameters in addition to the associated training difficulties (Salimans et al., 2016) . This calls for a fresh analysis of the requirements beyond the solutions found in literature.

In a general DA scenario, with access to source samples in the Deployment stage (specifically for Open-set or Partial DA), a widely adopted approach is to learn domain invariant features. In such approaches the placement of source category clusters is learned in the presence of unlabeled target samples which obliquely provides a supervision regarding the relationship between C s and C t . For instance, in case of Open-set DA, the source clusters may have to disperse to make space for the clusters from target private C t (see Fig. 2a to 2b ). Similarly, in partial DA, the source clusters may have to rearrange themselves to keep all the target shared clusters (C = C t ) separated from the source private C s (see Fig. 2a to 2c ). However in a complete source-free framework, we do not have the liberty to leverage such information as source and target samples never coexist together during training. Motivated by the adversarial discriminative DA technique (Tzeng et al., 2017) , we hypothesize that, inculcating the ability to reject samples that are out of the source data distribution can facilitate future source-free domain alignment using this discriminatory knowledge. Therefore, in the Procurement stage the overarching objective is two-fold.

• Firstly, we must aim to learn a certain placement of source clusters best suited for all kinds of category-gap scenarios acknowledging the fact that, a source-free scenario does not allow us to modify the placement in the presence of target samples during adaptation (see Fig. 2d ). • Secondly, the learned embedding must have the ability to reject out-of-distribution samples, which is an essential requirement for unsupervised adaptation in the presence of domain-shift.

3.1.2. Solution.:

In the presence of source data, we aim to restrain the model's domain and category bias which is generally inculcated as a result of the over-confident supervised learning paradigms (see Fig. 4A ). To achieve this goal, we adopt two regularization strategies viz. i) regularization via generative modeling and ii) utilization of a labeled simulated negative source dataset to generalize for the latent regions not covered by the given positive source samples (see Fig. 4C ).

How to configure the negative source dataset? While configuring D n , the following key properties have to be met. Firstly, latent clusters formed by the negative categories must lie in-between the latent clusters of positive source categories to enable a higher degree of intra-class compactness with interclass separability (Fig. 4C ). Secondly, the negative source samples must enrich the source domain n by randomly compositing local regions between a pair of images drawn from the positive source dataset D s (see Fig. 3A and appendix, Algo. 2). Intuitively, composite samples x n created on image pairs from different source categories are expected to lie in-between the two positive source clusters in the latent space, thereby introducing a combinatorial amount of new class labels i.e.

|C n | = |Cs| C 2 . b) Latent-simulated negative dataset D (b)

n . As an alternative approach, in the absence of domain knowledge (e.g. non-image datasets, or for tasks beyond image-recognition such as pose estimation), we propose to sample virtual negative instances, u n from the latent space which are away from the high confidence regions (3-sigma) of positive source clusters (Fig. 4B ). For each negative sample, we assign a negative class label (one of |C n | = |Cs| C 2 ) corresponding to the pair of most confident source classes predicted by the classifier. Thus, we obtain D

(b) n = {(u n , y n ) : u n ∼ p u n , y n ∈ C n } where p u n

is the distribution of negative samples in the latent u-space (more details in appendix Algo. 3).

Training procedure. The generative source classifier is divided into three stages; i) backbone-model M , ii) feature extractor F s , and iii) classifier D (see Fig. 3B ). Output of the backbone-model is denoted as v = M (x), where x is drawn from either D s or D n . Following this, the output of F s and D are represented as u and d respectively. D outputs a K-dimensional logit denoted as

d (k) for k = 1, 2...K; K = |C s | + |C n |.

The individual class probabilities, ŷ(k) are obtained by applying softmax over the logits i.e.

ŷ(k) = exp(d (k) )/ K k=1 exp(d (k) ) = σ (k) (D • F s • M (x))

. Additionally, we define priors of only positive source classes as P (u Lp = -log(exp(P (us|c ks ))/ |Cs| i=1 exp(P (us|ci))), where P (us|ci) = N (us|µc i , Σc i ) 8:

s |c i ) = N (u s |µ ci , Σ ci ) for i = 1, 2...|C s | at

Update θF s , θD, θG by minimizing LCE, Lv, Lu, and Lp alternatively using separate optimizers. 9:

if (iter % U pdateIter == 0) then 10:

Recompute the sample mean (µc i ) and covariance

(Σc i ) of Fs • M (xs) for xs from class ci; i = 1, 2...|Cs| (For D (b)

n : generate fresh latent-simulated negative samples using the updated priors)

the intermediate embedding

u s = F s • M (x s ).

Here, parameters of the normal distributions are computed during training as shown in line-10 of Algo. 1. A cross-entropy loss over these prior distributions is defined as L p (line-7 in Algo. 1), to effectively enforce intra-class compactness with inter-class separability (progression from Fig. 4B to 4C ). Motivated by generative variational auto-encoder (VAE) setup (Kingma & Welling, 2013), we introduce a feature decoder G, which aims to minimize the cyclic reconstruction loss selectively for the samples from positive source categories v s and randomly drawn samples u r from the corresponding class priors (i.e. L v and L u , line-6 in Algo. 1). This along with a lower weightage α for the negative source categories (i.e. at the cross-entropy loss L CE , line-6 in Algo. 1) is incorporated to deliberately bias F s towards the positive source samples, considering the level of unreliability of the generated negative dataset.

3.2. LEARNING IN THE DEPLOYMENT STAGE:

3.2.1 Challenges. We hypothesize that, the large number of negative source categories along with the positive source classes i.e. C s ∪ C n can be interpreted as a universal source dataset, which can subsume label-set C t of a wide range of target domains. Moreover, we seek to realize a unified adaptation algorithm, which can work for a wide range of category-gaps. However, a forceful adaptation of target samples to positive source categories will cause target private samples to be classified as an instance of the source private or the common label-set, instead of being classified as "unknown", i.e. one of the negative categories in C n .

In contrast to domain agnostic architectures (You et a) Source Similarity Metric (SSM). We define a weighting factor (SSM) for each target sample x t , as w(x t ). A higher value of this metric indicates x t 's similarity towards the positive source categories, specifically inclined towards the common label space C. Similarly, a lower value of this metric indicates x t 's similarity towards the negative source categories C n , showing its inclination towards the private target labels C t . Let, p s, qt be the distribution of source and target samples with labels in C s and C t respectively. We define, p c and q c to denote the distribution of samples from source and target domains belonging to the shared label-set C. Then, the SSM for the positive and negative source samples should lie on the two extremes, forming the following inequality:

EQUATION

To formalize the SSM criterion we rely on the class probabilities defined at the output of source model only for the positive class labels, i.e. 

We hypothesize that, the above definition will satisfy Eq. 1, as a result of the generative learning strategy adopted in the Procurement stage. In Eq. 2 the exponent is used to further amplify separation between target samples from the shared C and those from the private C t label-set (see Fig. 5A ).

b) Source-free domain adaptation. To perform domain adaptation, the objective function aims to move the target samples with higher SSM value towards the clusters of positive source categories and vice-versa at the frozen source embedding, u-space (from the Procurement stage). To achieve this, parameters of only F t network are allowed to be trained in the Deployment stage. However, the decision of weighting the loss on target samples towards the positive or negative source clusters is computed using the source feature extractor F s i.e. the SSM in Eq. 2. We define, the deployment model as h = D • F t • M (x t ) using the target feature extractor, with softmax predictions over K categories obtained as ẑ(k) = σ(h (k) ). Thus, the primary loss function for adaptation is defined as,

EQUATION

Additionally, in the absence of label information, there would be uncertainty in the predictions ẑ(k) as a result of distributed class probabilities. This leads to a higher entropy for such samples. Entropy minimization (Grandvalet & Bengio, 2005 ; Long et al., 2016) is adopted in such scenarios to move the target samples close to the highly confident regions (i.e. positive and negative cluster centers from the Procurement stage) of the classifier's feature space. However, it has to be done separately for positive and negative source categories based on the SSM values of individual target samples to effectively distinguish the target-private set from the full target dataset. To achieve this, we define two different class probability vectors separately for the positive and negative source classes denoted as,

z(i) s = exp(h (i) )/ |Cs| j=1 exp(h (j) ) and z(i) n = exp(h (i+|Cs|) )/ |Cn| j=1 exp(h (j+|Cs|)

) respectively (see Fig. 3B ). Entropy of the target samples in the positive and negative regimes of the source classifier is obtained as

H s (x t ) = - |Cs| i=1 z(i) s log z(i) s and H n (x t ) = - |Cn| i=1 z(i) n log z(i)

n respectively. Consequently, the entropy minimization loss is formalized as,

L d2 = w(xt)Hs(xt) + w (xt)Hn(xt) (4)

Thus, the final loss function for adapting the parameters of F t is presented as

L d = L d1 + βL d2 .

Here β is a hyper-parameter controlling the importance of entropy minimization during adaptation.

4. EXPERIMENTS:

We perform a thorough evaluation of the proposed source-free, universal domain adaptation framework against prior state-of-the-art models across multiple datasets. We also provide a comprehensive ablation study to establish generalizability of the approach across a variety of label-set relationships and justification of the various model components. (You et al., 2019) relies on a sensitive hyperparameter, as a threshold on the sample-level weighting, to mark a target sample as "unknown". Also note that, our method is completely source-free during the Deployment stage, while all other methods have access to the full source-data.

Accuracy on Target-Unknown data, T unk . We evaluate the target unknown accuracy, T unk , as the proportion of actual target private samples (i.e. {(x t , y t ) : y t ∈ C t }) being classified as "unknown" after adaptation. Note that, UAN (You et al., 2019) does not report T unk which is a crucial metric to evaluate the vulnerability of the model after its deployment in the target environment. The T avg metric fails to capture this as a result of class-imbalance in the Open-set scenario (Saito et al., 2018b) . Hence, to realize a common evaluation ground, we train the UAN implementation provided by the authors (You et al., 2019) and denote it as UAN* in further sections of this paper. We observe that, the UAN (You et al., 2019) training algorithm is often unstable with a decreasing trend of T unk and T avg over increasing training iterations. We thus report the mean and standard deviation of the peak values of T unk and T avg achieved by UAN*, over 5 separate runs on Office-31 dataset (see Table 7 ).

Implementation Details. We implement our network in PyTorch and use ResNet-50 (He et al., 2016) as the backbone-model M , pre-trained on ImageNet (Russakovsky et al., 2015) inline with UAN (You et al., 2019) . The complete architecture of other components with fully-connected layers is provided in the Supplementary. A sensitivity analysis of the major hyper-parameters used in the proposed framework is provided in Fig. 5B-C , and Appendix Fig. 8B . In all our ablations across the datasets, we fix the hyperparameters values as α = 0.2 and β = 0.1. We utilize Adam optimizer (Kingma & Ba, 2014) with a fixed learning rate of 0.0001 for training in both Procurement and Deployment stage (see Appendix for the code). For the implementation of UAN*, we use the hyper-parameter value w 0 = -0.5, as specified by the authors for the task A→D in Office-31 dataset.

4.3. DISCUSSION:

a) Comparison with prior arts. We compare our approach with UAN You et Relative freq.

SSM w(x t ):

x t from target-private x t from target-shared b) Does SSM satisfy the expected inequality? Effectiveness of the proposed learning algorithm, in case of source-free deployment, relies on the formulation of SSM, which is expected to satisfy Eq. 1. Fig. 5A shows a histogram of the SSM separately for samples from target-shared (blue) and target-private (red) label space. The success of this metric is attributed to the generative nature of Procurement stage, which enables the source model to distinguish between the marginally more negative target-private samples as compared to the samples from the shared label space.

P

c) Sensitivity to hyper-parameters. As we tackle DA in a source-free setting simultaneously intending to generalize across varied category-gaps, a low sensitivity to hyperparameters would further enhance our practical usability. To this end, we fix certain hyperparameters for all our ablations (also in Fig. 6C ) even across datasets (i.e. α = 0.2, β = 0.1). e) DA in absence of shared categories. In universal adaptation, we seek to transfer the knowledge of "class-separability criterion" obtained from the source domain to the deployed target environment. More concretely, it is attributed to the segregation of data samples based on some expected characteristics, such as classification of objects according to their pose, color, or shape etc. To quantify this, we consider an extreme case where

C s ∩ C t = ∅ (A→D in Office-31 with |C s | = 15, |C t | = 16).

Allowing access to a single labeled target sample from each category in C t = C t , we aim to obtain a one-shot recognition accuracy (assignment of cluster index or class label using the one-shot samples as the cluster center at F t • M (x t )) to quantify the above metric. We obtain 64.72% accuracy for the proposed framework as compared to 13.43% for UAN* (You et al., 2019) . This strongly validates our superior knowledge transfer capability as a result of the generative classifier with labeled negative samples complementing for the target-private categories.

f) Dependency on the simulated negative dataset. Conceding that a combinatorial amount of negative labels can be created, we evaluate the scalability of the proposed approach, by varying the number of negative classes in the Procurement stage by selecting 0, 4, 8, 64, 150 and 190 negative classes as reported in the X-axis of Fig. 5C . For the case of 0 negative classes, denoted as |C n | * = 0 in Fig. 5C , we synthetically generate random negative features at the intermediate level u, which are at least 3-sigma away from each of the positive source priors P (u s |c i ). We then make use of these feature samples along with positive image samples, to train a (|C s | + 1) class Procurement model with a single negative class. The results are reported in Fig. 5C on the A→D task of Office-31 dataset with category relationship inline with the setting in Table 7 . We observe an acceptable drop in accuracy with decrease in number of negative classes, hence validating scalability of the approach for large-scale classification datasets (such as ImageNet). Similarly, we also evaluated our framework by combining three or more images to form such negative classes. An increasing number of negative classes ( |Cs| C 3 > |Cs| C 2 ) attains under-fitting on positive source categories (similar to Fig. 5C , where accuracy reduces beyond a certain limit because of over regularization).

5. CONCLUSION:

We have introduced a novel source-free, universal domain adaptation framework, acknowledging practical domain adaptation scenarios devoid of any assumption on the source-target label-set relationship. In the proposed two-stage framework, learning in the Procurement stage is found to be highly crucial, as it aims to exploit the knowledge of class-separability in the most general form with enhanced robustness to out-of-distribution samples. Besides this, success in the Deployment stage is attributed to the well-designed learning objectives effectively utilizing the source similarity criterion. This work can be served as a pilot study towards learning efficient inheritable models in future.

A APPENDIX:

This appendix is organized as follows,

• Implementation details -Procurement Stage.

-Deployment Stage.

• Ablation Studies and Additional Results:

-Pretraining the backbone network on Places instead of ImageNet.

-Space and Time Complexity Analysis.

-Varying label-set relationship.

-Sensitivity analysis.

-Closed-set adaptation.

-Accuracy on source dataset post Procurement.

-Incremental one-shot classification.

-Feature Space Visualization.

• Miscellaneous:

-Specification of Computing Resources.

-References to code

B IMPLEMENTATION DETAILS:

In this section, we describe the architecture and the training process used for the Procurement and Deployment stages of our approach.

B.1 PROCUREMENT STAGE a) Design of classifier D used in the Procurement stage. Keeping in mind the possibility of an additional domain shift after performing adaptation (e.g. encountering domain W after performing the adaptation A → D in Office-31 dataset), we design the classifier's architecture in a manner which allows for dynamic modification in the number of negative classes post-procurement. We achieve this by maintaining two separate classifiers during Procurement -D src , that operates on the positive source classes, and, D neg that operates on the negative source classes (see architecture in Table 5 ). The final classification score is obtained by computing softmax over the concatenation of logit vectors produced by D src and D neg . Therefore, the model can be retrained on a different number of negative classes post deployment (using another negative class classifier D neg ), thus preparing it for a subsequent adaptation step to another domain.

b) Negative dataset generation. We propose two methods to generate negative samples for the Procurement stage, and name the models trained subsequently as USFDA-a and USFDA-b. Here, we describe the two processes:

• Using image-composition for D (a)

n (USFDA-a). In the presence of domain knowledge (knowledge of the task at hand, i.e. object recognition using images), we generate the negative dataset D n by compositing images taken from different classes, as described in Algo. 2. We generate random masks using quadratic splines passing through a central image region (lines 3-9). Using these masks, we merge alternate regions of the images, both horizontally and vertically, resulting in 4 negative images for each pair of images (lines 10-13). To effectively cover the inter-class negative region, we randomly sample image pairs from D s belonging to different classes, however we do not impose any constraint on how the classes are selected (for e.g. one can composite images from an animal and a non-animal class). We choose 5000 pairs for tasks on Office-31, Office-Home and VisDA datasets, and 12000 for ImageNet-Caltech. Since the input source distribution (p) is fixed we first synthesize a negative dataset offline (instead of creating them on the fly) to ensure finiteness of the training set. The training algorithm for USFDA-a is given in Algo. 1. Let λ cj and l cj be the maximum eigen value and the corresponding eigen vector of Σ cj , for each class c j 5:

ũr ∼ N (µ, Σ) n (USFDA-b): Here, we perform rejection sampling as given in Algorithm 3. Here, we obtain a sample from the global source prior P (u s ) = N (u s |µ, Σ), where µ and Σ are the mean and covariance computed at u-space over all the positive source image samples.We reject the sample if it lies within the 3-sigma bound of any class (i.e. we keep the sample if it is far away from all source class-priors, N (µ ci , Σ ci )), as shown in lines 6 to 11 in Algo. 3. A sample selected in this fashion is expected to lie in an intermediate region between the source class priors. The two classes in the vicinity of the sample are then determined by obtaining the two most confident class predictions given by the classifier D src (lines 7 and 8). Using this pair of classes, we assign a unique negative class label to the sample which corresponds to the intermediate region between the pair of classes. Note, to learn the arrangement of positive and negative clusters, the feature extractor F s must be trained using negative samples. We do this by passing the sampled latent-simulated negative instance (ũ r ) through the decoder-encoder pair, (i.e. D • F s • G(ũ r )), and enforcing the cross-entropy loss to classify them into the respective negative class. The training algorithm for USFDA-b is given in Algo. 4. c) Justification of L p . The cross-entropy loss on the likelihoods (referred as L p in the paper) not only enforces intra-class compactness but also ensures inter-class separability in the embedding space, u. Since the negative samples are only an approximation of future target private classes expected to be encountered during deployment, we choose not to employ this loss for them. Such a training procedure, eventually results in a natural development of bias towards the confident positive source classes. This subsequently leads to the placement of source clusters in a manner which enables source-free adaptation (See Fig. 4 ). We find that widely adopted standard domain adaptation datasets such as Office-31 and VisDA often share a part or all of their label-set with ImageNet. Therefore, to validate our method's applicability when initialized from a network pretrained on an unrelated dataset, we attempt to solve the adaptation task A→D in Office-31 dataset by pretraining the ResNet-50 backbone on Places dataset (Zhou et al., 2017) . In Table 3 it can be observed that our method outperforms even source-dependent methods (e.g. UAN (You et al., 2019) , which is also initialized a ResNet-50 backbone pretrained on Places dataset). In contrast to our method, the algorithm in UAN involves ResNet-50 finetuning. Therefore, we also compare against a variant of UAN with a frozen backbone network, by inserting an additional feature extractor that operates on the features extracted from ResNet-50 (similar to F s in the proposed method). The architecture of the feature extractor used for this variant of UAN is outlined in Table 6 . We observe that our method significantly outperforms this variant of UAN with lesser number of trainable parameters (see Table 3 ).

C.2 SPACE AND TIME COMPLEXITY ANALYSIS.

On account of keeping the weights of the backbone network frozen throughout the training process, and devoid of ad-hoc networks such as adversarial discriminator our method makes use of significantly lesser trainable parameters when compared to previous methods such as UAN (See Table 3 ). Devoid of adversarial training, the proposed method also has a significantly lesser total training time for adaptation: 44 sec versus 280 sec in UAN (for the A→D task of Office-31 and batch size of 32). Therefore, the proposed framework offers a much simpler adaptation pipeline, with a superior time and space complexity and at the same time achieves state-of-the-art domain adaptation performance across different datasets, even without accessing labeled source data at the time of adaptation (See Table 3 ). This corroborates the superiority of our method in real-time deployment scenarios.

C.3 VARYING LABEL-SET RELATIONSHIP:

In addition to the T avg reported in Fig. 6 in the paper, we also compare the target-unknown accuracy T unk for UAN* and our pipeline. The results are presented in Figure 7 . Refer the link to the code provided in the submission for details of the chosen class labels for each adaptation scenario shown in Figure 7 . Clearly, our method achieves a statistically significant improvement on most of the label-set 

:



Figures and Tables:

Figure 1: Various label-set relationships (category-gap).

Figure 2: Latent space cluster arrangement during adaptation (see Section 3.1.1).

Figure 3: A) Simulated labeled negative samples using randomly created spline segments (in pink), B) Proposed architecture, C) Procurement stage yields compact source clusters on experimental data.

Figure 4: Achieving intra-class compactness and inter-class separability using negative dataset D n . distribution without forming a new domain by themselves. This rules out the use of Mixup (Zhang et al., 2018a) or adversarial noise (Shu et al., 2018) as negative samples in this scenario. Thus, we propose the following two ways to synthesize the desired negative source dataset. a) Image-composition as negative dataset D (a) n . One of the key characteristics shared between the samples from source and unknown target domain is the semantics of the local part-related features specifically for image-based object recognition tasks. Relying on this assumption, we propose a systematic procedure to simulate the samples of D

Training algorithm in the Procurement stage 1: input: (xs, ys) ∈ Ds, (xn, yn) ∈ Dn; θF s , θD, θG: Parameters of Fs, D and G respectively. 2: initialization: pretrain {θF s , θD} using cross-entropy loss on (xs, ys) followed by initialization of the sample mean µc i and covariance Σc i (at u-space) of Fs • M (xs) for xs from class ci; i = 1, 2, ...|Cs| 3: for iter < M axIter do 4: vs = M (xs); us = Fs(vs); vs = G(us); ur ∼ N (µc i , Σc i ) for i = 1, 2, ...|Cs|; ûr = Fs • G(ur) 5: ŷ(ks) s = σ (ks) (D • Fs • M (xs)), and ŷ(kn) n = σ (kn) (D • Fs • M (xn)) where ks and kn are the index of ground-truth label ys and yn respectively. Lv = |vs -vs|; Lu = |ur -ûr| 7:

EXPERIMENTAL SETUP Datasets. For all the following datasets, we resort to the experimental settings inline with the recent work by You et al. (2019) (UAN). Office-Home (Venkateswara et al., 2017) dataset consists of images from 4 different domains -Artistic (Ar), Clip-art (Cl), Product (Pr) and Real-world (Rw). Alphabetically, the first 10 classes are selected as C, the next 5 classes as C s , and the rest 50 as C t . VisDA2017 (Peng et al., 2018) dataset comprises of 12 categories with synthetic images as the source domain and natural images as the target domain, out of which, the first 6 are chosen as C, the next 3 as C s and the rest as C t . Office-31 (Saenko et al., 2010) dataset contains images from 3 distinct domains -Amazon (A), DSLR (D) and Webcam (W). We use the 10 classes shared by Office-31 and Caltech-256 (Gong et al., 2012) to construct the shared label-set C and alphabetically select the next 10 as C s , with the remaining 11 classes contributing to C t . To evaluate scalability, ImageNet-Caltech is also considered with 84 common classes inline with the setting in You et al. (2019). Simulation of labeled negative samples. To simulate negative labeled samples for training in the Procurement stage, we first sample a pair of images, each from different categories of C s , to create

Figure 5: Ablative analysis on the task A→D in Office-31 dataset. A) Histogram of SSM values of x t separately for target-private and target-shared samples at the Procurement iteration 100 (top) and 500 (bottom). B) The sensitivity curve for β shows marginally stable adaptation accuracy for a wide-range of values. C) A marginal increase in T avg is observed with increase in |C n |.

Figure 6: Comparison across varied label-set relationships for the task A→D in Office-31 dataset. A) Visual representation of label-set relationships and T avg at the corresponding instances for B) UAN* (You et al., 2019) and C) ours source-free model. Effectively, the direction along x-axis (blue horizontal arrow) characterizes increasing Open-set complexity. The direction along y-axis (red vertical arrow) shows increasing complexity of Partial DA scenario. The pink diagonal arrow denotes the effect of decreasing shared label space.

Image-composition algorithm 1: input: Image pair (I1, I2) ∈ Ds. (image shape HxW x3= 224x224x3) 2: k ← -30 3: x1, x2, y1, y2 ←rand(0, W ), rand(0, W ), rand(0, H), rand(0, H) 4: cx, cy ←rand(W/2 -k, W/2 + k), rand(H/2 -k/3, H/2 + k/3) 5: dx, dy ←rand(W/2 -k/3, W/2 + k/3), rand(H/2 -k, H/2 + k) 6: s1 ← -quadratic_interpolation([(0,y1), (cx, cy), (223, y2)]) horizontal splicing 7: s2 ← -quadratic_interpolation([(x1, 0), (dx, dy), (x2, 223)]) vertical splicing 8: m1 ←mask region below s1 9: m2 ←mask region to the left of s2 10: Ia ← -m1 * I1 + (1 -m1) * I2 11: I b ← -m2 * I1 + (1 -m2) * I2 12: Ic ← -m1 * I2 + (1 -m1) * I1 13: I d ← -m2 * I2 + (1 -m2) * I1 14: return Ia, I b , Ic, I d Dataset generation using latent-simulated negative samples 1: input: class-wise source priors N (µ cj , Σ cj ), global source prior N (µ, Σ), number of required samples n, source classifier D src || signifies an Append Operation 2: Ũ ← {}; Ỹ ← {} 3: while | Ũ| ≤ n do 4:

if P (ũ r |c j ) < P (µ cj + 3 * λ cj * l cj | c j ) for all class c j then 7: ŷ ← σ(D src (ũ r )) 8: ỹr ← assign the negative class based on the top-2 confident classes in ŷ 9: Ũ ← -Ũ || ũr ; Ỹ ← -Ỹ || ỹr Using latent-simulated negative samples for D (b)

Training algorithm for USFDA-b in the Procurement stage 1: input: (xs, ys) ∈ Ds; θF s , θD, θG: Parameters of Fs, D and G respectively. 2: initialization: pretrain {θF s , θD} using cross-entropy loss on (xs, ys), then, compute the sample mean µc i and covariance Σc i of Fs • M (xs) for xs from class ci, for i = 1, 2, ...|Cs| 3: for iter < M axIter do 4: vs = M (xs); us = Fs(vs); vs = G(us); ur ∼ N (µc i , Σc i ) for i = 1, 2, ...|Cs|; ûr = Fs • G(ur) 5: (ũr, ỹr) = sample latent-simulated negative instances from D σ (ks) (D • Fs • M (xs)), and ŷ(kn) n = σ (kn) (D • Fs • G(ũr)) where ks and kn are the index of ground-truth label ys and yn respectively, and σ is the softmax activation. Lv = |vs -vs|; Lu = |ur -ûr| 8: Lp = -log(exp(P (us|c ks ))/

Figure 7: Comparison of target-unknown accuracy T unk across varied label-set relationships for the task A→D in Office-31 dataset. A) Visual representation of label-set relationships and T unk at the corresponding instances for B) UAN* (You et al., 2019) and C) ours source-free model. Effectively, the direction along x-axis (blue horizontal arrow) characterizes increasing Open-set complexity. The direction along y-axis (red vertical arrow) shows increasing complexity of Partial DA scenario. And the pink diagonal arrow denotes the effect of decreasing shared label space.

This is an essential requirement to retain the task-dependent knowledge gathered from the source dataset. Thus, we introduce a domain specific feature extractor denoted as F t , whose parameters are initialized from the fully trained F s (see Fig.3B). Further, we aim to exploit the learned generative classifier from the Procurement stage to complement for the purpose of separate ad-hoc networks (critic or discriminator) as utilized by the prior works(You et al., 2019; Cao et al., 2018b).

Table content: <table/>

Average per-class accuracy (T avg ) for universal-DA tasks on Office-Home dataset (with |C|/|C s ∪ C t | = 0.15). Scores for the prior works are directly taken from UAN(You et al., 2019). in C n . Note that, we impose no restriction on how the hypothetical classes are created (e.g. one can composite non-animal with animal). A random mask is defined which splits the images into two complementary regions using a quadratic spline passing through a central image region (see Appendix Algo. 2). Then, the negative image is created by merging alternate mask regions as shown in Fig.3A. For the I→C task of ImageNet-Caltech, the source domain (ImageNet), consisting of 1000 classes, results in a large number of possible negative classes (i.e. |C n | = |Cs| C 2 ). We address this by randomly selecting only 600 of these negative classes for ImageNet(I), and 200 negative classes for Caltech(C) in the task C→I. In a similar fashion, we generate latent-simulated negative samples only for the selected negative classes in these datasets. Consequently, we compare

Table content: <table><tr><td>Method</td><td colspan="3">Office-Home Ar→Cl Ar→Pr Ar→Rw Cl→Ar Cl→Pr Cl→Rw Pr→Ar Pr→Cl Pr→Rw Rw→Ar Rw→Cl Rw→Pr Avg</td></tr><tr><td>ResNet (He et al., 2016)</td><td>59.37 76.58 87.48 69.86 71.11 81.66 73.72 56.30 86.07</td><td>78.68</td><td>59.22 78.59 73.22</td></tr><tr><td colspan="2">IWAN (Zhang et al., 2018b) 52.55 81.40 86.51 70.58 70.99 85.29 74.88 57.33 85.07</td><td>77.48</td><td>59.65 78.91 73.39</td></tr><tr><td colspan="2">PADA (Zhang et al., 2018b) 39.58 69.37 76.26 62.57 67.39 77.47 48.39 35.79 79.60</td><td>75.94</td><td>44.50 78.10 62.91</td></tr><tr><td>ATI (Busto et al., 2017)</td><td>52.90 80.37 85.91 71.08 72.41 84.39 74.28 57.84 85.61</td><td>76.06</td><td>60.17 78.42 73.29</td></tr><tr><td colspan="2">OSBP (Saito et al., 2018b) 47.75 60.90 76.78 59.23 61.58 74.33 61.67 44.50 79.31</td><td>70.59</td><td>54.95 75.18 63.90</td></tr><tr><td>UAN (You et al., 2019)</td><td>63.00 82.83 87.85 76.88 78.70 85.36 78.22 58.59 86.80</td><td>83.37</td><td>63.17 79.43 77.02</td></tr><tr><td/><td>Source-free adaptation</td><td/><td/></tr><tr><td>Ours USFDA-a</td><td>63.35 83.30 89.35 70.96 72.34 86.09 78.53 60.15 87.35</td><td>81.56</td><td>63.17 88.23 77.03</td></tr><tr><td>Ours USFDA-b</td><td>62.46 82.71 88.26 71.10 70.88 85.75 78.21 59.18 86.05</td><td>82.17</td><td>63.22 87.68 76.47</td></tr><tr><td colspan="2">unique negative classes</td><td/><td/></tr></table>

al. (2019), and other prior methods. The results are presented in Table 1 and Table 2. Clearly, our framework achieves state-T avg on Office-31 (with |C|/|C s ∪ C t | = 0.32), VisDA (with |C|/|C s ∪ C t | = 0.50), and ImageNet-Caltech (with |C|/|C s ∪ C t | = 0.07). Here, SF denotes support for source-free adaptation. T avg 83.00±1.8 94.17±0.3 95.40±0.5 83.43±0.7 86.90±1.0 87.18±0.6 88.34 54.21 74.77 71.51 Ours USFDA-a T avg 85.56±1.6 95.20±0.3 97.79±0.1 88.47±0.3 87.50±0.9 86.61±0.6 90.18 63.92 76.85 72.13 Ours USFDA-b T avg 83.21±1.2 95.33±0.3 96.37±0.3 86.84±0.4 87.91±0.6 86.74±0.5 89.40 62.77 76.74 72.25 UAN* T unk 20.72±11.7 53.53±2.4 51.57±5.0 34.43±3.3 51.88±4.8 43.11±1.3 42.54 19.68 33.43 31.24 Ours USFDA-a T unk 73.98±7.5 85.64±2.2 80.00±1.1 82.23±2.7 78.59±3.2 75.52±1.5 79.32 36.25 51.21 48.76 Ours USFDA-b T unk 70.22±8.8 85.89±2.3 78.29±1.7 84.66±3.1 76.22±2.8 73.91±1.6 78.19 34.84 51.10 48.20

Table content: <table><tr><td>Method</td><td>SF</td><td>A→W</td><td>D→W</td><td>W→D</td><td>Office-31 A→D</td><td>D→A</td><td>W→A</td><td>VisDA ImNet-Caltech Avg S → R I → C C → I</td></tr><tr><td>ResNet (He et al., 2016)</td><td/><td>75.94</td><td>89.60</td><td>90.91</td><td>80.45</td><td>78.83</td><td>81.42</td><td>82.86 52.80 70.28 65.14</td></tr><tr><td>IWAN (Zhang et al., 2018b)</td><td/><td>85.25</td><td>90.09</td><td>90.00</td><td>84.27</td><td>84.22</td><td>86.25</td><td>86.68 58.72 72.19 66.48</td></tr><tr><td>PADA (Zhang et al., 2018b)</td><td/><td>85.37</td><td>79.26</td><td>90.91</td><td>81.68</td><td>55.32</td><td>82.61</td><td>79.19 44.98 65.47 58.73</td></tr><tr><td>ATI (Busto et al., 2017)</td><td/><td>79.38</td><td>92.60</td><td>90.08</td><td>84.40</td><td>78.85</td><td>81.57</td><td>84.48 54.81 71.59 67.36</td></tr><tr><td>OSBP (Saito et al., 2018b)</td><td/><td>66.13</td><td>73.57</td><td>85.62</td><td>72.92</td><td>47.35</td><td>60.48</td><td>67.68 30.26 62.08 55.48</td></tr><tr><td>UAN (You et al., 2019)</td><td/><td>85.62</td><td>94.77</td><td>97.99</td><td>86.50</td><td>85.45</td><td>85.12</td><td>89.24 60.83 75.28 70.17</td></tr><tr><td colspan="8">0.6 0.7 1.0 0.9 0.8 UAN* 0.5 1.0 1.2 1.4 0.04 0.00 0.04 0.08</td><td/></tr><tr><td>0.00</td><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>



Table content: <table><tr><td>-58.53 66.67</td><td>-78.Ours (source-free)</td></tr><tr><td>74.09 82.22 67.61</td><td/></tr><tr><td>82.56 83.87 85.77 63.79</td><td/></tr><tr><td>85.29 84.42 88.54 87.97 -</td><td/></tr></table>

|Cs| i=1 exp(P (us|ci))), where P (us|ci) = N (us|µc i , Σc i ) 9:Update θF s , θD, θG by minimizing LCE, Lv, Lu, and Lp alternatively using separate optimizers. 10:if (iter % U pdateIter == 0) then 11:Recompute µc i , Σc i for each source class ci ; Generate D We create an unbiased batch of training samples for a training iteration by sampling equal number of positive and negative samples from the dataset. For USFDA-a we sample 32 positive images (b +ve = 32) and 32 negative images per training iteration (b -ve = 32). Similarly, for USFDA-b we sample 32 positive images and 32 latent-simulated negative samples. This gives an effective batch size of b +ve + b -ve = 64. e) Use of multiple optimizers for training. In the presence of multiple loss terms, we subvert a time-consuming loss-weighting scheme search by making use of multiple Adam optimizers during training. Essentially, we define a separate optimizer for each loss term, and optimize only one of the losses (chosen in a round robin fashion) in each iteration of training. We use a learning rate of 0.0001 during training. Intuitively, the higher order moment parameters in the Adam optimizer adaptively scale the gradients as required by the loss landscape. From the remaining classes, in alphabetical order, we choose the first 10 classes as source-private (C s ) classes, and the rest 11 as target-private (C t ) classes. For VisDA, alphabetically, the first 6 classes are considered C, the next 3 as C s and the last 3 comprise C t . The Office-Home dataset has 65 categories, of which we use the first 10 classes as C, the next 5 for C s , and the rest 50 classes as C t .B.2 DEPLOYMENT STAGEThe details of the architecture used during the Deployment stage are given in Table7. Note that the Feature Decoder G used during the Procurement stage, is not available during the Deployment stage, restricting complete access to the source data.Training during the Deployment stage. The only trainable component is the Feature Extractor F t , which is initialized from F s at Deployment. Here, the SSM is calculated by passing the target images through the network trained on source data (source model), i.e for each image x t , we calculate ŷ = softmax(D • F s • M (x t )). Note that the softmax is calculated over all |C s | + |C n | classes. This is done by concatenating the outputs of D src and D neg , and then calculating softmax. Then, the SSM is determined by the exponential confidence of a target sample, where confidence is the highest softmax value in the categories in |C s |.

Table content: <table><tr><td>(b) n using the updated priors.</td></tr></table>

Evaluation of the proposed method on A→D task of Office-31 (Saenko et al., 2010) dataset, pretraining the ResNet-50 backbone (M ) on Places instead of Imagenet. Note that, we set |C|/|C s ∪ C t | = 0.32, similiar to the setting used in Table2of the main paper. Additionally, the last two columns of the table show a comparison between our method and UAN(You et al., 2019) with regard to the number of trainable parameters and total training time for adaptation

Table content: <table><tr><td>Method</td><td>ResNet-50 finetuning</td><td colspan="2">Avg. per-class accuracy, T avg</td><td colspan="2">Number of Trainable Parameters</td><td>Training time for Adaptation</td></tr><tr><td>UAN*</td><td/><td/><td>60.98</td><td colspan="2">26.7 Million</td><td>280s</td></tr><tr><td>UAN*</td><td/><td/><td>52.48</td><td colspan="2">5.6 Million</td><td>125s</td></tr><tr><td>USFDA-a</td><td/><td/><td>62.74</td><td colspan="2">3.5 Million</td><td>44s</td></tr><tr><td/><td/><td>-</td><td/><td/><td>-</td></tr><tr><td/><td/><td>-</td><td>33.33</td><td/><td>-</td><td>32.35</td></tr><tr><td/><td/><td>-</td><td colspan="2">49.02 35.22</td></tr><tr><td/><td/><td>-</td><td colspan="2">17.65 43.72 27.59</td></tr><tr><td/><td/><td>-</td><td colspan="2">14.71 28.74 43.6</td><td>-</td></tr></table>

Network architecture for Procurement stage. Hyperparameter α = 0.2

Table content: <table><tr><td>Component</td><td>Trainable?</td><td>Operation</td><td colspan="2">Notation Features Batch Norm? Non-Linearity</td></tr><tr><td>Resnet-50 (Upto AvgPool layer)</td><td/><td/><td>M</td><td>2048</td></tr><tr><td>Feature Extractor</td><td/><td/><td>F s</td><td>256</td></tr><tr><td/><td/><td>Input</td><td/><td>2048</td></tr><tr><td/><td/><td>Fully connected</td><td/><td>1024</td><td>ELU</td></tr><tr><td/><td/><td>Fully connected</td><td/><td>1024</td><td>ELU</td></tr><tr><td/><td/><td>Fully connected</td><td/><td>256</td><td>ELU</td></tr><tr><td/><td/><td>Fully connected</td><td/><td>256</td><td>ELU</td></tr><tr><td>Feature Decoder</td><td/><td/><td>G</td><td>2048</td></tr><tr><td/><td/><td>Input</td><td/><td>256</td></tr><tr><td/><td/><td>Fully connected</td><td/><td>1024</td><td>ELU</td></tr><tr><td/><td/><td>Fully connected</td><td/><td>1024</td><td>ELU</td></tr><tr><td/><td/><td>Fully connected</td><td/><td>2048</td><td>ELU</td></tr><tr><td/><td/><td>Fully connected</td><td/><td>2048</td><td>-</td></tr><tr><td>Classifier</td><td/><td/><td>D</td><td>|C s | + |C n |</td></tr><tr><td/><td/><td>Input</td><td/><td>256</td></tr><tr><td/><td/><td>Fully connected</td><td>D src</td><td>|C s |</td></tr><tr><td/><td/><td>Input</td><td/><td>256</td></tr><tr><td/><td/><td>Fully connected</td><td>D neg</td><td>|C n |</td></tr></table>

Feature Extractor Architecture used for training UAN (You et al., 2019) under the "no ResNet-50 finetuning" case (Refer Table 3 and Section C.1)

Table content: <table><tr><td>Operation</td><td colspan="2">Features Non-Linearity</td></tr><tr><td>Input</td><td>2048</td><td/></tr><tr><td>Fully connected</td><td>512</td><td>ReLU</td></tr><tr><td>Fully connected</td><td>256</td><td>ReLU</td></tr><tr><td>Fully connected</td><td>512</td><td>ReLU</td></tr><tr><td>Fully connected</td><td>2048</td><td>ReLU</td></tr></table>

Network architecture for Deployment stage. Hyperparameter β = 0.1

Table content: <table><tr><td>Component</td><td>Trainable?</td><td>Operation</td><td colspan="2">Notation Features Batch Norm? Non-Linearity</td></tr><tr><td>Resnet-50 (Upto AvgPool layer)</td><td/><td/><td>M</td><td>2048</td></tr><tr><td>Feature Extractor</td><td/><td/><td>F t</td><td>256</td></tr><tr><td/><td/><td>Input</td><td/><td>2048</td></tr><tr><td/><td/><td>Fully connected</td><td/><td>1024</td><td>ELU</td></tr><tr><td/><td/><td>Fully connected</td><td/><td>1024</td><td>ELU</td></tr><tr><td/><td/><td>Fully connected</td><td/><td>256</td><td>ELU</td></tr><tr><td/><td/><td>Fully connected</td><td/><td>256</td><td>ELU</td></tr><tr><td>Classifier</td><td/><td/><td>D</td><td>|C s | + |C n |</td></tr><tr><td/><td/><td>Input</td><td/><td>256</td></tr><tr><td/><td/><td>Fully connected</td><td>D src</td><td>|C s |</td></tr><tr><td/><td/><td>Input</td><td/><td>256</td></tr><tr><td/><td/><td>Fully connected</td><td>D neg</td><td>|C n |</td></tr></table>

Specifications of the machine used for both Procurement and Deployment stages CPU GPU RAM VRAM CUDA Intel i7-7700K NVIDIA GeForce GTX 1080 Ti 32 GB 11 GB V8.0.61

Table content: <table/>

Peer review (iclr2020):
In this paper, the authors studied a so-called source-free domain adaptation problem. They aimed to propose a universal DA method for different relationships between the target label set and the source label set, without accessing the source domain data in the adaptation phase. The problem studied is an interesting and practical problem for domain adaptation. The proposed solution looks promising at the high level. However, there are some major concerns. 1. The term *source-free* is misleading. Indeed, a *source domain* is needed in the procurement stage. Based on the the proposed method, there is still an assumption on the source domain, i.e., there is an overlapping label set between the source domain and the target domain. Given a target domain, if there are not overlapping labels between the target domain and the source domain, the proposed method will fail to work. In this sense, the term *universal* is also misleading. 2. In the proposed procurement stage, a key step is to generate the so-called *negative* source instances. The authors proposed two heuristic approaches, one for image data, and the other for non-image data. These two approaches looks okay, but not convincing as no deeper analysis is provided. Moreover, in experiments, only the image domain is used. The effectiveness of the approach for non-image negative instances generation is not verified. 3. In prediction, for the case where the target domain has a set of private labels, the proposed method can only predict all instances of private labels as *unknown* if successful. This may not be useful in practice. Here are the minor concerns: 1. Authors attempted to compress much content into 9 pages, which makes to the paper a little difficult to follow. 2. In the literature of transfer learning or domain adaptation, a similar problem setup was studied around 10 years ago, which was focused on the application of web text mining. *Xiang et al., Source-Selection-Free Transfer Learning, IJCAI-11*