Title: LEARNING FACTORIZED REPRESENTATIONS FOR OPEN-SET DOMAIN ADAPTATION

Abstract: Domain adaptation for visual recognition has undergone great progress in the past few years. Nevertheless, most existing methods work in the so-called closed-set scenario, assuming that the classes depicted by the target images are exactly the same as those of the source domain. In this paper, we tackle the more challenging, yet more realistic case of open-set domain adaptation, where new, unknown classes can be present in the target data. While, in the unsupervised scenario, one cannot expect to be able to identify each specific new class, we aim to automatically detect which samples belong to these new classes and discard them from the recognition process. To this end, we rely on the intuition that the source and target samples depicting the known classes can be generated by a shared subspace, whereas the target samples from unknown classes come from a different, private subspace. We therefore introduce a framework that factorizes the data into shared and private parts, while encouraging the shared representation to be discriminative. Our experiments on standard benchmarks evidence that our approach outperforms the state of the art in open-set domain adaptation.

1. INTRODUCTION:

In many practical machine learning scenarios, the test samples are drawn from a different distribution from the training ones, due to varying acquisition conditions, such as different data sources, illumination conditions and cameras, in the context of visual recognition. Over the years, great progress has been achieved to tackle this problem, known has the domain shift. In particular, many methods aim to align the source (i.e., training) and target (i.e., test) distributions by learning domaininvariant embeddings (Pan et al., 2011; Gong et al., 2012; Fernando et al., 2013; Sun et al., 2016) , the most recent approaches relying on deep networks (Ganin & Lempitsky, 2014; Long et al., 2015; Bousmalis et al., 2016; Tzeng et al., 2017; Long et al., 2016a; Yan et al., 2017) .

While effective, these methods work under the assumption that the source and target data contain exactly the same classes. In practice, however, this assumption may easily be violated, as the target data will often contain additional classes that were not present within the source data. For example, when training a model to recognize office objects from images, as with the popular Office dataset (Saenko et al., 2010) , one should still expect to see new objects, unobserved during training, when deploying the model in the real world. While one should not expect the model to recognize the specific class of such objects, at least in unsupervised domain adaptation where no target labels are provided, it would nonetheless be beneficial to identify these objects as unknown instead of misclassifying them. This was the task addressed by Busto & Gall (2017) in their so-called open-set domain adaptation approach. This method aims to learn a mapping from the source samples to a subset of the target ones corresponding to those identified as coming from known classes. While reasonably effective, this procedure involves alternatively solving for the mapping and the assignment of the samples to known/unknown classes, which, as shown in our experiments, can be costly. Recently, Saito et al. (2018) introduced a deep learning framework for open-set domain adaptation, relying on adversarial training to separate the samples from the known classes from the unknown ones.

In this paper, we introduce a novel approach to open-set domain adaptation based on learning a factorized representation of the source and target data. In essence, we seek to model the samples from the known classes with a low-dimensional subspace, shared by the source and target domains, and the target samples from unknown classes with another subspace, specific to the target domain. We then make use of group sparsity to encourage each target sample to be reconstructed by only one of these subspaces, which in turns lets us identify if this sample corresponds to a known or unknown class. We further show that we can obtain a more discriminative shared representation by jointly learning a linear classifier within our framework. Ultimately, our approach therefore allows us to jointly separate the target samples between known and unknown classes and represent the source and target samples within a consistent, shared latent space. Note that our approach is more intuitive than (Bousmalis et al., 2016) for the open-set DA scenario in the sense that we model each target sample as being generated by either the shared subspace or the private one, which is crucial to identify the target samples depicting unknown classes. By contrast, in (Bousmalis et al., 2016) , each sample is encoded as a mixture of shared and private representations, which does not provide information to discriminate samples from unknown classes.

We demonstrate the effectiveness of our approach on several open-set domain adaptation benchmarks for visual object recognition. Our method consistently and significantly outperforms the technique of Busto & Gall (2017) on all benchmarks, as well as the end-to-end learning approach of Saito et al. (2018) on the Office dataset, thus showing the benefits of learning shared and private representations corresponding to the known and unknown classes, respectively. Furthermore, it is faster than the algorithm of Busto & Gall (2017) by an order of magnitude.

2. RELATED WORK:

Domain adaptation for visual recognition has become increasingly popular over the past few years, in large part thanks to the benchmark Office dataset of Saenko et al. (2010) . A natural approach to tackling the domain shift consists of learning a transformation of the data such that the distributions of the source and target samples are as similar as possible in the resulting space (Baktashmotlagh et al., 2014; 2013; Sun et al., 2016) . Instead of learning a transformation of the data, other methods have been proposed to re-weight the source samples, so as to rely more strongly on those that look similar to the target ones (Quiñonero C. et al., 2009; Gong et al., 2013) .

With the advent of deep learning for visual recognition, domain adaptation research also eventually turned to exploiting deep networks. While it was initially shown that deep features were more robust than handcrafted ones to the domain shift (Donahue et al., 2013) , translating the abovementioned distribution-matching ideas to end-to-end learning proved even more effective (Tzeng et al., 2014; Long et al., 2015; 2016b; Rozantsev et al., 2018; Sun & Saenko, 2016) . In this context, other ideas were introduced, such as learning intermediate representations to interpolate between the source and target domains (Chopra S. & R., 2013; Tzeng et al., 2015) , the use of adversarial domain classifiers (Ganin & Lempitsky, 2014; Tzeng et al., 2017) , and additional reconstruction loss terms (Ghifary et al., 2016) .

Despite achieving great progress to tackle the domain shift, all the aforementioned methods are designed for the closed-set scenario, where the source and target data depict the exact same set of classes. Inspired by recent advances in open-set recognition (Bendale & Boult, 2015; Scheirer et al., 2014) , the work of Busto & Gall (2017) constitutes the first attempt to address the more realistic case where the target data contains samples from new, unknown classes. To achieve this, Busto & Gall (2017) proposed to jointly learn the assignments of the target samples to known/unknown classes and a mapping from the source data to the target samples depicting known classes. The resulting learning problem was solved by alternatively optimizing for the assignments and for the mapping, which can be costly. Very recently, a deep learning approach was proposed for open-set domain adaptation (Saito et al., 2018) , relying on adversarial training to separate the unknown target samples from the known ones.

Here, we introduce a new solution to the open-set domain adaptation problem, where we model the source and target data with subspaces. Subspace-based representations have proven effective for domain adaptation (Gong et al., 2012; Gopalan et al., 2014; Fernando et al., 2013) . Here, however, we exploit them in a different manner, based on the intuition that source samples and target samples from the known classes can be generated by a shared subspace, whereas target samples from unknown classes come from a private subspace. While the notion of shared-private representations has been exploited in the past, e.g., for multiview learning (Jia et al., 2010) and for closed-set domain adaptation (Bousmalis et al., 2016) , the resulting techniques all use them to encode each sample as a mixture of shared and private information. By contrast, here, we aim to model each target sample as being generated by either the shared subspace or the private one, which is crucial to identify the target samples depicting unknown classes.

Our experiments evidence that our open-set domain adaptation approach, based on shared-private representations, is more effective than the one of Busto & Gall (2017) , consistently outperforming it on several datasets, and also faster by an order of magnitude. We also outperform the recent deep learning open-set domain adaptation framework of Saito et al. (2018) on the Office benchmark.

3. OUR APPROACH:

The key idea behind our formulation is to find low-dimensional representations of the data, factorized into a subspace shared by the source samples and the target ones coming from known classes and another subspace specific to the target samples from unknown classes. Note that, when referring to target samples from known classes, we do not mean that these samples are labeled, but rather that they belong to the same set of classes as the source data. As a matter of fact, throughout the paper, we focus on the unsupervised domain adaptation scenario, where no target annotations are provided. In the remainder of this section, we first introduce the optimization problem at the heart of our approach, and then discuss two extensions of this basic formulation.

3.1. FRODA: FACTORIZED REPRESENTATIONS FOR OPEN-SET DOMAIN ADAPTATION:

Given n s source samples, grouped in a matrix X s ∈ R D×ns and n t target samples represented by X t ∈ R D×nt , our goal is to estimate a low-dimensional representation of each sample, such that the source and target samples coming from the same classes are generated by a shared subspace, whereas the target data from new, unknown classes are generated by a different, specific subspace. To this end, let V ∈ R D×d be the matrix encoding the shared subspace, with d D, and U ∈ R D×d the one representing the private subspace. A naive approach to finding the low-dimensional representations of the data would involve solving

EQUATION

where α sets the relative influence of both terms, B = [V , U ] ∈ R D×2d , and T and S encode the low-dimensional representations of the target and source data, respectively. This simple formulation, however, does not aim to separate the target samples belonging to known classes from the unknown ones, and thus will represent each target sample as a mixture of shared and private information.

Intuitively, we would rather like each target sample to be generated by either the shared subspace V , or the private one U . To address this, we propose to make use of a group sparsity regularizer on the coefficients of the target samples. Specifically, we split the coefficient vector T i for target sample i into a part T v i that corresponds to the shared subspace and a part T u i that corresponds to the private one. We then encourage that either of these two parts goes to zero for each sample. To this end, we therefore write the optimization problem

EQUATION

where the constraints prevent the basis vectors of the subspaces from growing while the coefficients decrease (Lee et al., 2007) , and where λ 1 is a scalar controlling the strength of the group sparsity regularizer. In essence, this formulation allows each target sample to be reconstructed from either the shared subspace or the private one, which reduces the influence of the samples from unknown classes on learning the shared representation.

Optimization. To solve equation 2 efficiently, we alternatively update one variable at a time while keeping the other ones fixed. Below, we describe the different updates.

Algorithm 1 : FRODA: Factorized Representations for Open-set Domain Adaptation Input: X s ∈ R D×ns : the source samples X t ∈ R D×nt : the target samples d D: the dimensionality of the subspaces Output: Compute U from equation 3 by the Lagrange dual method of (Lee et al., 2007) 5:

S ∈ R d×ns , T ∈ R 2d×nt Initialize: V ← P CA(X s ) U ← N ull(V ) (i.e.,

Compute V from equation 4 by the Lagrange dual method of (Lee et al., 2007) 6:

Compute T from equation 5 by proximal gradient descent 7:

Compute S by solving the linear least-squares problem min S X s -V S 2 F 8: until convergence B-minimization: Given the coefficients S and T , we update the tuple (U , V ) by solving

EQUATION

with A = X t -V T v , and

EQUATION

with A = X t -U T u . These two sub-problems can be solved efficiently using the Lagrange dual formulation introduced in (Lee et al., 2007) to update the basis in a standard sparse coding context.

T -minimization: Minimizing equation 2 with respect to T , with all other parameters fixed, yields

EQUATION

which can be solved efficiently via the proximal gradient method (Mairal et al., 2014) . In other words, for each target sample, we obtain T using group sparse coding to determine to which representation, shared or private, each target sample belongs.

S-minimization: Solving equation 2 with respect to S, with all the other parameters fixed, reduces to a linear least-squares problem, which has a closed-form solution.

To start optimization, we initialize V as the PCA subspace of the source data, and take U as its truncated null space. We then obtain the corresponding T and S as described above and start iterating. The pseudo-code of FRODA is provided in Algorithm 1. The steps are repeated until convergence, which typically occurs around 50 iterations, with each iteration taking roughly 0.05s.

Inference. After obtaining the final 2d-dimensional representation of T for the target samples, we determine if each sample i belongs to known classes or unknown ones based on the coefficients T u i and T v i . More specifically, given a threshold ε, a target sample is assigned to the unknown classes if T v i / T u i ≤ ε, which suggests that it can be well-reconstructed by the private subspace. In the presence of C known classes, we then train a (C + 1)-way classifier by augmenting the ddimensional source data S, i.e. S ∈ R d×ns , with the target samples T u identified as unknown.

3.2. D-FRODA: DISCRIMINATIVE FRODA:

The formulation above does not make use of the source labels at all during the representation learning stage. As such, it does not encourage the representation to be discriminative. To overcome this, we extend our basic formulation to further account for the classification task at hand. Specifically, let L = [l 1 . . . l ns ] ∈ R C×ns be the matrix containing the source labels, where l i ∈ R C represents the one-hot encoding of the label of sample i. We then write our D-FRODA formulation as

EQUATION

where W ∈ R C×d is the matrix containing the parameters of a linear classifier for the source data.

Optimization. To optimize equation 6, we follow a similar alternating strategy as before. The Bminimization and T -minimization steps are unchanged, but the S-minimization now incorporates a new term and we further need to solve for the classifier parameters W . This translates to:

S-minimization: Minimizing equation 6 with respect to S, with all the other parameters fixed, still reduces to a linear least-squares problem. The two terms involving S can be grouped into a single one of the form

X new -V new S 2 F , where X new = √ αX s √ βL and V new = √ αV √ βW

, and thus S can be obtained in closed form.

W -minimization: With all the other parameters fixed, finding W corresponds to a linear leastsquares problem, with a closed-form solution.

Inference. The same inference strategy as before can be followed to label the target samples. Another option here is to make use of W to classify the samples identified as belonging to known classes. We compare these two strategies in our experiments.

3.3. D-FRODA-U: D-FRODA WITH UNKNOWN SOURCE CLASSES:

Until now, we have tackled the scenario where there are no unknown classes in the source data, which we believe corresponds to the typical application scenario, since the source data can in general be fully annotated. Nevertheless, to match the scenario of Busto & Gall (2017) , who assume to have access to additional source samples from unknown classes, yet different from the target unknown classes, we introduce a modified version of our approach that takes such auxiliary data into account.

Note that, since one knows which source samples are from unknown classes, it is also possible to simply discard them from training. To nonetheless handle them, we re-write equation 6 as

EQUATION

where X s contains the source samples from both known and unknown classes, and B = [V , U ] ∈ R D×2d denotes the source transformation matrix with U ∈ R D×d the private subspace for the source data. Note that, similarly to the target coefficients, we have now separated the source coefficients for each sample S i into a part corresponding to the shared subspace S v i and a part corresponding to the private one S u i . Note also that the classifier parameters W now account for C + 1 classes, the additional class corresponding to the unknown samples.

Optimization. We follow a similar iterative procedure to the one used before, with modifications to update B and S , as discussed below. (Sun et al., 2016) 68.8 ± 3.3 60.9 ± 2.6 27.2 ± 3.9 40.7 ± 1.5 64.0 ± 2.6 31.4 ± 0.8 ATI (Busto & Gall, 2017) 71.4 ± 2.3 69.0 ± 2.8 37.4 ± 2.6 45.7 ± 3.0 67.9 ± 4.2 37.5 ± 2.7 AODA (Saito et al., 2018) 76.2 ± 1.7 70.9 ± 3.2 57.3 ± 1.1 63.5 ± 2.1 73.5 ± 0.8 60.5 ± 0.8 FRODA 73.8 ± 6.1 71.0 ± 2.0 54.7 ± 2.9 67.5 ± 1.4 74.5 ± 1.7 61.6 ± 2.2 D-FRODA 74.6 ± 5.5 71.4 ± 2.0 55.4 ± 2.7 67.6 ± 1.2 75.0 ± 1.8 61.7 ± 2.1 (Pan et al., 2011) 40.9 ± 2.9 68.6 ± 1.8 34.5 ± 3.8 19.4 ± 2.1 32.0 ± 3.9 31.1 ± 4.6 42 ± 3.04 GFK (Gong et al., 2012) 42.6 ± 2.4 73.3 ± 3.6 32.7 ± 3.6 16.9 ± 1.5 28.6 ± 3.8 26.4 ± 1.1 41.6 ± 2.5 SA (Fernando et al., 2013) 

Method I → B I → C I → S S → B S → C S → I Avg. TCA

EQUATION

)

The first two terms can be grouped into a single squared Frobenius norm, thus resulting in a sparse group lasso problem, which, as when updating T in FRODA, can be solved via proximal gradient descent (Mairal et al., 2014) .

B -minimization: Given the coefficients S v and S u , we can update U by solving

EQUATION

with A = X s -V S v , and V by solving

EQUATION

) with A = X t -U T u √ α(X s -U S u ) and C = T v √ αS v .

As in FRODA, these two sub-problems can be solved efficiently using the Lagrange dual formulation of Lee et al. (2007) .

4. EXPERIMENTS:

We evaluate our approach on the task of open-set visual domain adaptation using two benchmark datasets, and compare its performance against the state-of-the-art open-set domain adaptation methods on each dataset.1 Note that we also report the results of the methods used as baselines in (Busto & Gall, 2017) . For a dataset with C source classes, we report the accuracy on C + 1 classes, the additional one corresponding to the unknown case. Implementation details. Following Busto & Gall (2017) , we represent the source and target samples with 4096-dimensional DeCAF 7 features (Donahue et al., 2013) and first reduce their dimensionality by performing PCA jointly on the source and target data and keeping the components encoding 99% of the data variance. To then determine the dimensionality d of our shared and private subspaces, we make use of the subspace disagreement measure of (Gong et al., 2012) . For all our experiments, the hyperparameters of our approach were set as follows: α = 0.1, β = 0.01, λ 1 = 0.001, λ 2 = 0.001 and ε = 0.2. For recognition, for the comparison with (Busto & Gall, 2017) to be fair, we employ a linear SVM classifier in a one-vs-one fashion. Nevertheless, we also report results obtained with a k-Nearest-Neighbor classifier (with k = 3) and with our linear classifier with parameters W learnt during training. Results on the dense cross-dataset benchmark. We first evaluate our approach on the challenging cross-dataset benchmark of Tommasi & Tuytelaars (2014) . This dataset was built using images depicting 40 object categories and coming from four datasets, namely Bing (B), Caltech256 (C), ImageNet (I) and SUN (S), hence referred to as BCIS. Following Busto & Gall (2017) , we consider the samples from the first 10 classes as known instances, while the samples with class labels 11, 12, • • • , 25 and 26, 27, • • • , 40 are taken to be the unknown samples in the source and target domains, respectively. We follow the unsupervised protocol of Tommasi & Tuytelaars (2014) , which relies on 50 source samples per class and 30 target images per class, except when the target data is coming from SUN, in which case only 20 images per class are employed. Note that only the DeCAF 7 features are publicly available. To nonetheless evaluate the AODA method of Saito et al. (2018) , we made use of a network taking the DeCAF 7 features as input and processing them with two fully-connected layers, with 1024 and 128 units, respectively, and a final classification layer.

In Table 1 , we compare the results of our methods with those of the baselines on all 12 domain pairs of this dataset. Note that our algorithms (both with and without the discriminative term) outperform all the baselines, and in particular the state-of-the-art one of Busto & Gall (2017) by a large margin. For instance, the margin exceeds 32, resp. 26, percentage points when going from SUN to Bing and ImageNet, respectively. This, we believe, clearly evidences the benefits of our factorized representations, which allow us to separate the unknown target samples from the ones coming from known classes, thus yielding a better representation for the known classes. In Table 2 , we compare different versions of our method, corresponding to using different classifiers and to using additional unknown source data. Note that the linear SVM classifier, when used with our framework, tends to perform the best, followed by the NN one and finally the learnt linear classifier. This, we believe, can be explained by the fact that, while the linear classifier helps to learn a more discriminative representation, it remains less powerful than the other two classifiers to label the target samples. Note also that the use of unknown source data does not consistently help in our framework. Nevertheless, the corresponding results still outperform those of Busto & Gall (2017) .

We further evaluate the robustness of our approach to the choice of threshold ε to separate the target samples from known/unknown classes. In Fig. 1 , we plot the average accuracy over all 12 pairs of the BCIS dataset as a function of the value of ε. Note that, once a sufficiently large threshold is reached, the results are quite stable. This indicates that our algorithm is robust to the specific value of this hyperparameter.

Results on the Office dataset. We further evaluate our approach on the slightly less challenging, although standard Office benchmark (Saenko et al., 2010) . This dataset contains three different domains, namely Amazon (A), DSLR (D) and Webcam (W), sharing 31 object categories, but differing in data acquisition process. As in (Busto & Gall, 2017) , we take all the samples from the first 10 classes to represent the known ones, and all the samples with class labels 11, 12, • • • , 20 and 21, 22, • • • , 31 as unknown source and target data, respectively.

We report the results of our algorithms and of the baselines for all 6 domain pairs of this dataset in Table 3 . As before, note that we outperform the baselines in this open-set scenario. This includes the end-to-end AlexNet-based approach of Saito et al. (2018) for open-set domain adaptation, as well as the state-of-the-art UDA methods of Tzeng et al. (2017) and Bousmalis et al. (2016) , the latter of which is closest in spirit to our approach. In Table 4 , we compare the different variants of our approach. The conclusions that one can draw from these results are similar to those for the BCIS dataset, thus showing that our method generalizes well across different domain adaptation datasets.

To evaluate the robustness of our method to the hyper-parameters α, β, and λ 1 , in Fig. 2 , we plot the average accuracy of D-FRODA-NN (with k = 3) over all 6 pairs of the Office dataset as a function of the value of α, β, and λ 1 . Note that our results are stable for large ranges of these values.

Runtimes. As mentioned in Section 3, one iteration of our approach takes on average 0.05 second, and our algorithm typically takes around 50 iterations to converge. This yields a total runtime of roughly 2.5 seconds. By contrast, the publicly available implementation of the method of Busto & Gall (2017) takes on average 8 seconds per iteration and typically converges in 4 iterations, leading to a total runtime of roughly 32 seconds. Note that these runtimes were measured on the same computer and that both methods rely on the same input features. Therefore, our approach is not only significantly more accurate than (Busto & Gall, 2017) , but also faster by an order of magnitude. Further discussion: The experimental setup used in (Saito et al., 2018; Busto & Gall, 2017) and our work for open-set DA relies on features extracted using a network pre-trained on ImageNet, which in fact can be argued to already contain semantic information about some of the unknown classes. To evidence that our approach does not crucially depend on this information, and thus validate our results, we observed that 4 of the unknown classes in the Office dataset, namely Tape dispenser, Stapler, Scissors, Punchers, do not appear in ImageNet. We therefore performed additional experiments with only these classes as unknown ones and the same 10 known classes as before. We compared the accuracy of our formulations against the SVM baseline and the open-set ATI method of Busto & Gall (2017) . The gap with respect to both baselines remains large: SVM: 76.01%, ATI: 77.4%, D-FRODA: 78.2%, and FRODA: 78.5%. This confirms that our method applies to truly never-seen-before classes.

Note that among the 15 unknown classes in the setup for BCIS, 8 of them are not shared with ImageNet, namely Windmill, Steering wheel, Can-soda, Sneaker, Skyscraper, Ladder, Motorcycle, and Palm tree. This further confirms that our method handles the cases where no categorical or semantic information is available in the extracted features.

:

Among the different variants of ATI(Busto & Gall,

2017), we report the best one in each experiment.

CONCLUSIONWe have introduced a novel approach to open-set domain adaptation, based on the intuition that source and target samples coming from the same, known classes can be represented by a shared subspace, while target samples from unknown classes should be modeled with a private subspace. Each step of the resulting algorithms can be solved efficiently. As demonstrated by our experiments, our method outperforms the state of the art in open-set domain adaptation and is one order of magnitude faster than the technique ofBusto & Gall (2017). We believe that this clearly evidences the benefits of learning factorized representations, which allows us to jointly discard the unknown target samples and learn a better shared representation. In the future, we will investigate ways to make better use of unknown source data, and to exploit more effective classifiers, such as SVM, directly within our D-FRODA formulation.

Figures and Tables:

truncated null space of V ) 1: Compute T from equation 5 by proximal gradient descent 2: Compute S by solving the linear least-squares problem min S X s -V S 2

Figure 1: Sensitivity to ε

Figure 2: Sensitivity to α, β, and λ1.

Recognition accuracies on the 12 source/target pairs of the BCIS dataset(Tommasi & Tuytelaars, 2014) using a linear SVM classifier. B: Bing, C: Caltech256, I: ImageNet, S: SUN.

Table content: <table><tr><td>Method</td><td>B → C</td><td>B → I</td><td>B → S</td><td>C → B</td><td>C → I</td><td>C → S</td></tr><tr><td>TCA (Pan et al., 2011)</td><td>62.8 ± 3.8</td><td>56.6 ± 4.5</td><td>29.6 ± 4.2</td><td>38.9 ± 1.9</td><td>60.2 ± 1.4</td><td>29.7 ± 1.6</td></tr><tr><td>GFK (Gong et al., 2012)</td><td>66.2 ± 4.0</td><td>58.3 ± 3.1</td><td>23.8 ± 2.0</td><td>40.2 ± 1.8</td><td>62.2 ± 1.5</td><td>28.5 ± 1.0</td></tr><tr><td colspan="2">SA (Fernando et al., 2013) 66.0 ± 3.4</td><td>57.8 ± 3.2</td><td>24.3 ± 2.6</td><td>40.3 ± 1.7</td><td>62.5 ± 0.8</td><td>29.0 ± 1.5</td></tr><tr><td>CORAL</td><td/><td/><td/><td/><td/><td/></tr></table>

Recognition accuracies of variants of our approach using linear SVM, nearest neighbor (NN) and our linear classifier (W ) on the 12 source/target pairs of the BCIS dataset(Tommasi & Tuytelaars, 2014). B: Bing, C: Caltech256, I: ImageNet, S: SUN.

Table content: <table><tr><td>Method</td><td>B → C</td><td>B → I</td><td>B → S</td><td/><td colspan="2">C → B</td><td colspan="2">C → I</td><td>C → S</td></tr><tr><td>FRODA-SVM</td><td>73.8 ± 6.1</td><td>71.0 ± 2.0</td><td colspan="2">54.7 ± 2.9</td><td colspan="2">67.5 ± 1.4</td><td colspan="2">74.5 ± 1.7</td><td>61.6 ± 2.2</td></tr><tr><td>FRODA-NN</td><td>67.7 ± 2.8</td><td>64.1 ± 2.4</td><td colspan="2">54.4 ± 3.5</td><td colspan="2">65.1 ± 2.9</td><td colspan="2">72.9 ± 1.7</td><td>60.5 ± 2.0</td></tr><tr><td>D-FRODA-SVM</td><td colspan="8">74.6 ± 5.5 71.4 ± 2.0 55.4 ± 2.7 67.6 ± 1.2 75.0 ± 1.8 61.7 ± 2.1</td></tr><tr><td>D-FRODA-W</td><td>61.2 ± 1.2</td><td>59.3 ± 1.1</td><td colspan="2">53.3 ± 3.0</td><td colspan="2">63.1 ± 0.9</td><td colspan="2">66.2 ± 1.7</td><td>60.2 ± 2.1</td></tr><tr><td>D-FRODA-NN</td><td>67.7 ± 3.3</td><td>63.3 ± 2.8</td><td colspan="2">54.0 ± 3.5</td><td colspan="2">65.4 ± 2.8</td><td colspan="2">73.4 ± 1.5</td><td>60.3 ± 2.4</td></tr><tr><td colspan="2">D-FRODA-U-SVM 71.9 ± 3.8</td><td colspan="5">69.2 ± 2.7 56.7 ± 3.3 64.8 ± 1.8</td><td colspan="2">72.8 ± 2.7</td><td>59.4 ± 2.3</td></tr><tr><td>D-FRODA-U-W</td><td>55.9 ± 3.6</td><td>55.5 ± 4.5</td><td colspan="2">41.7 ± 4.8</td><td colspan="2">52.6 ± 4.7</td><td colspan="2">61.4 ± 3.6</td><td>49.5 ± 4.1</td></tr><tr><td>D-FRODA-U-NN</td><td>57.6 ± 9.1</td><td>52.2 ± 5.0</td><td colspan="2">47.5 ± 6.5</td><td colspan="2">51.8 ± 5.7</td><td colspan="2">64.0 ± 5.7</td><td>57.7 ± 5.6</td></tr><tr><td>Method</td><td>I → B</td><td>I → C</td><td>I → S</td><td/><td>S → B</td><td colspan="2">S → C</td><td>S → I</td><td>Avg.</td></tr><tr><td>FRODA-SVM</td><td>66.0 ± 1.9</td><td>79.9 ± 1.7</td><td>59.2 ± 2.1</td><td colspan="2">55.7 ± 2.5</td><td colspan="2">61.2 ± 1.8</td><td>59.4 ± 1.9</td><td>65.4</td></tr><tr><td>FRODA-NN</td><td>60.9 ± 3.7</td><td>77.7 ± 2.8</td><td>58.0 ± 2.2</td><td/><td>± 2.2</td><td colspan="2">61.2 ± 1.4</td><td>58.1 ± 1.5</td><td>62.8</td></tr><tr><td>D-FRODA-SVM</td><td colspan="5">66.4 ± 1.7 80.5 ± 1.6 59.8 ± 2.0 55.5 ± 2.4</td><td colspan="2">61.2 ± 1.9</td><td>59.6 ± 2.2 65.7</td></tr><tr><td>D-FRODA-W</td><td>62.2 ± 1.6</td><td>70.5 ± 2.5</td><td colspan="5">58.1 ± 1.7 56.4 ± 1.9 58.9 ± 1.5</td><td>58.5 ± 0.7</td><td>60.7</td></tr><tr><td>D-FRODA-NN</td><td>60.9 ± 4.1</td><td>78.7 ± 2.8</td><td>57.7 ± 2.0</td><td colspan="2">53.0 ± 2.3</td><td colspan="2">61.2 ± 1.2</td><td>57.9 ± 1.6</td><td>62.8</td></tr><tr><td colspan="2">D-FRODA-U-SVM 66.0 ± 1.2</td><td>76.8 ± 1.9</td><td>57.2 ± 4.5</td><td colspan="5">56.3 ± 1.9 61.8 ± 3.0 59.9 ± 1.7 64.4</td></tr><tr><td>D-FRODA-U-W</td><td>54.6 ± 4.4</td><td>66.2 ± 3.8</td><td>43.9 ± 5.4</td><td colspan="2">47.6 ± 3.8</td><td colspan="2">53.5 ± 4.9</td><td>51.3 ± 3.7</td><td>52.8</td></tr><tr><td>D-FRODA-U-NN</td><td>58.4 ± 6.1</td><td>70.9 ± 4.5</td><td>52.8 ± 9.0</td><td colspan="2">55.4 ± 2.0</td><td colspan="2">61.5 ± 2.0</td><td>60.1 ± 1.7</td><td>57.5</td></tr></table>

Recognition accuracies on the 6 source/target pairs of the Office dataset(Saenko et al., 2010) using a linear SVM classifier. A: Amazon, W: Webcam, D: DSLR.

Table content: <table><tr><td>Method</td><td colspan="7">A → D A → W W → A W → D D → A D → W Avg.</td></tr><tr><td>LSVM</td><td>72.6</td><td>57.5</td><td>49.2</td><td>98.8</td><td>45.1</td><td>88.5</td><td>68.6</td></tr><tr><td>DAN (Long et al., 2016a)</td><td>77.6</td><td>72.5</td><td>60.8</td><td>98.3</td><td>57</td><td>88.4</td><td>75.8</td></tr><tr><td>RTN (Long et al., 2016b)</td><td>76.6</td><td>73</td><td>62.4</td><td>98.8</td><td>57.2</td><td>89</td><td>76.2</td></tr><tr><td>BP (Ganin &amp; Lempitsky, 2014)</td><td>78.3</td><td>75.9</td><td>64</td><td>98.7</td><td>57.6</td><td>89.8</td><td>77.4</td></tr><tr><td>ADDA (Tzeng et al., 2017)</td><td>52.5</td><td>58.3</td><td>54.1</td><td>89.1</td><td>45.3</td><td>79.1</td><td>63.1</td></tr><tr><td>DSN (Bousmalis et al., 2016)</td><td>58.3</td><td>57.2</td><td>55.1</td><td>79.3</td><td>58.1</td><td>70.2</td><td>63.0</td></tr><tr><td>ATI (Busto &amp; Gall, 2017)</td><td>79.8</td><td>78.4</td><td>76.7</td><td>98.8</td><td>71.3</td><td>94.4</td><td>83.2</td></tr><tr><td>AODA (Saito et al., 2018)</td><td>76.6</td><td>74.9</td><td>81.2</td><td>96.9</td><td>62.3</td><td>94.6</td><td>81.1</td></tr><tr><td>FRODA</td><td>88.0</td><td>78.7</td><td>76.5</td><td>98.0</td><td>73.7</td><td>94.6</td><td>84.9</td></tr><tr><td>D-FRODA</td><td>87.4</td><td>78.1</td><td>77.1</td><td>98.5</td><td>73.6</td><td>94.4</td><td>84.9</td></tr></table>

Recognition accuracies of variants of our approaches using a linear SVM, nearest neighbor (NN) and our linear classifier (W ) on the 6 source/target pairs of the Office dataset(Saenko et al., 2010).

Table content: <table><tr><td>Method</td><td colspan="7">A → D A → W W → A W → D D → A D → W Avg.</td></tr><tr><td>FRODA-SVM</td><td>88.0</td><td>78.7</td><td>76.5</td><td>98.0</td><td>73.7</td><td>94.6</td><td>84.9</td></tr><tr><td>FRODA-NN</td><td>83.9</td><td>69.5</td><td>75.0</td><td>97.7</td><td>69.0</td><td>83.9</td><td>79.8</td></tr><tr><td>D-FRODA-SVM</td><td>87.4</td><td>78.1</td><td>77.1</td><td>98.5</td><td>73.6</td><td>94.4</td><td>84.9</td></tr><tr><td>D-FRODA-NN</td><td>83.9</td><td>70.1</td><td>75.1</td><td>96.8</td><td>69.2</td><td>84.5</td><td>79.9</td></tr><tr><td>D-FRODA-W</td><td>71.1</td><td>65.3</td><td>68.1</td><td>83.0</td><td>67.3</td><td>79.1</td><td>72.3</td></tr><tr><td>D-FRODA-U-SVM</td><td>81.9</td><td>83.5</td><td>75.5</td><td>96.2</td><td>70.6</td><td>94.2</td><td>83.7</td></tr><tr><td>D-FRODA-U-NN</td><td>78.1</td><td>72.1</td><td>69.1</td><td>93.6</td><td>67.0</td><td>75.7</td><td>75.9</td></tr><tr><td>D-FRODA-U-W</td><td>73.4</td><td>65.9</td><td>62.8</td><td>88.0</td><td>61.7</td><td>65.1</td><td>69.5</td></tr></table>

Peer review (iclr2019):
 Pros: - Paper proposes a somewhat complicated but easy to understand idea for open set classification. Formulation is quite intriguing. - Outperforming recent baselines on most scenarios, despite being a linear classifier on fixed CNN features. Cons: - Experiment setup somewhat flawed (but the same flaw is in prior work too) To elaborate: DeCAF7 is trained on ImageNet, which gives the underlying network extra categorical information of the 1000 classes. Some of these clases are arguably in the "unknown classes" in the open set setting. This may jeopardize the premise since the feature knows those classes are semantically different from known classes. Unfortunately (Busto & Gall, 2017) and (Saito et al., 2018) do this too. This is especially problematic since DeCAF7 has a near-linear relationship to the final sigmoid logits, which are the 1000-way ImageNet class scores. This makes the authors formulation (separate subspaces for known and unknown classes) more easily exploit this leaked information. This is because the 1000-way scores obviously have subspaces for all 1000 ImageNet classes, and by extension, the "known" and "unknown" classes in the open set setting. If this is true and is the main reason that the proposed method outperforms, I would not consider the conclusion of the paper very informative. Instead, its signifies the need of a better experiment setup for the problem. A way to strengthen the paper is to use a network pre-trained on other datasets (e.g. Places, or a subset of ImageNet) to verify the findings of the paper. - Lacks clarity for what is being done at test time. I cannot find whether the final SVM is trained on original DeCAF features, or S and T. If it is the latter, how are the representations of target domain data obtained at test time? Are they d dimentional or 2d dimentional? Can you clarify that the test samples are not used for unsupervised training? - Experiment elaborate but feels incomplete. It feels like the authors are proposing 3 variations of the method, and there is not one of them that consistently outperform the others. If so, the paper would lack some ablation analysis that provides insights of what makes the FRODA-SVM outperform prior art. For example, how much do the hyperparameters matter? What happens if e.g. d or lambda1 is very large/small? Clarity: - Abstract spends too much time on defining problem setup - "Faster than prior work" refers to the training time, and excludes the DeCAF feature extraction. Originality: I am not familiar with the related work. Significance: It is quite impressive that a linear model on fixed CNN activations outperforms prior art. However, see the first point in the cons. ----------- Edit: most of the issues listed in "cons" are addressed. Although the additional experiments are not very comprehensive, they can better support the claims. I am bumping up the rating to 7.