LEARNING FACTORIZED REPRESENTATIONS FOR OPEN-SET DOMAIN ADAPTATION

Abstract: Domain adaptation for visual recognition has undergone great progress in the past few years. Nevertheless, most existing methods work in the so-called closed-set scenario, assuming that the classes depicted by the target images are exactly the same as those of the source domain. In this paper, we tackle the more challenging, yet more realistic case of open-set domain adaptation, where new, unknown classes can be present in the target data. While, in the unsupervised scenario, one cannot expect to be able to identify each specific new class, we aim to automatically detect which samples belong to these new classes and discard them from the recognition process. To this end, we rely on the intuition that the source and target samples depicting the known classes can be generated by a shared subspace, whereas the target samples from unknown classes come from a different, private subspace. We therefore introduce a framework that factorizes the data into shared and private parts, while encouraging the shared representation to be discriminative. Our experiments on standard benchmarks evidence that our approach significantly outperforms the state-of-the-art in open-set domain adaptation.

Reviews and author responses:

Thank you for your response.

We re-ran the experiments on truly-unknown classes by further removing the Phone class in the Office experiment and compared the accuracy of our approach against the SVM baseline and the open-set ATI method of (Busto & Gall, 2017) using remaining 4 truly-unknown classes. The conclusions essentially remain unchanged, with the new results being: SVM: 76.01%, ATI: 77.4%, and D-FRODA: 78.2%. Note that, in this case, FRODA performs even a bit better than D-FRODA with 78.5% accuracy.
 
The results in Fig. 2 of the revised version were computed with k=1 nearest-neighbors, instead of k=3 for those in Table 4. Thank you for noticing this. We recomputed all the curves with k=3 and will revise the paper accordingly. The updated plots are available at: https://ibb.co/DG1k13Z . Note that we now also evaluate the influence of the value k on the results, although this parameter is only required in the NN version of our algorithm.

Pros:
- Paper proposes a somewhat complicated but easy to understand idea for open set classification. Formulation is quite intriguing.
- Outperforming recent baselines on most scenarios, despite being a linear classifier on fixed CNN features.

Cons:
- Experiment setup somewhat flawed (but the same flaw is in prior work too)
    To elaborate: DeCAF7 is trained on ImageNet, which gives the underlying network extra categorical information of the 1000 classes. Some of these clases are arguably in the "unknown classes" in the open set setting. This may jeopardize the premise since the feature knows those classes are semantically different from known classes. Unfortunately (Busto & Gall, 2017) and (Saito et al., 2018) do this too.
    This is especially problematic since DeCAF7 has a near-linear relationship to the final sigmoid logits, which are the 1000-way ImageNet class scores. This makes the authors formulation (separate subspaces for known and unknown classes) more easily exploit this leaked information. This is because the 1000-way scores obviously have subspaces for all 1000 ImageNet classes, and by extension, the "known" and "unknown" classes in the open set setting. 
    If this is true and is the main reason that the proposed method outperforms, I would not consider the conclusion of the paper very informative. Instead, its signifies the need of a better experiment setup for the problem.
    A way to strengthen the paper is to use a network pre-trained on other datasets (e.g. Places, or a subset of ImageNet) to verify the findings of the paper.
- Lacks clarity for what is being done at test time. 
    I cannot find whether the final SVM is trained on original DeCAF features, or S and T. If it is the latter, how are the representations of target domain data obtained at test time? Are they d dimentional or 2d dimentional?
    Can you clarify that the test samples are not used for unsupervised training?
- Experiment elaborate but feels incomplete.
    It feels like the authors are proposing 3 variations of the method, and there is not one of them that consistently outperform the others. If so, the paper would lack some ablation analysis that provides insights of what makes the FRODA-SVM outperform prior art. For example, how much do the hyperparameters matter? What happens if e.g. d or lambda1 is very large/small?

Clarity:
- Abstract spends too much time on defining problem setup
- "Faster than prior work" refers to the training time, and excludes the DeCAF feature extraction.

Originality:
I am not familiar with the related work.

Significance:
It is quite impressive that a linear model on fixed CNN activations outperforms prior art. However, see the first point in the cons.


-----------
Edit: most of the issues listed in "cons" are addressed. Although the additional experiments are not very comprehensive, they can better support the claims. I am bumping up the rating to 7.


- Unfortunately some of the classes you listed are still in ImageNet. Namely,
Phone: 487, 528, 707
Tshirt: 610
People: 981, 982, 983
Arguably, if subcategories of a class are seen, then the class is not really new, since the features are trained to distinguish them from the rest of the categories too.
But I have some confidence that your method can still perform better than compared methods if you leave these out. Please kindly include the experiment in the appendix on any updated versions, and either remove the classes I listed or argue that they are indeed new classes.

- Apologies for not seeing the reference of Fig. 2. Can you update the text and its figure caption so that it is clearer? 
Also, Figure 2 plots seem to peak at around 78%, while in Table 4 D-FRODA-NN's average performance is 79.9%. Are you sure this is the experiment you ran for Figure 2?

- Thank you for the update. Please kindly include the variance in Table 1 in a future version.

The paper addresses the problem of Domain Adaptation (DA) in an open setting (OSDA): while traditional DA assumes that the set of classes of the source and the target are identical, in Open-set DA, there are samples in the target which do not belong to any class in the source (unknown classes that I will outliers in this review). The main difficulty of Open-set DA is to simultaneously discard outliers and correctly classify other samples in the target. There are only two papers on Open-set DA so far, Busto'17 and Saito'18.
The method proposed by the authors can be summarized in a single equation, eq. 2, where they aim at learning a linear mapping to a latent space, which can be separated into two sub-spaces U (private space) and V (shared space) such that target outliers will be mapped to 0 in V while source and target non-outliers will be mapped to 0 in U, and hence separate outliers with non-outliers. To solve eq. 2, the authors convert it to Eqs. 3, 4, and 5 and apply techniques in Lee'07 and Mairal'14. The authors propose an extension for learning a linear classifier simultaneously and an extension for incorporating also unknown source classes (i.e. source outliers) when appropriate. An experimental evaluation on 2 datasets show the good performance of the method.

Pros:
-A novel method for a rather new and understudied so far, the work is then interesting for this setting
-Good results reported

Cons:
-The criterion used for choosing when examples are outliers seems heuristic, more discussion would be welcomed as well as some qualitative analysis for showing the interest of the method
-Existing baseline of Saito'18 not used in the 1st experiment
-Some parts require more justification

*Comments:

-The idea of the method is similar to the one of Jia'10 (Eq.6) for multi view learning, but this is rather new for Open-set DA.

-In order to separate target samples to either private or shared, the authors "encourage that either of these two parts (i.e. vectors T_i^u and T_i^v) goes to zero for each sample", which is reasonable. To achieve this the authors use sparse coding method coming from Lee'07. However, this does not make sense to me, because the sparse coding algorithm will encourage both T_i^u and T_i^v to be sparse, but nothing forces one of them to go to the zero-vector.
The authors should then better justify this choice. In particular, I wonder if adding explicitly the criterion used for identifying outliers as a new constraint to satisfy. Then, the optimization problem considered would make more sense to me.

Anyway, the authors could perform additional experiments to show the effectiveness of their method: (i) apply on a classic DA problem where we will expect that ||T^u|| or ||U|| (private subspace for outliers) should be close to zero. 
Add a qualitative analysis on the values of  |T^u|| and |T^v|| - both in Open-set DA and classic DA - showing that the results are as expected. 

- The 1st method (Eq.2) learns the latent space without using any label in the source (i.e there are only two labels: outlier or non-outlier, and all source samples are labeled non-outlier). Thus, the authors resort to the assumption that outliers are farther from source samples than non-outliers. This assumption is strong and may not hold in practice for two reasons: (1) the domain shift can be large and (2) without clustering techniques, many outliers can easily fall into the safe non-outliers zone (consider 0-4 for outliers and 5-9 for non-outliers, high chance this method will incorrectly classify 0 or 3 as non-outliers since 6,8,9 are already non-outliers). 

- The Lagrange dual method (Lee'07, Eq. 6) solves an optimization problem with multiple quadratic constraints, i.e. ||U_j||^2 \le c for every j. However, the authors apply it to solve a problem (eq. 3 and 4) with a single linear constraint which is not quadratic: \sum ||U_j|| \le 1. Please explain:
(i) Why do you use that constrain instead of the one in Lee'07?
(ii) With your constrain, does the Lagrange dual method still work? 

-The authors mention that they reported the results reported by Busto'17 in their experiment. Does this mean that the experiments were not reproduced? If so this seems rather unfair for other baselines since they may have worked on different instances. 
Many baselines are not specific to Open-set DA, so it is rather expected to see bad results.
Since OSDA is new, it is true that there exists only two true baselines: Busto'17 and Saito'18. However, Saito'18 does not appear in BCIS benchmark (although appears in Office benchmark). Please add Saito'18 to the BCIS benchmark.

-The authors use fixed parameters for all the subproblems, I am a bit surprised by this choice, I would rather expect a parameterization task-dependent. Does this mean that the method is hard to tune ?

-The method seems complex, is there any convergence guarantee?

--
After rebuttal: thanks many points were answered.

Thank you for your response.

- We extracted the truly-new classes in Office and BCIS by comparing their classes with those of ImageNet (https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a). The truly-new classes in Office are:
Tape dispenser, Stapler, Scissors, Punchers, and Phone. Those in BCIS are: Windmill, Tshirt, Steering wheel, Can_soda, sneaker, skyscraper, Ladder, Motorcycle, Palmtree, and People. Computing results on these truly-new classes does not involve only evaluation because all methods work in the transductive setting and were thus re-trained for the truly-new class scenario. Note that, despite our best efforts at tuning hyper-parameters, we were unable to obtain reasonable results for AODA for this experiment.

We initially thought of incorporating these results to the main paper, but felt it would be confusing for the reader unless a rather long explanation was provided. Following the reviewer’s suggestion, we will include and discuss them in detail in the appendix. 

- Figure 2 is referenced just above the paragraph Runtimes on Page 8, but we will clarify the details of this experiment. In particular, Fig. 2 evaluates the robustness of our method to the hyper-parameters of α, β, and λ. To this end, we plot the average accuracy of D-FRODA-NN from Table 4 over all 6 pairs of the Office dataset as a function of the value of α, β, and λ.

- The variance of the average for AODA and our approach in Table 1 are 1.7 and 2.2, respectively. While we agree that we only have a small advantage over AODA on BCIS, the gap is larger on Office. We will clarify this in the final version of the paper.

Thank you for your response! I am glad that you did an extra evaluation on truly new samples. However, I am having a hard time interpreting the additional results, because there is not enough explanation in the rebuttal and the updated pdf.

- Specifically, have you updated the truly-new samples' results in the paper, or at least as part of the appendix? Can you list the truly-new classes in Office that are not in ImageNet? Why are there classes in BCIS that are not in ImageNet, considering that ImageNet is actually one of the B,C,*I*,S datasets? Which source/target pair is your extra results performed on? Why not compare to all the compared methods, since it is only evaluation and is not computationally expensive?

- Another hard-to-tell update is Figure 2 -- it is not referenced in the text, and I have no idea which experiment (which column of Table 3, if it is about an experiment in Table 3) this figure is referring to. What does "revised version of Office" mean? Is it on the 5 truly-new classes settings (there is no table or any reference in the text about this setting)?

- Regarding the "average" column that Reviewer 2 has mentioned -- it lacks a variance. Considering Saito (2018) is close to your performance, I would like to see the variance of the average of both methods' performance.


As requested by AnonReviewer2, we have now added a column to each table showing the average for all methods. This column further evidences that we outperform the baselines.

Regarding the AODA results, in our submission, we reported the numbers from the arXiv version of this paper, which was the only available version then. We have now updated the AODA results according to the final version of the paper. Note that the conclusions remain unchanged.

To evaluate the robustness of our method to the hyper-parameters \alpha, \beta and \lambda, we have included a figure plotting the accuracy, averaged over all pairs of the Office dataset, as a function of the values of these hyper-parameters. As shown in Figure 2 of the revised version, our results are stable for a wide range of values.

To determine the dimensionality d of our subspaces, we make use of the subspace disagreement measure of Gong’12, as explained in the implementation details of the paper.

AnonRev3 is concerned that the experimental setup used in (Busto & Gall, 2017), (Saito et al., 2018) and our work for open-set DA is flawed because the methods rely on features extracted using a network pre-trained on ImageNet, which in fact has seen the unknown classes. To address this and validate our results, we observed that 5 of the unknown classes in the Office dataset do not appear in ImageNet. We therefore performed additional experiments with only these classes as unknown ones and the same 10 known classes as before. We compared the accuracy of our D-FRODA formulation against the SVM baseline and the open-set ATI method of (Busto & Gall, 2017). The gap with respect to both baselines remains large: SVM: 76.8%, ATI: 77.01%, and D-FRODA: 79.2%. This confirms that our method applies to truly never-seen-before classes. Note that among the 15 unknown classes in the current setup for BCIS, only 5 are shared with ImageNet.

At test time, we use the low-dimensional representations S and T for classification. Specifically, we first determine whether each target sample belongs to a known or unknown class based on the ratio of ||T_i^v|| to ||T_i^u||, as discussed in the Inference paragraph below Eq. 5. In the presence of C known classes, we then train a (C+1)-way classifier by augmenting the low-dimensional source data S, with the target samples T^u identified as unknown. We have clarified this in the paper.
As requested by the reviewer, we performed an ablation study to evaluate the influence of the hyper-parameters on our results. As shown in Figure 2 of the revised version for the Office dataset, our results are stable for a wide range of values.

Our runtime comparison was done with respect to the approach of (Busto & Gall, 2017), which uses the same DeCAF features. We have clarified this in the paper.

Our approach differs from that of Jia’10 and, for the open-set DA scenario, is more intuitive; we model each target sample as being generated by either the shared subspace or the private one, which is crucial to identify the target samples depicting unknown classes. By contrast, in Jia'10, each sample is encoded as a mixture of shared and private representations. This would make separating known and unknown target samples difficult. This is explained in the related work section of our paper.

Regarding the sparsity of T_i^u and T_i^v, there seems to be a misunderstanding. We make use of group sparsity on these vectors, as indicated by the L_{1,2} norm in, e.g., Eq. 2. This differs from standard sparsity, which uses the L_1 norm. As a consequence, we encourage the entire vectors T_i^u and T_i^v to go to zero. The reconstruction term then prevents that they both go to 0 simultaneously. Because we use group sparsity and not standard sparsity, we cannot use the entire algorithm of Lee’07. However, we can use a part of this algorithm to update the bases U and V, since this step does not involve the group sparsity terms. Obtaining T is then done by proximal gradient descent. This is discussed in the Optimization paragraph below Eq. 2.

Our formulation in Eq. 2 includes reconstruction terms based on the subspace representations. Our assumption is that the appearance variations between the known and unknown classes are more important than those across the domains for a given class. As such, it will be easier for our method to use a separate subspace for the unknown samples instead of using the source one, which is dedicated to modeling the appearance of the known classes. We believe that the good results of our FRODA approach confirm this intuition. We acknowledge, however, that adding a discriminative term (D-FRODA) typically improves the results.

There was a typo in our paper. We also use quadratic constraints of the form ||U_j||^2 \leq 1. As such, the Lagrange dual algorithm used in Lee’07 to update the basis also applies for the same purpose in our case. We thank the reviewer for noticing this and fixed this in the revised version.

Comparison to Saito’18 on BCIS. We have now incorporated these results in Table 1. Note that, as mentioned in our earlier comment to the reviewer, for the comparison to be fair, we used DeCAF-fc7 features, which are used by all other methods, including ours, and are the only data available for this benchmark. We proposed to make use of a network with two Fully Connected (FC) layers, with 1024 and 128 units, respectively, and a final classification layer, and took the DeCAF-fc7 features as input. As suggested by Saito’18, we used Leakly-ReLU activation functions and batch normalisation for stable training. As shown in Table 1, we also outperform Saito’18 on this dataset.

We believe that the fact that we use fixed hyper-parameters for all our experiments rather shows the robustness of our approach. Note that, in domain adaptation, one does not have access to validation target data, since the target examples are unlabeled. Nevertheless, in Figure 2 of the revised paper, we now evaluate the influence of the hyper-parameters on the results for the Office dataset. Note that our method is robust to the values of these hyper-parameters.

The overall method is non-convex, because we learn both the bases and the coefficients. However, the individual subproblems that we solve are convex and solved to optimality. As such each step is guaranteed not to increase the objective function, and the method thus converges to a stationary point.

We thank the reviewers for their constructive feedback and their overall positive comments. They mention that “the paper is well organized and easy to read”, and the mathematical formulation of the method is “sound”, and “clearly explained”. They also highlight that we propose a “novel method” for an “understudied” problem.

In the following, we summarize the changes we made to the paper:
    •	We have added a column to each table showing the average for all methods.
    •	To evaluate the robustness of our method to the hyper-parameters \alpha, \beta and \lambda, we have included a figure plotting the accuracy, averaged over all pairs of the Office dataset, as a function of the values of these hyper-parameters.
   •	We have incorporated the results of Saito’18 on the BCIS dataset in Table 1.
   •	We have added more details about our approach at test time.
   •	We have taken all minor comments of the reviewers into account.

In addition to this, we have compared the accuracy of our D-FRODA formulation against the SVM baseline and the open-set ATI method of (Busto & Gall, 2017) using as unknown classes the 5 classes in the Office dataset that do not appear in ImageNet, thus demonstrating the effectiveness of our approach in the presence of truly unknown classes.

We address the reviewers' main concerns in more detail in the individual responses.

Yes, what is important is to follow the principle of the main structure of Saito's architecture, for the layers it is up to you but you should tune them a bit to get an objective result. 

For the BCIS dataset, the only data that is publicly available, and that was used for all results in Table 1 of our paper, is the 4096-dimensional DeCAF-fc7 features. As such, to compare our results with Saito’18, which relies on deep learning, we need to define a neural network that takes these features as input. We propose to make use of a network with two Fully Connected (FC) layers, with 1024 and 128 units, respectively, with ReLU activation functions, and a final classification layer. Would AnonReviewer1 be satisfied with this comparison?

This paper tackles the problem of open-set unsupervised domain adaptation with a method based on 
subspace learning. Specifically the proposed approach searches for two low-dimensional spaces, one shared 
by the known source and target categories while the other is specific for the unknown classes. 

Overall the paper is well organized and easy to read. The mathematical formulation of the method is sound and
clearly explained in all its variants.

I have few concerns 
- it would be good to have the "average" columns in the tables reporting the experimental results. This will help to have an overall idea on the performance of the different proposed and baseline methods.
- it is not clear whether the authors are reporting the results of AODA from the original paper or if they re-ran the code to get the recognition accuracies. For instance in table 3 the result 70.1 for A->W is lower than those reported in the original paper for this setting.
- the paper does not discuss how the hyperparameters of the methods are chosen. Only an analysis on epsilon is provided. It would be very helpful to understand the procedure used to select the values of alpha, beta and lambda and to evaluate the robustness of the method to those parameters. Moreover,  the value of the dimensionality d is not explicitly indicated in the text. This should be added together with a discussion about if and how the subspace disagreement measure (that was introduced for closed set domain adaptation) is reliable in the open set condition.






Meta-review: This paper proposes a new approach to domain adaptation based on sub-spacing, such that outliers are filtered out. While similar ideas have been used e.g. in multi-view learning, their application to domain adaptation makes it a novel and interesting approach. 

While the above is considered by the AC an adequate contribution to ICLR, the authors are encouraged to investigate further the implications of the assumptions made, in a way that the derived criteria seem less heuristic, as R1 pointed out.

There had been some concerns regarding the experiments, but the authors have been very active in the rebuttal period and addressed these concerns satisfactorily.


Decision: accepted-poster-papers