One-Shot Adaptation of Supervised Deep Convolutional Models
Authors: Trevor Darrell, Eric Tzeng, Yangqing Jia, Judy Hoffman, Kate Saenko, Jeff Donahue
ICLR 2014

Abstract: Dataset bias remains a significant barrier towards solving real world computer vision tasks. Though deep convolutional networks have proven to be a competitive approach for image classification, a question remains: have these models have solved the dataset bias problem? In general, training or fine-tuning a state-of-the-art deep model on a new domain requires a significant amount of data, which for many applications is simply not available. Transfer of models directly to new domains without adaptation has historically led to poor recognition performance. In this paper, we pose the following question: is a single image dataset, much larger than previously explored for adaptation, comprehensive enough to learn general deep models that may be effectively applied to new image domains? In other words, are deep CNNs trained on large amounts of labeled data as susceptible to dataset bias as previous methods have been shown to be? We show that a generic supervised deep CNN model trained on a large dataset reduces, but does not remove, dataset bias. Furthermore, we propose several methods for adaptation with deep models that are able to operate with little (one example per category) or no labeled domain specific data. Our experiments show that adaptation of deep models on benchmark visual domain adaptation datasets can provide a significant performance boost.


Peer review:
[Review]:
This paper presents an extensive empirical evaluation of various methods and frameworks for domain adaptation, that is, when training (i.e. source) and test (i.e. target) data are expected to be sampled from different - but related - distributions. The study is carried out for the image labeling problem, with the CNN of Krizhevsky et al. as base classifier for the source domain. The main goal is to explore various strategies for applying this network trained for 1,000 categories on ImageNet to other images of the same categories but taken in different conditions (basically taken from a webcam here). The main ideas is to use features from the CNN architecture (different layers are considered) and feed them to various domain adaptation techniques, supervised or not. For the supervised case, the setting is drastic and allows only a single labeled example per category from the target domain.

----

The paper tackles an important issue and is sound; the numerous experiments appear to be reliable. It is clearly written despite some typos. But it also raises some questions.

In the beginning of Section 3, it is written 'We propose a generic framework to selectively adapt the parameters of a deep network', which is rather ambitious. The experimental study is extensive and covers many methods but I'm unsure that this actually provides a generic framework for deep learning because (1) this only covers CNNs and (2) it is more the application of existing methods than the definition of a new scheme. The main conclusion of the paper, which is that transferring from ImageNet is more efficient than from Amazon, is fine but does not lead to a framework definition. 

A very interesting point of the paper is to provide some elements about which features from a CNN should be fed to adaptation methods (which layer basically). I think this is a key problem and I regret that the paper does not elaborate too much on this point. There is some discussion but it seems that the quality of adaption given features from a certain layer can also indicate the abstraction level of the features learned by the CNN. Perhaps, one could elaborate on this.

When labeled data is unavailable, which is the case here for the target domain, training is complicated (and that's what's addressed in the paper with at most 1 example per category) but model selection is also tricky. For instance, setting the C for the SVMs of the Daume III method or the alpha of the linear interpolation can be complex. This crucial problem is not really tackled in the paper since hyperparameters values are either set through unjustified heuristics or left somewhat undecided. The alpha of the linear interpolation seems to be chosen while looking at the curves of Figure 1 (a), that is, by looking at the evaluation set.. Nothing is said on how the C used for the SVMs of Daume III is chosen and its value is not given. This is problematic since these are the 2 best performing methods.

It is said a couple times that adaption methods based on SVMs (such as Daume III) might suffer from long training duration because of the large number of training examples. I disagree. I suppose that the SVM is using a linear kernel (it should be stated). If this is the case, then it has been shown many times that linear SVMs can be very efficient in training time and memory usage. I suspect the SVM training time to be somewhat negligible compared to that of the CNN.

The test set is rather small, only 160 images, which are split into 20 random splits of 8 images. I wonder if averaging test results obtained on such small sets, which hence never contains all 16 categories, makes sense. I would rather like some statistical significance paired tests for instance.


Minor remarks:
- Section 2: typo: reported in 4 -> reported in Section 4
- Section 4.2: how many unlabeled examples from the target are used for the unsupervised adaptation approaches, is it 10 per categories as in the test set or 20 as for the source domain train set?
- Section 4.3: typo: make use a subspace -> make use of a subspace .
- Section 4.6: the discussion on the size of the subspace is useless. It is pretty obvious that a dimension lower than the number of categories is detrimental.