Multi-View Priors for Learning Detectors from Sparse Viewpoint Data
Authors: Bojan Pepik, Michael Stark, Peter Gehler, Bernt Schiele
ICLR 2014

Abstract: While the majority of today's object class models provide only 2D bounding boxes, far richer output hypotheses are desirable including viewpoint, fine-grained category, and 3D geometry estimate. However, models trained to provide richer output require larger amounts of training data, preferably well covering the relevant aspects such as viewpoint and fine-grained categories. In this paper, we address this issue from the perspective of transfer learning, and design an object class model that explicitly leverages correlations between visual features. Specifically, our model represents prior distributions over permissible multi-view detectors in a parametric way -- the priors are learned once from training data of a source object class, and can later be used to facilitate the learning of a detector for a target class. As we show in our experiments, this transfer is not only beneficial for detectors based on basic-level category representations, but also enables the robust learning of detectors that represent classes at finer levels of granularity, where training data is typically even scarcer and more unbalanced. As a result, we report largely improved performance in simultaneous 2D object localization and viewpoint estimation on a recent dataset of challenging street scenes.


Peer review:
[Review]:
The paper presents a method to learn a quadratic regularizer that improves the performance of multi-view object detectors when very little training data is available for each object or view.  The regularizer is computed using a sparse correlation matrix to identify similarity amongst feature weights in the detectors for different views of the same object (in some cases relying on a 3D model to help establish correspondence between features).  The similarity matrix is then used to define a Laplace regularization term for a standard SVM, thus requiring that a new multi-view detector trained with the regularizer share similar structure across views.  Results are presented on several detection tasks using the KITTI  (driving/urban) dataset and 3D Object Classes dataset.  It is shown that the main advantage is that the method can learn to detect views of novel target objects even when some views of these objects have few or no training examples.

Overall the paper is clearly written and the experiments are extensive.  The authors broke out all of the different cases [number of examples per view, and which views are missing] to make their point.  In the regime where the method is intended to help (where very few examples are available for a particular view of a target object) the regularizer does help for the multi-view DPM model considered in the paper.

The main caveat here is that the method seems to assume the use of separate detectors for every subclass and view (and thus each detector requires its own dataset).  That really exacerbates the problem of too little data where other methods might not have an issue.  This is a nice trick, but since the one-detector-per-object-per-view approach has multiple scalability issues, it looks like the proposed solution also suffers from those barriers.  If this approach to multi-view detection is not workable in the near future, what is the high-level idea that we should take from this work?  I do not see it.

Also, the “dense” sparsity pattern appears to be by far the best performer.  This is somewhat interesting by itself, but also detracts from the “sparse” proposals in the paper (and, of course, the dense approach is not very scalable).  Some candid discussion on the consequences of this result might clarify what parts of the system are important.

Pros:
A relatively simple idea to exploit knowledge of regularity across views in multi-view detectors.
Primary value is in cases where there are few or no examples for a particular view, in which case it does appear to help over baseline approaches.
Cons:
Relies on a fair amount of prior geometric knowledge.
It is unclear how to apply this to cases where the number of features is much larger than DPM-style models [e.g., deep neural nets] or has deeper difficult-to-interpret layers where geometric information cannot be leveraged.
Other:
The SVM regularization penalty might need to be tuned for complete fairness of comparisons.  Since the regularizer is fundamental to performance and the number of training examples is varied, the penalty setting could alter the testing numbers.  (It is possible that the experiments/implementation are set up in such a way that this does not matter much;  a note to this effect, if true, would be helpful.)