Multi-View Priors for Learning Detectors from Sparse Viewpoint Data
Authors: Bojan Pepik, Michael Stark, Peter Gehler, Bernt Schiele
ICLR 2014

Abstract: While the majority of today's object class models provide only 2D bounding boxes, far richer output hypotheses are desirable including viewpoint, fine-grained category, and 3D geometry estimate. However, models trained to provide richer output require larger amounts of training data, preferably well covering the relevant aspects such as viewpoint and fine-grained categories. In this paper, we address this issue from the perspective of transfer learning, and design an object class model that explicitly leverages correlations between visual features. Specifically, our model represents prior distributions over permissible multi-view detectors in a parametric way -- the priors are learned once from training data of a source object class, and can later be used to facilitate the learning of a detector for a target class. As we show in our experiments, this transfer is not only beneficial for detectors based on basic-level category representations, but also enables the robust learning of detectors that represent classes at finer levels of granularity, where training data is typically even scarcer and more unbalanced. As a result, we report largely improved performance in simultaneous 2D object localization and viewpoint estimation on a recent dataset of challenging street scenes.


Peer review:
[Review]:
This work aims to perform simultaneous detection and viewpoint estimation in the face of having few training examples for many classes or viewpoints, but where many examples are available for a smaller or related set of source instances.  To accomplish this, two new types of weight regularization are described, for use with deformable parts models.  Both regularizers form a quadratic penalty on the weights by means of a covariance matrix, constructed using models trained on the sources.  The first, SVM-MV, constructs the covariance by averaging across all pairs of HoG cells that overlap between adjacent object viewpoints; overlaps are found by projecting to a (provided or guessed) 3D object model.  The second, SVM-Sigma, constructs an explicit all-to-all covariance of the weights by averaging across different model instances.  SVM-MV extends the ideas of Gao et al. to work across views, while SVM-Sigma breaks from these careful constructions and uses all pairwise interactions that arise.  The authors evaluate their effectiveness using two datasets, 3D-objects and KITTI, concluding that such regularization enables good performance in these tasks, with SVM-Sigma generally outperforming the other methods.

Unfortunately, the exposition is rather dry and can be hard to follow.  Illustrative examples and diagrams would help a lot here, though, particularly sketches of the projections and overlaps for SVM-MV.  I think it also would help to instantiate model early on, using one of the datasets from the experiments (i.e., in sec 3, linking sim_n, w^s, w^t to concrete instances).

I'm also a bit confused on what exactly comprised the source vs target data.  For sec 4.1 (3D-objects), how was the data divided between sources (used for the priors) and the targets?  Was there any overlap between these, either by datapoint or by object instance?  For sec 4.2, p.8 para 1 seems to say that for KITTI, the priors were trained from source data drawn from either from 3D-objects or KITTI (i.e. there are two different cases).  In the former case, did viewpoints need to be mapped to transfer between the datasets, and which object classes were used?  In the latter case, did the prior data overlap the target data?


Pros:

- Presents new regularizers that exploit structure relations, learned from the data in cases where there are dense subsets or aggregates

- Experiments are detailed

Cons:

- Dry and hard to grasp
- Could use more illustrations of the method and problem setup
- Results presentation confusing at times



Minor comments:

- Fig 1 right, TD2ND:  the labels along the rows (y-axis) appear swapped:  The text indicates the block with ones should connect with-data to no-data.

- I found the italics on occurrences of 'target' and 'source' somewhat distracting; it tended to take my eyes away from the parts of paragraphs I wanted to concentrate on much of the time.

- p.5 last para:  says there are 9 object classes, but the webpage for this dataset says there are 8?

- p.6 (Experiments sec):  k in {1,5,10,all} -- would be nice if this said how many 'all' is as well.

- Fig 2:  figures could use titles

- Fig 3:  I'm confused about which views were included/excluded for each plot -- are the included views progressive subsets?  It looks like the differences are more than that.  Maybe a key with on/off bitmap listing each view would help.