Towards Geometry-aware Deep Learning Models for Single-view 3 D Reconstruction

Single view reconstruction of objects/scenes, one of the most fundamental problems in computer vision, despite significant advances in recent times, is still far from being solved. My research is primarily focused on extending deep learning models with naive data-driven reasoning, to geometrically reason scenes or objects, for reconstructing them in 3D. The first part of my research tackles the problem of 3D object detection from single view. Owing to the ill-posed nature of the problem, most techniques, in recent times, try to solve them jointly, so that one can be used to resolve the ambiguities that arise due to the other. My research departs from the beaten path of using plain regression-based models (including deep learning methods) for estimating pose/shape, to geometrically reason an object (as constellation of parts) in 3D, to simultaneously detect, estimate continuous pose and recover its underlying 3D shape. Rest of my research, including the current work, focuses on Monocular depth prediction on video sequences of street scenes, using geometry-aware end-to-end trained, deep learning models. We propose Long-Short Term Memory (LSTM) and Generative Adversarial Network (GAN) based deep learning models, that can exploit the sequential nature of the data to reason pose geometrically, obtaining state-of-the art results for the problem. 1. Category specific 3D Object Reconstruction from Single View Formally stated, given an image containing an object of a specific class, my goal is to simultaneously detect and reason the object in 3D, to recover its shape, from 2D appearance cues. The ill-posed nature of the problem leads to a complex solution landscape with several local minima. My research is focused on part based reasoning where we model the object’s shape as a set of part positions in 3D as opposed to a wholistic reasoning. This allows us to model the intra-class appearance and shape variation better, and reason the object in 3D, instead of regressing pose as a function of 2D appearances (wholistic). My work differs from existing part-based approaches in two ways. (1) Most part-based approaches [1, 2] use publicly available 3D CAD models to reason the object’s part geometry. But manually generating CAD models for objects is expensive, painstaking to design, and are limited in their capability to capture the high levels of intra-class variation in terms of the object’s appearance and shape geometry. Instead, we learn SfM based class-specific shape and appearance models from real image sequences. (2) Even existing part-based approaches view the problem of viewpoint/pose estimation a fine-grained categorization problem with strong appearance regressors and better cost functions to penalize deformations [4, 1], ultimately binning them into a fixed pose interval. Alternatively, we geometrically model an object, by reasoning the physical projection of the underlying shape, effectively estimating the continuous viewpoint of the object. Given a set of image sequences taken around an object of a specific class, our objective is to learn a deformable shape model that captures the object’s appearance and 3D shape geometry using a set of 3D parts. Parts are, in general, semantically meaningful and salient regions (in 2D images), obtained via manual annotations or automated techniques. In [5], we use images taken around instances of cars, to compute 3D shape instances using