3D-RelNet: Joint Object and Relational Network for 3D Prediction

1 INTRODUCTION . A single 2D image can induce a rich 3D perception . When we look at an image , we can reason about its 3D layout , the objects in the image , their shape , extent , relationships etc .. This is really surprising given that going from a 2D projection to a 3D model is inherently ill-posed . How are we able to solve this problem ? Humans rely on regularities in the 3D world in order to do so – this helps us discard many improbable solutions in 3D and reason about more likely ones . This regularity exists at the scene level - indoor scenes have roughly perpendicular walls ; object level - chairs have similar shapes ; and in local object relationships - chairs are close to tables , monitors are on top of tables etc .. A decade ago , a lot of work in computer vision focused on using all the three levels of regularities . For example , a lot of work focused on object-centered 3D prediction ( Fidler et al. , 2012 ) , scene-level 3D prediction ( Hedau et al. , 2009 ) , and multi-object 3D reasoning ( Jiang et al. , 2012 ) . However , in recent years , since the advent of ConvNets , a vast majority of computer vision approaches do not leverage these object-object relationships , and instead reason about each object independently . In this paper , we attempt to take a holistic view of the 3D prediction problem and note that solving the 3D prediction problem would require incorporation of all the three cues . We believe there are three fundamental questions that need to be answered to design this holistic architecture : ( a ) What is the right representation for object level 3D prediction ? ; ( b ) How do we represent object-object relationships and how do we predict them from pixels ? ; ( c ) Finally , how to incorporate objectobject relationships with object-level modules . This paper builds upon the recent success in ( a ) and investigates how to model relationships and incorporate them into our 3D prediction framework . So , how do we model relationships and estimate them from pixels ? There is a whole spectrum of possible approaches . On one end of the spectrum is a complete end-to-end approach . Some examples of these include Interaction Networks ( Battaglia et al. , 2016 ) or Graph Convolutional Networks ( Kipf & Welling , 2016 ) . Both these methods provide a mechanism for object features in the scene to effect each other , thereby allowing an implicit modeling of relationships among them . However , as we show in our experiments , these end-to-end approaches disregard the structural information which might be crucial for modeling the relationships . The other end of spectrum is to use category-based image-agnostic pairwise priors ( Zhao & Zhu , 2011 ) to model relationships . A drawback is that these priors are often too strong to generalize and it is better to learn them ( Jiang et al. , 2012 ) . When it comes to the final question of how does one incorporate relationships to improve 3D prediction , the answer is even murkier . One classical approach is to use graphical models such as CRFs ( Lafferty et al. , 2001 ; Schwing et al. , 2012 ) . However , these classical approaches have usually provided little improvements over object-based approaches . Our key insight is to incorporate structural information in end-to-end systems . Specifically , we model and predict pairwise relationships in the translation , rotation and scale space . One advantage of using this structured relationship space is that the incorporation of relationships into object-level estimates is simple yet effective . But how do we predict these pairwise relationships from pixels ? Our paper investigates several design choices and proposes a simple architecture . Our method demonstrates significant improvement in performance across multiple metrics and datasets . As we show in our experiments , this modeling of relationships in this structured space provides a huge 6 point AP improvement in detection settings over current state-of-the-art 3D approaches . We will release our code for reproducibility . 2 RELATED WORK . Dating back to the first thesis written on computer vision ( Roberts , 1963 ) , inferring the underlying 3D structure of an image has been a long-standing goal in the field . While we have explored several 3D scene representations over the years , e.g. , depth ( Saxena et al. , 2009 ; Eigen et al. , 2014 ) , qualitative 3D ( Hoiem et al. , 2005 ; Gupta et al. , 2010 ) , manifolds ( Osadchy et al. , 2007 ) or volumetric 3D ( Choy et al. , 2016 ; Girdhar et al. , 2016 ; Song et al. , 2017 ) , the prevalent paradigm is still the one followed in Roberts ’ seminal work – that of inferring a 3D scene in terms of the shape and pose of the underlying objects . The initial attempts under this paradigm ( Guzmán , 1968 ; Huttenlocher & Ullman , 1990 ) focused on placing known object instances to match image evidence , relying on matching edges , corners etc . to fit the known shape templates to images . Subsequent approaches have focused on a more general setting of reconstructing scenes comprising of novel objects , and leverage either explicit or implicitly learned category level priors for pose and shape estimation , typically relying on a deformable shape space ( Cashman & Fitzgibbon , 2013 ; Kar et al. , 2015 ) or template CAD models ( Lim et al. , 2013 ; Aubry et al. , 2014 ; Li et al. , 2015 ; Bansal et al. , 2016 ; Izadinia et al. , 2017 ) for the latter inference . Current CNN based incarnations of these approaches , driven by the abundant success of deep learning and availability of annotated data , have further improved the results for pose estimation ( Tulsiani & Malik , 2015 ; Pavlakos et al. , 2017 ) , and have also been extended to joint shape and pose inference of the objects present in a scene ( Kundu et al. , 2018 ; Tulsiani et al. , 2018 ) . A common characteristic amongst these approaches is the reasoning at a per-object level . While the object-centric nature is certainly desirable as a representation , we argue that reasoning independently for each object to infer this representation is not , as it does not allow leveraging the relationships between the entities in a scene . We propose a method that also uses object-centric representations but goes beyond independent reasoning per object . We are of course not the first to pursue reasoning about relationships between entities in a scene . Several previous approaches focus on the goal of predicting various relations e.g . human-object interactions ( Gupta & Malik , 2015 ; Gkioxari et al. , 2018 ) , object-object interactions ( Gupta et al. , 2010 ; Lu et al. , 2016 ) , object-scene interactions ( Lee et al. , 2010 ) etc . While these works pursue relation inference as the end goal , we instead aim to leverage these for a per-instance prediction task . In the context of incorporating relations for such per-instance prediction , there are two alternate ideologies . On the one hand , approaches pursuing 3D scene inference or generation ( Zhao & Zhu , 2011 ; Lin et al. , 2013 ; Choi et al. , 2013 ; Fisher et al. , 2012 ; 2011 ; Huang et al. , 2018 ; Jiang et al. , 2018 ) typically incorporate pairwise ( or higher order ) relations via explicit class-based priors regarding possible configurations and optimize predictions to adhere to these . This approach of explicitly modeling relations as a prior imposes the same constraints across all scenes , independent of the structure in the image , and is therefore not flexible enough and has difficulties in scaling up to arbitrary relations across arbitrary objects . The alternate ideology for incorporating relationships is to eschew any explicit structure for these relations , and instead implicitly capture these via architectural changes to the CNNs , thereby allowing the features of objects to influence one another ( Battaglia et al. , 2016 ; Kipf & Welling , 2016 ) . While this a generally applicable mechanism , it does not leverage several aspects regarding the structure of the problem – for 3D inference , specific relations like relative position , orientation are very relevant , and can be used in specific ways to influence per-instance predictions . Our approach leverages some aspects of both these ideologies – unlike the classical prior based approaches , we learn and infer these relations in a image-dependent context via a CNN , and unlike the purely implicit methods , we are more explicit about the structure and meaning of these relations . 3 APPROACH . Our goal is to predict the 3D pose and shape for all the objects in a scene . We observe that in addition to the visual cues per object , reasoning about relationships across them can further help our predictions , in particular for the 3D pose – a chair would be in front of a table , and of a compatible relative size , and therefore even if we are uncertain about the pose of one of these objects , e.g. , due to occlusion , these relationships can enable us to make accurate predictions . We operationalize this insight in our method ( see figure 1 ) that leverages both - independent perobject predictions alongwith predictions regarding relationships between them . We infer the final estimates for all the objects in the scene by integrating these two . We first formally describe the object-centric representations pursued and briefly review a recent per-instance prediction approach in section 3.1 . We then introduce the relative representations in section 3.2 and present our network architecture that enables predicting these in section 3.3 . In section 3.4 we discuss how these relative predictions are combined with the independent per-object predictions to yield the final 3D estimates for the objects . We show that optimizing the combination of these estimates ( i.e. , final estimate ) in a differentiable end-to-end framework helps improve the final per-object predictions . 3.1 INSTANCE SPECIFIC REPRESENTATIONS AND INFERENCE . We output the 3D pose of an object by predicting its shape in a canonical frame , and its scale , translation and rotation in camera frame . The shape is parametrized as a 323 volumetric occupancy grid in a canonical space where the objects are upright , front-facing , normalized to a unit cube . The translation t ∈ R3 and ( logarithm of ) anisotropic scale s ∈ R3 , and normalized quaternion q indicate the position , size and orientation of the object respectively . Following prior work ( Tulsiani et al. , 2018 ) , we parametrize the rotation prediction as a classification task among fixed bins . We build upon the recent work by ( Tulsiani et al. , 2018 ) : they pursue a similar per-object representation , but make independent predictions across objects . We use their approach to obtain the independent predictions for each object . We briefly review their prediction framework but refer the reader to the paper for details about the representation and architecture . Their method uses an architecture similar to Fast R-CNN ( Girshick , 2015 ) , where the input image is encoded via convolutional layers , and for each object bounding box , an RoI pooling layer crops the corresponding features . These per-object features , in conjunction with coarse image feature and an encoding of the bounding box coordinates , are encoded to an instance-specific bottleneck representation from which the corresponding shape and pose are predicted . We adopt a similar architecture , illustrated in figure 2 , with the introduction of spatial coordinates as additional feature channels ( see section 3.3 ) to obtain per-object ( unary ) predictions . We note that these predictions are obtained independently across objects , and do not incorporate reasoning across them . However , unlike previous work , these are subsequently integrated with relative predictions ( section 3.4 ) to obtain final estimates .

Summary: <Summary>: This paper presented a method for incorporating binary relationship between objects (relative location, rotation and scale) into single object 3d prediction. It is built on top of previously published work of [a] and used same network architecture and loss as of [a] and only added the binary relations between objects for object 3d estimation. The results are shown on SUNCG synthetic dataset and only *4 image* instances of NYUv2 dataset which is very small for a computer vision task.