NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation

1 INTRODUCTION . Object pose estimation is a fundamentally important task in computer vision with a multitude of realworld applications , e.g . in self-driving cars , or partially autonomous surgical systems . Advances in the architecture design of deep convolutional neural networks ( DCNNs ) Tulsiani & Malik ( 2015 ) ; Su et al . ( 2015 ) ; Mousavian et al . ( 2017 ) ; Zhou et al . ( 2018 ) increased the performance of computer vision systems at 3D pose estimation enormously . However , our experiment shows current 3D pose estimation approaches are not robust to partial occlusion and when objects are viewed from a previously unseen pose . This lack of robustness can have serious consequences in real-world applications and therefore needs to be addressed by the research community . In general , recent works follow either of two approaches for object pose estimation : Keypoint-based approaches detect a sparse set of keypoints and subsequently align a 3D object representation to the detection result . However , due to the sparsity of the keypoints , these approaches are highly vulnerable when the keypoint detection result is affected by adverse viewing conditions , such as partial occlusion . On the other hand , rendering-based approaches utilize a generative model , that is built on a dense 3D mesh representation of an object . They estimate the object pose by reconstructing the input image in a render-and-compare manner ( Figure 1 ) . While rendering-based approaches can be more robust to partial occlusion Egger et al . ( 2018 ) , their core limitation is that they model objects in terms of image intensities . Therefore , they pay too much attention to object details that are not relevant for the 3D pose estimation task . This makes them difficult to optimize Blanz & Vetter ( 2003 ) ; Schönborn et al . ( 2017 ) , and also requires a detailed mesh representation for every shape variant of an object class ( e.g . they need several types of sedan meshes instead of using one prototypical type of sedan ) . In this work , we introduce NeMo a rendering-based approach to 3D pose estimation that is highly robust to partial occlusion , while also being able to generalize to previously unseen views . Our key idea is to learn a generative model of an object category in terms of neural feature activations , instead of image intensities ( Figure 1 ) . In particular , NeMo is composed of a prototypical mesh representation of the object category and feature representations at each vertex of the mesh . The feature representations are learned to be invariant to instance specific details ( such as shape and color variations ) that are not relevant for the 3D pose estimation task . Specifically , we use contrastive learning He et al . ( 2020 ) ; Wu et al . ( 2018 ) ; Bai et al . ( 2020 ) to ensure that the extracted features of an object are distinct from each other ( e.g . the features of the front tire of a car are different from those of the back tire ) , while also being distinct from non-object features in the background . Furthermore , we train a generative model of the feature activations at every vertex of the mesh representation . During inference , NeMo estimates the object pose by reconstructing a target feature map with using render-and-compare and gradient-based optimization w.r.t . the 3D object pose parameters . We evaluate NeMo at 3D pose estimation on the PASCAL3D+ Xiang et al . ( 2014 ) and the ObjectNet3D Xiang et al . ( 2016 ) dataset . Both datasets contain a variety of rigid objects and their corresponding 3D CAD models . Our experimental results show that NeMo outperforms popular approaches such as Starmap Zhou et al . ( 2018 ) at 3D pose estimation by a wide margin under partial occlusion , and performs comparably when the objects are not occluded . Moreover , NeMo is exceptionally robust when objects are seen from a viewpoint that is not present in the training data . Interestingly , we also find that the mesh representation in NeMo can simply approximate the true object geometry with a cuboid , and still perform very well . Our main contributions are : 1 . We propose a 3D neural mesh model of objects that is generative in terms of contrastive neural network features . This representation combines a prototypical geometric representation of the object category with a generative model of neural network features that are invariant to irrelevant object details . 2 . We demonstrate that standard deep learning approaches to 3D pose estimation are highly sensitive to out-of-distribution data including partial occlusions and unseen poses . In contrast , NeMo performs 3D pose estimation with exceptional robustness . 3 . In contrast to other rendering-based approaches that require instance-specific mesh representations of the target objects , we show that NeMo achieves a highly competitive 3D pose estimation performance even with a very crude prototypical approximation of the object geometry using a cuboid . 2 RELATED WORK . Category-Level Object Pose Estimation . Category-Level object pose estimation has been well explored by the research community . A classical approach as proposes by Tulsiani & Malik ( 2015 ) and Mousavian et al . ( 2017 ) was to formulate object pose estimation as a classification problem . Another common category-level object pose estimation approach involves a two-step process Szeto & Corso ( 2017 ) ; Pavlakos et al . ( 2017 ) : First , semantic keypoints are detected interdependently and subsequently a Perspective-n-Point problem is solved to find the optimal 3D pose of an object mesh Lu et al . ( 2000 ) ; Lepetit et al . ( 2009 ) . Zhou et al . ( 2018 ) further improved this approach by utlizing depth information . Recent work Wang et al . ( 2019 ) ; Chen et al . ( 2020 ) introduced render-andcompare to for category-level pose estimation . However , both approaches used pixel-level image synthesis and required detailed mesh models during training.In contrast , NeMo preforms renderand-compare on the level of contrastive features , which are invariant to intra-category nuisances , such as shape and color variations . This enables NeMo to achieve accurate 3D pose estimation results even with a crude prototypical category-level mesh representation . Pose Estimation under Partial Occlusion . Keypoint-based pose estimation methods are sensitive to outliers , which can be caused by partial occlusion Pavlakos et al . ( 2017 ) ; Sundermeyer et al . ( 2018 ) . Some rendering-based approaches achieve satisfactory results on instance-level pose estimation under partial occlusion Song et al . ( 2020 ) ; Peng et al . ( 2019 ) ; Zakharov et al . ( 2019 ) ; Li et al . ( 2018 ) . However , these approaches render RGB images or use instance-level constraints , e.g . pixellevel voting , to estimate object pose . Therefore , these approaches are not suited for category-level pose estimation . To the best of our knowledge , NeMo is the first approach that performs categorylevel pose estimation robustly under partial occlusion . Contrastive Feature Learning . Contrastive learning is widely used in deep learning research . Hadsell et al . ( 2006 ) proposed an intuitive tuple loss , which was later extended to triplets and NPair tuples Schroff et al . ( 2015 ) ; Sohn ( 2016 ) . Recent contrastive learning approaches showed high potential in unsupervised learning Wu et al . ( 2018 ) ; He et al . ( 2020 ) . Oord et al . ( 2018 ) ; Han et al . ( 2019 ) demonstrated the effectiveness of feature-level contrastive losses in representation learning . Bai et al . ( 2020 ) proposed a keypoint detection framework by optimizing the feature representations of keypoints with a contrastive loss . In this work , we use contrastive feature learning to encourage the feature extraction backbone to extract locally distinct features . This , in turn , enables 3D pose estimation by simply optimizing the neural reconstruction loss with gradient descent . Robust Vision through Analysis-by-Synthesis . In a broader context , our work relates to a line of work in the computer vision literature , which demonstrate that the explicit modeling of the object structure significantly enhances the robustness of computer vision models , e.g . at 3D pose estimation Zeeshan Zia et al . ( 2013 ) , face reconstruction Egger et al . ( 2018 ) and human detection Girshick et al . ( 2011 ) under occlusion . More specifically , our work builds on a recent line of work that introduced a neural analysis-by-synthesis approach to vision Kortylewski et al . ( 2020b ) and demonstrated its effectiveness in occlusion-robust image classification Kortylewski et al . ( 2020a ; c ) and object detection Wang et al . ( 2020 ) . Our work significantly extends neural analysis-by-synthesis to include an explicit 3D object representation , instead of 2D template-like object representations . This enables our model to achieve state-of-the-art robustness at pose estimation through neural render-and-compare . 3 NEMO : A 3D GENERATIVE MODEL OF NEURAL FEATURES . We denote a feature representation of an input image I as Φ ( I ) = F l ∈ RH×W×D . Where l is the output of a layer l of a DCNN Φ , with D being the number of channels in layer l. f li ∈ RD is a feaure vector in F l at position i on the 2D lattice P of the feature map . In the remainder of this section we omit the superscript l for notational simplicity because this is fixed a-priori in our model . 3.1 NEURAL RENDERING OF FEATURE MAPS . Similar to other graphics-based generative models , such as e.g . 3D morphable models Blanz & Vetter ( 1999 ) ; Egger et al . ( 2018 ) , our model builds on a 3D mesh representation that is composed of a set of 3D vertices Γ = { r ∈ R3|r = 1 , . . . , R } . For now , we assume the object mesh to be given at training time but we will relax this assumption in later sections . Different from standard graphics-based generative models , we do not store RGB values at each mesh vertex r but instead store feature vectors Θ = { θr ∈ RD|r = 1 , . . . , R } . Using standard rendering techniques , we can use this 3D neural mesh model N = { Γ , Θ } to render feature maps : F̄ ( m ) = < ( N , m ) ∈ RH×W×D , ( 1 ) where m are the camera parameters for projecting the neural mesh representation ( Figure 2 ) . 3.2 NEURAL MESH MODELS . Neural Mesh Models are probabilistic generative models of neural feature activations . Hence , our goal is to learn a generative model p ( F |Ny ) of the real-valued feature activations F of an object class y by leveraging a 3D neural mesh representation Ny . Assuming that the 3D pose m of the object in the input image is known , we define the likelihood of the feature representation F as : p ( F |Ny , m , B ) = ∏ i∈FG p ( fi|Ny , m ) ∏ i′∈BG p ( fi′ |B ) . ( 2 ) The foreground FG is the set of all positions on the 2D lattice P of the feature map F that are covered by the rendered neural mesh model . We compute FG by projecting the 3D vertices of the mesh model Γy into the image using the ground truth camera pose m to obtain the 2D locations of the visible vertices in the image FG = { st ∈ R2|t = 1 , . . . , T } . We define foreground feature likelihoods to be Gaussian distributed : p ( fi|Ny , m ) = 1 σr √ 2π exp ( − 1 2σ2r ‖fi − θr‖2 ) . ( 3 ) Note that the correspondence between the feature vector fi in the feature map F and the vector θr on the neural mesh model is given by the 2D projection of Ny with camera parameters m. Those features that are not covered by the neural mesh model BG = P \ { FG } , i.e . are located in the background , are modeled by a Gaussian likelihood : p ( fi′ |B ) = 1 σ √ 2π exp ( − 1 2σ2 ‖fi′ − β‖2 ) , ( 4 ) with mixture parameters B = { β , σ } .

Summary: This paper tackles the task of pose prediction and takes a render-and-compare approach. However, instead of rendering pixel colors, the key insight is to render features -- each mesh vertex is associated with 3D (learned) features which are encouraged to match computed 2D image features. This 'neural mesh' representation allows pose inference via SGD as one can optimize for pose s.t. the rendered features best match the image features, and is also robust to foreground occlusion. The paper demonstrates results on ObjectNet3d and PASCAL3D+ where the proposed approach is shown to be more robust to occlusion and also better at precise pose estimation.