To The Point: Correspondence-driven monocular 3D category reconstruction

We present To The Point ( TTP ) , a method for reconstructing 3D objects from a single image using 2D to 3D correspondences learned from weak supervision . We recover a 3D shape from a 2D image by first regressing the 2D positions corresponding to the 3D template vertices and then jointly estimating a rigid camera transform and non-rigid template deformation that optimally explain the 2D positions through the 3D shape projection . By relying on 3D-2D correspondences we use a simple per-sample optimization problem to replace CNN-based regression of camera pose and non-rigid deformation and thereby obtain substantially more accurate 3D reconstructions . We treat this optimization as a differentiable layer and train the whole system in an end-to-end manner . We report systematic quantitative improvements on multiple categories and provide qualitative results comprising diverse shape , pose and texture prediction examples . Project website : https : //fkokkinos.github.io/to_the_point/ . 1 Introduction . Monocular 3D reconstruction of general categories is a task that humans perform with ease , yet remains challenging for computer vision due to its inherently ill-posed nature : the observed 2D image is the result of a confluence of multiple sources of variation , including non-rigid intra-category shape variation , rigid transforms due to camera pose , as well as appearance variation . CNNs can easily learn to discard appearance variation , yet the treatment of the geometric sources of variability remains elusive . Even though strongly-supervised approaches have delivered compelling results e.g . for human reconstruction [ 1 ] , for general categories we need to rely on weaker forms of supervision as well as self-supervision stemming from the know-how of computer vision . 3D vision has traditionally relied on correspondences to recover both rigid scenes from 2D images for the Structure-from-Motion ( SFM ) problem [ 2 , 3 , 4 ] as well as the more challenging problem of recovering Non-Rigid structure from 2D point tracks ( NR-SFM ) [ 5 , 6 , 7 , 8 ] . In all those problems 3D reconstruction is accomplished by minimizing the reprojection error between the 3D positions of the inferred 3D scene and their 2D image correspondences . While these solutions have been developed for the ( potentially deformable ) single-instance case , the idea of relying on correspondences to supervise monocular 3D reconstruction has transpired in recent deep learning works . CNN-driven monocular 3D category reconstruction [ 9 , 10 , 11 , 12 ] has largely relied on selfsupervision for 3D recovery expressed in terms of correspondence-based loss terms . For instance the geometric cycle loss terms of [ 13 , 14 , 15 ] are explicitly phrased in terms of correspondence established from UV maps while the texture-driven loss terms of [ 9 , 13 , 12 , 16 , 11 ] are implicitly relying on pixel correspondence . The common ground of such loss terms is that if the 3D shape is predicted correctly , it should project to the image in a way that is consistent with the 2D observations , as measured in a pixel-by-pixel sense . These correspondence terms are typically used in 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . tandem with explicit geometric priors such as 3D symmetry [ 10 , 9 ] , predefined camera viewpoint ranges [ 14 , 15 , 10 ] , or predefined object scales [ 14 , 15 , 10 , 12 ] in order to tackle the ill-posed nature of the problem and the presence of multiple local minima in the associated learning problem . Local minima however emerge even in the simpler single-instance case of NR-SFM , while highly sophisticated optimization schemes have been introduced to address them , e.g . [ 17 ] . Current CNNbased approaches seem to ignore this problem and further exacerbate it by delegating the solution of 3D reconstruction to back-propagation with SGD : separate network heads are tasked with regressing the camera pose and non-rigid deformation given an image and are trained in an end-to-end manner , aiming to minimize the correspondence-driven losses . We argue that this is making optimization harder : network training aims at simultaneously establishing the association between images and rigid and non-rigid pose parameters as well as solving the 3D reconstruction problem in terms of these parameters . Each of these problems is hard enough in isolation and putting them together makes the optimization even harder . This challenge is reflected in the complicated numerical schemes currently used to mitigate local minima ; for instance [ 10 , 12 ] use multiple camera hypotheses during both training and testing . The number of hypotheses can range from 8 to up to 40 for a single reconstruction and the hypotheses have to be accompanied with a probabilistic method to select the most accurate pose either predicted by an MLP [ 14 ] or using heuristic loss-based weighting schemes [ 10 ] . Another example of brittle optimization , even when keypoint supervision is available , are the works of [ 9 , 18 ] where in a first stage SFM/NR-SFM is used to get the camera pose right based on keypoint supervision , which is then followed by optimization with image-based losses to recover a mesh . This challenge has been observed also in the strongly-supervised case of human pose estimation , and the use of per-sample numerical optimization [ 19 ] was shown to improve performance in [ 20 , 21 , 22 , 23 , 24 ] . In this work we deviate from the current practice of using a CNN to regress camera and mesh deformation estimates . Instead , during both training and testing we solve a per-sample optimization problem that explicitly aims at providing a 3D reconstruction that projects “ To The Point '' ( TTP ) . We take as input the 2D coordinates corresponding to the 3D vertices of a mesh and recover the 3D vertex positions by optimizing with respect to the rigid and non-rigid pose parameters through differentiable optimization [ 25 , 26 ] . We obtain the 2D points required by our layer by only relying on mask annotations and optionally a small number of 2D semantic keypoint annotations , as well as self-supervision coming from the 3D reprojection loss . We jointly learn the 2D point regression and the 3D modes of shape variability through end-to-end optimization , while treating the per-instance rigid and non-rigid pose parameters as latent variables that are optimized on-the-fly , per sample . We claim that predicting the correspondences is not only sufficient , but also more appropriate for driving monocular 3D reconstruction : it spares us from the use of any additional geometric priors and also yields state-of-the-art results while only relying on a single camera hypothesis . We evaluate our approach on 3D shape , pose and texture reconstruction on four objects categories using realworld datasets CUB [ 27 ] and PASCAL3D+ [ 28 ] . We demonstrate competitive 3D reconstruction quality to previous state-of-the-art methods and our ablation study confirms the importance of the self-supervised losses we employ . 2 Related Work . Monocular 3D reconstruction Recent works on this problem [ 29 , 9 , 14 , 15 , 16 , 12 , 30 ] have relied on varying forms of supervision . Earlier approaches [ 9 , 29 ] treat the problem of 3D reconstruction from single images using known masks and manually labelled keypoints from single viewpoint image collections . Recent works [ 10 , 16 , 14 ] have removed the need for keypoints but introduced multiple viewpoint and deformation hypotheses accompanied with a probabilistic method to select the most accurate pose . [ 13 , 31 , 32 ] jointly recover cameras and non-rigid 3D meshes with single hypothesis-based networks , but limit themselves to simpler , almost planar categories like faces , or exploit symmetry priors , limiting their broader applicability . Closer to our work is Canonical Surface Mapping ( CSM ) and Articulated Canonical Surface Mapping ( ACSM ) [ 14 , 15 ] where the 3D representation is produced in the form of a rigid or articulated template using a 2D-to-3D cycle-consistency loss . Non-rigid structure from motion ( NR-SfM ) The aim of NR-SfM is the recovery of the 3D shape and accompanying camera pose given only 2D landmarks without any explicit 3D supervision [ 7 , 8 , 33 , 34 ] . Lately , several deep learning [ 13 , 35 , 36 ] methods have been proposed that surpassed the performance of traditional methods while being considerably faster . All of the aforementioned methods employ different priors to tackle the under-constrained problem of NR-SfM . The priors are embedded into the methods using low-rank subspaces [ 37 , 7 , 17 ] , spatio-temporal domains [ 34 , 38 ] , equivariance constraints [ 35 ] or sparse basis coefficients using L1 constraints [ 36 , 39 , 40 ] . Learnable Optimization : Common methods for incorporating optimization as layers in deep neural networks include implicit function differentiation [ 25 , 26 , 41 , 42 , 43 ] and optimization unrolling [ 44 , 45 , 36 ] ; we refer to [ 26 , 25 ] for a survey . In 3D reconstruction recent works address the challenge of incorporating RANSAC in an end-to-end trainable pipeline for camera pose estimation based on the Perspective-n-Point ( PnP ) problem , such as differentiable blind PnP [ 42 , 43 ] or DSAC [ 46 ] . Unlike these works , we do not have to address the combinatorial nature of correspondence , but rather focus on regressing the 2D image positions of a 3D template with a fixed number of vertices . 3 To-The-Point Monocular 3D Mesh Reconstruction . We start in Sec . 3.1 by introducing the 2D quantities predicted by our network , we then present our differentiable camera and mesh optimization layer in Sec . 3.2 , and finally present the losses driving our end-to-end training in Sec . 3.4 . 3.1 Predicting 3D to 2D Correspondences . Our method assumes that the template of our object category can be described in 3D in terms of N points . As shown in Fig . 1 , given an image , we use a CNN , φθ , to regress the 2D coordinates u ∈ RN×2 corresponding to these N 3D points . We also predict a visibility vector v where every vi ∈ [ 0 , 1 ] indicates whether the 2D to 3D correspondence is occluded in the image . Recognition of occluded points allows for accurate camera pose estimation by eliminating the influence of noisy predicted points belonging to the non-visible areas of the object . 3.2 Estimation of Pose and Deformation . Our aim is to estimate the camera pose and object deformation using only the predicted 2D points u , the visibility vector v and the template mesh T. We first introduce our assumptions about the rigid and non-rigid part of the shape and then turn to the resulting optimization problem . Firstly , as in [ 9 , 10 , 14 , 15 , 16 , 11 ] , we model the 3D-to-2D projection through weak perspective [ 2 ] . This involves a 2× 3 3D-to-2D “ scaled orthographic ” projection matrix of the following form : C = [ s 0 0 0 s 0 ] , ( 1 ) where the scaling factor s accounts for global scaling due to depth variation ; given a set of 2D image points this is set to their standard deviation , yielding invariance of the pose parameters to similarity transforms of the 2D and 3D coordinate frames . The rigid pose parameters comprise a rotation R and translation t that account for viewing a 3D object V from a given camera position . The parametric estimates for the 2D projections of a 3D object can thus be obtained as follows : û ( R , t ) = C ( RV + t ) ( 2 ) where V is 3×N , u is 2×N , and we overload notation for + assuming that t is replicated N times . Having covered the rigid object case , we now turn to the modeling of non-rigid categories . For this we rely on the deformable template paradigm [ 47 ] commonly used also in the NR-SFM literature [ 7 , 8 ] , and obtain a shape estimate V by adding offsets ∆V = Bc to a template shape T , yielding V = ∆V +T . Combined with Eq . 2 , we have the following parametric estimate of the 2D positions : û ( C , R , t ) = C ( R ( T+Bc ) + t ) ( 3 ) which is bilinear in R , t and c. So far we do not deviate substantially from recent works [ 9 , 10 , 16 , 11 ] in terms of modelling : these also rely on the morphable model paradigm [ 47 ] and regress a shape update ∆V and a rotation matrix through network heads . A minor modelling difference is that we have a low-rank model for Bc , while their shape update is driven by a high-dimensional latent vector . Our main difference is that rather than delegating to the network the task of predicting the ‘ right ’ values of R , t , V , we directly optimize for them through a lightweight and differentiable optimization scheme by exploiting our regressed 2D correspondences . This ‘ optimizes out ’ these parameters and ensures our 3D inference will project as accurately as possible to the 2D points , rather than delegating the optimization to backprop and the network heads . Our “ To-The-Point ” approach aims at minimizing the following re-projection error between the predicted 2D points u and the 2D point estimates deliver by the parametric , 3D-based prediction : l ( R , t , c ) = N∑ i=1 vi ‖ui − ûi ( C , R , t ) ‖ 2 2 + γ ‖c‖ 2 2 , ( 4 ) where we weigh the discrepancy between the two quantities by the regressed visibility , ensuring that the reconstruction process is robust to occluded points . We also add a regularization weight γ on the expansion coefficients to avoid instabilities in the first stages of training , when the basis B is still unknown and can lead to prematurely committing to large arbitrary deformations . To minimize the loss term in Eq . ( 4 ) , we use an alternating optimization scheme and perform separate updates for camera pose estimation and mesh deformation : R̂t , t̂t = argmin R , t l ( R , t , ĉt ) , subject toR ∈ SO ( 3 ) ( 5 ) ĉt+1 = argmin c l ( R̂t , t̂t , c ) ( 6 ) where at each step we use some of the previously estimated quantities ( denoted by hat ) as fixed and optimize with respect with the remaining ones to update their estimates . Starting from the optimization in Eq . 5 , we satisfy the constraint that R ∈ SO ( 3 ) by using the angleaxis representation r of the rotation Rr such that Rr = exp [ r ] × where [ · ] × is the skew symmetric operator . The underlying non-linear problem is solved using the L-BFGS optimizer [ 48 ] and when t > 1 is initialized with the estimate of the previous iteration . To backpropagate through L-BFGS we use the implicit function theorem as described in [ 42 , 43 ] . Given the rotation matrix R , we solve Eq 6 in closed form as follows : ct+1 = ( ΩTXΩ+ γI ) −1ΩTXΥ . ( 7 ) Each row of Ω ∈ R2N×K is defined as Ωi = CR t+1Bi and X ∈ R 2N×2N is a diagonal matrix containing the visibility vector v. Finally , the vector Υ ∈ R2N is defined as Υ = ui −CR t+1Ti − Ctt+1 . For clarity we provide the analytic derivation of the closed form solution in the supplementary material . We also backpropagate through matrix inversion , meaning that the chaining of the update steps can be treated as a differentiable layer and be used in tandem with end-to-end training . Even though the only quantities regressed by our CNN are the 2D positions and associated visibilities , the basis B that represents the shape variability of our category in Eq.3 is a parameter of this layer , is randomly initialized , and is estimated through back-propagation . As shown in the supplemental material , the basis elements learned this way can be intuitively understood , while our experimental results indicate that they suffice for the accurate recovery of intricate mesh deformations .

Summary: The paper tackles the problem of 3D shape texture and pose estimation from a single image without 3D, multi-view nor camera calibrations. This is an extremely challenging problem. Concurrent works are CMR, U-CMR, CSM and A-CSM. **The key novelty** is to avoid learning pose, texture and shape with several networks heads. Instead, the author propose to learn the 2D positions corresponding to the 3D template vertices, then solve a per-sample optimisation to estimate the pose and template deformation, enabling texture recovery through differentiable rendering. They use implicit differentiation to differentiate through the L-BFGS solver of their inner PnP problem. They achieve successful reconstruction of cars, birds and motorcycles. They outperform their competitors on the Cub dataset and Pascal3D+ dataset, showing exciting successes even without keypoint supervision.