To The Point: Correspondence-driven monocular 3D category reconstruction

We present To The Point ( TTP ) , a method for reconstructing 3D objects from a single image using 2D to 3D correspondences learned from weak supervision . We recover a 3D shape from a 2D image by first regressing the 2D positions corresponding to the 3D template vertices and then jointly estimating a rigid camera transform and non-rigid template deformation that optimally explain the 2D positions through the 3D shape projection . By relying on 3D-2D correspondences we use a simple per-sample optimization problem to replace CNN-based regression of camera pose and non-rigid deformation and thereby obtain substantially more accurate 3D reconstructions . We treat this optimization as a differentiable layer and train the whole system in an end-to-end manner . We report systematic quantitative improvements on multiple categories and provide qualitative results comprising diverse shape , pose and texture prediction examples . Project website : https : //fkokkinos.github.io/to_the_point/ . 1 Introduction . Monocular 3D reconstruction of general categories is a task that humans perform with ease , yet remains challenging for computer vision due to its inherently ill-posed nature : the observed 2D image is the result of a confluence of multiple sources of variation , including non-rigid intra-category shape variation , rigid transforms due to camera pose , as well as appearance variation . CNNs can easily learn to discard appearance variation , yet the treatment of the geometric sources of variability remains elusive . Even though strongly-supervised approaches have delivered compelling results e.g . for human reconstruction [ 1 ] , for general categories we need to rely on weaker forms of supervision as well as self-supervision stemming from the know-how of computer vision . 3D vision has traditionally relied on correspondences to recover both rigid scenes from 2D images for the Structure-from-Motion ( SFM ) problem [ 2 , 3 , 4 ] as well as the more challenging problem of recovering Non-Rigid structure from 2D point tracks ( NR-SFM ) [ 5 , 6 , 7 , 8 ] . In all those problems 3D reconstruction is accomplished by minimizing the reprojection error between the 3D positions of the inferred 3D scene and their 2D image correspondences . While these solutions have been developed for the ( potentially deformable ) single-instance case , the idea of relying on correspondences to supervise monocular 3D reconstruction has transpired in recent deep learning works . CNN-driven monocular 3D category reconstruction [ 9 , 10 , 11 , 12 ] has largely relied on selfsupervision for 3D recovery expressed in terms of correspondence-based loss terms . For instance the geometric cycle loss terms of [ 13 , 14 , 15 ] are explicitly phrased in terms of correspondence established from UV maps while the texture-driven loss terms of [ 9 , 13 , 12 , 16 , 11 ] are implicitly relying on pixel correspondence . The common ground of such loss terms is that if the 3D shape is predicted correctly , it should project to the image in a way that is consistent with the 2D observations , as measured in a pixel-by-pixel sense . These correspondence terms are typically used in 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . tandem with explicit geometric priors such as 3D symmetry [ 10 , 9 ] , predefined camera viewpoint ranges [ 14 , 15 , 10 ] , or predefined object scales [ 14 , 15 , 10 , 12 ] in order to tackle the ill-posed nature of the problem and the presence of multiple local minima in the associated learning problem . Local minima however emerge even in the simpler single-instance case of NR-SFM , while highly sophisticated optimization schemes have been introduced to address them , e.g . [ 17 ] . Current CNNbased approaches seem to ignore this problem and further exacerbate it by delegating the solution of 3D reconstruction to back-propagation with SGD : separate network heads are tasked with regressing the camera pose and non-rigid deformation given an image and are trained in an end-to-end manner , aiming to minimize the correspondence-driven losses . We argue that this is making optimization harder : network training aims at simultaneously establishing the association between images and rigid and non-rigid pose parameters as well as solving the 3D reconstruction problem in terms of these parameters . Each of these problems is hard enough in isolation and putting them together makes the optimization even harder . This challenge is reflected in the complicated numerical schemes currently used to mitigate local minima ; for instance [ 10 , 12 ] use multiple camera hypotheses during both training and testing . The number of hypotheses can range from 8 to up to 40 for a single reconstruction and the hypotheses have to be accompanied with a probabilistic method to select the most accurate pose either predicted by an MLP [ 14 ] or using heuristic loss-based weighting schemes [ 10 ] . Another example of brittle optimization , even when keypoint supervision is available , are the works of [ 9 , 18 ] where in a first stage SFM/NR-SFM is used to get the camera pose right based on keypoint supervision , which is then followed by optimization with image-based losses to recover a mesh . This challenge has been observed also in the strongly-supervised case of human pose estimation , and the use of per-sample numerical optimization [ 19 ] was shown to improve performance in [ 20 , 21 , 22 , 23 , 24 ] . In this work we deviate from the current practice of using a CNN to regress camera and mesh deformation estimates . Instead , during both training and testing we solve a per-sample optimization problem that explicitly aims at providing a 3D reconstruction that projects “ To The Point '' ( TTP ) . We take as input the 2D coordinates corresponding to the 3D vertices of a mesh and recover the 3D vertex positions by optimizing with respect to the rigid and non-rigid pose parameters through differentiable optimization [ 25 , 26 ] . We obtain the 2D points required by our layer by only relying on mask annotations and optionally a small number of 2D semantic keypoint annotations , as well as self-supervision coming from the 3D reprojection loss . We jointly learn the 2D point regression and the 3D modes of shape variability through end-to-end optimization , while treating the per-instance rigid and non-rigid pose parameters as latent variables that are optimized on-the-fly , per sample . We claim that predicting the correspondences is not only sufficient , but also more appropriate for driving monocular 3D reconstruction : it spares us from the use of any additional geometric priors and also yields state-of-the-art results while only relying on a single camera hypothesis . We evaluate our approach on 3D shape , pose and texture reconstruction on four objects categories using realworld datasets CUB [ 27 ] and PASCAL3D+ [ 28 ] . We demonstrate competitive 3D reconstruction quality to previous state-of-the-art methods and our ablation study confirms the importance of the self-supervised losses we employ . 2 Related Work . Monocular 3D reconstruction Recent works on this problem [ 29 , 9 , 14 , 15 , 16 , 12 , 30 ] have relied on varying forms of supervision . Earlier approaches [ 9 , 29 ] treat the problem of 3D reconstruction from single images using known masks and manually labelled keypoints from single viewpoint image collections . Recent works [ 10 , 16 , 14 ] have removed the need for keypoints but introduced multiple viewpoint and deformation hypotheses accompanied with a probabilistic method to select the most accurate pose . [ 13 , 31 , 32 ] jointly recover cameras and non-rigid 3D meshes with single hypothesis-based networks , but limit themselves to simpler , almost planar categories like faces , or exploit symmetry priors , limiting their broader applicability . Closer to our work is Canonical Surface Mapping ( CSM ) and Articulated Canonical Surface Mapping ( ACSM ) [ 14 , 15 ] where the 3D representation is produced in the form of a rigid or articulated template using a 2D-to-3D cycle-consistency loss . Non-rigid structure from motion ( NR-SfM ) The aim of NR-SfM is the recovery of the 3D shape and accompanying camera pose given only 2D landmarks without any explicit 3D supervision [ 7 , 8 , 33 , 34 ] . Lately , several deep learning [ 13 , 35 , 36 ] methods have been proposed that surpassed the performance of traditional methods while being considerably faster . All of the aforementioned methods employ different priors to tackle the under-constrained problem of NR-SfM . The priors are embedded into the methods using low-rank subspaces [ 37 , 7 , 17 ] , spatio-temporal domains [ 34 , 38 ] , equivariance constraints [ 35 ] or sparse basis coefficients using L1 constraints [ 36 , 39 , 40 ] . Learnable Optimization : Common methods for incorporating optimization as layers in deep neural networks include implicit function differentiation [ 25 , 26 , 41 , 42 , 43 ] and optimization unrolling [ 44 , 45 , 36 ] ; we refer to [ 26 , 25 ] for a survey . In 3D reconstruction recent works address the challenge of incorporating RANSAC in an end-to-end trainable pipeline for camera pose estimation based on the Perspective-n-Point ( PnP ) problem , such as differentiable blind PnP [ 42 , 43 ] or DSAC [ 46 ] . Unlike these works , we do not have to address the combinatorial nature of correspondence , but rather focus on regressing the 2D image positions of a 3D template with a fixed number of vertices . 3 To-The-Point Monocular 3D Mesh Reconstruction . We start in Sec . 3.1 by introducing the 2D quantities predicted by our network , we then present our differentiable camera and mesh optimization layer in Sec . 3.2 , and finally present the losses driving our end-to-end training in Sec . 3.4 . 3.1 Predicting 3D to 2D Correspondences . Our method assumes that the template of our object category can be described in 3D in terms of N points . As shown in Fig . 1 , given an image , we use a CNN , φθ , to regress the 2D coordinates u ∈ RN×2 corresponding to these N 3D points . We also predict a visibility vector v where every vi ∈ [ 0 , 1 ] indicates whether the 2D to 3D correspondence is occluded in the image . Recognition of occluded points allows for accurate camera pose estimation by eliminating the influence of noisy predicted points belonging to the non-visible areas of the object . 3.2 Estimation of Pose and Deformation . Our aim is to estimate the camera pose and object deformation using only the predicted 2D points u , the visibility vector v and the template mesh T. We first introduce our assumptions about the rigid and non-rigid part of the shape and then turn to the resulting optimization problem . Firstly , as in [ 9 , 10 , 14 , 15 , 16 , 11 ] , we model the 3D-to-2D projection through weak perspective [ 2 ] . This involves a 2× 3 3D-to-2D “ scaled orthographic ” projection matrix of the following form : C = [ s 0 0 0 s 0 ] , ( 1 ) where the scaling factor s accounts for global scaling due to depth variation ; given a set of 2D image points this is set to their standard deviation , yielding invariance of the pose parameters to similarity transforms of the 2D and 3D coordinate frames . The rigid pose parameters comprise a rotation R and translation t that account for viewing a 3D object V from a given camera position . The parametric estimates for the 2D projections of a 3D object can thus be obtained as follows : û ( R , t ) = C ( RV + t ) ( 2 ) where V is 3×N , u is 2×N , and we overload notation for + assuming that t is replicated N times . Having covered the rigid object case , we now turn to the modeling of non-rigid categories . For this we rely on the deformable template paradigm [ 47 ] commonly used also in the NR-SFM literature [ 7 , 8 ] , and obtain a shape estimate V by adding offsets ∆V = Bc to a template shape T , yielding V = ∆V +T . Combined with Eq . 2 , we have the following parametric estimate of the 2D positions : û ( C , R , t ) = C ( R ( T+Bc ) + t ) ( 3 ) which is bilinear in R , t and c. So far we do not deviate substantially from recent works [ 9 , 10 , 16 , 11 ] in terms of modelling : these also rely on the morphable model paradigm [ 47 ] and regress a shape update ∆V and a rotation matrix through network heads . A minor modelling difference is that we have a low-rank model for Bc , while their shape update is driven by a high-dimensional latent vector . Our main difference is that rather than delegating to the network the task of predicting the ‘ right ’ values of R , t , V , we directly optimize for them through a lightweight and differentiable optimization scheme by exploiting our regressed 2D correspondences . This ‘ optimizes out ’ these parameters and ensures our 3D inference will project as accurately as possible to the 2D points , rather than delegating the optimization to backprop and the network heads . Our “ To-The-Point ” approach aims at minimizing the following re-projection error between the predicted 2D points u and the 2D point estimates deliver by the parametric , 3D-based prediction : l ( R , t , c ) = N∑ i=1 vi ‖ui − ûi ( C , R , t ) ‖ 2 2 + γ ‖c‖ 2 2 , ( 4 ) where we weigh the discrepancy between the two quantities by the regressed visibility , ensuring that the reconstruction process is robust to occluded points . We also add a regularization weight γ on the expansion coefficients to avoid instabilities in the first stages of training , when the basis B is still unknown and can lead to prematurely committing to large arbitrary deformations . To minimize the loss term in Eq . ( 4 ) , we use an alternating optimization scheme and perform separate updates for camera pose estimation and mesh deformation : R̂t , t̂t = argmin R , t l ( R , t , ĉt ) , subject toR ∈ SO ( 3 ) ( 5 ) ĉt+1 = argmin c l ( R̂t , t̂t , c ) ( 6 ) where at each step we use some of the previously estimated quantities ( denoted by hat ) as fixed and optimize with respect with the remaining ones to update their estimates . Starting from the optimization in Eq . 5 , we satisfy the constraint that R ∈ SO ( 3 ) by using the angleaxis representation r of the rotation Rr such that Rr = exp [ r ] × where [ · ] × is the skew symmetric operator . The underlying non-linear problem is solved using the L-BFGS optimizer [ 48 ] and when t > 1 is initialized with the estimate of the previous iteration . To backpropagate through L-BFGS we use the implicit function theorem as described in [ 42 , 43 ] . Given the rotation matrix R , we solve Eq 6 in closed form as follows : ct+1 = ( ΩTXΩ+ γI ) −1ΩTXΥ . ( 7 ) Each row of Ω ∈ R2N×K is defined as Ωi = CR t+1Bi and X ∈ R 2N×2N is a diagonal matrix containing the visibility vector v. Finally , the vector Υ ∈ R2N is defined as Υ = ui −CR t+1Ti − Ctt+1 . For clarity we provide the analytic derivation of the closed form solution in the supplementary material . We also backpropagate through matrix inversion , meaning that the chaining of the update steps can be treated as a differentiable layer and be used in tandem with end-to-end training . Even though the only quantities regressed by our CNN are the 2D positions and associated visibilities , the basis B that represents the shape variability of our category in Eq.3 is a parameter of this layer , is randomly initialized , and is estimated through back-propagation . As shown in the supplemental material , the basis elements learned this way can be intuitively understood , while our experimental results indicate that they suffice for the accurate recovery of intricate mesh deformations .

Summary: The paper proposes a method for category-level 3D mesh reconstruction from single image. The key innovation is a correspondence-driven optimization layer that produces camera pose and object deformation given a single image, as opposed to using conventional pose CNNs. Overall, the idea is sensible, and the results are better than prior methods under the same setup, either given keypoint annotation or without keypoint annotation. My concerns is mostly on significance and writing.