MeshInversion: 3D textured mesh reconstruction with generative prior

1The code will be available upon paper acceptance . 1 INTRODUCTION . Single-view 3D object reconstruction aims at recovering the shape and texture of the object from a single image . This long-standing problem is fundamental to various applications such as robotics navigation , 3D scene understanding , and augmented/virtual reality . A key challenge is the lack of 3D or multi-view supervision due to the prohibitive cost of data collection and annotation for object instances in the wild . Leveraging more readily available 2D supervisions such as keypoints , Kanazawa et al . ( 2018b ) first introduce CMR , a deep learning framework of textured mesh reconstruction that is trained with a collection of real-world monocular images of an object category . To further relax the supervision constraint , several follow-up studies propose to learn the 3D manifold in a self-supervised manner , only requiring single-view images and their corresponding masks for training ( Li et al. , 2020 ; Goel et al. , 2020 ; Bhattad et al. , 2021 ; Hu et al. , 2021 ) . This further complicates the problem as minimizing the reconstruction error in the 2D domain often tends to ignore the overall 3D geometry and back-side appearance , which may end up with a shortcut solution that looks plausible only from the input viewpoint . While these methods compensate the relaxed supervision by exploiting various forms of prior information in the CMR-like framework , e.g. , semantic invariance in the UV space ( Li et al. , 2020 ) and interpolated consistency of the predicted 3D attributes ( Hu et al. , 2021 ) , this task remains challenging and worth further exploration . In this work , we propose an alternative approach that is built upon generative prior possessed by Generative Adversarial Networks ( GANs ) , named MeshInversion . GANs are typically known for their exceptional ability to capture comprehensive knowledge ( Brock et al. , 2019 ; Karras et al. , 2019 ; Shu et al. , 2019 ) , empowering the success of GAN inversion in various image restoration tasks ( Gu et al. , 2020 ; Pan et al. , 2020b ) and in point cloud completion ( Zhang et al. , 2021 ) . By training a GAN to synthesize 3D shapes in the form of a topology-aligned texture and deformation map , one could enable the generator to capture rich prior knowledge of a certain object category , including high-level semantics , object geometries , and texture details . We , therefore , propose to exploit this appealing generative prior through GAN inversion . Formally , our framework finds the latent code of the GAN manifold that best recovers the 3D object corresponding to the input image with a pre-trained 3D GAN . Given the single-view observation , the latent code is updated towards minimizing 2D reconstruction losses by rendering the 3D object onto the image plane . Hence , the 3D GAN manifold implicitly constrains the reconstructed 3D shape within the realistic boundaries , whereas minimization of the existing 2D losses explicitly drives the 3D shape towards a faithful reflection of the input image . Searching for the optimal latent code in the GAN manifold for single-view 3D object reconstruction is not trivial . Specifically , reprojection misalignment easily leads to a local minimum , and the impact of discretization-induced information loss is amplified by the smooth manifold . To address the misalignment issue , we propose a Chamfer Texture Loss , which relaxes the one-to-one pixel correspondences in existing low-level losses and allows the match to be found within a local region . By jointly considering the appearance and positions of image pixels or feature vectors , it provides a robust appearance distance despite inaccurate camera poses and in the presence of high-frequency textures . To avoid information loss , we propose a Chamfer Mask Loss , which intercepts the rasterization process and computes the Chamfer distance between the projected vertices before discretization to retain information , with the foreground pixels of the input image projected to the same continuous space . Hence , it becomes more sensitive to small variations in shape and offers a more accurate gradient for geometric learning . Our proposed method demonstrates compelling performance for 3D reconstruction from real-world monocular images . Overall , MeshInversion gives highly plausible and faithful 3D reconstruction in terms of both appearance and 3D shape . It achieves state-of-the-art results on the perceptual metric , i.e. , FID , when evaluating the textured mesh from various viewpoints , and is on-par with the existing CMR-based methods in terms of geometric accuracy . In addition , MeshInversion benefits from a holistic understanding of the objects given the generative prior . As a result , it not only gives a realistic recovery of the back-side texture but also generalizes well in the presence of occlusion . Furthermore , MeshInversion also shows fairly remarkable generalization for 3D shapes that are less commonly seen , such as birds with open wings and long tails . Some representative reconstruction results are depicted in Fig . 1 . 2 RELATED WORK . Single-view 3D Reconstruction . It recovers the 3D information of an object , such as its shape and texture , from a single-view observation . Several methods use image-3D object pairs ( Wang et al. , 2018 ; Pan et al. , 2019 ; Mescheder et al. , 2019 ; Rematas et al. , 2021 ) or multi-view images ( Niemeyer et al. , 2020 ; Liu et al. , 2019 ; Yariv et al. , 2020 ; Wang et al. , 2021 ; Oechsle et al. , 2021 ) for training , which limit the scenarios to synthetic data . Another line of work fits the parameters of a 3D prior morphable model , e.g. , SMPL for humans and 3DMM for faces ( Gecer et al. , 2019 ; Sanyal et al. , 2019 ; Kanazawa et al. , 2018a ) , which are expensive to build and difficult to extend to many different natural object categories . To relax the constraints on supervision , CMR ( Kanazawa et al. , 2018b ) reconstructs category-specific textured mesh by training with a collection of monocular images and associated 2D supervisions , i.e. , 2D key-points , camera poses , and silhouette masks . Thereafter , several follow-up works further relax the supervision , e.g. , masks only , and improves the reconstruction results mainly by exploiting forms of prior information across the object category or specific to the CMR-like framework . Specifically , they incorporate the prior by enforcing various types of cycle consistencies , such as texture cycle consistency ( Li et al. , 2020 ; Bhattad et al. , 2021 ) , rotation adversarial cycle consistency ( Bhattad et al. , 2021 ) , and interpolated consistency ( Hu et al. , 2021 ) . Some of these methods also leverage external information , e.g. , category-level mesh templates ( Goel et al. , 2020 ; Bhattad et al. , 2021 ) , and semantic parts provided by an external SCOPS model ( Li et al. , 2020 ) . For texture modeling , a direct regression of pixel values in the UV texture map often leads to blurry images , e.g. , Goel et al . ( 2020 ) . Therefore , the mainstream approach is to regress pixel coordinates , i.e. , learning texture flow from the input image to the texture map . Although texture flow is easier to regress and usually provides a vivid front view result , it is often unable to capture the high-level semantics , and thus fails to generalize well to novel views or occluded regions . Our approach directly predicts the texture pixel values by incorporating a pre-trained GAN . In contrast to the texture flow approach , it benefits from a holistic understanding of the objects given the generative prior and offers high plausibility and fidelity at the same time . GAN Inversion . A well-trained GAN usually captures useful statistics and semantics underlying the training data . In the 2D domain , GAN prior has been explored extensively in various image restoration and editing tasks ( Bau et al. , 2019b ; Pan et al. , 2020b ; Gu et al. , 2020 ) . GAN inversion , the common method in this line of work , finds a latent code that best reconstructs the given image using the pre-trained generator . Typically , the target latent code can be obtained via gradient descent ( Ma et al. , 2018 ; Lipton & Tripathi , 2017 ) , projected by an additive encoder that learns the inverse mapping of a GAN ( Bau et al. , 2019a ) , or a combination of them ( Zhu et al. , 2020 ) . Recently , there are attempts to apply GAN inversion in the 3D domain . Zhang et al . ( 2021 ) use a pre-trained point cloud GAN to address shape completion in the canonical pose , giving remarkable generalization for out-of-domain data such as real-world partial scans . Pan et al . ( 2020a ) first explore to recover the geometric cues from pre-trained 2D GANs and achieve exceptional reconstruction results , but the reconstructed shapes are limited to 2.5D due to limited poses that 2D GANs can synthesize . In this work , we directly exploit the prior from a 3D GAN to reconstruct the shape and texture of complete 3D objects . 3 APPROACH . Preliminaries . We represent a 3D object as a textured triangle mesh O ≡ ( V , F , T ) , where V ∈ R|v|×3 represents the location of the vertices , F represents the faces that define the fixed connectivity of vertices in the mesh , and T represents the texture map . An individual mesh is isomorphic to a 2-pole sphere , and thus we model the deformation ∆V from the initial sphere template , and then obtain the final vertex positions by V = Vsphere+∆V . Previous methods ( Kanazawa et al. , 2018b ; Li et al. , 2020 ; Goel et al. , 2020 ) typically regress individual vertices via a fully connected network ( MLP ) . In contrast , recent studies have found that using a 2D convolutional neural network ( CNN ) to learn a deformation map in the UV space would benefit from consistent semantics across the entire category ( Pavllo et al. , 2020 ; Bhattad et al. , 2021 ) . In addition , the deformation map S and the texture map T are topologically aligned , so both the values can be mapped to the mesh via the same UV mapping . We assume a weak-perspective camera projection , where the camera pose c is parameterized by scale s ∈ R , translation t ∈ R2 , and rotation in the form of quaternion r ∈ R4 . We use the DIB-R by Chen et al . ( 2019 ) as our differentiable renderer . We denote I = R ( S , T , c ) as the image rendering process . Under review as a conference paper at ICLR 2022 G formulas LD ( 1 ) LCD ( 2 ) z 2 Rd ( 3 ) d✓ ( 4 ) dz ( 5 ) xin ( 6 ) xc ( 7 ) xp ( 8 ) Copyright c 2020 , Association for the Advancement of Artificial Intelligence ( www.aaai.org ) . All rights reserved . formulas LD ( 1 ) LCD ( 2 ) z 2 Rd ( 3 ) d✓ ( 4 ) dz ( 5 ) xin ( 6 ) xc ( 7 ) xp ( 8 ) Copyright c 2020 , Association for the Advancement of Artificial Intelligence ( www.aaai.org ) . All rights reserved . Texture Deformation Differentiable Renderer Chamfer Texture Loss Chamfer Mask Loss Mesh Mask Spatial Textural Image Rendered Image Figure 2 : MeshInversion framework . We exploit a pre-trained 3D GAN that synthesizes plausible textured meshes in the form of topology-aligned deformation map and texture map in the UV space . Given a single-view image and the mask , we search for the latent code that minimizes reconstruction losses in the 2D domain via gradient descent . We address the intrinsic challenges associated with rendering with two Chamfer-based losses : 1 ) Chamfer Texture Loss ( Sec 3.2 ) relaxes the one-to-one pixel correspondences between two images or their respective feature maps and allows the match to be found within a local region , in which a pairwise distance is factorized into a spatial distance term and a textural distance term . We illustrate the distance maps between one anchor point from the rendered image to the target image , where brighter regions correspond to smaller distances . 2 ) Chamfer Mask Loss ( Sec 3.3 ) intercepts the discretization process and computes the chamfer distance between the projected vertices and the foreground pixels projected to the same continuous space .

Summary: This paper addresses the problem of learning single view reconstruction for a specific category using an image collection. This is related to the CMR framework by Kanazawa et al, ECCV 2018. The main contribution of this paper is using a GAN, and at test time searching for the latent code that best resembles the object in the input image. This is done with test-time optimization, and two losses, a Chamfer Texture Loss and a Chamfer Mask Loss, are proposed for this.