MeshInversion: 3D textured mesh reconstruction with generative prior

1The code will be available upon paper acceptance . 1 INTRODUCTION . Single-view 3D object reconstruction aims at recovering the shape and texture of the object from a single image . This long-standing problem is fundamental to various applications such as robotics navigation , 3D scene understanding , and augmented/virtual reality . A key challenge is the lack of 3D or multi-view supervision due to the prohibitive cost of data collection and annotation for object instances in the wild . Leveraging more readily available 2D supervisions such as keypoints , Kanazawa et al . ( 2018b ) first introduce CMR , a deep learning framework of textured mesh reconstruction that is trained with a collection of real-world monocular images of an object category . To further relax the supervision constraint , several follow-up studies propose to learn the 3D manifold in a self-supervised manner , only requiring single-view images and their corresponding masks for training ( Li et al. , 2020 ; Goel et al. , 2020 ; Bhattad et al. , 2021 ; Hu et al. , 2021 ) . This further complicates the problem as minimizing the reconstruction error in the 2D domain often tends to ignore the overall 3D geometry and back-side appearance , which may end up with a shortcut solution that looks plausible only from the input viewpoint . While these methods compensate the relaxed supervision by exploiting various forms of prior information in the CMR-like framework , e.g. , semantic invariance in the UV space ( Li et al. , 2020 ) and interpolated consistency of the predicted 3D attributes ( Hu et al. , 2021 ) , this task remains challenging and worth further exploration . In this work , we propose an alternative approach that is built upon generative prior possessed by Generative Adversarial Networks ( GANs ) , named MeshInversion . GANs are typically known for their exceptional ability to capture comprehensive knowledge ( Brock et al. , 2019 ; Karras et al. , 2019 ; Shu et al. , 2019 ) , empowering the success of GAN inversion in various image restoration tasks ( Gu et al. , 2020 ; Pan et al. , 2020b ) and in point cloud completion ( Zhang et al. , 2021 ) . By training a GAN to synthesize 3D shapes in the form of a topology-aligned texture and deformation map , one could enable the generator to capture rich prior knowledge of a certain object category , including high-level semantics , object geometries , and texture details . We , therefore , propose to exploit this appealing generative prior through GAN inversion . Formally , our framework finds the latent code of the GAN manifold that best recovers the 3D object corresponding to the input image with a pre-trained 3D GAN . Given the single-view observation , the latent code is updated towards minimizing 2D reconstruction losses by rendering the 3D object onto the image plane . Hence , the 3D GAN manifold implicitly constrains the reconstructed 3D shape within the realistic boundaries , whereas minimization of the existing 2D losses explicitly drives the 3D shape towards a faithful reflection of the input image . Searching for the optimal latent code in the GAN manifold for single-view 3D object reconstruction is not trivial . Specifically , reprojection misalignment easily leads to a local minimum , and the impact of discretization-induced information loss is amplified by the smooth manifold . To address the misalignment issue , we propose a Chamfer Texture Loss , which relaxes the one-to-one pixel correspondences in existing low-level losses and allows the match to be found within a local region . By jointly considering the appearance and positions of image pixels or feature vectors , it provides a robust appearance distance despite inaccurate camera poses and in the presence of high-frequency textures . To avoid information loss , we propose a Chamfer Mask Loss , which intercepts the rasterization process and computes the Chamfer distance between the projected vertices before discretization to retain information , with the foreground pixels of the input image projected to the same continuous space . Hence , it becomes more sensitive to small variations in shape and offers a more accurate gradient for geometric learning . Our proposed method demonstrates compelling performance for 3D reconstruction from real-world monocular images . Overall , MeshInversion gives highly plausible and faithful 3D reconstruction in terms of both appearance and 3D shape . It achieves state-of-the-art results on the perceptual metric , i.e. , FID , when evaluating the textured mesh from various viewpoints , and is on-par with the existing CMR-based methods in terms of geometric accuracy . In addition , MeshInversion benefits from a holistic understanding of the objects given the generative prior . As a result , it not only gives a realistic recovery of the back-side texture but also generalizes well in the presence of occlusion . Furthermore , MeshInversion also shows fairly remarkable generalization for 3D shapes that are less commonly seen , such as birds with open wings and long tails . Some representative reconstruction results are depicted in Fig . 1 . 2 RELATED WORK . Single-view 3D Reconstruction . It recovers the 3D information of an object , such as its shape and texture , from a single-view observation . Several methods use image-3D object pairs ( Wang et al. , 2018 ; Pan et al. , 2019 ; Mescheder et al. , 2019 ; Rematas et al. , 2021 ) or multi-view images ( Niemeyer et al. , 2020 ; Liu et al. , 2019 ; Yariv et al. , 2020 ; Wang et al. , 2021 ; Oechsle et al. , 2021 ) for training , which limit the scenarios to synthetic data . Another line of work fits the parameters of a 3D prior morphable model , e.g. , SMPL for humans and 3DMM for faces ( Gecer et al. , 2019 ; Sanyal et al. , 2019 ; Kanazawa et al. , 2018a ) , which are expensive to build and difficult to extend to many different natural object categories . To relax the constraints on supervision , CMR ( Kanazawa et al. , 2018b ) reconstructs category-specific textured mesh by training with a collection of monocular images and associated 2D supervisions , i.e. , 2D key-points , camera poses , and silhouette masks . Thereafter , several follow-up works further relax the supervision , e.g. , masks only , and improves the reconstruction results mainly by exploiting forms of prior information across the object category or specific to the CMR-like framework . Specifically , they incorporate the prior by enforcing various types of cycle consistencies , such as texture cycle consistency ( Li et al. , 2020 ; Bhattad et al. , 2021 ) , rotation adversarial cycle consistency ( Bhattad et al. , 2021 ) , and interpolated consistency ( Hu et al. , 2021 ) . Some of these methods also leverage external information , e.g. , category-level mesh templates ( Goel et al. , 2020 ; Bhattad et al. , 2021 ) , and semantic parts provided by an external SCOPS model ( Li et al. , 2020 ) . For texture modeling , a direct regression of pixel values in the UV texture map often leads to blurry images , e.g. , Goel et al . ( 2020 ) . Therefore , the mainstream approach is to regress pixel coordinates , i.e. , learning texture flow from the input image to the texture map . Although texture flow is easier to regress and usually provides a vivid front view result , it is often unable to capture the high-level semantics , and thus fails to generalize well to novel views or occluded regions . Our approach directly predicts the texture pixel values by incorporating a pre-trained GAN . In contrast to the texture flow approach , it benefits from a holistic understanding of the objects given the generative prior and offers high plausibility and fidelity at the same time . GAN Inversion . A well-trained GAN usually captures useful statistics and semantics underlying the training data . In the 2D domain , GAN prior has been explored extensively in various image restoration and editing tasks ( Bau et al. , 2019b ; Pan et al. , 2020b ; Gu et al. , 2020 ) . GAN inversion , the common method in this line of work , finds a latent code that best reconstructs the given image using the pre-trained generator . Typically , the target latent code can be obtained via gradient descent ( Ma et al. , 2018 ; Lipton & Tripathi , 2017 ) , projected by an additive encoder that learns the inverse mapping of a GAN ( Bau et al. , 2019a ) , or a combination of them ( Zhu et al. , 2020 ) . Recently , there are attempts to apply GAN inversion in the 3D domain . Zhang et al . ( 2021 ) use a pre-trained point cloud GAN to address shape completion in the canonical pose , giving remarkable generalization for out-of-domain data such as real-world partial scans . Pan et al . ( 2020a ) first explore to recover the geometric cues from pre-trained 2D GANs and achieve exceptional reconstruction results , but the reconstructed shapes are limited to 2.5D due to limited poses that 2D GANs can synthesize . In this work , we directly exploit the prior from a 3D GAN to reconstruct the shape and texture of complete 3D objects . 3 APPROACH . Preliminaries . We represent a 3D object as a textured triangle mesh O ≡ ( V , F , T ) , where V ∈ R|v|×3 represents the location of the vertices , F represents the faces that define the fixed connectivity of vertices in the mesh , and T represents the texture map . An individual mesh is isomorphic to a 2-pole sphere , and thus we model the deformation ∆V from the initial sphere template , and then obtain the final vertex positions by V = Vsphere+∆V . Previous methods ( Kanazawa et al. , 2018b ; Li et al. , 2020 ; Goel et al. , 2020 ) typically regress individual vertices via a fully connected network ( MLP ) . In contrast , recent studies have found that using a 2D convolutional neural network ( CNN ) to learn a deformation map in the UV space would benefit from consistent semantics across the entire category ( Pavllo et al. , 2020 ; Bhattad et al. , 2021 ) . In addition , the deformation map S and the texture map T are topologically aligned , so both the values can be mapped to the mesh via the same UV mapping . We assume a weak-perspective camera projection , where the camera pose c is parameterized by scale s ∈ R , translation t ∈ R2 , and rotation in the form of quaternion r ∈ R4 . We use the DIB-R by Chen et al . ( 2019 ) as our differentiable renderer . We denote I = R ( S , T , c ) as the image rendering process . Under review as a conference paper at ICLR 2022 G formulas LD ( 1 ) LCD ( 2 ) z 2 Rd ( 3 ) d✓ ( 4 ) dz ( 5 ) xin ( 6 ) xc ( 7 ) xp ( 8 ) Copyright c 2020 , Association for the Advancement of Artificial Intelligence ( www.aaai.org ) . All rights reserved . formulas LD ( 1 ) LCD ( 2 ) z 2 Rd ( 3 ) d✓ ( 4 ) dz ( 5 ) xin ( 6 ) xc ( 7 ) xp ( 8 ) Copyright c 2020 , Association for the Advancement of Artificial Intelligence ( www.aaai.org ) . All rights reserved . Texture Deformation Differentiable Renderer Chamfer Texture Loss Chamfer Mask Loss Mesh Mask Spatial Textural Image Rendered Image Figure 2 : MeshInversion framework . We exploit a pre-trained 3D GAN that synthesizes plausible textured meshes in the form of topology-aligned deformation map and texture map in the UV space . Given a single-view image and the mask , we search for the latent code that minimizes reconstruction losses in the 2D domain via gradient descent . We address the intrinsic challenges associated with rendering with two Chamfer-based losses : 1 ) Chamfer Texture Loss ( Sec 3.2 ) relaxes the one-to-one pixel correspondences between two images or their respective feature maps and allows the match to be found within a local region , in which a pairwise distance is factorized into a spatial distance term and a textural distance term . We illustrate the distance maps between one anchor point from the rendered image to the target image , where brighter regions correspond to smaller distances . 2 ) Chamfer Mask Loss ( Sec 3.3 ) intercepts the discretization process and computes the chamfer distance between the projected vertices and the foreground pixels projected to the same continuous space .

Summary: The paper proposes MeshInversion, a method for single-view 3D shape reconstruction. Given a test image with silhouette, it optimizes the 3D shape represented as the input latent code to a pre-trained 3D GAN (specifically ConvMesh, NeurIPS 20) via the proposed chamfer losses. The idea of leveraging the 3D GAN shape priors to regularize the reconstruction of unobservable surface is novel and sensible. The method is quantitatively evaluated on CUB and performs better than some recent methods on FID scores and user study.