MeshInversion: 3D textured mesh reconstruction with generative prior
Authors: Junzhe Zhang
Daxuan Ren
Zhongang Cai
Chai Kiat Yeo
Bo Dai
Chen Change Loy
Venue: ICLR 2022 Submitted (2022)
Keywords: Single-view 3D object reconstruction
GAN inversion

Abstract: Recovering a textured 3D mesh from a single image is highly challenging, particularly for in-the-wild objects that lack 3D ground truths. Prior attempts resort to weak supervision based on 2D silhouette annotations of monocular images. Since the supervision lies in the 2D space while the output is in the 3D space, such in-direct supervision often over-emphasizes the observable part of the 3D textured mesh, at the expense of the overall reconstruction quality. Although previous attempts have adopted various hand-crafted heuristics to reduce this gap, this issue is far from being solved. In this work, we present an alternative framework, \textbf{MeshInversion}, that reduces the gap by exploiting the \textit{generative prior} of a 3D GAN pre-trained for 3D textured mesh synthesis. Reconstruction is achieved by searching for a latent space in the 3D GAN that best resembles the target mesh in accordance with the single view observation. Since the pre-trained GAN encapsulates rich 3D semantics in terms of mesh geometry and texture, searching within the GAN manifold thus naturally regularizes the realness and fidelity of the reconstruction. Importantly, such regularization is directly applied in the 3D space, providing crucial guidance of mesh parts that are unobserved in the 2D space. Experiments on standard benchmarks show that our framework obtains faithful 3D reconstructions with consistent geometry and texture across both observed and unobserved parts. Moreover, it generalizes well to meshes that are less commonly seen, such as the extended articulation of deformable objects.

Under review as a conference paper at ICLR 2022
MESHINVERSION: 3D TEXTURED MESH RECONSTRUCTION WITH GENERATIVE PRIOR
Anonymous authors
Paper under double-blind review
Inpur
UMR
SMR (Ht
2020)
Inpuí view
Novel view
Texture
a)
c)
d)
Figure 1: MeshInversion is alternative approach to single-view textured mesh reconstruction by exploiting generative prior encapsulated in a pre-trained GAN. Our method has three major advantages: 1) It reconstructs highly faithful and realistic 3D objects, even when observed from novel views; 2) The reconstruction is robust against occlusion in (b), where red arrows indicate textural errors; 3) The method generalizes reasonably well to less common shapes, such birds with (c) extended tails or (d) open wings, Note that both MeshInversion and SMR have access to silhouettes during inference, whereas UMR does not.
ABSTRACT
Recovering textured 3D mesh from single image is highly challenging, par- ticularly for in-the-wild objects that lack 3D ground truths. Prior attempts resort weak supervision based 2D silhouette annotations of monocular images. Since the supervision lies in the 2D space while the output is in the 3D space, such in-direct supervision often over-emphasizes the obscrvable part of the 3D textured mesh, at the expense of the overall reconstruction quality. Although previ- ous attempts have adopted various hand-crafted heuristics to reduce this gap, this issue is far from being solved. In this work, we present an alternative framework, Ieshinversion, that reduces the gap by exploiting the generative prior a 3D GAN pre-trained for 3D textured mesh synthesis. Reconstruction is achieved by searching for a latent space in the 3D GAN that best resembles the target mesh in accordance with the single view observation. Since the pre- mre-trained GAN en- capsulates rich 3D semantics terms of mesh geometry and texture, scarching within the GAN manifold thus naturally regularizes the realness and fidelity of the reconstruction. Importantly, such regularization is directly applied in the 3D space, providing crucial guidance of mesh parts that are unobserved in the 2D space. Experiments on standard benchmarks show that our framework obtains faithful 3D reconstructions with consistent geometry and texture across both observed and unobserved parts. Moreover, it generalizes well to meshes that are less commonly seen, such as the extended articulation of deformable objects! [
'The code will be available upon paper acceptance.

---

Under review as a conference paper at ICLR 2022
INTRODUCTION
Single-view 3D object reconstruction aims at recovering the shape and texture of the object from single image. This navigation, 3D scene understanding, and augmented/virtual reality. A key challenge is the lack of 3D or multi-view supervision due to the prohibitive cost of data collection and annotation for object instances in the wild.
problem fundamental to various applications such as robotics
Leveraging more readily available 2D supervisions such as keypoints, Kanazawa et al. (201 first introduce CMR, deep learning framework of textured mesh reconstruction that is trained with collection of real-world monocular images of an object category. To further relax the supervision constraint, several follow-up studies propose to learn the 3D manifold in self-supervised manner. only requiring single-view images and their corresponding masks for training (Li et al., 2020; Goel et al. 2020 Bhattad et al.) 2021 Hu et al. 2021]. This further complicates the problem as minimizing the reconstruction error in the 2D domain often tends to ignore the overall 3D geometry and back-side appearance, which may end up with shorteut solution that looks plausible only from the input viewpoint. While these methods compensate the relaxed supervision by exploiting various forms of prior information in the CMR-like framework, e.g., semantic invariance in the UV space (Li et al.) 2020) and interpolated consistency of the predicted 3D attributes (Hu et al. 202T). this task remains challenging and worth further exploration.
In this work, we propose an alternative approach that is built upon generative prior possessed by Generative Adversarial Networks (GANs), named MeshInversion. GANs are typically known for their exceptional ability to capture comprehensive knowledge (Brock et al. 2019 Karras et al., 2019 Shu et al. 2019), empowering the success of GAN inversion in various image restoration tasks Gu et al. 2020 Pan et al., 2020b) and in point cloud completion (Zhang et al. (021). By training a GAN to synthesize 3D shapes in the form of a topology-alee -liigned texture and deformation map, one could enable the generator to capture rich prior knowledge of certain object category, including high-level semantics, object geometries, and texture details. We, therefore, propose to exploit this appealing generative prior through GAN inversion. Formally, our framework finds the latent code of the GAN manifold that best recovers the 3D object corresponding to the input image with a prented 3D GAN. Given the single-view observation, the latent code is updated towards minimizing 2D reconstruction losses by rendering the 3D object onto the image plane. Hence, the 3D GAN manifold implicitly constrains the reconstructed 3D shape within the realistic boundaries. whereas minimization of the existing 2D losses explicitly drives the 3D shape towards a faithful reflection of the input image.
Searching for the optimal latent code in the GAN manifold for hinglevview 3D object reconstruction not trivial. Specifically, reprojection misalignment easily leads to a local minimum, and the impact of discretization-inddeee information loss is amplified by the smooth manifold. To address the misalignment issue, we propose a Chamfer Texture Loss, which relaxes the one-to-one pixel correspondences in existing low-level losses and allows the match to be found within local region. By jointly considering the appearance and positions of image pixels or feature vectors, it provides a robust appearance distance despite inaccurate camera poses and in the presence of high-frequency textures. To avoid information loss, we propose a Chamfer Mask Loss, which intercepts the rasterization process and computes the Chamfer distance between the projected vertices before discretization to retain information, with the foreground pixels of the input image projected to the same continuous space. Hence, it becomes more sensitive to small variations in shape and offers more accurate gradient for geometric learning.
Our proposed method demonstrates compelling performance for 3D reconstruction from real-world monocular images. Overall, MeshInversion gives highly plausible and faithful 3D reconstruction in terms of both appearance and 3DD shape. It achieves state-of-the-art results on the perceptual metric, i.., FID, when evaluating the textured mesh from various viewpoints, and is on-par with the existing CMR-based methods terms of geometric accuracy. In addition, MeshInversion benefits from a holistic understanding of the objects given the generative prior. As result, it not only gives a realistic recovery of the side texture but also generalizes well in the presence of occlusion. Furthermore, MeshInversion also shows fairly remarkable generalization for 3D shapes that are less commonly seen, such as birds with open wings and long tails. Some representative reconstruction results are depicted in Fig. [
2

---

Under review as a conference paper at ICLR 2022
2 RELATED WORK
Single-view 3D Reconstruction. It recovers the 3D information of an object, such as its shape and texture, from a single-view observation. Several methods use image-3D object pairs (Wang et al.
2018 Pan et al. 2019
Mescheder et al., 2019 Rematas et al.
2021 or multi-view images (Nimmeyer
et al. 2020 Liu et al.
2019] Yariv et al. 20.00 Wang et al.
2021 Oechsle et al. 2021 for training.
which limit the scenarios to synthetic data. Another line of work fits the parameters of 3D prior
morphable model, e.g., SMPL for humans and 3DMM for faces (Gecer et al. 2019 Sanyal et al. 2019)) Kanazawa et al., 20T8a), which are expensive to build and difficult to extend to many different natural object categories. To relax the constraints on supervision, CMR (Kanazawa et al. 8b) reconstructs category-speciffe textured mesh by training with a collection of monocular images and associated 2D supervisions, i.e., 2D key-points, camera poses, and silhouette masks. Thereafter, several follow-up works further relax the supervision, e.g., masks only, and improves the reconstruction results mainly by exploiting forms of prior information across the object category or specific to the CMR-like framework. Specifically, they incorporate the prior by enforcing various types of cycle consistencies, such as texture cycle consistency (Li et al.) 2020] [Bhattad et al.; (202T), rotation adversarial cycle consistency (Bhattad et al. (02T), D. and interpolated consistency (Hu al. 2021). Some of these methods also leverage external information, e.g., category-level mesh templates (Goel et al. 2020
Bhattad et al. (02T), and semantic parts provided by an external SCOPS model (Li et al. 2020).
For texture modeling, a direct regression of pixel values in the UV texture map often leads to blurry images, e.g., e.g.,Goel et al. (2020). Therefore, the mainstream approach is to regress pixel coordinates, i.e., learning Texture flow from the input image to the texture map. Although texture flow is easier to regress and usually provides a vivid front view result, it is often unable to capture the high-level semantics, and thus fails to generalize well to novel views occluded regions. Our approach directly predicts the texture pixel values by incorporating a pre-trained GAN. In contrast to the texture flow approach, it benefits from holistic understanding of the objects given the generative prior and offers high plausibility and fidelity at the same time.
GAN Inversion. A well-trained GAN usually captures useful statistics and semantics underlying the training data. In the 2D domain, GAN prior has been explored extensively in various image restoration and editing tasks (Bau et al. 2019b Pan et al. 2020b Gu et al. 2020). GAN inversion, the common method in this line of work, finds a latent code that best reconstructs the given image using the pre-trained generator. Typically. the target latent code can be obtained via gradient descent (Ma et al.) 2018 Lipton & Tripathi 2017). projected by an additive encoder that learns the inverse mapping of a GAN (Bau et al.) 2019a),, or combination of them (Zhu et al. 2020). Recently, there are attempts to apply GAN inversion in the 3D domain. Zhang et al.22202) use pre-trained point cloud GAN to address shape completion in the canonical pose, giving remarkable generalization for off-oind data such as real-world partial scans. Pan et al. (020a) first explore to recover the geometric cues from pre-trained 2D GANs and achieve exceptional reconstruction results, but the reconstructed shapes are limited to 2.5D due to limited poses that 2D GANs can synthesize. In this work, directly exploit the prior from a 3D GAN to reconstruct the shape and texture of complete 3D objects.
3 APPROACH
Preliminaries. We represent a 3D object as a textured triangle mesh O = (V, where V E R represents the location of the vertices, F represents the faces that define the fixed connectivity of vertices in the mesh, and T represents the texture map. An individual mesh is iso- morphic to a 2-pole sphere, and thus we model the deformation AV from the initial sphere template, and then obtain the final vertex positions by V sphere + AV. Previous methods (Kanazawa et al., 2018b Li et al. 2220; Goel et al., 2020) typically regress individual vertices via a fully connected network (MLP). contrast, recent studies have found that using a 2D convolutional neural network (CNN) to learn deformation map in the UV space would benefit from consistent semantics across the entire category (Pavllo et al. 2020; Bhattad et al. 202IJ. In addition, the deformation map S and the texture map T are topologically aligned, so both the values can be mapped to the mesh via the same UV mapping. We assume a weak- perukpeespeettve camera projection, where the camera pose c is parameterized by scale s R, translation E R and rotation in the form of quaternion r R-. We use the DIB-R by Chen et al. (2019) as our differentiable renderer. We denote I = R(S, T, c) as the image rendering process.
3

---

Under review as a conference paper at ICLR 2022
dz
Mask
Deformation
Mesh
IRd
Mask
Differntiable Renderer
Spuial
Rendered
Texture
Image
Chamfer Texture Loes
Image
Figure 2: Insriivnrsoon framework. We exploit pre-trained 3D GAN that synthesizes plausible textured meshes in the form topology-aligned deformation map and texture map in the UV space. Given single-view image and the mask, search for the latent code that minimizes reconstruction losses in the 2D domain via gradient descent. We address the intrinsic challenges associated with rendering with two Chamfer-basee losses: Chamfer Texture Loss (Sec(3.2) relaxes the one-to-one pixel correspondences between two images or their respective feature maps and allows the match to be found within local region, in which pairwise distance is factorized into spatial distance term and a textural distance term. We illustrate the distance maps between one anchor point from the rendered image to the target image, where brighter regions correspond to smaller distances. Chamfer Mask Loss (Sec 3.3) intercepts the discretization process and computes the chamfer distance between the projected vertices and the foreground pixels projected to the same continuous space.
3.1 MESHINVERSION
In this work, we propose to exploit generative prior in the task of single-view 3D reconstruction. Pavllo et al. (2000) recently propose CNN-based GAN framework, ConyMesh, which generates 3D objects in the form of eppoogyyaaligned texture maps and deformation maps in the UV space. Despite being trained with purely 2D supervisions from single-view natural images, ConvMesh captures meaningful prior knowledge of the object category (such as high-level semantics, object geometries, and texture details), and is able generate objects with wide range of plausible shapes and high-fidelity textures. Motivated by the success of GAN inversion in various image restoration tasks, we incorporate the prior stored in pre-trained ConvMesh achieve plausible and faithful recovery of 3D shape and appearance through GAN inversion.
Pre-training Stage. Prior to GAN inversion, first pre-train the textured GAN on the training split to capture desirable prior knowledge for 3D reconstruction. Pavllo et al. 2020) train ConvMesh in the UV space, where pseudo ground truths texture map and deformation map are obtained by inverse rendering from the single-view images. In addition to the UV space discrimination, we further enhance the photorealism of the generated 3D objects by introducing discriminator the image space, following the architecture of PatchGAN in Isola et al.) (017)) The loss functions for the pre-training stage are shown as follows. Deo refers to the discriminator in the UV space; Dy refers to the discriminator in the image space; Auv and A are the weights. We use least-squares losses following (Mao et al. 2017). Ablation study on image space discrimination, and more details on ConyMesh and pseudo ground truths preparation be found in the Supplementary Material.
LG Buu [(D (G(z)) 1) B, (.(x) (D), (D) 1)
LD.. = ES.T~ [D (S, T) 1) E ([
(2)
LD, = EI-Paa. (D) - 1) (2)
(3)
Inversion Stage. We now formally introduce GAN inversion for single-view 3D reconstruction. Given a pre-trained ConvMesh that generates a textured mesh from a latent code, S,T G(z), we aim find the that best recovers the 3D object from the input image I,n and its silhouette mask M Specifically, we achieve this by test-time optimization, where is optimized towards minimizing the overall reconstruction loss Line via gradient descent. In general, the inversion stage can be denoted by Eq. 4 and illustrated in Fig. 2
z. = arg min Limv (R(G(z), Inn)
(4)
2
4

---

Under review as a conference paper at ICLR 2022
Given the single-view image and the associated mask, we would need to project the reconstructed 3D object to the observation space for computing Line. However, such 3D-to-2D degradation is non-trivial. In existing image- tasks involving GAN inversion, we always assume pixel-wise image correspondence in the observation space. This is because both the appearance and pose of the scene can be controlled in the latent space of 2D GANs (Shen al. 2020; [Shen & Zhou (202T). In contrast, we generate 3D objects in the form of a pose-invariant UV texture map and deformation map, and explicitl control the object pose during differentiable rendering. While concurrently optimizing the latent code and the camera pose is plausible approach, this often suffers from canera-shape ambiguity (Li et al., 2020) and leads to erroneous reconstruction. In addition, the presence of high- nigh-frreuency textures, e.g., complex bird feathers, often leads to blurry appearance even with slight discrepancies pose. Consequently, is infeasible to assume a perfect alignment between the rendered image and the input one, which calls for robust form of appearance loss in Sec3.2
3.2 CHAMFER TEXTURE LOSS
To facilitate search in the GAN manifold without worrying about blurry reconstructions, reconsider the appearance loss by relaxing the pixel-aligned assumption in existing low-level losses. Taking inspiration from the point cloud data structure, treat 2D image as of 21D colored points, which have both appearance attributes, Le., RGB values, and spatial attributes, the values of which relate their coordinates in the image grid. Thereafter, we aim measure the dissimilarity between the two colored point sets via Chamfer distance,
1 S2
LCD(S S2)
min DE
min D xE8;
(5)
S
WE
Intuitively, defining the pairwise distance between pixel x and pixel y in the two respective images should jointly consider their appearance and location. In this regard, we factorize the overall pairwise distance Day into an appearance term D By and a spatial term D ay both of which are L2 distance. Like conventional Chamfer distance, single-sided pixel correspondences are determined by column- wise row-wise minimum in the distance matrix D.
Importantly, we desire the loss be tolerant and only tolerant of local misalignment, as large misalignment will potentially introduce noisy pixel correspondences that may jeopardize appearance learning. Inspired by the focal loss for detection (Lin et al. 2017), we introduce an exponential operation in the spatial term to penalize those spatially distant pixel pairs. Therefore, we define the overall distance matrix D as follows
D max (D* +e,"
(6)
where D and D are the appearance distance matrix and spatial distance matrix respectively; es and 6a are residual terms to avoid incorrect matches with identical location or identical pixel value respectively; is the scaling factor for flexibility. Specifically, we let e, < 1 so that the spatial term remains one when two pixels are slightly misaligned. Note that the spatial term is not differentiable and it only serves as a weight matrix for appearance learning.
By substituting the resulting D into Eq. [ we thus have the final formulation of our proposed Chamfer Texture Loss, denoted as LoT. Such relaxed formulation provides robust measure of texture distance, which effectively eases searching of the target latent code while preventing blurry reconstructions: in return, although Ler only concerns about local patch statistics but not photorealism, the use of GAN prior is sufficient to give realistic predictions. Besides, the GAN prior also allows computing Ler with a nown-sampled size of colored points. In practice, we randomly select 8096 pixels from each image as a point set.
Moreover, the proposed formulation gives flexible control between appearance and spatial attributes. which is readily extendable to misaligned feature maps to achieve more semantically faithful 3D reconstruction. Specifically, we apply the Chamfer texture loss between the (foreground) feature maps extracted with a pre-trained VGG-19 network (Simonyan & Zisserman, 2014) from the rendered image and the input image. It is worth noting that the feature-level Chamfer texture loss is somewhat related to the contextual loss (Mechrez et al., 2018). which addresses the misalignment issue for image transfer. The key difference is that the contextual loss only considers the feature distances but ignores their locations. We compare against the contextual loss in the experiment.
5

---

Under review as a conference paper at ICLR 2022
CMR
U-CMR
UMR
SMR
View-geni 0.629
Ours
IoU1
0.703
0.701
0.734
0.8
0.708
Table 1: Quantitative results on
FID:
140.9
65.0
40.0
52.9
38.6
CUB. 1; We report results from
10 4
176.2
314.9
72.8
63.2
38.6
Bhattad et al. (202T) as the im-
11 4
180.1
315.2
86.9
79.2
70.3
56.6
plementation is not available.
3.3
CHAMFER MASK LOSS
Conventionally, the geometric distance is usually computed between two binary masks in terms of L.1 or loU loss (Kanazawa et al. 2018b] Goel et al., 2020) ei al., 2020) Hu et al. (021). However, obtaining the mask of the reconstructed mesh usually involves rasterization that discretizes the mesh into grid of pixels. This operation inevitably introduces information loss and thus inaccurate supervision signals. This is particularly harmful to a well-trained ConvMesh, the shape manifold of which is typically smooth. Specifically, a small perturbation in z usually corresponds to slight variation in the 3D shape. which may translate to an unchanged binary mask. This usually leads to an insensitive gradient for paok-prop,gation, which undermines geometric learning.
To this end, we propose a Chamfer Mask Loss, LCM, to compute the geometric distance in an unquantized 2D space. Instead of rendering the mesh into a binary mask, we directly project the 31) obtain
vertices of the mesh onto the image plane, S, P(S, T, c). For the foreground mask, the positions of the foreground pixels by normalizing their pixel coordinates in the range of [-1, 1]. Thereafter, compute the Chamfer distance between S, and Sy. which is the set of normalized coordinates of the input mask. Note that the bidirectional Chamfer distance between the sparse set S, and dense set S would regularize the vertices from highly uneven deformation.
3.4 OVERALL OBJECTIVE FUNCTION
We apply the pixel-level Chamfer texture loss LCT-p and the feature-level one LCT-f as our appearance losses, and the Chamfer mask loss LCM as geometric loss. Besides, we also introduce two regularizers: the smooth loss Lsmooth that encourages neighboring faces to have similar normals, i.e., low cosine; the latent space loss Lg that regularizes the L2 norm of z to ensure Gaussian distribution. The overall object function is shown in Eq. 7 where ACT-p- XCT-S, ÀCM. Asmooth. and A2 are the weights, g is the pre-trained VGG-19 feature extractor, and denotes element-wise product.
Line -CT -CT--fcT-ppp R[(S) T, I) O M,n) + (on-s-er-y(((RR) T, 9(T, @ M,n)
(PMS T c), Min) + Asmooth Eymooth ((V, F)) + Xg -22000000 (z)
4 EXPERIMENTS
Datasets and Experimental Setting. We primarily evaluate method on CUB-200-201 dataset (Wah et af., 2011). It consists of 200 species of birds with a wide range of shapes and feathers, making an ideal benchmark evaluate 3D reconstruction in terms of both geometric and texture fidelity. Apart from the organic shapes like birds, we also validate our method on 11 man-made rigid car categories from PASCAL3D+ (Xiang et al. 2014).
We use the same train-validation-tes split as provided by CMR (Kanazawa et al. 2018b). The images in both datasets are annotated with foreground masks and camera poses. Specifically, we pre-train ConvMesh on the pscudo ground truths derived from the training split following class conditional setting (Pavllo et al., 2020 During inference, MeshInversion has access to the input image and the associated mask and camera pose. Since the focus of this work is to study the effectiveness of exploiting GAN prior in 3D reconstruction, we base our experiments mainly on silhouette ground truths and cameras estimated by structure-from-
(SIM), following the setting of SMR (Hu et al. 2021 and Chen et al. 2019) respectively. In the Supplementary Material, we also validate that our method is reasonably robust to inaccurate camera poses predicted by off-the-shel camera pose estimator from Kanazawa et al. (2018b), and invariant under masks predicted by instance segmentation method PointRend (Kirillov al. 2020) pre-trained on COCO (Lin et al. 2014).

---

Under review as a conference paper at ICLR 2022
Evaluation Strategy. Since there 3D ground truths available, evaluate method against various baselines from the following three aspects: 1) We evaluate the geometry accuracy in the 2D domain by IoU between the input mask and the rendered mask. 2) We evaluate the appearance quality by image synthesis metric FID (Heusel et al.) 2017), which compares the distribution of test images and the render of reconstruction. We report both single-view FID (FID)) and multi-view FIDs. Specifically, we report 12 following the setting in SMR, which covers azimuth from 0" to 360" at an interval of 30% However, we notice that the exact front view (90%) and the exact back view (2709) are rarely seen in CUB, so we mainly report FID10 that excludes these two viewpoints. 3) Apart from extensive qualitative results, we conduct user study to evaluate human preferences comparing our method against existing baselines, in terms of both shape and appearance.
Input
CMR
U-CMR
UMR
SMR
Ours
a)
b)
c)
d)
e)
Figure 3: Qualitative results on CUB. MeshInversion achieves highly faithful and realistic 3D reconstruction. In particular, it exhibits superior generalization under various challenging scenarios, including with occlusion (b, c) and extended articulation (d, e).
4.1 COMPARISON WITH BASELINES
We compare MeshInversion with various CMR-based methods on the CUB dataset. Among these methods, only U-CMR (Goel et al. 2020) directly predicts texture maps. CMR (Kanazawa et al.) 2018b). UMR (Li et al.) 2020), and SMR (Hu al. (02I)) all use texture flow. We provide quantitative results in Tab. Overall, our method achieves state-of-the-arl results on perceptual metrics, particularly multi- FIDs, and on par with existing baselines in terms of IoU.
We provide single-view and novel-view qualitative results in Fig. 3 and Fig. respectively. It is observed that our method achieves highly faithful and realistic 3D reconstruction, particularly when observed from novel views. In contrast, although texture flow-based methods generally give superior texture reconstruction for visible regions, they tend to give incorrect predictions for invisible regions, e.g., abdomen or back. Moreover, our method benefits from holistic understanding of the objects and gives remarkable performance the presence of occlusion, while texture flow-based methods only learn to copy from the foreground pixels including the obstacles, e,g., twig, from the bird. Due to the same reason, these methods also tend to copy background pixels onto the reconstructed object when the shape prediction is inaccurate. Moreover, our method generalizes reasonably well to highly articulated shapes, such as birds with long tails and open wings, where many of the existing methods fail to give satisfactory reconstructions. See the Supplementary Material for more novel-view results.
7

---

Under review as a conference paper at ICLR 2022
Input
CMR
U-CMR
UMR
Ours
b)
C)
Figure 4: Novel-view rendering results on CUB for MeshInversion and various baselines. general, our method gives plausible 3D reconstruction even when observed from novel views, whereas many of these baseline methods fail generalize well invisible regions and challenging articulations.
Criterion Texture
CMR
U-CMR
UMR
SMR
2.7%
15.7%
13.2%
6.6%
61.8%
Table
User preference study on CUB in terms of the quality and faithfulness of texture, shape, and overall reconstruction.
Shape
2.7%
19.6%
14.6%
4.2%
58.9 %
Overall
2.5%
19.8%
12.8%
3.3%
61.5 %
To further evaluate the 3D reconstruction results, we conduct user preference study on 30 selected images from the test set. The reconstructed 3D) objects by different methods are rendered from three different viewpoints, for the users choose the most realistic and faithful reconstruction in terms of texture, shape, and overall 3D reconstruction. The evaluation results based on 40 users' responses in Tab. 2 show that MeshInversion achieves the most preferred 3D reconstruction, whereas all texture flow-based methods give poor results mainly due their incorrect prediction for unseen regions.
However, it is worth noting that all the baseline methods are regression based, and all except for SMR have no access to silhouettes test time. For fair comparison, also conduct test-time optimization for existing baselines with access to silhouettes during inference in the Supplementary Material, and the results show that method remains highly competitive especially for FIDs.
4.2 TEXTURE TRANSFER
As the shape and texture topologically and semantically aligned in the UV space, it allows to modify the surface appearance across bird instances. As shown in Fig. 5 we sample pairs of instances and transfer the texture between another, swapping their texture maps. Note that even for extended articulations like open wings and long tails, the resulting new 3D objects remain highly faithful and realistic.
Figure 5: MeshInver- sion enables faithful and realistic texture transfer between bird instances even with highly articu- lated shapes.
Input
Reconstruction
Texture transter
Iapul
Reconstraction Texture transter
4.3 EVALUATION ON PASCAL3D+ CAR
We also evaluate our method on the man-made rigid car category. As demonstrated Fig. 6 our method performs reasonably well across different car models and appearances. Unlike Bhattad et al. (02T) and Goel et al. (2020) that explicitly use one or multiple mesh templates provided by 3ASCAL3D+, the appealing GAN prior implicitly provides rich number of templates that
8

---

Under review as a conference paper at ICLR 2022
Mask loss
Texture loss
IoU f
FID, 4
1ID
Table 3:
Ablation study. Our proposed Chamfer texture losses LcT-p and LCT-S- and Chamfer mask loss LCM are effective to address the misalignment and quantiza- tion challenges induced by ren- dering.
LI loss
0.705
71.2
97.3
loss (MSE)
0.708
75.2
108.1
LCM
perceptual loss
0.701
49.8
52.3
contextual loss
0.699
65.2
72.5
L1 percep loss
0.707
47.1
50.8
loss
LCT-p LCT-f
0.582
51.7
IoU loss
LCT-P LCT-f
0.605
51.2
50.8
LCM
LCT-p LCT-r
0.708
38.6
38.6
makes it possible to reconstruct cars of various models. More novel-view results can be found in the Supplementary Material.
Inpac
Predicted mesh Predicted testure
Reodered images
Iapat
Predicted mesh Predicted testure
Readered images
Figure 6:
Qualitative results PASCAL3D+ Car. Our method gives reasonably good performance across different car models and appearances.
4.4 ABLATION STUDY
We evaluate the effectiveness of the proposed losses in our MeshInversion framework by comparing with conventional texture and mask losses, as reported in Tab. 3. We compare Chamfer texture loss, LCT, with pixel-to-pixel LI and L2 losses, perceptual loss (Johnson et al. 2016), and contextual loss (Mechrez et al. 2018). Both L1 and 1.2 losses tend give blurry reconstructions, as perfect image pixel-wise alignment is not guaranteed even with the not-so-accurate ground truth cameras estimated by moneture-frommmotiee but they are usually not discriminative enough to reflect the appearance details between the two images. In particular, although the contextual loss is designed address the misalignment issue in translation tasks, it only considers feature distances while ignoring their positions in the image. contrast, our proposed pixel-level and feature-level Chamfer texture losses achieve highly faithful and realistic texture reconstruction, and gives the best single-view and multi-view FII scores. We compare the Chamfer mask loss, LCM, with mask loss and IoU loss (negative intersection over union loss). LcM avoids quantization during 3D-to-2D degradation, resulting in more accurate gradients for geometric learning. Consequently, it yields the highest IoU result. More ablation studies can be found in the Supplementary Material.
Feature- based losses generally more robust to misalignment,
DISCUSSION
We have presented MeshInversion for single-view 3D object reconstruction. It exploits generative prior encapsulated pre-trained GAN and reconstructs textured shapes through GAN inversion. To address reprojection misalignment and diseretizationiinduee degradation, we propose two Chamfer-basee based losses in the 2D space. i.e., Chamfer texture loss and Chamfer mask loss. By efficiently incorporating the GAN prior, MeshInversion achieves highly realistic and faithful 3D reconstruction, and exhibits superior generalization power for challenging cases, such as in the presence occlusion or extended articulations. However, this challenging problem is far from being solved. In particular, although we can faithfully reconstruct flying birds with open wings, the wings are only represented by a few vertices due to semantic consistency across the entire category, which strictly limits the representation power in terms of geometry and texture details. Therefore, future work may explore more flexible solutions, for instance, an adaptive number of vertices can be assigned to articulated regions to accommodate richer details.
information loss due to 3D-to-2D

---

Under review as a conference paper at ICLR 2022
REFERENCES
David Bau, Hendrik Strobelt, William Peebles, Bolei Zhou, Jun- Yan Zhu, Antonio Torralba, et al. Semantic photo manipulation with generative image prior. In SIGGRAPH, 2019a.
David Bau, Jun-Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Torralba. Seeing what GAN cannot generate. ICCV, 2019b.
Anand Bhattad, Aysegul Dundar, Guilin Liu, Andrew Tao, and Bryan Catanzaro. View generalization for single image textured 3d models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6081-6090, 2021.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. ICLR, 2019. URL htttt::/ net fortmmormmm ?id-
B1x
Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3d objects with an interpolation- differentiable renderer. Advances in Neural Information Processing Systems, 32:9609-9999 9619, 2019.
Baris Gecer, Stylianos Ploumpis. Irene Kotsia, and Stefanos Zafeiriou. Ganfit: Generative adver- sarial network fitting for high fidelity 3d face reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1155-1164, 2019.
Shubham Goel, Angjoo Kanazawa, and Jitendra Malik. Shape and viewpoint without keypoints. In European Conference on Computer Vision, pp. 88-104. Springer, 2020.
Jinjin Gu, Yujun Shen, and Bolei Zhou. Image processing using multi-code GAN prior. In CVPR. 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Tao Hu, Liwei Wang, Xiaogang Xu, Shu Liu, and Jiaya Jia. Self-supervised 3d mesh reconstruction from single images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6002-6011 2021.
Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 104-711..Srrineee 2016.
Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7122-7131, 2018a.
Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category- specific mesh reconstruction from image collections. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 371-386. 2018b.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019.
Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. PointRend: Image segmenta- tion as rendering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9799-9808, 2020.
Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Hsuan Yang, and Jan Kautz. Self-supervised single-view 3d reconstruction via semantic consistency. In European Conference Computer Vision, pp. 677-693. Springer, 2020.
10

---

Under review as a conference paper at ICLR 2022
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitmick. Microsoft COCO: Common objects in context. In European conference computer vision, pp. 740-755. Springer, 2014.
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Zachary C Lipton and Subarna Tripathi. Precise recovery of latent vectors from generative adversarial networks. arXiv preprint 10X.::772.04482, 2017.
Shichen Liu, Weikai Chen, Tianye Li, and Hao Li. Soft rasterizer: Differentiable rendering for unsupervised single-view mesh reconstruction. arXiv preprint arXiv: 05567, 2019.
Fangchang Ma, Ulas Ayaz, and Sertac Karaman, Invertibility of convolutional generative networks from partial measurements. In NeurlPS, 2018.
Xudong Mao, Qing Li. Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2794 2802, 2017.
Roey Mechrez, Itamar Talmi, and Lihi Zelnik-Manor. The contextual loss for image transformation with non-aligned data. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 768-783, 2018.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4460 4470, 2019.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3504-3515, 2020.
Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. arXiv preprint arXiv:2104.1 10078, 2021.
Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, and Kui Jia. Deep mesh reconstruction from single rgb images via topology modification networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9964-9973, 2019.
Xingang Pan, Bo Dai, Ziwei Liu, Chen Change Loy, and Ping Luo. Do 2d gans know 3d shape? unsupervised 3d shape reconstruction from 2d image gans. arXiv preprint arXiv:201 000::21110000000 100044, 2020a.
Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. In ECCV, 2020b.
Dario Pavllo, Graham Spinks, Thomas Hofmann, Marie-Francine Moens, and Aurelien Lucchi. Convolutional generation of textured 3d meshes. arXiv preprint arXiv:2006.0766) 2020.
Konstantinos Rematas, Ricardo Martin-Brually and Vittorio Ferrari. ShaRF: Shape-conditioned radiance fields from a single view. arXiv preprint arXiv:2102.0886/ 2021.
Soubhik Sanyal, Timo Bolkart, Haiwen Feng. and Michael J Black. Learning to regress 3d face shape and expression from an image without 3d supervision. In Proceedings of the IERE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7763-7772, 2019.
Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1532-1540, 2021.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9049923-9999 92522, 2020.

---

Under review as a conference paper at ICLR 2022
Dong Wook Shu, Sung Woo Park, and Junseok Kwon. 3D point cloud generative adversarial network based on tree structured graph convolutions. In ICCV, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1555 2014.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Nanyang Wang. Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. Proceedings the European Conference Computer Vision (ECCV), pp. 52-67, 2018.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint 1.6::21066.1068) 2021.
Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: benchmark for 3d object detection in the wild. In IEEE winter conference on applications of computer vision, pp. 75-82.
IEFE, 2014.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. arXiv preprint arXiv:2003.09852. 2020.
Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai, and Chen Change Loy. Unsupervised 3d shape completion through gan inversion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1768-1777, 2021.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain GAN inversion for real image editing. In ECCV, 2020.
12


Reviews:
[{"review_id": "yEOX6EIcGjU", "reviewer": "Reviewer_i1Mn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n- The idea of optimizing the 3D shape via a pre-trained 3D GAN is interesting and sensible. It effectively leverages 3D priors for the task of single-view reconstruction of 3D shapes. Prior work mostly use explicit regularizations, such as as-rigid-as-possible or L1 losses to avoid irregular deformation of the shapes. In contrast, MeshInversion regularizes the deformation in a implicit but meaningful way -- the deformed shape has to lie on the 3D GAN manifold.\n- The writing is mostly clear and is easy to follow. \n\nWeakness\n- One major weakness is the evaluation protocol. As described, the proposed method takes a test RGB and silhouette image as input to optimize the 3D shape. However, the same inputs are used as the \"ground-truth\" to evaluate the optimized 3D shapes (Table 2). This does not seem to be a good evaluation metric, considering a method could overfit to the ground-truth masks and RGB image without reconstructing a meaningful 3D shape. A standard way to quantitatively evaluate the method would be using ground-truth 3D data as did in CMR. Another way is to evaluate keypoint transfer accuracy as in CSM [1].\n- Another weakness is the unfair comparison. For instance in Table 2, the baseline methods (such as CMR, UCMR) do not take test time image and object silhouette as inputs, but MeshInversion does. A fair comparison would be also test-time optimize CMR, UCMR, etc. Considering MeshInversion has access to test silhouette image but the baselines do not, Figure 1 caption may also needs to reflect this.\n- I'm also not convinced that the proposed chamfer loss is more powerful than the widely adopted differentiable rendering-based reconstruction losses. As in Sec 3.3, how would differentiable rasterization \"introduce information loss\"? Moreover, how would the proposed chamfer loss deal with occlusion, which should not be a problem for differentiable rendering? For instance, the proposed chamfer texture loss would project both the front and back surface points to the image, and compare with the observed pixels. Would it color the back surface with observed pixels and is this desired?\n\nMinor comments / questions\n- When optimizing the latent shape/texture code on a single image, are the weights of the generative network frozen? How much does the accuracy of the pre-trained network affect the test-time optimization? The extreme case is using a randomly initialized network for optimization.\n- The ablation result (Table 1) should normally go after the results (Table 2)\n\n[1] Kulkarni, Nilesh, Abhinav Gupta, and Shubham Tulsiani. \"Canonical surface mapping via geometric cycle consistency.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.", "summary_of_the_paper": "The paper proposes MeshInversion, a method for single-view 3D shape reconstruction. Given a test image with silhouette, it optimizes the 3D shape represented as the input latent code to a pre-trained 3D GAN (specifically ConvMesh, NeurIPS 20) via the proposed chamfer losses. The idea of leveraging the 3D GAN shape priors to regularize the reconstruction of unobservable surface is novel and sensible. The method is quantitatively evaluated on CUB and performs better than some recent methods on FID scores and user study.", "summary_of_the_review": "The paper proposes an interesting idea for single-view 3D shape reconstruction using 3D shape priors built-in a GAN model. However, the experiments and comparisons are not convincing enough to support the major claims. A few technical details also needs to be clarified.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6EP7BNWKOzr", "reviewer": "Reviewer_LKvE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n#### S1 - Novel Chamfer texture loss\n- The novel Chamfer texture loss seems to be quite effective in maintaining high-fidelity textures in the presence of pixel misalignments. This can potentially be useful in many cases.\n\n#### S2 - High quality results\n- This test time optimization approach seems to produce better shape and texture reconstructions than existing single pass inference methods.\n- By leveraging the prior captured during pretraining, the model is able to filter out out of distribution noise in the input image, eg. still producing a full textured bird in the presence of occlusion.\n\n### Weaknesses\n#### W1 - Core method similar to simple test-time optimization\n- I do not see how the proposed MeshInversion is different from test time optimization. I think similar test time optimization can be easily done with existing methods to obtain higher fidelity results.\n- The major claim is to leverage generative priors to achieve 3D GAN inversion. The only benefit of the generative prior that is evident here is perhaps the texture prior, ie., the fact that the model can produce full-textured bird under occlusion.\n- However, the geometric prior is much more under-utilized. The model still requires camera/object poses and masks as input during inference.\n\n#### W2 - Unfair comparisons, insufficient evaluation\n- I do not find the comparisons convincing enough. The proposed method is based on test time optimization, whereas all compared methods perform inference with only a single forward pass. I think similar test time optimization could also be easily done with other methods, for example using BGFS search.\n- 2D mask projection IoU is not indicative of the quality of 3D shape reconstruction. Especially, it is directly optimized as an objective during test time optimization.\n- It is also unclear to me whether multi-view FID is really indicative of the 3D shapes, rather than being primarily sensitive to the texture. Table 1 shows the ablation models with a much lower mask IoU (0.57 vs. 0.71) still achieves relatively high multi-view FID score (55.7 vs. 45.4). I suspect the this metric might be more sensitive to texture than to the shape.\n- Moreover, most of the reconstruction results, including the ones in the supplementary material, are visualized from the same viewpoint as the input image. Only few two-view examples are provided in Fig. 1 and Fig. 4. It is difficult to judge the quality of the 3D shapes. Extensive multi-view visualizations including untextured meshes should be provided, in order to judge the shape quality.\n- The reconstructed meshes of cars in Fig. 6 also do not look very convincing. The paper claims that the proposed method predicts the mesh by deforming a sphere, whereas other methods requires template meshes for training, which I do not agree with. The proposed method requires pretraining the ConvMesh, which is essentially akin to a template mesh. And the method also assumes camera poses available, unlike U-CMR.\n- The baseline with L1 and IoU mask losses seems to lead to poor reconstructions, indicated by the 0.57 mask IoU. This is surprising to me as it should be quite straightforward to minimize the mask loss during optimization. If the proposed Chamfer mask loss can lead to better fitting, I do not see why L1 or IoU loss cannot, with a reasonable weight and sufficient iterations.\n- Regarding the user study, it is a great effort, but why not present the three meshes to the users, rather than only three views?\n\n### Clarifications\n- Are the camera poses fixed or jointly refined during inference?\n", "summary_of_the_paper": "This paper presents a test-time optimization method that is able to produce a textured 3D mesh by fitting to a single input image, given a pretrained category-specific generative model of textured meshes and the estimated camera/object pose as input.\n\nTo enable high fidelity texture and shape reconstruction, the paper introduces a novel Chamfer texture loss that takes into account potential geometric misalignments, as well as a Chamfer mask loss that is said to encourage uniform deformation.", "summary_of_the_review": "Overall, I do not find the major claim of exploiting 3D generative priors for single image mesh prediction clearly validated in this paper. The presented method still requires 2D mask and camera pose given as input, and ends up being similar to test-time optimization with existing methods. Moreover, the evaluation and comparisons are not entirely convincing. Therefore, I recommend reject.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2paTLDn5Z_M", "reviewer": "Reviewer_owCV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n- GAN-inversion for 3D generative models is a very interesting research area to the community which is not yet well explored. \n-  The qualitative results in the paper show impressive results on single-view 3D reconstruction.\n\n### Weaknesses:\n- The experiments do not sufficiently motivate the necessity of the proposed method: \n    - Does the shape really need to be learned by inversion? ConvMesh already trains an encoder/decoder to estimate the shape from a single image – why not simply use this and optimize only for the texture? Did you try this and it did not work? Wouldn’t this greatly simplify the task? Are the proposed terms even needed in this scenario?\n  - The baselines are not optimized specifically per evaluated image while the proposed method requires test-time optimization for every image. This difference should be made more clear. Also, is there a way to modify the baselines such that the comparison becomes more equal?\n  - The quantitative comparison to the baselines shows mixed results. For the qualitative results, it is unclear how samples were selected. From this, it is hard to judge if the proposed method indeed outperforms existing methods.\n- The limitations of this work are not discussed.\n- The paper lacks structure and is not self-contained. Also, the motivation for the approach is not clear enough. See comments below. \n\n### Missing related work\n- Single-view 3D reconstruction: \nusing multi-view images: [1,2,3]\nusing radiance fields: [4]\n\n> [1] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, Yaron Lipman. Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance. NeurIPS 2020.\n>\n>[2] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, Wenping Wang. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. arXiv preprint arXiv:2106.10689\n>\n>[3] Michael Oechsle, Songyou Peng, Andreas Geiger.UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction. ICCV 2021 \n>\n>[4] Konstantinos Rematas, Ricardo Martin-Brualla, Vittorio Ferrari. ShaRF: Shape-conditioned Radiance Fields from a Single View. ICML 2021\n\n### Additional questions / comments:\n- To me, the motivation is not quite clear. If I understand correctly, the main argument is that, in contrast to existing methods that rely on 2D data, this method can leverage a 3D prior. However, this prior is, similarly to existing works, learned from only 2D supervision, no? Should the key question then not rather be if a generative model provides a better prior than the auto-encoder framework which is used in most existing methods?\n-  The main approach this work builds on [Pavllo et al.] is missing from the related work section entirely and it is not clear enough why a direct comparison to this method is not possible. I had to carefully read [Pavllo et al.] first to really understand this work, which ideally should not be the case for a self-contained paper.\n- In the results in Figure 1 of the supplementary it looks like the prior does not always match the images well, particularly the shape e.g. in b), e), f) h) in which UMR appears to work better. Please discuss these limitations in the paper.\n - Baseline for Table 1: It would be interesting to combine perceptual + image based (L1/L2) as a potentially stronger baseline. This is particularly interesting since the perceptual loss already gets quite close to the performance of the proposed loss functions.\n - Please provide more qualitative results for Pascal 3D, particularly different viewpoints in the supplementary, as the ones in the paper are fairly close to the input views.\n  - L_CM and L_CT are not formally defined in the paper, please provide explicit formulas for both loss terms as these are main contributions.\n- It is unclear to me what ground truth from SfM means? If it is inferred with SfM it is not the ground truth of the image and if ground truth is available one does not need SfM?\n- Could you explain how the camera pose estimator is trained ‘individually’ in your setting? \n- Please add qualitative results for the ablation in Table 1 to the appendix rather than only describing it in the text.\n- General: please add references to the appendix where appropriate, e.g. for the ablation on the discriminator in image space in sec 3.1.\n- Sec 3.3 S_f is not defined\n\n### Misc:\n- Introduction, paragraph 4: “Chamfar Texture Loss”\n- Related Work: ‘’It aims to recover”: informal\n- Sec 3.4 “is shown”\n- Sec 3.1 “ in the form of a pose-invariant”", "summary_of_the_paper": "The work proposes inversion of a generative model for single-view 3D reconstruction. \nInterestingly, the inversion is applied to a generative model that generates a 3D representation of an object rather than only a (2D) image.\nTo improve the inversion process from 2D to 3D, two Chamfer-distance inspired losses are proposed for texture and shape, respectively.", "summary_of_the_review": "The problem setting is interesting and the qualitative results in the paper are impressive. However, the experiments appear not sound enough to support the necessity of the approach and important baselines seem to be missing. Hence I lean towards rejecting the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-CdgRN7wO70", "reviewer": "Reviewer_pz7V", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper proposes a very interesting alternative to the popular CMR formulation, which achieves good results, that look very realistic, both in terms of the texture and in terms of the shape of the objects (particularly birds). Quantitative results are also strong. Results on perceptual studies are also included.\n\n- The setting of GAN inversion, to the best of my knowledge, is novel for the CMR setting. I find the idea interesting and I think it is well executed here.\n\n- The code will be released, which further helps comparisons with future works.\n\n- Most CMR methods are regression-based at test time, without any test-time optimization. It would be helpful to clarify this again at the experimental section, because this is a significant shift in the methodology. For example the mask is used as input at test time, which happens for SMR, but not CMR. I think the authors should highlight these differences more clearly.\n\n- I would like some clarifications on the setting of the ablation study of Table 1. For the rows with Texture Losses, do the authors use their Chamfer Mask Loss (and vice versa, for the rows corresponding to the Mask Losses, do they also use their Texture Mask Loss)? Similarly, the last line is using the \"predicted camera poses\" or not?\n\n- I would encourage the authors to provide more results from novel views. Currently, most results are from the side view of the bird. It would be useful to see more visualizations from other views as well (top/frontal).", "summary_of_the_paper": "This paper addresses the problem of learning single view reconstruction for a specific category using an image collection. This is related to the CMR framework by Kanazawa et al, ECCV 2018. The main contribution of this paper is using a GAN, and at test time searching for the latent code that best resembles the object in the input image. This is done with test-time optimization, and two losses, a Chamfer Texture Loss and a Chamfer Mask Loss, are proposed for this.", "summary_of_the_review": "Overall, I am positive about the work and I find the idea novel in the CMR setting. I am rating this paper with a 6, since I would like the authors to clarify some experimental details, but all in all, I expect to keep my positive score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received 4 diverging ratings: 3, 5, 5, 6. On the positive side, reviewers appreciated the novelty of the approach and strong empirical performance. At the same time, all negatively-inclined reviewers mentioned unfair comparisons with baselines (which was partially addressed in the rebuttal), flaws in the evaluation protocols and ablations not fully supporting claims made in the paper. After discussions with the authors most reviewers decided to stick with their original ratings.
AC agrees that the remaining open questions around empirical validation will need to be answered more clearly before the paper can be accepted. The final recommendation is reject.
Decision: Reject