Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of learning single view reconstruction for a specific category using an image collection. This is related to the CMR framework by Kanazawa et al, ECCV 2018. The main contribution of this paper is using a GAN, and at test time searching for the latent code that best resembles the object in the input image. This is done with test-time optimization, and two losses, a Chamfer Texture Loss and a Chamfer Mask Loss, are proposed for this.
main_review: - The paper proposes a very interesting alternative to the popular CMR formulation, which achieves good results, that look very realistic, both in terms of the texture and in terms of the shape of the objects (particularly birds). Quantitative results are also strong. Results on perceptual studies are also included.

- The setting of GAN inversion, to the best of my knowledge, is novel for the CMR setting. I find the idea interesting and I think it is well executed here.

- The code will be released, which further helps comparisons with future works.

- Most CMR methods are regression-based at test time, without any test-time optimization. It would be helpful to clarify this again at the experimental section, because this is a significant shift in the methodology. For example the mask is used as input at test time, which happens for SMR, but not CMR. I think the authors should highlight these differences more clearly.

- I would like some clarifications on the setting of the ablation study of Table 1. For the rows with Texture Losses, do the authors use their Chamfer Mask Loss (and vice versa, for the rows corresponding to the Mask Losses, do they also use their Texture Mask Loss)? Similarly, the last line is using the "predicted camera poses" or not?

- I would encourage the authors to provide more results from novel views. Currently, most results are from the side view of the bird. It would be useful to see more visualizations from other views as well (top/frontal).
summary_of_the_review: Overall, I am positive about the work and I find the idea novel in the CMR setting. I am rating this paper with a 6, since I would like the authors to clarify some experimental details, but all in all, I expect to keep my positive score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The work proposes inversion of a generative model for single-view 3D reconstruction. 
Interestingly, the inversion is applied to a generative model that generates a 3D representation of an object rather than only a (2D) image.
To improve the inversion process from 2D to 3D, two Chamfer-distance inspired losses are proposed for texture and shape, respectively.
main_review: ### Strengths:
- GAN-inversion for 3D generative models is a very interesting research area to the community which is not yet well explored. 
-  The qualitative results in the paper show impressive results on single-view 3D reconstruction.

### Weaknesses:
- The experiments do not sufficiently motivate the necessity of the proposed method: 
    - Does the shape really need to be learned by inversion? ConvMesh already trains an encoder/decoder to estimate the shape from a single image – why not simply use this and optimize only for the texture? Did you try this and it did not work? Wouldn’t this greatly simplify the task? Are the proposed terms even needed in this scenario?
  - The baselines are not optimized specifically per evaluated image while the proposed method requires test-time optimization for every image. This difference should be made more clear. Also, is there a way to modify the baselines such that the comparison becomes more equal?
  - The quantitative comparison to the baselines shows mixed results. For the qualitative results, it is unclear how samples were selected. From this, it is hard to judge if the proposed method indeed outperforms existing methods.
- The limitations of this work are not discussed.
- The paper lacks structure and is not self-contained. Also, the motivation for the approach is not clear enough. See comments below. 

### Missing related work
- Single-view 3D reconstruction: 
using multi-view images: [1,2,3]
using radiance fields: [4]

> [1] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, Yaron Lipman. Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance. NeurIPS 2020.
>
>[2] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, Wenping Wang. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. arXiv preprint arXiv:2106.10689
>
>[3] Michael Oechsle, Songyou Peng, Andreas Geiger.UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction. ICCV 2021 
>
>[4] Konstantinos Rematas, Ricardo Martin-Brualla, Vittorio Ferrari. ShaRF: Shape-conditioned Radiance Fields from a Single View. ICML 2021

### Additional questions / comments:
- To me, the motivation is not quite clear. If I understand correctly, the main argument is that, in contrast to existing methods that rely on 2D data, this method can leverage a 3D prior. However, this prior is, similarly to existing works, learned from only 2D supervision, no? Should the key question then not rather be if a generative model provides a better prior than the auto-encoder framework which is used in most existing methods?
-  The main approach this work builds on [Pavllo et al.] is missing from the related work section entirely and it is not clear enough why a direct comparison to this method is not possible. I had to carefully read [Pavllo et al.] first to really understand this work, which ideally should not be the case for a self-contained paper.
- In the results in Figure 1 of the supplementary it looks like the prior does not always match the images well, particularly the shape e.g. in b), e), f) h) in which UMR appears to work better. Please discuss these limitations in the paper.
 - Baseline for Table 1: It would be interesting to combine perceptual + image based (L1/L2) as a potentially stronger baseline. This is particularly interesting since the perceptual loss already gets quite close to the performance of the proposed loss functions.
 - Please provide more qualitative results for Pascal 3D, particularly different viewpoints in the supplementary, as the ones in the paper are fairly close to the input views.
  - L_CM and L_CT are not formally defined in the paper, please provide explicit formulas for both loss terms as these are main contributions.
- It is unclear to me what ground truth from SfM means? If it is inferred with SfM it is not the ground truth of the image and if ground truth is available one does not need SfM?
- Could you explain how the camera pose estimator is trained ‘individually’ in your setting? 
- Please add qualitative results for the ablation in Table 1 to the appendix rather than only describing it in the text.
- General: please add references to the appendix where appropriate, e.g. for the ablation on the discriminator in image space in sec 3.1.
- Sec 3.3 S_f is not defined

### Misc:
- Introduction, paragraph 4: “Chamfar Texture Loss”
- Related Work: ‘’It aims to recover”: informal
- Sec 3.4 “is shown”
- Sec 3.1 “ in the form of a pose-invariant”
summary_of_the_review: The problem setting is interesting and the qualitative results in the paper are impressive. However, the experiments appear not sound enough to support the necessity of the approach and important baselines seem to be missing. Hence I lean towards rejecting the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a test-time optimization method that is able to produce a textured 3D mesh by fitting to a single input image, given a pretrained category-specific generative model of textured meshes and the estimated camera/object pose as input.

To enable high fidelity texture and shape reconstruction, the paper introduces a novel Chamfer texture loss that takes into account potential geometric misalignments, as well as a Chamfer mask loss that is said to encourage uniform deformation.
main_review: ### Strengths
#### S1 - Novel Chamfer texture loss
- The novel Chamfer texture loss seems to be quite effective in maintaining high-fidelity textures in the presence of pixel misalignments. This can potentially be useful in many cases.

#### S2 - High quality results
- This test time optimization approach seems to produce better shape and texture reconstructions than existing single pass inference methods.
- By leveraging the prior captured during pretraining, the model is able to filter out out of distribution noise in the input image, eg. still producing a full textured bird in the presence of occlusion.

### Weaknesses
#### W1 - Core method similar to simple test-time optimization
- I do not see how the proposed MeshInversion is different from test time optimization. I think similar test time optimization can be easily done with existing methods to obtain higher fidelity results.
- The major claim is to leverage generative priors to achieve 3D GAN inversion. The only benefit of the generative prior that is evident here is perhaps the texture prior, ie., the fact that the model can produce full-textured bird under occlusion.
- However, the geometric prior is much more under-utilized. The model still requires camera/object poses and masks as input during inference.

#### W2 - Unfair comparisons, insufficient evaluation
- I do not find the comparisons convincing enough. The proposed method is based on test time optimization, whereas all compared methods perform inference with only a single forward pass. I think similar test time optimization could also be easily done with other methods, for example using BGFS search.
- 2D mask projection IoU is not indicative of the quality of 3D shape reconstruction. Especially, it is directly optimized as an objective during test time optimization.
- It is also unclear to me whether multi-view FID is really indicative of the 3D shapes, rather than being primarily sensitive to the texture. Table 1 shows the ablation models with a much lower mask IoU (0.57 vs. 0.71) still achieves relatively high multi-view FID score (55.7 vs. 45.4). I suspect the this metric might be more sensitive to texture than to the shape.
- Moreover, most of the reconstruction results, including the ones in the supplementary material, are visualized from the same viewpoint as the input image. Only few two-view examples are provided in Fig. 1 and Fig. 4. It is difficult to judge the quality of the 3D shapes. Extensive multi-view visualizations including untextured meshes should be provided, in order to judge the shape quality.
- The reconstructed meshes of cars in Fig. 6 also do not look very convincing. The paper claims that the proposed method predicts the mesh by deforming a sphere, whereas other methods requires template meshes for training, which I do not agree with. The proposed method requires pretraining the ConvMesh, which is essentially akin to a template mesh. And the method also assumes camera poses available, unlike U-CMR.
- The baseline with L1 and IoU mask losses seems to lead to poor reconstructions, indicated by the 0.57 mask IoU. This is surprising to me as it should be quite straightforward to minimize the mask loss during optimization. If the proposed Chamfer mask loss can lead to better fitting, I do not see why L1 or IoU loss cannot, with a reasonable weight and sufficient iterations.
- Regarding the user study, it is a great effort, but why not present the three meshes to the users, rather than only three views?

### Clarifications
- Are the camera poses fixed or jointly refined during inference?

summary_of_the_review: Overall, I do not find the major claim of exploiting 3D generative priors for single image mesh prediction clearly validated in this paper. The presented method still requires 2D mask and camera pose given as input, and ends up being similar to test-time optimization with existing methods. Moreover, the evaluation and comparisons are not entirely convincing. Therefore, I recommend reject.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes MeshInversion, a method for single-view 3D shape reconstruction. Given a test image with silhouette, it optimizes the 3D shape represented as the input latent code to a pre-trained 3D GAN (specifically ConvMesh, NeurIPS 20) via the proposed chamfer losses. The idea of leveraging the 3D GAN shape priors to regularize the reconstruction of unobservable surface is novel and sensible. The method is quantitatively evaluated on CUB and performs better than some recent methods on FID scores and user study.
main_review: Strength
- The idea of optimizing the 3D shape via a pre-trained 3D GAN is interesting and sensible. It effectively leverages 3D priors for the task of single-view reconstruction of 3D shapes. Prior work mostly use explicit regularizations, such as as-rigid-as-possible or L1 losses to avoid irregular deformation of the shapes. In contrast, MeshInversion regularizes the deformation in a implicit but meaningful way -- the deformed shape has to lie on the 3D GAN manifold.
- The writing is mostly clear and is easy to follow. 

Weakness
- One major weakness is the evaluation protocol. As described, the proposed method takes a test RGB and silhouette image as input to optimize the 3D shape. However, the same inputs are used as the "ground-truth" to evaluate the optimized 3D shapes (Table 2). This does not seem to be a good evaluation metric, considering a method could overfit to the ground-truth masks and RGB image without reconstructing a meaningful 3D shape. A standard way to quantitatively evaluate the method would be using ground-truth 3D data as did in CMR. Another way is to evaluate keypoint transfer accuracy as in CSM [1].
- Another weakness is the unfair comparison. For instance in Table 2, the baseline methods (such as CMR, UCMR) do not take test time image and object silhouette as inputs, but MeshInversion does. A fair comparison would be also test-time optimize CMR, UCMR, etc. Considering MeshInversion has access to test silhouette image but the baselines do not, Figure 1 caption may also needs to reflect this.
- I'm also not convinced that the proposed chamfer loss is more powerful than the widely adopted differentiable rendering-based reconstruction losses. As in Sec 3.3, how would differentiable rasterization "introduce information loss"? Moreover, how would the proposed chamfer loss deal with occlusion, which should not be a problem for differentiable rendering? For instance, the proposed chamfer texture loss would project both the front and back surface points to the image, and compare with the observed pixels. Would it color the back surface with observed pixels and is this desired?

Minor comments / questions
- When optimizing the latent shape/texture code on a single image, are the weights of the generative network frozen? How much does the accuracy of the pre-trained network affect the test-time optimization? The extreme case is using a randomly initialized network for optimization.
- The ablation result (Table 1) should normally go after the results (Table 2)

[1] Kulkarni, Nilesh, Abhinav Gupta, and Shubham Tulsiani. "Canonical surface mapping via geometric cycle consistency." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
summary_of_the_review: The paper proposes an interesting idea for single-view 3D shape reconstruction using 3D shape priors built-in a GAN model. However, the experiments and comparisons are not convincing enough to support the major claims. A few technical details also needs to be clarified.
review_title: null

Meta-review: This submission received 4 diverging ratings: 3, 5, 5, 6. On the positive side, reviewers appreciated the novelty of the approach and strong empirical performance. At the same time, all negatively-inclined reviewers mentioned unfair comparisons with baselines (which was partially addressed in the rebuttal), flaws in the evaluation protocols and ablations not fully supporting claims made in the paper. After discussions with the authors most reviewers decided to stick with their original ratings.
AC agrees that the remaining open questions around empirical validation will need to be answered more clearly before the paper can be accepted. The final recommendation is reject.
Decision: Reject