Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduce a 3D-aware generative model that can control 3D shape, camera pose, object appearance and background independently. The author connect 3D morphable modelling with deferred neural rendering by performing an implicit surface rendering of TOCS, and construct a purely geometric, deformation-equivariant 2D signal that reflects the compounded geometric effects of non-rigid shape, pose, and perspective projection. The experiments of 4 datasets shows the good disentanglement capability of the proposed approach. 

strength_and_weaknesses: Strengths:
1. To disentangle the key elements such as camera pose, apperance et al. is a important research topic, and the proposed approach is interesting and achieves good performance on this topic. Some illustrated results are impressive. 
2. Most of the quantitative results outperform existing works, demonstrating the superiority of the proposed approach. The experiments are conducted on four datasets, which are sufficient. 

Weaknesses and questions:
1. Some works like [1] provide more than the four choices in this paper for manipulation, like illumination, expression et al, while this work only covers the four(camera pose, back gournd, appearence, shape). In other word, I think the shape/appearance manipulation cannot support fine-grained manipulation, and this paper lacks detailed manipulation capability.  
[1] A. Tewari et al, StyleRig: Rigging StyleGAN for 3D Control over Portrait Images, CVPR 2020
2. The quality of the reconstructed 3D shapes seems coarse compared with EG3D [2], i.e., Figure 2 in supplementary file.
[2] ER Chan et al. EG3D: Efficient Geometry-aware 3D Generative Adversarial Networks, CVPR 2022
3. (1) Is TOCS map provide all necessary information for stage 2 training? 
(2) How about train the whole framework end-to-end? 
(3) How does the GAN loss in stage 1 helps stage 2 trainin? It seems only TOCS goes into stage 2 but the generated image in stage 1 not goes to stage 2. 
clarity,_quality,_novelty_and_reproducibility: Quality: I think the paper achieves good quality and the proposed approach is interesting. 
Clarity: I think there are some motivations that should be further introduced, e.g., why we need two stage training, why deformable field is really necessary? 
Novelty: Good. I think the deformable field, the way to disentangle the four elements of image are interesting. 
Reproducibility: Will you release code after publication? I think it is hard to implement the approach due to the complex pipeline and some missing details. 
summary_of_the_review: Overall, I think it is a good paper. Though I am not experted in 3D-aware GANs, some results in the paper impressed me and I tend to give positve rating to this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a 3D-aware generative model to disentangle 3D shapes, camera poses, object appearance, and background appearance when synthesizing high-quality images, which could realize non-rigid shape variation in an object category exclusively from 2D image supervision.  To achieve a clear disentanglement, it proposes Template Object Coordinates (TOCS) to provide a powerful, deformation-equivariant descriptor of 3D shapes.  It provides impressive qualitative results with disentangled controls, and competitive quantitative results on the evaluated benchmark datasets.
strength_and_weaknesses: Strength:
1. It is an interesting paper that could generate 3D-aware and high-quality images with disentangled controls over 3D shape, camera pose, foreground appearance, and background appearance.
2. They utilize 3D morphable models for synthesizing high-quality images by introducing a canonical coordinate system, and therefore they could deform the 3D shape by morphing the 3D canonical template.
3. They show impressive synthetic images with disentangled controls.

Weakness:
The current manuscript seems to be a technical report, and the proposed method seems to be a comprehensive combination of existing methods.  I think it is an interesting work, but I am not sure if it is good enough for an ICLR publication.
clarity,_quality,_novelty_and_reproducibility: 1. The writing is generally clear. But I would suggest highlighting the research gaps as well as the methodological contributions. The current manuscript reads like a technical report, and the current methods seem to be a comprehensive combination of existing methods.

2. The contribution list could be reorganized.  The current version is not clear and kind of wordy.  Introducing TOCS could be summarized into one contribution, and so is the disentangled control.

3. The reference formats used in the manuscript are not consistent.
summary_of_the_review: In general, I think it is an interesting work, showing impressive results.  Although I am a little worried about its contributions and novelties, I tend to support its acceptance at the current stage.  I would reconsider my rating when the other reviews are available.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a novel 3D-aware generative image model, StyleMorph, which can disentangle 3D shape, camera pose, appearance and background for high-quality image synthesis.  By bridging 3D morphable models with GAN synthesis and a canonical coordinate system, dense correspondences among generated objects can be provided. Experiments show disentangled control over pose, shape, object appearance, and background appearance for high-quality image synthesis. 
strength_and_weaknesses: Pros:
1. It is a good idea to bride the 3D morphable models with GAN synthesis, enabling dense correspondence between generated objects.
2. The method is clearly explained and the text is well-written. 

 I don't have major concerns about this paper. In general, I like the idea of 3D control of 2D generative models. A few minor comments are listed below.
1) Can the proposed model be applied to generic object image generation, such as cars, and buildings? Would the method work for objects with large intra-class deformations? 
2) More qualitative and quantitative evidence of the 3D disentanglement is needed.
clarity,_quality,_novelty_and_reproducibility: The paper is well-motivated and presented. A good attempt is made in this paper to bridge the gap between 2D images and 3D physical reality. 
summary_of_the_review: Overall, I think the idea is interesting and the proposed "template object coordinates" is a good contribution in image synthesis.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present a method to synthesize photo-realistic images of objects from a given category while the approach allows for disentangled representation in terms of the camera pose, crude object shape, and foreground/background appearance. The approach introduces the concept of template object coordinates (TOCS) which both sidestep the need to perform 3D shape UV unwrapping and also allow for representing deformed instances of the shape in the common canonical form. The authors show competitive results while allowing for more semantic control over the synthesized images.
strength_and_weaknesses: # Strengths
While the approach builds on many existing components, it combines them in a creative way to achieve a disentangled representation of the camera/shape/appearance while being able to synthesize compelling images. The method is unsupervised.
The paper is very well structured and written in a comprehensive way where it is easy to follow the core ideas.
The experiments are compelling and the authors justify their design choices through a thorough ablation study.

# Weaknesses
## Related Work
While the Related Work section is comprehensive, I would suggest that the authors still consider reviewing and citing the following two publications, which use very similar concepts as the authors of this paper. Specifically, Nerfies [1] makes use of the concept of deforming the camera rays to support a deforming scene and NPMs [2] employ a pose network which also uses a deformation field to express the correspondence between a canonical SDF space and its deformed (posed) version.
Probably should cite Nerfies and NPMs as both use the similar idea of warping the camera rays and learning the deformation field of a template.

## Methodology
- Fig. 4: It would help the reader if the variables from the text of section 2.2 (e.g. T_{FG}, I_{BG} etc.) were indicated as the inputs/outputs of the individual blocks in the image.
- Section 2.1: It would help the reader to provide the intuition behind how w() of Eq. 2 is computed and what does it represent. Similarly, the concept of TOCS could be very briefly introduced directly in the Methodology on top of linking to the paper [Wang et al. 2019] which introduced the related NOCS.
- Section 2.3: Similarly to the comment above, it would help the reader if the authors explicitly noted the mathematical formulas for the used loss function.
- Fig. 3: The figure indicates that the Shape and Appearance Mapping modules share weights but this is never described in the text. Could the authors elaborate?

## Typos:
- pg. 3: "encapsulates all geometr information." -> "encapsulates all geometry information."
- pg. 4: "In a first/second stage" -> "In the first/second stage"

## Questions for the authors
Since the 2D TOCS map is immediately passed through a spatial encoder (which produces a 64D feature vectors), one could imagine that the network is not really incentivized to produce a meaningful 3D shape (encoded by the 2D TOCS). The Figure 7 indicates that the TOCS are reasonable shapes given the corresponding RGB images, but it is also clear that there is quite some high-frequency noise on the surface geometry. Could the authors comment on why this is happening? Alternatively, have the authors experimented with regularizing the shape which the TOCS represent (to me closer to a more plausible 3D shape) and what were the observations?

- [1] K. Park. et al. Nerfies: Deformable Neural Radiance Fields. ICCV'21
- [2] P. Palafox. et al. NPMs: Neural Parametric Models for 3D Deformable Shapes. ICCV'21

clarity,_quality,_novelty_and_reproducibility: The paper is written in a clear way and seems to provide sufficient details for reproducibility. The authors promised to release their code. The main idea smartly combines multiple existing components and introduces a notion of the TOCS used to condition the DNR.
summary_of_the_review: Thanks to the creativity of the method, compelling results and good quality of the presentation I am leaning towards acceptance. There are a couple of minor remarks which I would like the authors to comment on (see the weaknesses section).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces StyleMorph, a high-resolution 3D-aware generative model with morphable geometry and disentangled appearance. 
The network is composed of two components, where a Morphable Renderer is trained to yield the TOCS map, and a Deferred Neural Renderer learns to translate the TOCS map to a high-resolution image.
strength_and_weaknesses: Strength:
1. The image quality in the paper looks good.
2. The supplementary materials are detailed.

Weaknesses:
1. Morphable Renderer lacks novelty. The main idea of the Morphable Renderer (disentangling the geometry and appearance + warping the learned template space to modify the geometry) is too similar to the Disentangled3d[1]. Actually, in my opinion, they are almost the same.
2. Deferred Neural Render lacks novelty. The proposed DNR takes the TOCS (a sort of geometry representation) as input and yields a corresponding high-resolution image, which is similar to StyleNeRF[2], SofGAN[3], etc. and has little novelty from my perspective. 
3. The experiments are not sufficient.
    - Efficiency of TOCS. Although the TOCS outperforms NOCS, there lacks a comparison with some naive methods, e.g. directly feeding the Deferred Neural Render with the low-resolution image/depth outputs of Morphable Renderer, or using an on-the-shelf super-resolution model, e.g. GFPGAN[4], to enhance the image quality.
    - No expression control experiments.
4. The writing is exaggerated and not clear:
    - "we disentangle shape (e.g. gender, expression, hair style), camera pose, object appearance, and background". No gender or expression disentanglement was found. And the background disentanglement here is trivial, where they use a pretrained 2D generator to synthesise the plain background. It is strange to see the background keeping static while the foreground is rotated, as shown in the YouTube video.
    - "yielding a similar level of control to that enjoyed by 3D morphable models (3DMM)". I think this paper cannot control facial expressions as in the 3DMM. And the warping field is hard to edit.
    - "show that we can improve the FID of the most competitive methods". I did not find experiment results to show this general improvement.
    - "changing 3D expression, pose etc". Did not find experiment results.
    - "but their hybrid nature makes it harder to have a clear separation of geometry and appearance". why? The hybrid architecture does not conflict with the disentangled design.
5. Appearance-geometry misalignment. In Fig.1 in the main paper, the geometry of column "foreground" and column "background" is the same, which suggests the corresponding images have the same shape. However, when looking at the images, you will notice a change of expression when adjusting the appearance, indicating the appearance and geometry are not aligned well semantically.

[1] Ayush Tewari, Xingang Pan, Ohad Fried, Maneesh Agrawala, Christian Theobalt, et al. Disentangled3d: Learning a 3d generative model with disentangled geometry and appearance from monocular images. arXiv preprint arXiv:2203.15926, 2022. \
[2] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint arXiv:2110.08985, 2021. \
[3] Anpei Chen, Ruiyang Liu, Ling Xie, Zhang Chen, Hao Su, and Jingyi Yu. Sofgan: A portrait image
generator with dynamic styling. ACM Transactions on Graphics (TOG), 41(1):1–26, 2022. \
[4] Wang, Xintao, et al. "Towards real-world blind face restoration with generative facial prior." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
clarity,_quality,_novelty_and_reproducibility:  - Clarity: 5 out of 10. I think the clarity of the method part is okay, which tells what it is doing. But the introduction is exaggerated.
 - Quality: 6 out of 10. As I have mentioned in the Strength And Weaknesses, the image quality is good, while the experiments are not sufficient. And the writing makes me upset.
 - Novelty: 1 out of 10. I see some similarities with previous works. Please let me know if I misunderstood.
 - Reproducibility: 6 out of 10. No code was submitted, but they provide details including hyper-parameters and loss functions. So I think it could be reproduced given enough time.
summary_of_the_review: In general, I think this paper has good qualitative results, but lacks novelty, and the writing is not clear and objective enough.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This submission introduces a 3D generative model with a disentangled representation of shape, pose, and texture for image synthesis. Reviewers in general like the formulation and the good results; however, there are concerns about the novelty and the inappropriate overclaims in the paper. There is a lot of discussion about the paper post-rebuttal, with four reviewers being positive and one being negative. 

The AC has been following the discussion and also invited two external evaluators who work exactly on this topic to provide additional feedback. The consensus is that this paper has value in terms of its TOCS formulation and its good results. However, current overclaims are inappropriate, related works should be better explained and novelty claims should be revised, and additional ablation studies especially on the benefits of TOCS would certainly be helpful.

The AC decides to recommend acceptance due to the potential value of the paper, but urges the authors to revise the paper accordingly.  In particular, reviewer cRkf provided the following candid feedback during the discussion, which the AC, reviewers, and additional evaluators all agree on.

"The main concerns are the novelty.

* Morphable Renderer Similar to Disentangle3D[1]. Other reviewers mainly praise the disentanglement of this paper. However, the Morphable Renderer, which accounts for the disentanglement is technically similar to the Disentangle3D[1], a CVPR'22 paper. I point this out as the first point of weakness and the authors did not tell the difference between the Morphable Renderer and Disentangle3D[1]. Instead, they strengthen that their image quality is better, which is misleading because the enhancement of image quality comes from the Deferred Neural Renderer, rather than the Morphable Renderer. In my opinion, in order to demonstrate their novelty on Morphable Renderer, they ought to:

1. directly tell the difference between the Morphable Renderer and Disentangle3D[1],
2. report the FID of the first stage output of the Morphable Renderer to show the superiority of the Morphable Renderer in terms of image quality.
* Deferred Neural Renderer Similar to StyleNeRF[2], EG3D[3], etc. The Deferred Neural Renderer is quite common for the NeRF-based generative model to enhance image quality. Although there might be some differences in implementation, I think the main idea is similar. And the use of TOCS as the super-resolution input is too small a point to be the main contribution of an ICLR paper.

In rebuttal, the authors go around that Disentangle3D[1] has poor image quality while other high-res methods are not disentangled, implying this work is actually a combination of Disentangle3D[1] and a super-resolution module. "

justification_for_why_not_higher_score: There are significant presentation issues in particular regarding the novelty claims
justification_for_why_not_lower_score: The results are strong and the TOCS formulation may have some value
Decision: Accept: poster