Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper targets at stylizing a mesh with a text prompt. It represents appearance of a mesh as spatially varying BRDF, normal and lighting, and matches the CLIP feature of rendered image and given text prompt by learning the parameters of these components. For evaluation, the paper compares with the state-of-the-art text prompt stylization method Text2Mesh and achieves better performance.
strengths_and_weaknesses: **Strengths.**
- This paper is well written and easy to follow.
- The idea of representing the appearance of a given mesh as SVBRDF, normal and lighting is interesting, although its has certain limitations.

**Weaknesses.**
- Modeling appearance as a lighting model certainly performs better in complex realistic scenes. However, it also limits the capacity of the model without geometric displacements modeling. For instance, in Fig.3 in the Text2Mesh paper, the surface of the given mesh can be changed to match the text prompt, while the proposed method cannot do so. This limitation is not discussed in the paper.
questions: **Method.**
I am actually surprised this model can predict correct lighting parameters given only CLIP latent space as supervision. Have the authors observed any ambiguity in disentangling lighting and reflectance? If not, what is the key to make the model learn correct disentanglement?

**Implementation details.**
- How to find reasonable anchor view in line 145?
- How to find the first intersection point and intersection face in line 150?

**Novelty.**
Although not using text prompt as supervision, a similar lighting model has been explored in the NeRD [4] paper, while the text prompt stylization has been used by Text2Mesh. Thus although the idea is interesting, the novelty and contribution of the proposed method is limited, in my opinion.
limitations: The authors have discussed the limitations of the chosen lighting model. As stated in the weakness, I encourage the authors to include 
 the limitation of not modeling geometric displacement.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a 3D mesh stylization method that performs both appearance and geometric styles by optimizing with a differentiable rendering pipeline. Compared with existing methods that simply optimize per-vertex color and displacement, it adopts a more physics-aware rendering model that explicitly models geometry (normal map), BRDF (diffuse, roughness, specular), and lighting (SG), which is claimed to generate more realistic results (better shading appearance and details) and allows for editing such as relighting. Results are compared with Text2Mesh.
strengths_and_weaknesses: ++ Indeed very promising idea to do physics-aware modeling of mesh stylization.

++ Interesting visual results are presented, which look good.

++ The proposed rendering formulation is technically sound and practically reasonable.

++ Very good paper-writing, except for some redundancy that could be made more concise.

-- All results are just final renderings, lacking results to show the individual components of normal maps, BRDFs, and lightings, which are critical to show the quality of the system and its advantage over Text2Mesh.

-- No results on manipulating the rendering, such as relighting (the relighting results on the shoe are less interesting for stylization) and material editing, which are not achievable by Text2Mesh.

-- Approximating geometry with normal maps turns the method into a pure appearance stylization method. There are no actual geometry changes but just a way to model geometry variation for shading. But Text2Mesh does achieve joint geometry and appearance stylization, which seems better than this method although the geometric details have limitations.

-- Limited contributions, the rendering pipeline, and CLIP-guided stylization are mostly based on existing works.
questions: I have put my major concerns in the weaknesses section. Overall, I think it is an interesting work that combines neural rendering with mesh stylization that could produce meaningful assets that are more compatible with current rendering pipelines and allow for more wide editing applications. However, the presented results, unfortunately, do not fully the potential of the proposed method -- there are no results showing the quality of individual components and the possibility to edit one of them for editable stylization, without these I am not convinced that this method has achieved significant improvement over Text2Mesh except some better shading details. Also considering that this method is not able to produce real geometric details, and the visual results on single stylization images are not clearly better than Text2Mesh, I cannot say this method advances the state-of-the-art. Finally, given the limited technical contributions, I am leaning a bit negative unless more convincing evidence could be presented.
limitations: There are discussions on limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on the task of 3D mesh stylization according to the text prompt. The key idea is to jointly learn three disentangled components, ie, the spatially varying bidirectional reflectance distribution function, the local geometric variation, and the lighting condition, under the supervision of CLIP Loss with a spherical Gaussians-based differentiable renderer. Experiments show that the proposed method achieves better visualization quality and 3D geometry consistency on stylized meshes compared to the state-of-the-art. Besides, this method demonstrates robustness when stylizing low-quality meshes.
strengths_and_weaknesses: Strengths:
1. This is an interesting idea that may inspire future research in related fields. It employs a professional shading model to help learn more photorealistic appearance and geometry, which is intuitively reasonable and practically effective.
2. The paper is well-organized and easy to follow.
3. The experiments look convincing since it supports the major contributions claimed in the introduction that the realism of rendered images can be improved by the explicit modeling of light and materials. Moreover, the ablation study also demonstrates that each disentangled component plays an important role in the improvement of the rendering quality.
4. The authors also provide some reasonable limitations of the current method, which may help researchers to build new frameworks based on current results.

Weakness:
1. This paper proposes to learn materials, normals, and light separately. How can **a good disentanglement** of light and physical reflectance be guaranteed? In the whole pipeline, only a CLIP loss is used. It seems like the network here has no motivation to learn a good disentanglement. In Fig4, the relighting results are so weird since the materials are also modified here. How about changing the light only?
2. The quality of 3D stylization results may not be impressive enough in Fig.2. It is not easy to evaluate the performance of the proposed method from the target text prompt. Such as “A shoe made of brick”, I know the proposed method produces more details, but it seems that the result of test2mesh is more natural. A similar situation occurs in "A vase made of colorful crochet". It is hard to judge between this method and Text2Mesh. Of course, this method can preserve better 3D geometry. However, is geometry preservation necessary in 3D stylization? By the way, there are a lot of clear artifacts in the case of "A candle wearing jeans". I think more convincing visualization results can be provided.
3. The quantitative result in Table 1 is unclear. As this work splits the style into geometry and appearance and claims the improvement in both, user studies should be conducted separately in terms of both aspects.
questions: 1. I wonder why the authors adopt “randomly cropped images” rather than “multi-view images” that text2mesh adopts in the phase of data augmentation. Is it because that normal map helps the model focus on the geometry and cropped images guide the model to focus on the detail? If text2mesh adopts cropped images in data augmentation, will it produce more details? An ablation study should be conducted here.
2. What is the purpose of adding a random background to the rendered image in data augmentation? Is it necessary? I do not see the ablation study about the background in experiments.
3. Is it easy to explicitly control the lighting intensity and direction? How should we determine the light distribution?
limitations: CLIP model may produce semantic ambiguity. It seems that the target text has to contain the category of input mesh for content preservation, which may limit the application of this method in real life.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper describes a method for generating a reflectance map (and lighting environment) for a given 3D shape, based on a text prompt, using a CLIP-guided loss.
strengths_and_weaknesses: The problem being tackled is novel, and several of the results are plausibly-good. The impact of the work is incremental, but the general direction of text-guided geometry/texture generation is a significant one. 

One big gap in the paper for me is a clear explanation for the motivation for using the SVBRDF/env. map rather than just fitting a texture map. It's clear that the results look better than just fitting a texture map (as shown in the ablations), so this aspect is clear. But, by fitting materials+lighting, the paper gives the possible impression that one gets a good materials+lighting in general, not just for a single view. Is this true? It hasn't been demonstrated or tested. Specifically: can the model be relighted and the viewpoint moved with sensible results? For example, do shiny materials have highlights that move in sensible ways, and diffuse materials do not? etc.

I think the description of the method is unnecessarily mathematical and opaque, with many different terms. However, I don't have any suggestions for improving it and it might be that this is necessary level of detail. Eq 1 has a lot of terms but, in a way, it all boils down to one integral with a bunch of parameters.

Note that "text2mesh" was published at CVPR 2022 after the NeurIPS deadline, so I don't think it "counts" as prior work. Nonetheless, it's great that it's included as a baseline, which is very informative.
questions: Are there any constraints on the normal fitting, e.g., do the normals need to be consistent with the geometry or can they be totally unrelated? Presumably there is a limit to how far normals can vary from the normals of the input geometry (or the nearby vertex normals).


limitations: One limitation is that stylization may involve creating new geometry, not just texture. "text2mesh" does create new geometry, which has visible pros and cons in the results.

As discussed above, it's unclear whether good materials+lighting have been recovered, or just enough to render views with static lighting.
review_title: null

Meta-review: metareview: 
This paper presents a new CLIP-driven stylization method given an input mesh and text description. Compared to previous works Text2Mesh, the paper introduces a more expressive rendering model based on learnable SVBRDF and normal maps. Many reviewers found the paper easy to follow, the idea promising, and the results visually appealing. They also expressed their concerns regarding the similarity to Text2Mesh, the limitations of the normal maps approach (compared to changing geometry explicitly), and the relighting and material editing of the stylized object. The rebuttal has addressed most of the concerns. The AC agreed with most of the reviewers and recommended accepting the paper. 

Please revise the papers according to the reviewer’s comments: (1) change the title according to Reviewer kpuS, (2) add relighting/material editing/view synthesis results, and (3) highlight the pros and cons of the proposed method w.r.t. Text2Mesh. 


confidence: Certain
Decision: Accept