TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decomposition
Authors: Yongwei Chen
Rui Chen
Jiabao Lei
Yabin Zhang
Kui Jia
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D content creation
CLIP
Text-to-3D

Abstract: Creation of 3D content by stylization is a promising yet challenging problem in computer vision and graphics research. In this work, we focus on stylizing photorealistic appearance renderings of a given surface mesh of arbitrary topology. Motivated by the recent surge of cross-modal supervision of the Contrastive Language-Image Pre-training (CLIP) model, we propose TANGO, which transfers the appearance style of a given 3D shape according to a text prompt in a photorealistic manner. Technically, we propose to disentangle the appearance style as the spatially varying bidirectional reflectance distribution function, the local geometric variation, and the lighting condition, which are jointly optimized, via supervision of the CLIP loss, by a spherical Gaussians based differentiable renderer. As such, TANGO enables photorealistic 3D style transfer by automatically predicting reflectance effects even for bare, low-quality meshes, without training on a task-specific dataset. Extensive experiments show that TANGO outperforms existing methods of text-driven 3D style transfer in terms of photorealistic quality, consistency of 3D geometry, and robustness when stylizing low-quality meshes. Our codes and results are available at our project webpage https://cyw-3d.github.io/tango/.

TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decomposition
Yongwei Chen Rui Chen', Jiabao Lei', Yabin Zhang", Kui Jia South China University of Technology "The Hong Kong Polytechnic University
'DexForce Co. Ltd. *Peng Cheng Laboratory eecywêmail scut. edu. en, kuijia@scut. edu.
Abstract
Creation of 3D content by stylization is a promising yet challenging problem in com- puter vision and graphics research. this work, we focus on stylizing photorealistic appearance renderings of given surface mesh of arbitrary topology. Motivated by the recent surge of cross-modal supervision of the Contrastive Language-Image Pre-training (CLIP) model, propose TANGO, which transfers the appearance style of given 3D shape according to text prompt Technically, propose disentangle the appearance style as the spatially varying bidirectional reflectance distribution function, the local geometric variation, and the lighting condition, which are jointly optimized, via supervision of the CLIP loss, by a spherical Gaussians based differentiable renderer. As such, TANGO enables photorealistic 3D style transfer by automatically predicting reflectance effects even for bare, low-quality meshes, without training on a task-specific dataset. Extensive experiments show that TANGO outperforms existing methods of text-driven 3D style transfer in terms of photorealistic quality, consistency of 3D geometry, and robustness when stylizing lualquality meshes. Our codes and results are available at our project webpage
photorealistic manner.
github.io/tango
Introduction
3D content creation by stylization (e.g., stylized according to text prompts 33.. images [44, or 3D shapes (50) has important applications in computer vision and graphics areas. The problem is yet challenging and traditionally requires manual efforts from experts of professional artists and large amount of time cost. In the meanwhile, many online 3D repositories 66, 47 41
can be easily accessed on the Internet whose contained surface meshes are bare contents without any styles. is be achieved given these raw 3D 18 33 (21, we consider style as the particular appearance of object, which is determined by color (albedo) and physical reflective effect of the object surface, while considering content as the global shape structure and topology prescribed by an explicit 3D mesh other implicit representation of the object surface.
thus promising if automatic, diverse, and realistic stylization contents. We note that similar to recent 3D stylization works [3 50
Stylization is usually guided by some sources of styling signals, such as a text prompt [31] a reference target 3D shape [50]. In this work, we choose work with stylization by text prompts, motivated by the surprising effects recently achieved in many applications [44] 19 [0], by using the cross-modal supervision model of Contrastive Language- Image Pre-training (CLIP) [37] The goal of this paper is to devise an transfer, guided by text prompt, the style of given 3D shape of arbitrary topology. Note that TANGO can be directly applied to arbitrary meshes with arbitrary target styles, without additional leaming/optimizztionn procedures task-specific dataset; Figure I gives the illustration.
image 44.
neural architecture system, named TANGO, that
*Conresooding Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

The most relevant work so far is Text2Mesh [1], which is the first to use the CLIP loss in mesh stylization; given an input mesh and a text prompt, it predicts stylized color and displacement for each mesh vertex, and the stylized mesh is just a result of this colored vertex displacement procedure. However, Text2Mesh does not support the stylization of lighting conditions, reflectance properties, and local geometric variations, which are necessary in order to produce a photorealistic appearance of 3D surface. To this end, we formulate the problem of mesh stylization as learning three unknowns of the surface, i.e., the spatially varying Bidirectional Reflectance Distribution Function (SVBRDF), the local geometric variation (normal map), and the lighting condition. We show that TANGO is able to learn the above unknowns supervised only by CLIP-bridged text prompts. and generate photorealistic rendering effects by approximating these unknowns during inference. Technically, by jointly encoding the text prompt and images of the given mesh rendered by a spherical Gaussians based differentiable renderer, TANGO is able to compare the embeddings of text and mesh style and then backpropagate the gradient signals to update the style parameters. Our approach disentangles the style into three components represented by learnable neural networks, namely continuous functions respectively of the point, its normal, and the viewing direction on the surface. Note that due to the learning of neural normal map. TANGO can produce fine-grained geometric details even on low-quality meshes with only few faces; in contrast, vertex displacement-basee methods, such as B1. often fail in such cases.
We finally summarize contributions follows.
We propose a novel, end-to-end architecture (TANGO) for text-driven photorealistic 3D style transfer; our model learns assign advanced physical lighting appearance and local geometric details to a raw mesh, by leveraging an off-the-shelf,, pre-trained CLIP model.
TANGO can automatically predict material, normal map, and lighting condition for any bare mesh prescribed by text description, and can also handle low-quality meshes with only a few faces, due to our prediction of colors and reflectance properties for every intersection point of a camera ray.
We conduct extensive ablation studies and experiments that show the advantages of TANGO. Notably, except for more photorealistic renderings, our results have no self inferstttssttoon in local geometry details; in contrast, such imperfections would appear in results from existing methods.
2 Related Work
Text-Driven Generation and Manipulation. Our work is primarily inspired by [1 and other
20 22] which use text prompt to guide 3D generation manipulation. The [37]. Specifically, CLIP- NeRF [44] proposes unified framework that allows manipulating NeRF, guided by either short text prompt or an examplar image. Another work Dreamfields [1] chooses to generate 3D scenes from scratch in NeRF representation. Different from those NeRF-based methods, CLIP-Forge 40) presents a method for shape generation represented by voxels. Text2Mesh (31) predicts color and geometric details for given template mesh according the text prompt. Concurrently, Khalid et al. [22] generate stylized mesh from a sphere guided by CLIP. Mcanwhile, apart from 3D content creation, there are other works focusing on text-guided image generation and manipulation. StyleCLIP (3) and VQGAN-CLIP [0) use CLIP-loss to optimize latent codes in GAN's s style space. GLIDE 35 is introduced for more photorealistic image generation and editing with text-guided diffusion models, while DALL-E (38) explores to generate images using transformer. Compared to these methods, we focus on improving the realism of text-driven 3D mesh stylization.
methods 44
19
40
optimization procedures of these methods are guided by the CLIP model
3D Style Transfer. Generating or editing 3D content according given style a long-standing task in computer graphics [12 14 48
NeuralCage [49] implements traditional cage-based defor- mation as differentiable network layers and generates natture-deserving deformations. Furthermore, 3DStyleNet [50] conducts 3D object stylization informed by a reference textured 3D shape. Other works are specific to styles of furniture (27), 3D colleges
[12], LEGO 23], and portraits [17] Com- pared to these global geometric transfer approaches, some works investigate style transfer on more local level. Specifically, Hertz et al. [8 propose generative framework to learn local structures
from the style of a given mesh while Liu et al. [25] subdivide the given mesh for coarse-to-fin

---

geometric modeling. Unlike these methods, we aim to synthesize a wide range of realistic styles specified by text prompt, making it more convenient to guide the process by high-level semantics.
BRDF and Lighting Estimation. Previously, BRDF and lighting can only be estimated under complicated settings in a laboratory environment 2 28 46 before the deep learning era. With the help of neural networks, some techniques can simpler settings to estimate BRDF and lighting beyond geometry [33 IM They assume that scenes are under specific illumination, like one [T] or multiple [13] flashlights and linear light [5]. However, they often assume that the geometry is plane [33 or known 3 These requirements are difficult to be met in reality. Furthermore, recent works 7 [8 26 53 exploit neural networks to jointly predict geometry, BRDF and in some cases, lighting via 2D image loss. These methods can be split into two categories according to the geometry representation. Among them, those who use explicit representation 126 to obtain final shapes, which cannot represent arbitrary topology. Another pipeline of methods that use implicit geometry representation like SDF occupancy function 30 NeRF
usually deform sphere
42
can
get higher quality shapes, but these representations are not handy to be processed by contemporary game engines. Unlike them, our task is to change the style of a given shape, so the explicit triangle mesh is preferred due to its convenience. As for BRDF and lighting representation, PhySG [5]]
and NeRD 4 use mixtures of spherical Gaussians to represent illumination; NeRFactor [52 and nvdiffrec
34 choose low-resolution envmaps and split sum lighting model, respectively. Different
from all these 21) supervision methods, we jointly estimate spatially varying BRDF, lighting, and local geometry (normal map) supervised by text prompts. With the help of estimated BRDF and
lighting, we can represent complicated light reflection in the real world on stylized meshes.
3 Method
We contribute TANGO, an end-to-end architecture that enables text-driven photorealistic 3D style transfer for any bare mesh, which is supervised by semantic loss. The heart of TANGO is to disentangle the style of the input mesh as reflectance properties and scene lighting. Then, given target style specified by a text prompt, we could learn the corresponding style parameters by leveraging the pre-trained CLIP model, and then the stylized images could be generated with the learned style parameters through spherical Gaussians (SG) differentiable renderer. There three unknowns in our defined style parameters: (i) spatially varying BRDF (SVBRDF), including diffuse, roughness, specular, represented by parameters & E Rmm (ii) normal, represented by parameters E R"; (iii) lighting, represented by parameters T E R'.
We use explicit triangle mesh representation to represent the input 3D shape. The input template mesh M consists of vertices V E ex and u faces F E: {1, .... n "xx and is fixed throughout training. The object's 's style is described by a text prompt, and the aforementioned SVBRDF, normal and lighting parameters are optimized to fit the description. Compared to other implicit representation [5T 4) the explicit representation here is more convenient to obtain and easy to use by current game engines and other applications.
Forward model. Given an object mesh M. we first scale it in a unit sphere and then randomly sample points near M as camera positions indexed by p. let Rp = {cp + tup tt 2 0} denotes the ray through pixel p. where c, is randomly sampled on a sphere containing the object and Up denotes the direction of the ray (i.e. the vector pointing from Cp to p). Let Ip denotes the first intersection of the ray R2 and the mesh M, npp denotes the ground truth normal of point Tp. n. = II(Ng. point Ep.. For each surface point Tp with the estimated normal Ωn we suppose that Li(w; r) the incident light intensity from direction Wi, and SVBRDF fr(P,, w;, &pi {) are the surface reflectance coefficients of the material at location Pp from viewing direction U, and incident light direction wi Then the observed light intensity Lp(up, Xp. np) is an integral over the hemisphere Ω = {wi wi ñp 2 0}:
render images. For each pixel in rendered images,
is the estimated normal on surface
Lp Fp, noi E 7. / (((wi))) wp, Tpi (w, II(,, (p:)))) dww,
(I)
For each image / E [0, 10 XxWx3 corresponding to camera position e, we calculate the pixel color by Equation (L). The rendered image is encoded into a latent code by pre-trained CLIP model,
3

---

Inputs
Forward Model
Mesh
Ray Casting
SVBRDF Network
Rendering
Diffuse
Roughness
Normal Network
Specular
Normal
Image
Lighting
Text
"a shoe made
gold"
CLIP Loss
Image
Augmentation
Encoder
Encoder
Figure 1: Given a scaled bare mesh and a text prompt (e.g.. "a shoe made gold" as this figure), we first specify camera location and cast rays R to intersect with the given mesh. For each intersecting camera ray, we obtain a surface point Tp and normal n2p, which are used for predicting spatially varying BRDF parameters and normal variation by two MIPs, i.e. the SVBRDF Network and Normal Network. Meanwhile, the lighting is represented using a series of spherical Gaussians parameters {pR, Ak, ag). For each iteration, encode the augmented image using the image encoder of CLIP model, which backpropagates the gradients update all learnable parameters.
render the image using differentiable SG renderer and then
which is expected be aligned with the encoded prompt. Finally, loss measuring the divergence between encoded image and text representations is used to simultaneously train the model parameters, i.e., &, 7, and т, resulting in photorealistic renderings matching the text description. We summarize our approach in Figure [ and clarify each step in the following: Sec. outliness our intersection calculation, Sec. presentss our appearance style model, Sec. [3. illustrates the rendering procedure. and the loss function we use to train our network is described in Sec. 3.4
3.1 Intersection of camera rays and mesh
Given input 3D mesh M and target text, first scale M in unit sphere and find an anchor view on the sphere, then other camera positions are randomly sampled using Gaussian distribution centered around the anchor view, whose direction vectors are bounding sphere. Then for each sampled camera position e and pixel p. we get a camera ray R2 = + lvp 2 0). pointing from to p. After that, intersection point Zp and intersection face fp of Rp and mesh M. while the face normal np of fp is used surface normal at point Ep.
look the origin of the unit
ray casting (3) to find the first
3.2 Appearance modeling
To generate photorealistic style of object M, we use three components in Equation ( to model appearance: (1) environment map, (2) normal map, and (3) SVBRDF consists of diffuse term and specular term. To calculate Equation (T), some works use Monte Carlo sampling [24), split sum
approximation (34), spherical Gaussians [51] or spherical Harmonies (16) solve the integration. In this paper, we choose to use spherical Gaussians (SG) to efficiently approximate the rendering equation closed form.
An n--imennional spherical Gaussian spherical function that takes the following form
G(v; μ, (u ae (uru-11))
where E $2 is the function input, l E 8 is the lobe axis, À E R4 is the lobe sharpness, and R2 the lobe amplitude (a E T.. for RGB color). Spherical Gaussians have two nice properties to ensure the convenience of calculating integration:

---

(1) the multiplication of two spherical Gaussians is another spherical Gaussian:
Giv: (1: P2, X2, a2) = G(v;
Mmm
(A1 + A2)ll aia (0lleml-)))))) (3)
|
where Am ー1 + X2, Pm
42/22 A1
(2) the integration of single spherical Gaussian has closed-form solution:
Jo
G(v: μ. A, a)dd 2nT a
A [1 - 2
The above two properties guarantee that if we use SG to represent each multiplication term in Equation I.. the formula integral can be written as single SG, which can be calculated efficiently.
Environment map. We represent environment map (ws with multiple spherical Gaussians (SG):
L,(wi) = Σ
G(wii Mex as.
(5)
k=0
The light energy is:
a
E(L) - Σ Ax (1-1--22
(6)
When energy is too large or too small, the rendered image will be too bright or too dark, resulting in the optimization falling to local minimum. To alleviate this issue, we initialize the total energy to 6.25.
Normal map. To provide more geometry details for meshes when rendering, for each xp and n, {(1, pp. pp))l E (0, 2x), Pp E (0, x)), we estimate normal offset to n, get estimated
normal ú { {1, (p, Pp) 0p (0.2x). Pp E (0,
(0 pn) = (0p + A., 9p + Ap) = (I(3(pp Op pp); y): (40, Ax) = (3xp Op, pp):
(7)
(8)
where All is neural network (MLP) estimating the normal offset. For every input, we apply positional encoding layer ((1) [cos 2t Bl), (2m B1))² to synthesize high-frequency details, where B is random Gaussian matrix whose entry is randomly drawn from (N a²) or arithmetic
sequence ranging from N like NeRF 32
(0,, +p) is clamped to lie in reasonable ranges.
SVBRDE Below we take apart the SVBRDF fr (V, wi, xrp) into diffuse term fa(p)) and specular term (V wi, æp). For diffuse term, use an MLP o (ax,)) E1) to calculate spatially varying diffuse albedo, where (I) is positional encoding layer. Then diffuse term is
fa(x,) T (3(p)):
π
For the specular term, use the simplified Disney BRDF model 5
fv w;) = M(P, w. (,),
(10)
where M accounts for Fresnel and shadowing effects, and normal distribution. Following [5T] we spherical Gaussians to represent specular term, so the specular SVBRDF function can be written as:
N
X Y Mpaj),
f. wi, xp)) =
Σ
G(h: n.. 4h
(11)
1=0
where the half vector h vow i v). {A3, a,) are predicted by an MLP (B(x,)))
|e the approximated Fresnel and shadowing term Md & M(p, 2(v-
E2). The complete SVBRDF function is w/ (r w,, fa(x,) + (v, Following 29 we use an SG to approximate term Ω. = G(v; 0.0315. á,. 32. 7080) 30.7003.
(12)
Note that components in Equation [ are represented by SGs, so the integration can be calculated in closed form.
5

---

3.3 Rendering
Given our geometric and appearance components, we perform rendering of ray R 's color as follows: (1) find the intersection point 2p and normal nt, between Rd and the mesh using ray
casting; (2) predict the normal offset and get the estimated normal nn: (3) compute the diffuse albedo 4 (Tpp {1) and specular albedo (2 E2); (4) use estimated normal np. environment map (fy; Ae; ). SVBRDF (mp: E). and viewing direction u, to compute the color for R,; (5) conduct steps (1)-(4) repeatedly for every pixel and then finally gather them as the rendered image. The overall pipeline is illustrated in Figure [ optimizable parameters.
Note that our pipeline is fully differentiable w.r.t. all the
3.4 Text-based correspondence loss
Our optimization is guided by pre-trained CLIP 37 model. For each iteration, we sample a hatch of camera positions. For each camera position c,
render image I E [0, 1] xx whose pixel's color is calculated by Equation (1) and a mask O E {0, 1] M> Then we use the mask O add background to image I. We test black, white, and random Gaussian backgrounds, and observe similar stylized performance. Therefore, we use white and black backgrounds in TANGO for convenience. Next, in the 2D augmentation phase, we randomly crop some areas of I and resized them to (224, 224), resulting in the augmentations of Ia Subsequently. I and augmented images Ia are all encoded into the multimodal space by the image encoder CLIP, generating an averaged latent code L, E R2522 Mcanwhile, the input text prompt is encoded to L, E 18 by the text encoder of CLIP. Finally, we use the following cosine similarity as the overall objective:
Li Ce
Loss(C;, Ci) |,C,
(13)
Implementation details. The environment map is represented by 32 or 64 spherical Gaussians, and the specular BRDF is represented by SG. The normal estimation network [ p... 6p. Pp: 7) E R2
consists of 3 layers with hidden layers of width 256. The SVBRDF network, ( {) E R7, with hidden layers of width 256, predicts diffuse, specular and roughness per surface point; the three ones share the first 2 layers of (a p {) and the parameters in next 3 layers are exclusive. We use the pre-trained CLIP model with the ViT-B/32 backbone. We adopt the Adam W optimizer and the initial learning rate set as 5 x 10-4, The learning rate is decayed by 0.7 in every 500 iterations.
Experiment
Following (311 we examine TANGO in a variety of meshes and different text prompts.
The object meshes are collected from: [01 Thingil0K [ringil0K[54] (54). ShapeNet(o). Turbo Squid[3) and ModelNet[47]. To test TANGO's robustness to low-quality meshes, we downsample the number of meshes' vertices and faces in Sec. 4.2, while in other experiments we directly stylize the given mesh without any pre-processing, Except in Sec. 4.2, meshes used our paper contain an average of 73, 966 faces, 16% non-manifold edges, 0.2% non non-manifold vertices, and 12% boundaries. Before stylizing the given mesh, we scale the input into a unit sphere and move the center of its vertices to the origin. TANGO is trained on single Nvidia RTX 3090 GPU and takes about 10 minutes for 1500 iterations.
In Sec. 4.1 we examine TANGO on different meshes with text prompts and compare our results the text-driven mesh stylization approach Tex12Mesh(31]. After that, in Sec. 4.2 we degenerate the quality of meshes by downsampling the number of faces to 5000, verifying the robustness of TANGO sparse mesh data. Next in Sec. 4.3 we conduct the ablation study to explore the impact of our proposed modules.
4.1 Neural stylization and controls
TANGO generates photorealistic details with fine granularity and avoids self-intersection in the meantime. As presented in Figure 2 TANGO not only presents realistic reflective effects like highlight and shade in smooth materials like gold and silver but also tackles uneven surfaces by producing dark and bright shadings via the estimated normals. Furthermore, in Figure 3 our approach

---

Original Mesh|
Texd2Mesh
TANGO (ours)
Texd2Mesh
TANGOO (ours)
Tex22Mesh
TANGO (ours)
Candle
candle wearing jcans"
candle made bark""
"A candle made silver"
Shoe
"A shoc made of gold"
"A shoe made of chainmail"
"A shoc made of brick"
Vase
"A made of wicker"
"A made of wood"
made of colorful crochet"
stylization results of TANGO and Text2Mesh given the same mesh and text prompt. TANGO produces results of higher fidelity and more realistic rendering. See Sec. 4. for the details.
Figure 2:
Table 1: Mean opinion scores (1-5) for Q1-Q3 (cf.. Sec. 4.1 for TANGO and baseline.
(Q1): Overall
(Q2): Content
(Q3): Style
Text2Mesh
3.30(+0.75)
3.5.(=0.79)
3.42(##.66) 3.94(10.75)
Ours
4.02 +0077)
3.98 (10.74)
demonstrates a consistent 3D stylization in different views and exhibits natural variation in texture. Similar to Text2Mesh, incorporate 3D symmetry prior for those meshes that need style symmetry across an axis, like the person shown in Figure shape with bilateral symmetry across the X-Y plane, apply symmetry prior the positional encoder, i.e., replacing ((x, y, =) as (Ix, y: |=]). With this symmetry prior, the stylized Iron Man in Figure is symmetrical across the z-axis, which satisfies the real situation for person. It is surprising that we can generate such photorealistic renderings only supervised by CLIP loss. Besides the powerful pre-trained CLIP model, the key point of success the shading model and carefully designed disentangled elements in Equation (T). By predicting normal offsets and SVBRDF individually, optimizing the light SG parameters, and inputting these elements into the physical rendering equation, the individual module predicts physics-awart results, as illustrated with the disentangled components Figure 4 Meanwhile, because TANGO disentangles geometry, material and lighting representation, we can easily relight the stylized mesh, which can not be accomplished methods. In Figure 3, we conduct the relighting experiments with the example of "A vase made of wood". By replacing the estimated environment map with others downloaded from the Internet, the lighting of the rendering results could be successfully manipulated. Another benefit of our disentangled mesh style representation is that we can easily edit the physical elements of materials to get the preferred style, which is important for incorporating current graphic engines. As Figure 6 shows, the surface of the golden shoe becomes more diffuse roughness value of the material, while a larger specular value typically results in a more shiny surface, proving the capability of TANGO material editing.
Specifically, for each point xp = (I. y, 2) and
by vertex displacement-
enlarge the
Comparison with Text2Mesh. In this part we compare TANGO with pioneering work in text- driven 3D stylization, Text2Mesh (31, which operates on the point colors and vertex's displacement. We use the official implementation of Text2Mesh and also train it for 1500 iterations for a fair comparison. Compared to Text2Mesh, TANGO can handle both smooth mental appearance and rough materials, generating realistic renderings with highlight effects. For example in Figure 2 given
7

---

Person
"A 3D rendering of the Iron Man unreal engine"
Figure 3: Given a bare person mesh and text prompt, TANGO can stylize the entire 3D Iron Man appearance and therefore generates 3D-consistent renderings.
(a) Rendering (b) Lapul normal (c) Predicted normal (d) Diffuse
(e) Roughness
(() Specular
(g) Environment map
Figure 4: An illustration of the disentangled rendering components. This figure shows (a) our final rendering result of "A shoe made of brick" and individual components for the rendering including (b) the original normal map input mesh, (c) the normal map predicted by TANGO, (d) diffuse map predicted by TANGO, (c) spatially varying roughness map, (f) spatially varying specular map and (g) environment map. Note that the BRDF composed the diffuse map, roughness map, and specular map.
a bare shoe mesh and a text prompt "a shoe made of gold", TANGO can produce realistic golden reflectance that has light and shade contrast, which is not observed in the competitive Mesh2Mesh. For materials that need an apparent bump, for example, "a vase made of wicker", our approach successfully generates bump materials without incorporating local self-intersection. In addition, randomly chose 73 users to evaluate 9 source meshes and style text prompt combinations. Each of them asked three questions:
(QI) "How natural is the output results?" "How well does the output match the original content?" (Q3) "How well does the output match the target style?" We report the mean opinion scores with standard deviations for both Text2Mesh and ours in Table I TANGO outperforms the Text2Mesh baseline across all questions, with an advantage of 0.72, 0.45, and 0.52 for Q1-Q3, respectively. More in-depth discussions and comparisons with Text2Mesh appear in our appendices.
4.2 Robustness to low-quality meshes
To measure the robustness of our approach low-quality meshes, we show stylization results of original meshes and downsampled meshes in Figure 7 The original lamp mesh and alien mesh have 41, 160 faces and 68, 430 faces, respectively. To generate the low-quality meshes, we both
Ours
Relight
Relight
Relight.3
Figure 5: An illustration of the relighting performance. The left image shows our rendering result and estimated environment map of "A vase made of wood", while the following three ones are relighted results with different environment maps downloaded from the Internet.
8

---

Roughness
0.2
0.4
0.6
0.8
1.0
Specular
[0.2, 0.2, 0.2]
10.4. 0.4, 0.4]
[0.6, 0.6, 0.6]
[0.8, 0.8, 0.8]
[1.0, 1.0, 1.0]
Figure 6: An illustration the material editing performance. In each line, we increase the roughness value of the material, while increase the specular value in each column. We can observe that the surface becomes more diffuse as we enlarge the roughness, while increasing the specular value results in more shiny surfaces. This illustration justifies the capability of TANGO editing the material.
Lamp
"A lamp made of brick"
Alien
"An alien made chainmail"
Original Mesh
Downsampled Mesh
Text2Mesh
TANGO (owrs)
Texi2Mesh
TANGO (ows)
A robustness comparison of TANGO and Text2Mesh on high-quality meshes (cf. the top row) and downsampled meshes (cf. the second row). The results of Text2Mesh degenerate significantly as the mesh quality downgrades, while TANGO presents consistent results on meshes of various qualities.
Figure 7:
downsample them to 5, 000 faces using Meshlab 99 On low-quality meshes with fewer faces, TANGO still maintains similar performance the original mesh, while Text2Mesh is trapped in oeff-intersection and monotonous color prediction. The reason for the phenomenon is that we make a spatially varying prediction for each intersection point of the camera ray and mesh and the local geometric variation compensates for the lack of geometric details in low-quality meshes. As for displacement- methods like Text2Mesh, they rely on moving each vertex's position generate bumpy geometry, which makes them deviate with vertices number shrinking.

---

Candle
full
-spec. rough
-normal
-normal
-BRDF PE
-crop
Figure 8: Ablation experiments the proposed modules in TANGO, where the text prompt is "A candle made of wicker". Please refer to the Sec. 4.3 for the detailed definitions.
4.3 Ablation study
To verify the effectiveness each module in our approach, we conduct an ablation study by removing different components from TANGO. The results are shown in Figure Compared to the full model, when we remove spatially varying specular and roughness prediction (-spec, rough in Figure (). the whole SVBRDF is only represented by diffuse color and therefore it cannot generate realistic view-dependent highlight in renderings. we disable the normal estimation network (-normal in Figure()). the stylized appearance becomes smooth and fails generate enough geometric variations, verifying the effectiveness of the normal estimation network in generating geometric bump details. To provide enough high-frequency details when stylizing, we use the positional encoding layer to wrap the input points and normals for both SVBRDF prediction and the normal estimation network. we remove positional encoder (-PE, -normal PE and -RDDF PE in Figure D our results will lack in high-frequency variation in both color and geometry. In addition, local crop augmentation (-crop in Figure g also contributes obviously the stylized performance, since it makes our network focus on small areas on the surface and avoid blurred results.
Limitations and conclusions
Our main limitation is the simplified shading model to represent indirect lighting effects and shadow, impeding TANGO from representing complicated lighting in large scenes which need to simulate multiple light bounces. The reason for choosing spherical Gaussians to represent our shading is to accelerate the rendering and training process. To extend our model for large scene stylization, indirect lighting term and acceleration techniques can be incorporated to guarantee both quality and speed. Another limitation that the vertex displacement framework presents larger geometry capacity than ours in theory. However, the rendering capacity gap between frameworks of vertex displacement and our adopted normal displacement could be largely reduced with our introduced learnable SVBRDF and normal, meanwhile presenting more robustness and time efficiency. TANGO and the recent Text2Mesh conduct the initial attempts to this problem in different directions, which may inspire more in-depth following investigations.
In this paper, we propose TANGO, novel framework to generate photorealistic appearance style for an arbitrary 3D mesh according to text prompt. By disentangling the appearance style as SVBRDF; local geometric variation, and lighting condition, we jointly learn them under the supervision of the CLIP loss via spherical Gaussians based differentiable renderer. Compared to existing competitors, TANGO could automatically predict reflectance effects even for bare, low-quality meshes. However, on the other hand, it will possibly make junior artists responsible for creating simple texture and lighting for bare meshes out of employment.
Acknowledgements
This work is supported in part by Guangdong R&D key project of China (No.: 20198010155001) and the Program for Guangdong Introducing Innovative and Enterpreneurial Teams (No.: 20172T07X183).
10

---

References
[1] Miika Aittala. Timo Aila, and Jaakko Lehtinen. Reflectance modeling by neural texture synthesis. ACM Transactions on Graphics (TOG), 35(4):1-13, 2016.
[2] Miia Aittala, Tim Weyrich, and Jaakko Lehtinen. Practical SVBRDF capture in the frequency domain. ACM Transactions on Graphics (TOG). 32:4): 2013.
[3] Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold- Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv preprint wrXiv:2088.0322 2020.
[4] Mark Boss, Raphael Braun, Varun Jampani, Jonathan Barron, Ce Liu, and Hendrik Lensch. Nerd: Neural reflectance decomposition from image collections. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12684-12694, 2021.
[5] Brent Burley and Walt Disney Animation Studios. Physically- based shading at disney. ACM SIGGRAPH, volume 2012, pages 1-7, 2012.
[6] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012. 2015.
[7] Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith. Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning predict 3d objects with an interpolation-based differentiable renderer. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
[8] Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang. Clement Fuji Tsang, Sameh Khamis, Or Litany, and Sanja Fidler. Dib-r++: Learning to predict lighting and material with hybrid differentiable renderer. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[9] Paolo Cignoni, Marco Callieri, Massimiliano Corsini, Matteo Dellepiane, Fabio Ganovelli, Guido Ranzuglia, et al. Meshlab: an open-source mesh processing tool. In Eurographics Italian Chapter Conference, volume 2008, pages 129-136. Salemo, Italy, 2008.
[10] Katherine Crowson, Stella Biderman, Daniel Koriis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vqgan-clip: Open domain image generation and editing with natural language guidance. arXiv preprint arXiv:2204.000 08883, 2022.
[11] Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. Single- image svbrdf capture with rendering-aware deep network. ACM Transactions Graphics (TOG), 1-(5) 2018.
[12] Ran Gal, Olga Sorkine, Tiberiu Popa, Alla Sheffer, and Daniel Cohen-Or. 3d collage: Expressive non- realistie modeling. In Proceedings of the 5th international symposium on Photoreeeaaiiicc Animation and Rendering (NPAR), pages 7-14, 2007.
[13] Duan Gao, Xiao Li, Yue Dong, Pieter Peers, Kun Xu, and Xin Tong. Deep inverse rendering for high- resolution svbrdf estimation from an arbitrary number of images. ACM Transactions on Graphics (TOG), 38(4):134:1-13:155 15, July 2019.
[14] Lin Gao, Jie Yang. Yi-Ling Qiao, Yu-Kun Lai, Paul Rosin, Weiwei Xu, and Shihong Xia. Automatic unpaired shape deformation transfer. ACM Transactions on Graphics (TOG), 1-15) 2018.
[15] Andrew Gardner, Chris Tchou, Tim Hawkins, and Paul Debevec. Lincar light source reflectometry. ACM Transactions on Graphics (TOG), 22(3):749 758, 2003.
[16] Robin Green. Spherical harmonic lighting: The gritty details. In Archives of the Game Developers Conference (GDC), volume 56, page 4, 2003.
[17] Fangzhou Han, Shuquan Yc, Mingming Hc, Menglei Chai, and Jing Liao. Exemplar-based 3d portrait stylization. IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021.
[18] Amir Hertz, Rana Hanocka, Raja Giryes, and Daniel Cohen-Or. Deep geometric texture synthesis. ACM SIGGRAPH, 39(4). jul 2020.
[19] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text guided object generation with dream fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
11

---

[20] Nikolay Jetchev. Clipmatrix: Text-controlled creation of 3d textured meshes.
arXiv preprint
arXiv:2109 12922, 2021.
[21] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3907-3916, 2018.
[22] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Popa Tiheriu. Clip-mesh: Generating textured meshcs from text using pretrained image-text models. December 2022.
[23] Kyle Lennon, Katharina Fransen, Alexander O' Brien,, Yumeng Cao, Matthew Beveridge, Yamin Arefeen, Nikhil Singh, and Iddo Drori. Image2lego: Customized lego set generation from images. arXiv preprint arXiv:2108,0m 08477, 2021.
[24] Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. Differentiable monte carlo ray tracing through edge sampling. ACM SIGGRAPH Asia, (7(6):2221-2222) 2018.
[25] Hsueh-Ti Derek Liu, Vladimir G Kim, Siddhartha Chaudhuri, Noam Aigerman, and Alec Jacobson. Neural subdivision. ACM SIGGRAPH, 2020.
[26] Shichen I. Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 7778-7717, 2019.
[27] Zhaoliang Lun, Evangelos Kalogerakis, Rui Wang, and Alla Sheffer. Functionality preserving shape style transfer. ACM Transactions on Graphics (TOG), 35(6):1-14, 2016.
[28] Wojciech Matusik. A data-driven reflectance model. PhD thesis, Massachusetts Institute of Technology, 2003.
[29] Julian Meder and Beat Brüderlin. Hemispherical gaussians for accurate light integration. In International Conference on Computer Vision and Graphics (ICCVG), pages 3-15. Springer, 2018.
[30] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4460 4470, 2019.
[31] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[32] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), pages 421. Springer, 2020.
[33] Alexander Mordvintsey, Nicola Pezzotti, Ludwig Schubert, and Chris Olah. Differentiable image parame- terizations. Distill, 3(7):e 2018
[34] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
[35] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. Proceedings of Machine Learning Research (PMLR), 2022.
[36] Patashnik. Zongze Wu, Eli Shechtman, Daniel Cohen Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2085-2094, 2021.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh. Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 8748-8763. PMLR, 2021.
[38] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-imann generation. In International Conference on Machine Learning (ICML). pages
PMLR, 2021.
12

---

[39] Scott D Roth. Ray casting for modeling solids. Computer Graphics and Image Processing (CGIP). 18(2):109-144, 1982.
[40] Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin- Yi Cheng, and Marco Fumero. Clip-forge: Towards zero-shot text-to-shape generation. Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), 2022.
[41] Oana Sidi, Oliver Kaick, Yanir Kleiman, Hao Zhang, and Daniel Cohen-Or. Unsupervised segmentation a set shapes via descriptor space spectral clustering. In ACM SIGGRAPH Asia, pages 1-10, 2011.
[42] Pratul Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, Jonathan Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7495-7500 1994,, 2021.
[43] TurboSquid. Turbosquid 3d model repository. httpp:://ww, wttp::::wwwboruuu...ooouu com/ 2022.
[44] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, Jing Liao. Clip-nerf: Text-and-image driven manipulation of neural radiance fields. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[45] Jiaping Wang. Peiran Ren, Minmin Gong. John Snyder, and Baining Guo. All-frequency rendering of dynamic, spatially-varying reflectance. ACM Transactions Gruphics (TOG), 28(5):1-10, dec 2009.
[46] Hongzhi Wu, Julie Dorsey, and Holly Rushmeier, A sparse parametric mixture model for btf compression, editing and rendering. Computer Graphics Forum (Comput Graph Forum), 30(2):465 473, 2011.
[47] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1912-1920, 1920, 2015.
[48] Jie Yang, Kaichun Mo. Yu-Kun Lai, Leonidas Guibas, and Lin Gao. Dsm-net: Disentangled structured mesh net for controllable generation of fine geometry. arXiv preprint w0X5v:2008.05440. 2(3), 2020.
[49] Wang Yifan, Noam Aigerman, Vladimir G Kim, Siddhartha Chaudhuri, and Olga Corkine-Homune Neural cages for detail- preeeeiipen 3d deformations. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 75 83, 2020.
[50] Kangxue Yin, Jun Gao, Maria Shugrina, Sameh Khamis, and Sanja Fidler. 3dstylenet: Creating 3d shapes with geometric and texture style variations. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22456-1245 2021.
[51] Kai Zhang, Fujun Luan, Qianqian Wang. Kavita Bala. Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5453-5462, 2021.
[52] Xiuming Zhang. Pratul Srinivasan, Boyang Deng. Paul Debevec, William T Freeman, and Jonathan Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions Graphics (TOG), 40(6):1-18 18. 2021.
[53] Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering. International Conference on Learning Representations (ICLR), 2020.
[54] Qingnan Zhou and Alec Jacobson. Thingil0k: A dataset of 10,000 3d-printing models. arXiv preprint arXiv: 1o055:1605.0477 047977 2016.
13

---

1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] see Section
(c) Did you discuss any potential negative societal impacts your work? [Yes] see Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Our codes and results are available at our project webpage https:///yww3d.iihuub..b. io/tanoo
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Implementation details in Section 3
and Section 4
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] see Table
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] see Implementation details in Section 3
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "nxkKXHxut2m", "reviewer": "Reviewer_kpuS", "summary": "This paper presents a 3D mesh stylization method that performs both appearance and geometric styles by optimizing with a differentiable rendering pipeline. Compared with existing methods that simply optimize per-vertex color and displacement, it adopts a more physics-aware rendering model that explicitly models geometry (normal map), BRDF (diffuse, roughness, specular), and lighting (SG), which is claimed to generate more realistic results (better shading appearance and details) and allows for editing such as relighting. Results are compared with Text2Mesh.", "questions": "I have put my major concerns in the weaknesses section. Overall, I think it is an interesting work that combines neural rendering with mesh stylization that could produce meaningful assets that are more compatible with current rendering pipelines and allow for more wide editing applications. However, the presented results, unfortunately, do not fully the potential of the proposed method -- there are no results showing the quality of individual components and the possibility to edit one of them for editable stylization, without these I am not convinced that this method has achieved significant improvement over Text2Mesh except some better shading details. Also considering that this method is not able to produce real geometric details, and the visual results on single stylization images are not clearly better than Text2Mesh, I cannot say this method advances the state-of-the-art. Finally, given the limited technical contributions, I am leaning a bit negative unless more convincing evidence could be presented.", "limitations": "There are discussions on limitations.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "++ Indeed very promising idea to do physics-aware modeling of mesh stylization.\n\n++ Interesting visual results are presented, which look good.\n\n++ The proposed rendering formulation is technically sound and practically reasonable.\n\n++ Very good paper-writing, except for some redundancy that could be made more concise.\n\n-- All results are just final renderings, lacking results to show the individual components of normal maps, BRDFs, and lightings, which are critical to show the quality of the system and its advantage over Text2Mesh.\n\n-- No results on manipulating the rendering, such as relighting (the relighting results on the shoe are less interesting for stylization) and material editing, which are not achievable by Text2Mesh.\n\n-- Approximating geometry with normal maps turns the method into a pure appearance stylization method. There are no actual geometry changes but just a way to model geometry variation for shading. But Text2Mesh does achieve joint geometry and appearance stylization, which seems better than this method although the geometric details have limitations.\n\n-- Limited contributions, the rendering pipeline, and CLIP-guided stylization are mostly based on existing works.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZgOJ3FDGlJu", "reviewer": "Reviewer_8TXp", "summary": "This paper focuses on the task of 3D mesh stylization according to the text prompt. The key idea is to jointly learn three disentangled components, ie, the spatially varying bidirectional reflectance distribution function, the local geometric variation, and the lighting condition, under the supervision of CLIP Loss with a spherical Gaussians-based differentiable renderer. Experiments show that the proposed method achieves better visualization quality and 3D geometry consistency on stylized meshes compared to the state-of-the-art. Besides, this method demonstrates robustness when stylizing low-quality meshes.", "questions": "1. I wonder why the authors adopt “randomly cropped images” rather than “multi-view images” that text2mesh adopts in the phase of data augmentation. Is it because that normal map helps the model focus on the geometry and cropped images guide the model to focus on the detail? If text2mesh adopts cropped images in data augmentation, will it produce more details? An ablation study should be conducted here.\n2. What is the purpose of adding a random background to the rendered image in data augmentation? Is it necessary? I do not see the ablation study about the background in experiments.\n3. Is it easy to explicitly control the lighting intensity and direction? How should we determine the light distribution?", "limitations": "CLIP model may produce semantic ambiguity. It seems that the target text has to contain the category of input mesh for content preservation, which may limit the application of this method in real life.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This is an interesting idea that may inspire future research in related fields. It employs a professional shading model to help learn more photorealistic appearance and geometry, which is intuitively reasonable and practically effective.\n2. The paper is well-organized and easy to follow.\n3. The experiments look convincing since it supports the major contributions claimed in the introduction that the realism of rendered images can be improved by the explicit modeling of light and materials. Moreover, the ablation study also demonstrates that each disentangled component plays an important role in the improvement of the rendering quality.\n4. The authors also provide some reasonable limitations of the current method, which may help researchers to build new frameworks based on current results.\n\nWeakness:\n1. This paper proposes to learn materials, normals, and light separately. How can **a good disentanglement** of light and physical reflectance be guaranteed? In the whole pipeline, only a CLIP loss is used. It seems like the network here has no motivation to learn a good disentanglement. In Fig4, the relighting results are so weird since the materials are also modified here. How about changing the light only?\n2. The quality of 3D stylization results may not be impressive enough in Fig.2. It is not easy to evaluate the performance of the proposed method from the target text prompt. Such as “A shoe made of brick”, I know the proposed method produces more details, but it seems that the result of test2mesh is more natural. A similar situation occurs in \"A vase made of colorful crochet\". It is hard to judge between this method and Text2Mesh. Of course, this method can preserve better 3D geometry. However, is geometry preservation necessary in 3D stylization? By the way, there are a lot of clear artifacts in the case of \"A candle wearing jeans\". I think more convincing visualization results can be provided.\n3. The quantitative result in Table 1 is unclear. As this work splits the style into geometry and appearance and claims the improvement in both, user studies should be conducted separately in terms of both aspects.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UHul6mOdKy", "reviewer": "Reviewer_ojBL", "summary": "The paper describes a method for generating a reflectance map (and lighting environment) for a given 3D shape, based on a text prompt, using a CLIP-guided loss.", "questions": "Are there any constraints on the normal fitting, e.g., do the normals need to be consistent with the geometry or can they be totally unrelated? Presumably there is a limit to how far normals can vary from the normals of the input geometry (or the nearby vertex normals).\n\n", "limitations": "One limitation is that stylization may involve creating new geometry, not just texture. \"text2mesh\" does create new geometry, which has visible pros and cons in the results.\n\nAs discussed above, it's unclear whether good materials+lighting have been recovered, or just enough to render views with static lighting.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem being tackled is novel, and several of the results are plausibly-good. The impact of the work is incremental, but the general direction of text-guided geometry/texture generation is a significant one. \n\nOne big gap in the paper for me is a clear explanation for the motivation for using the SVBRDF/env. map rather than just fitting a texture map. It's clear that the results look better than just fitting a texture map (as shown in the ablations), so this aspect is clear. But, by fitting materials+lighting, the paper gives the possible impression that one gets a good materials+lighting in general, not just for a single view. Is this true? It hasn't been demonstrated or tested. Specifically: can the model be relighted and the viewpoint moved with sensible results? For example, do shiny materials have highlights that move in sensible ways, and diffuse materials do not? etc.\n\nI think the description of the method is unnecessarily mathematical and opaque, with many different terms. However, I don't have any suggestions for improving it and it might be that this is necessary level of detail. Eq 1 has a lot of terms but, in a way, it all boils down to one integral with a bunch of parameters.\n\nNote that \"text2mesh\" was published at CVPR 2022 after the NeurIPS deadline, so I don't think it \"counts\" as prior work. Nonetheless, it's great that it's included as a baseline, which is very informative.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Q1194HZhOBL", "reviewer": "Reviewer_qBos", "summary": "This paper targets at stylizing a mesh with a text prompt. It represents appearance of a mesh as spatially varying BRDF, normal and lighting, and matches the CLIP feature of rendered image and given text prompt by learning the parameters of these components. For evaluation, the paper compares with the state-of-the-art text prompt stylization method Text2Mesh and achieves better performance.", "questions": "**Method.**\nI am actually surprised this model can predict correct lighting parameters given only CLIP latent space as supervision. Have the authors observed any ambiguity in disentangling lighting and reflectance? If not, what is the key to make the model learn correct disentanglement?\n\n**Implementation details.**\n- How to find reasonable anchor view in line 145?\n- How to find the first intersection point and intersection face in line 150?\n\n**Novelty.**\nAlthough not using text prompt as supervision, a similar lighting model has been explored in the NeRD [4] paper, while the text prompt stylization has been used by Text2Mesh. Thus although the idea is interesting, the novelty and contribution of the proposed method is limited, in my opinion.", "limitations": "The authors have discussed the limitations of the chosen lighting model. As stated in the weakness, I encourage the authors to include \n the limitation of not modeling geometric displacement. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths.**\n- This paper is well written and easy to follow.\n- The idea of representing the appearance of a given mesh as SVBRDF, normal and lighting is interesting, although its has certain limitations.\n\n**Weaknesses.**\n- Modeling appearance as a lighting model certainly performs better in complex realistic scenes. However, it also limits the capacity of the model without geometric displacements modeling. For instance, in Fig.3 in the Text2Mesh paper, the surface of the given mesh can be changed to match the text prompt, while the proposed method cannot do so. This limitation is not discussed in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept