Title: ITEM3D: Illumination-Aware Directional Texture Editing for 3D Models
Venue: NeurIPS 2023
Accepted: no
Keywords: Texture editing

diffusion model

relative direction

direction adjustment

relighting
Abstract: Texture editing is a crucial task in 3D modeling that allows users to automatically manipulate the surface properties of 3D models. However, 
the inherent complexity of 3D models and the ambiguous text description lead to the challenge in this task. 
To address this challenge, we propose ITEM3D, an illumination-aware model for automatically 3D object editing according to the text prompts. 
Leveraging the power of diffusion model, ITEM3D takes the rendered images as the bridge of text and 3D representation and further optimizes the disentangled texture and environment map.
Previous methods adopt the absolute editing direction namely score distillation sampling (SDS) as the optimization objective, which unfortunately results in the noisy appearance and text inconsistency.
To solve the problem caused by the ambiguous text, we introduce a relative editing direction, a optimization objective defined by the noise difference between the source and target texts, to release the semantic ambiguity between the texts and images.
Additionally, we gradually adjust the direction during optimization to further address the unexpected deviation in texture domain. 
Qualitative and quantitative experiments show that our ITEM3D outperforms SDS-based methods on various 3D objects. We also perform text-guided relighting to show the explicit control over lighting.

Abstract

Texture editing is a crucial task in 3D modeling that allows users to automatically manipulate the surface properties of 3D models. However, the inherent complexity of 3D models and the ambiguous text description lead to the challenge in this task. To address this challenge, we propose ITEM3D, an illumination-aware model for automatic 3D object editing according to the text prompts. Leveraging the power of the diffusion model, ITEM3D takes the rendered images as the bridge of text and 3D representation, and further optimizes the disentangled texture and environment map. Previous methods adopt the absolute editing direction namely score distillation sampling (SDS) as the optimization objective, which unfortunately results in the noisy appearance and text inconsistency. To solve the problem caused by the ambiguous text, we introduce a relative editing direction, an optimization objective defined by the noise difference between the source and target texts, to release the semantic ambiguity between the texts and images. Additionally, we gradually adjust the direction during optimization to further address the unexpected deviation in the texture domain. Qualitative and quantitative experiments show that our ITEM3D outperforms SDS-based methods on various 3D objects. We also perform text-guided relighting to show explicit control over lighting.

1 Introduction

Texture editing is an important task in 3D modeling that involves manipulating the surface properties of 3D models to create a visually fantastic and appealing appearance according to the user's ideas. With the increasing applications of 3D models in entertainment and e-shopping, how to automatically generate and edit the texture of a 3D model without manual effort becomes an appealing task in the field of 3D vision. However, this task is challenging due to the complexity of 3D models and the special representation of the texture.

To sufficiently handle the above applications, it would be desirable if a texture editing method can fulfill the following aspects: 1) Realism : The generated textures should give rise to realistic and visually natural 2D images after rendering. It requires generative models to capture the complex patterns and structures present in the textures of the 3D model. 2) Relighting : The relighting ability allows adjusting the lighting conditions of the edited model to be consistent with the changes made to its texture. 3) Efficiency : Texture editing should be efficient and scalable. This requires the use of fast and memory-efficient generative models that can generate high-quality textures in a short time.

Recent advances have demonstrated the effectiveness of generative models in synthesizing high32 quality textures that are both visually pleasing and semantically meaningful. The use of generative 33 adversarial networks (GANs) (50; 2; 43; 7) has shown promising results in producing textures with 34 intricate patterns and complex structures. Other approaches, such as texture synthesis via direct 35 optimization (8; 9; 40; 54) or neural style transfer (3; 14; 48; 24), have also been explored for their 36

ability to generate textures with specific artistic styles. However, the capacity of these models is still 37 unable to meet the need of real-world applications, which requires high-quality and diverse textures. 38 Meanwhile, recent researches (6; 34; 25; 20; 21; 17; 23) on the diffusion models have emerged as a 39 powerful new family of generative methods, which achieve impressive generation results in natural 40 images and videos, inspiring us to introduce the awesome power into the task of 3D modeling. 41

42

43

44

45

46

47

48

49

50

However, directly applying the diffusion model to 3D objects is a non-trivial task due to the following reasons. 1) The gap between the 3D representation and natural images. Existing diffusion models are typically trained with natural images, making the pre-trained diffusion model lack prior knowledge in the 3D domain. Moreover, due to the complexity of the 3D model, it would be difficult to simultaneously edit shape, appearance, and shading, sometimes leading to conflicts in optimization goals. Therefore, directly editing the 3D representation may cause extreme semantic bias and destruction of inherent 3D topology. 2) The learning misdirection of text description. It is hard for text prompts to exactly describe the target images at the pixel level, leading to an ambiguous direction when taking the rendered images as the bridge.

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

To solve these problems, we present an efficient model, dubbed ITEM3D, which can generate visually natural texture corresponding to the text prompt generated by users. Instead of directly applying the diffusion model for texture editing in the 2D space, we adopt rendered images as the intermediary that bridges the text prompts and the appearance of 3D models. Apart from the appearance, the lighting and shading are also key components influencing the rendering results. Therefore, we represent the 3D model into a triangular mesh and a set of disentangled materials consisting of the texture and an environment map using nvdiffrec (29), which achieves a balance for representing both appearance and shading.

To optimize a texture and an environment map with the diffusion model, a naive idea is to adopt the score distillation sampling (SDS) like 2D diffusion-based editing methods, which represents the absolute direction. Unfortunately, the absolute direction often leads to noisy details and an inconsistent appearance, due to the ambiguous description of the text prompt for the target images. Inspired by the recent improvement (13), we replace the absolute editing direction led by the score distillation sampling with a relative editing direction determined by two predicted noises under the condition of the source text and the target text respectively, as illustrated in Fig. 1 (a). In this way, our model enables us to edit the texture in obedience to the text while bypassing the inconsistency problem by releasing the ambiguous description. It is ideal that the intermediate states between the source and target text can give relatively accurate descriptions for arbitrary rendered images during the optimization, like the green straight lines in Fig. 1 (b). However, the optimization in the texture domain actually shows an unexpected offset of the appearance in rendered images, leading to the deviation from the determined direction, like the red line in Fig. 1 (b). To reduce the deviation caused by the texture projection, we gradually adjust the editing direction during the optimization, as green fold lines shown in Fig. 1 (b). With the advent of the textural-inversion model, it can be easy to automatically correct the description as the change of the texture and its rendered images.

Thanks to the proposed solutions, our method overcomes the challenges of domain gap and learning misdirection, fulfilling all three requirements of texture editing. In summary, our contributions are:

We design an efficient optimization pipeline to edit the texture and environment map obedient to the text prompt, directly empowering the downstream application in the industrial pipeline.

We introduce the relative direction to the 3D texture optimization, releasing the problem of noisy details and inconsistent appearance caused by the semantic ambiguity between the texts and images.

We propose to gradually adjust the relative direction guided by the source and target texts which addresses the unexpected deviation from the determined direction caused by the texture.

2 Related Work

3D Model Representation. From the perspective of 3D representations, traditional methods typically 85 exploit point clouds or meshes to estimate depth maps (1; 38; 11) or employ a voxel grid and estimate 86 the corresponding occupancy and color (39; 4). However, these methods are often limited to the 87 memory requirement, which results in excessive runtime. With the development of computer vision, 88 neural implicit representations are brought up and leverage differentiable rendering to reconstruct 89

一

一

nbias!

adjustment

一

"A yellow toy duck"

"A red toy duck"

"A yellow toy duck

"A red toy duck"

(a)RelativeDirection

(b)DirectionAdjustment

3D geometry with appearance. Neural Radiance Field (27) and followup methods (52; 47; 28; 5; 90 49; 51; 33; 22), utilize volumetric representations and a neural encoded field to compute radiance 91 by ray marching. While these NeRF-based methods synthesize high-fidelity rendering results, the 92 quality of the generated geometry is limited due to the ambiguity of volume rendering. Meanwhile, 93 surface-based methods (30; 46; 10) optimize the underlying surface directly. These methods usually 94 rely on volumetric representation and utilize an implicit surface by converging the volumetric 95 representation (30) or constructing a field that converse SDF into density (46; 10). Though surface96 based methods achieve better geometry than NeRF-based methods, they require excessive computation 97 runtime since they rely too much on the ray-marching mechanism. Apart from implicit neural 98 representations, there also exist approaches that utilize explicit surface representations to estimate 99 explicit mesh from images. To extend such methods that originally built upon a fixed mesh topology, 100 DMTet (41) employs a differentiable marching tetrahedral layer and optimizes the surface mesh 101 directly. Nvdiffrec (29) further extends DMTet by jointly optimizing mesh topology, materials, and 102 lighting. ITEM3D leverages an explicit mesh representation and optimizes texture and environment 103 map. By supporting the decomposition of shape, materials and lighting, ITEM3D supports texture 104 editing while preserving the topology by design. Additionally, ITEM3D employs an efficient 105 differentiable rasterization pipeline for faster optimization. 106

107

108

109

110

111

112

113

114

115

116

117

118

119

2D Diffusion-based Image Editing. Owing to the remarkable generalization ability of the diffusion model, a growing number of works (23; 17; 35; 36; 42) emerged to create customized images with specific styles or objects, as well as stunning images based on text descriptions. All these methods rely on the diffusion process by either fine-tuning the diffusion model or refining the target embedding to reach the desired image domain. SDEdit (23) denoises the noisy image through a diffusion process under the given description. DDIB (42) first converts the input image into a latent representation using origin text and subsequently translates the latent into the desired image with the target text. ControlNet (53) trains a controlling module to augment images with additional conditions that improve the controllability of the editing process. DiffusionCLIP (18) fine-tunes the diffusion model, which translate the image from a pretrained domain to a target text domain. Imagic (17) fine-tunes both the text embeddings and the diffusion model to ensure more stable editing. Unlike these methods that optimize in 2D image space, our ITEM3D utilizes the pre-trained diffusion model as a prior for 3D texture optimization.

120

121

122

123

124

125

126

3D Text-guided Generation. With the advent of large text-image models, i . e ., the CLIP, recent works (45; 37; 16; 15; 26) have made impressive progress on 3D text-driven synthesis. The majority of methods adopt the optimization procedures supervised by the CLIP similarity (32). Specifically, CLIP-NeRF (45) proposes a unified framework to manipulate NeRF, guided by a text prompt or an example image. Similarly, CLIP-Mesh applies the explicit textured mesh as a 3D representation, able to deform the shape along with its texture corresponding to the text. Apart from the CLIP-based method, the diffusion model (35) recently inspires huge breakthroughs in 3D text-guided generation.

Latent-NeRF (25) utilizes the score distillation sampling to bring the NeRF representation to the 127 latent space, showing impressive generation results of the combination between diffusion model and 128 NeRF. TEXTure (34) takes an iterative scheme to paint a 3D model from different viewpoints based 129 on a pre-trained depth-to-image diffusion model. Fantasia3D (6) leverages the disentangled modeling 130 and learns the geometry and appearance supervised by the score distillation sampling. However, 131 these SDS-based methods often produce non-detailed and blurry outputs due to noisy gradients. In 132 contrast, our ITEM3D uses the relative direction to eliminate the semantic ambiguity of the target 133 prompt towards the texture. 134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

3 Method

3.1 Overview

Given a set of multi-view images I = { I 1 , ..., I n } , we aim to reconstruct the 3D model with both geometry and texture, and then edit the texture under the guidance of text prompts. To this end, we design a zero-shot differentiable framework that optimizes the disentangled materials of the object, i . e ., texture map and environment map. We first leverage a differentiable rendering model R to represent the 3D model as an accurate shape and surface materials with texture and environment map Sec. 3.2). For further editing of appearance, we utilize the diffusion model to guide the direction of the texture optimization given the target text prompt. To solve the problem of ambiguous and noisy details, we introduce the relative direction of source text and target text into the optimization (Sec. 3.3). Moreover, we gradually adjust relative direction to address the challenges of deviation caused by the unbalanced optimization in the texture domain (Sec. 3.4). The overview of our method is demonstrated in Fig. 2.

3.2 3D Model Representation

To accomplish editing the appearance of the 3D model via text prompt, we disentangle the 3D model into a triangular mesh and a set of spatially varying materials. The disentanglement thus allows us to edit the texture directly while keeping the geometry invariant. The material model we employed combines a diffuse term, a specular term and a normal term. A four-channel texture is provided for the diffuse parameters k d , where the optional fourth channel α represents the transparency. Meanwhile, the specular term is described by a roughness factor r , a metalness factor m and a sheen factor o that is unused in our model. These values ( o, r, m ) are stored in another texture map k orm . The normal term in our representation is a tangent space normal map n , which is utilized to capture the high-frequency details of the appearance. In order to handle texturing effectively during optimization, we utilize volumetric texturing and access our texture by the world space position x . We tackle the challenge of the impractical cubic growth in memory usage of volumetric textures for our target resolution by leveraging a multi-layer perceptron (MLP) to encode the material parameters into a compact representation. Specifically, given a world space position x , we compute the base color, k d , the specular parameters, k orm and a tangent space normal perturbation n , the mapping is thus formulated as x → ( k d , k orm , n ) . With the introduction of this mapping, for a fixed topology, the textures are initialized by sampling the MLP on the mesh surface and then optimized efficiently. Following the rendering equation of the image-based lighting model, we compute the radiance L in direction ω o by:

where L i is the incident radiance from direction ω i , the f is the BSDF and n is the intersection normal of the corresponding integral domain Ω . Specifically, we adopt the Cook-Torrance microfacet specular shading model as the BSDF in our rendering equation:

The term D here represents the GGX (44) normal distribution while the term G is the geometric atten170 uation and F is the Fresnel term respectively. Furthermore, we employ the split-sum approximation 171 for its efficiency and the rendering equation Eq. (1) can be formulated as: 172

197

"A green chair"

e(Zt, yo,t)

U-net

Relative

Texture

E

3+

Direction

Loss

Z

U-net

Differentiable

[RenderedImage

Renderer

Env. Lighting

"A green wooden chair"

e$(Zt, Ytgt,t)

Optimization

The first term of this product only relies on the parameters ( ω i · n ) and the roughness r of the BSDF, 173 which are precomputed and stored in a 2D lookup texture. Meanwhile, the second term is the integral 174 of the radiance with the specular normal distribution function D expressed in Eq. (2), which is also 175 precomputed and stored by a filtered cubemap. Owing to the precomputation and lookup mechanism, 176 the rendering process is then accelerated. In order to learn the environment lighting from 2D image 177 observations, we employ a differentiable shading model to represent this split-sum approximation. 178 The cube map in our case can be represented as trainable parameters, which are initialized as the 179 preintegrated lighting. 180

181

182

183

184

185

3.3 Relative Direction Based Optimization

Our goal is to enable users to edit the appearance of 3D models using natural language descriptions. To accomplish this, the directional idea is to utilize the diffusion model that has been pre-trained in 2D images as knowledge prior to guide the editing of texture. Naively, we could use Score Distillation Sampling (SDS) loss,

where x is the rendered images, t is the sampled time step, z t is the t time step latent, w ( t ) is the 186 weighting function that equals ∂ z t /∂ x , y is the text condition, ϵ ω ϕ ( z t ; y, t ) is the predicted noise 187 through classifier-free guidance, and ϵ ∈ N (0 , I ) is the noise added to the rendered images. The 188 gradient of SDS loss gives an editing direction for our texture optimization, determined by the text 189 prompt y . However, the SDS loss may cause the destruction of original image content with noisy 190 details, because the text prior typically cannot faithfully reflect the information of the image. It is 191 known that the entropy of an RGB image is significantly larger than that of a text prompt. As a 192 consequence, the misdescription inevitably arises when taking the text prompt as the prior to restore 193 the high-quality image from the same-scale noise. Therefore, even for a text prompt y 0 describing the 194 original images, there exists a deviation related to the optimized texture θ between the added noise ϵ 195 and the predicted noise ϵ ω ϕ ( z t ; y 0 , t ) , which can be simply expressed as, 196

Thus, the gradient leads to a bias term from the original input image, which can be expressed as,

Moreover, for an arbitrary text prompt y tgt describing the target editing texture, it could be considered 198 that there exists a term of expected editing direction and a term of bias discussed above, 199

200

201

202

203

204

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

As a result, the ⃗ n bias gives rise to the misdirection for the optimization procedure.

To address these issues, it is ideal to find the accurate editing direction ⃗ n tgt , while the term of ⃗ n bias is hard to estimate due to the diverse input images. To mitigate the gap, it is natural to take the text guidance as a relative direction rather than an absolute direction, enabling us to eliminate the term of ⃗ n bias . The absolute direction of the source ⃗ n src and the target ⃗ n tgt can be expressed as,

where ⃗ n src is actually the ⃗ 0 giving no extra information to the input images. Inspired by the CLIP205 directional loss improved by the StyleGAN-Nada (12) and the denoising loss proposed by the recent 206 work (31), we utilize the difference between the source ⃗ n src and the target ⃗ n tgt as the relative direction 207 of the target, which can be presented as, 208

Therefore, the final gradient utilized for optimizing the texture can be presented as,

3.4 Direction Adjustment

Different from the gradual transition in the nature image domain, the optimization of the texture domain unfortunately shows an unexpected offset of the appearance in rendered images, due to the complex projection in differentiable rendering. The inherent reason is that the complexity of rendering leads to unbalanced optimization for the texture, with some parts under-tuning and other parts over-tuning. This appearance offset can be seen in some parts of the rendered image, leading to the inconsistency between the source text and the rendered images in the median period of the optimization procedure. It is known that a source image with an inconsistent text description means an optimization misdirection which leads to an unknown change in the editing results. Similar to the known problem, if a rendered image during the median optimization hops out the direction between the source text and the target text, it can be considered as the inconsistent description for the source image when we take the current median point as a relative beginning point. The original editing direction is give by,

If the optimization continues along the original direction, a more severe deviation can be attached to the optimization procedure.

To avoid the misdirection caused by the texture domain, we propose to adjust the editing direction, specifically the source text prompt, during our optimization process of the texture map. The adjusted direction ∆ ˆ T i can be represented as,

where i is the optimization iteration and B ( · ) expresses the inverse text generated by a pre-trained language-image model BLIP-v2 (19).

As shown in the Fig. 1 (b), the direction is continually adjusted during the optimization so that the new global direction ⃗ n i can be written as,

By adjusting the optimization direction step by step, we achieve more delicate and controllable editing, which can be seen in Sec. 4.3.

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

Origin

ITEM3D

SDS-based

Origin

ITEM3D

SDS-based

Texture

Rendered

Image

"A ficus with blue pot"

"A red ficus"

Texture

Rendered

Image

"A red chair"

"A wooden chair"

4 Experiments

4.1 Implementation Details

Dataset. In the experiments, we mainly evaluate our model on the NeRF Synthetic (27) dataset. The NeRF synthetic dataset consists of 8 path-traced scenes with multi-view images which we reconstruct into our textural mesh-based representation via nvdiffrec (29). Besides, we adopt 3D objects from Keenan's 3D model repository.

Experiment Setup. We optimize the 3D model on one RTX A6000 GPU with 48G memory. The optimization procedure lasts about average 500 iterations with 8 minutes for each 3D model. We use the Adam optimizer for both the texture and the environment map with an initial learning rate of 0.01 which gradually decreases to 1/10 every 5k iterations during the training process.

4.2 Comparison with Baseline

Qualitative Comparison. We compare ITEM3D with the optimization method based on the SDS loss. Specifically, Fig. 3 shows the results of editing texture and rendered image with the guidance of text prompts. While SDS-based method could edit textures along the direction of text prompt, their rendered images show the unrealistic appearance, sometimes overfitting to the text. In contrast,

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

the texture edited by our ITEM3D can render realistic images with high quality, while remaining consistent with the input text prompt. The comparison indicates the effectiveness of the introduced relative direction of optimization and further direction adjustment. Besides, it can be noticed that our methods support segmentation-aware editing. Although the diffusion model lacks the capacity of recognizing the semantics in the texture map, it enables to edit the specific part of texture corresponding to a text prompt describing partial change. For example, with the prompt 'A ficus with blue pot', the change in the texture precisely reflects to the part of the pot in the rendered images. It proves that the gradients can accurately back-propagate to the corresponding parts of the texture map via the rendered images.

Quantitative Comparison. Moreover, we conduct a quantitative comparison in the Tab. 1. To evaluate the semantic consistency, we choose objects from Keenan's 3D Model Repository, render their 512 × 512 RGBimages after texture editing, and further compute the CLIP-Score of the rendered image and corresponding target text. CLIP-score contains two parts, i . e ., global score and directional score. Global score measures the similarity between the target text and the editing images, and directional score measures the similarity between two editing directions of text prompts and images, which are expressed as which can be presented as,

where T tgt and I tgt are the embedding of target text and edited image encoded by the CLIP encoder, and ∆ T and ∆ I are expressed as,

As illustrated in Tab. 1, our method achieves better results than the SDS-based method.

User Study. Additionally, we perform a user study in Tab. 2 to further assess the quality of editing objects. Users are required to rate on a scale of 1 to 5, based on the following questions: (1) Are the edited objects realistic and natural (Photorealism)? (2) Are the edited objects accurately reflect the target text's semantics (Text Consistency)? As presented in Tab. 2, the results demonstrate the superior quality with higher realism and more text consistency of our proposed method as compared to the baselines.

4.3 Direction Adjustment

In this section, we further study the necessity of direction adjustment. We perform the ablation study in Fig. 4. Without the adjustment for the relative optimization direction, the texture shows a wired change that the duck gradually generates two heads and the color seems partially yellow and partially red. When applying the gradual adjustment, the duck bypasses the unnatural change and smoothly achieves the target appearance. The example of cattle shows a similar trend. In this experiment, it can be noticed that there exists unbalanced optimization for different parts of the

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

A brassy cattle'

A red stone duck'

Origin

w/o adj.

w/ adj.

Origin

w/o adj.

w/adj.

Origin

"Hotdogs on the plate at sunrise"

"Hotdogs on the plate inbright light"

Origin

"Reflective metal ball in dazzling light” “Reflective metal ball under strong light'"'

texture. The optimization scheme of simple pieces of texture converges quickly, while more complex modifications require longer time, which in turn over-tunes easy parts leading to poor results. We also compute the two CLIP score for the results without direction adjustment in Tab. 1. It shows that the adjustment indeed helps to maintain the major semantics.

4.4 Illumination-aware Editing

The disentangled representation of environment map empowers ITEM3D to explicitly control the lighting under the guidance of a text prompt aiming to relight the 3D model. The results of illuminationaware editing are demonstrated in Fig. 5. As shown, given the prompt including lighting information such as 'sunrise', 'bright light', and 'dazzling light', ITEM3D enables to edit the environment map along the direction led by the prompt. It is valuable to prove that the lighting condition of a 3D model can be learned solely from the text through the bridge of rendered 2D images.

5 Conclusion and Limitations

In conclusion, our ITEM3D model presents an efficient solution to the challenging task of texture editing for 3D models. By leveraging the power of diffusion models, ITEM3D is capable to optimize the texture and environment map under the guidance of text prompts. To address the semantic ambiguity between text prompts and images, we replace the traditional score distillation sampling (SDS) with a relative editing direction. We further propose a gradual direction adjustment during the optimization procedure, solving the unbalanced optimization in the texture.

Despite the promising editing results, our ITEM3D still remains several limitations which should be solved in future work. The major limitation is that there remains irremovable noise in some samples. Because of the synthesis mechanism of the diffusion model, our ITEM3D extremely depends on the denoising ability of the pre-trained U-Net. Another limitation is that the adjustment by the source description is non-essential. Our further work aims to explore the learning scheme to solve the problem of unbalanced optimization in the texture.

References 305

Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M. Seitz, and 306 Richard Szeliski. Building rome in a day. Commun. ACM , 54(10):105-112, 2011. 307

Urs Bergmann, Nikolay Jetchev, and Roland Vollgraf. Learning texture manifolds with the periodic spatial 308 GAN. In ICML , volume 70, pages 469-477, 2017. 309

310

[3]

Sema Berkiten, Maciej Halber, Justin Solomon, Chongyang Ma, Hao Li, and Szymon Rusinkiewicz.

Learning detail transfer based on geometric features. Comput. Graph. Forum , 36(2):361-373, 2017. 311

Jeremy S De Bonet and Paul Viola. Poxels: Probabilistic voxelized volume reconstruction. In ICCV , 1999. 312

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In 313 ECCV , pages 333-350, 2022. 314

Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance 315 for high-quality text-to-3d content creation. arXiv preprint arXiv:2303.13873 , 2023. 316

Zhuo Chen, Xudong Xu, Yichao Yan, Ye Pan, Wenhan Zhu, Wayne Wu, Bo Dai, and Xiaokang Yang. 317 Hyperstyle3d: Text-guided 3d portrait stylization via hypernetworks. arXiv preprint arXiv:2304.09463 , 318 2023. 319

Alexei A. Efros and Thomas K. Leung. Texture synthesis by non-parametric sampling. In ICCV , pages 320 1033-1038, 1999. 321

Anna Frühstück, Ibraheem Alhashim, and Peter Wonka. Tilegan: synthesis of large-scale non-homogeneous 322 textures. ACM Trans. Graph. , 38(4):58:1-58:11, 2019. 323

Qiancheng Fu, Qingshan Xu, Yew Soon Ong, and Wenbing Tao. Geo-neus: Geometry-consistent neural 324 implicit surfaces learning for multi-view reconstruction. NeurIPS , 35:3403-3416, 2022. 325

Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IEEE Trans. Pattern 326 Anal. Mach. Intell. , 32(8):1362-1376, 2010. 327

Rinon Gal, Or Patashnik, Haggai Maron, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. Stylegan328 nada: Clip-guided domain adaptation of image generators. ACM Trans. Graph. , 41(4):141:1-141:13, 329 2022. 330

Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. arXiv preprint arXiv:2304.07090 , 331 2023. 332

Amir Hertz, Rana Hanocka, Raja Giryes, and Daniel Cohen-Or. Deep geometric texture synthesis. ACM 333 Trans. Graph. , 39(4):108, 2020. 334

Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object 335 generation with dream fields. In CVPR , pages 867-876, 2022. 336

Nikolay Jetchev. Clipmatrix: Text-controlled creation of 3d textured meshes. arXiv preprint 337 arXiv:2109.12922 , 2021. 338

Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal 339 Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276 , 340 2022. 341

Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for 342 robust image manipulation. In CVPR , pages 2426-2435, 2022. 343

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training 344 with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023. 345

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, 346 Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv 347 preprint arXiv:2211.10440 , 2022. 348

Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 349 Zero-1-to-3: Zero-shot one image to 3d object. arXiv preprint arXiv:2303.11328 , 2023. 350

Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and 351 Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. arXiv 352 preprint arXiv:2008.02268 , 2020. 353

Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image 354 synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073 , 2021. 355

Tom Mertens, Jan Kautz, Jiawen Chen, Philippe Bekaert, and Frédo Durand. Texture transfer using 356 geometry correlation. In Proceedings of the Eurographics Symposium on Rendering Techniques , pages 357 273-284, 2006. 358

Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape359 guided generation of 3d shapes and textures. arXiv preprint arXiv:2211.07600 , 2022. 360

Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural 361 stylization for meshes. In CVPR , pages 13492-13502, 2022. 362

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 363 Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , pages 99-106, 2020. 364

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives 365 with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) , 41(4):1-15, 2022. 366

Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, 367 and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. In CVPR , pages 368 8280-8290, 2022. 369

Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and 370 radiance fields for multi-view reconstruction. arXiv preprint arXiv:2104.10078 , 2021. 371

Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. 372 arXiv preprint arXiv:2209.14988 , 2022. 373

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish 374 Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from 375 natural language supervision. In ICML , pages 8748-8763, 2021. 376

Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance 377 fields with thousands of tiny mlps. arXiv preprint arXiv:2103.13744 , 2021. 378

Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text-guided 379 texturing of 3d shapes. arXiv preprint arXiv:2302.01721 , 2023. 380

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution 381 image synthesis with latent diffusion models. In CVPR , pages 10684-10695, 2022. 382

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar 383 Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to384 image diffusion models with deep language understanding. NeurIPS , 35:36479-36494, 2022. 385

Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Ka386 mal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In CVPR , pages 387 18603-18613, 2022. 388

Johannes L. Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view 389 selection for unstructured multi-view stereo. In ECCV , volume 9907, pages 501-518, 2016. 390

Steven M. Seitz and Charles R. Dyer. Photorealistic scene reconstruction by voxel coloring. Int. J. Comput. 391 Vis. , 35(2):151-173, 1999. 392

Omry Sendik and Daniel Cohen-Or. Deep correlations for texture synthesis. ACM Trans. Graph. , 393 36(5):161:1-161:15, 2017. 394

Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a 395 hybrid representation for high-resolution 3d shape synthesis. NeurIPS , 34:6087-6101, 2021. 396

Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to397 image translation. In ICLR , 2022. 398

Jingxiang Sun, Xuan Wang, Yichun Shi, Lizhen Wang, Jue Wang, and Yebin Liu. IDE-3D: interactive 399 disentangled editing for high-resolution 3d-aware portrait synthesis. ACM Trans. Graph. , 41(6):270:1400 270:10, 2022. 401

Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. Microfacet models for 402 refraction through rough surfaces. In Proceedings of the Eurographics Symposium on Rendering Techniques , 403 pages 195-206, 2007. 404

Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and-image 405 driven manipulation of neural radiance fields. In CVPR , pages 3835-3844, 2022. 406

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: 407 Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint 408 arXiv:2106.10689 , 2021. 409

Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance 410 fields without known camera parameters. arXiv preprint arXiv:2102.07064 , 2021. 411

Zhizhong Wang, Lei Zhao, Haibo Chen, Ailin Li, Zhiwen Zuo, Wei Xing, and Dongming Lu. Texture 412 reformer: Towards fast and universal interactive texture transfer. In AAAI , pages 2624-2632, 2022. 413

Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn. Nex: 414 Real-time view synthesis with neural basis expansion. In CVPR , pages 8534-8543, 2021. 415

Wenqi Xian, Patsorn Sangkloy, Varun Agrawal, Amit Raj, Jingwan Lu, Chen Fang, Fisher Yu, and James 416 Hays. Texturegan: Controlling deep image synthesis with texture patches. In CVPR 417

, pages 8456-8465, 2018. 418

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time 419 rendering of neural radiance fields. In ICCV , pages 5752-5761, 2021. 420

Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural 421 radiance fields. arXiv preprint arXiv:2010.07492 , 2020. 422

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv 423 preprint arXiv:2302.05543 , 2023. 424

Yang Zhou, Zhen Zhu, Xiang Bai, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Non-stationary 425 texture synthesis by adversarial expansion. ACM Trans. Graph. , 37(4):49, 2018. 426

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe research introduces ITEM3D, a model that enhances texture editing in 3D modeling, addressing challenges like complexity and text ambiguity. Leveraging a diffusion model, ITEM3D uses images to bridge text and 3D representations, optimizing texture and environment map. It employs a relative editing direction, reducing noise difference and semantic ambiguity between source and target texts, and adjusts direction during optimization to limit texture domain deviation. Experiments show ITEM3D outperforms previous methods and can effectively control lighting.\n\nSTRENGTHS:\n+ The research introduces an optimization pipeline for texture and environment map editing in 3D models, adhering to text prompts. \n+ This paper presents a relative direction diffusion-based approach for 3D texture optimization, mitigating issues of noisy details and inconsistent appearances resulting from semantic ambiguity between texts and images.\n\nWEAKNESSES:\n1. This paper asserts its realism and efficiency in generating new textures through optimization, yet its claims are evaluated solely on rendered images. It remains uncertain whether this method is applicable to in-the-wild objects. I suggest that the authors conduct additional experiments using real 3D objects, such as those in the DTU dataset. Furthermore, I find a lack of supportive material regarding its efficiency claims. Given that the method works based on optimization, I question whether it can truly generate high-quality textures swiftly, as stated in the introduction.\n\n2. The introduction lacks clarity and adequate context. The paper outlines two challenges when applying the diffusion model to 3D objects, but it doesn't discuss any related works aimed at addressing these challenges. This omission makes it difficult to gauge the novelty of the proposed method in relation to existing solutions.\n\n3. The proposed relative direction loss appears strikingly similar to that of NeRF-Art[1], albeit the latter employs the CLIP model to implement this constraint. The authors should highlight the distinguishing factors between their method and NeRF-Art in the introduction and add a direct comparison.\n\n\t[1] Wang, C., Jiang, R., Chai, M., He, M., Chen, D. and Liao, J., 2023. Nerf-art: Text-driven neural radiance fields stylization. IEEE Transactions on Visualization and Computer Graphics.\n\n4. In Figure 4, the examples provided are not representative enough. It would be beneficial to showcase results with brown, golden, and porcelain materials. The current examples make it difficult to evaluate the method's performance.\n\nQUESTIONS:\n1. The reconstruction results presented in Figure 5 seem inferior to those achieved by the original NeRF. The results appear quite coarse. Could the authors elaborate on the reasons for this?\n\nLIMITATIONS:\nThis paper has acknowledged its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGiven a set of multi-view calibrated images, the authors aim to first reconstruct the 3D objects based on the input images and then edit them via a pre-trained diffusion model. To accomplish this, they developed ITEM3D, which utilizes a differentiable marching tetrahedron (DMTet) as its 3D implicit representation. To achieve the editing task, the authors proposed two methods: (1) relative direction based optimization that respects both descriptive and editive prompts, and (2) a gradient direction adjustment that integrates changes from previous iterations to improve network robustness. Qualitative and quantitative experiments demonstrated ITEM3D's superior capabilities compared to current state-of-the-art methods.\n\nSTRENGTHS:\nThe strengths of this paper can be summarized as:\n1. The paper employs both descriptive and editive directions to effectively address the editing task. Although it's adapted from the recent DDS paper, I believe the tricks proposed by the authors are useful.\n2. Additionally, the authors integrate changes to adjust the gradient direction, enhancing the network's robustness.\n\nWEAKNESSES:\nThe weaknesses of this paper can be summarized as:\n1. The paper overlooks several noteworthy recent studies, such as Make-it-3D, DreamBooth3D, DreamAvatar, Instruct-NeRF2NeRF, Rodin, and so on, which should be included in the references.\n2. The authors fail to provide a thorough discussion and qualitative comparisons with Instruct-NeRF2NeRF, the most recent and relevant paper.\n3. the ablation study presented in Figure 4 may not be immediately apparent. The \"w/o adj\" may be a better result for the \"A brassy cattle\" scenario.\n\nQUESTIONS:\nBesides the weaknesses listed above, I have several questions for the authors to consider in the rebuttal:\n1. The descriptive prompt may not perfectly describe the object since the 3D DMTet is initially optimized via the input multi-view images, which could potentially impact the final results in a negative manner.\n2. Following the approach proposed in HeadSculpt, it may be beneficial to input the rendered images from the original DMTet instead of the optimized DMTet. I am wondering whether the IESD loss proposed in HeadSculpt would achieve comparable or superior results compared to the approaches in ITEM3D\n3. Figure 2 currently depicts prompts as \"A green chair\" and \"A green wooden chair.\" It may be worth exploring the possibility of changing the prompts to \"A green chair\" and \"make it wooden,\" as proposed in HeadSculpt.\n\nLIMITATIONS:\nLimitations and potential impacts have been discussed in the submission.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ITEM3D, a model for automatic 3D object editing according to text prompts. ITEM3D bridges the gap between 3D representation and natural images using rendered images. It optimizes disentangled texture and environment maps using a relative editing direction to bypass ambiguous text descriptions. It gradually adjusts the editing direction to reduce deviation caused by texture projection. Results demonstrate ITEM3D outperforms SDS-based methods on various 3D objects. It also allows for explicit control over lighting with text-guided relighting. This paper also suggests directly applying the diffusion model to 3D objects is challenging due to conflicts in optimization goals and extreme semantic bias.\n\nSTRENGTHS:\nOptimizing disentangled texture and environment maps using a relative editing direction to bypass ambiguous text descriptions looks interesting. Results demonstrate ITEM3D outperforms SDS-based methods on various 3D objects. The explicit control over lighting with text also looks interesting.\n\nWEAKNESSES:\nOne major weakness of the proposed method is the lack of comparisons with other important related works, such as CLIPNeRF, ARF[1], and SINE[2]. While CLIPNeRF is cited, the others are not. Additionally, there are several related works that do not use text descriptions for editing NeRF, including NeRV[3], NeRD[4], NerFactor[5], and EditNeRF[6], that are also missing from the comparison. Another related line of work, SDFusion[7], is not discussed at all. Although the authors may have overlooked recent related works in the crowded field of text-based editing with NeRF + Diffusion models, it is crucial to compare the proposed method to CLIPNeRF and ARF to understand its advantages. Comparisons to intrinsic image decomposition-based methods are also important, especially for relighting, to determine the strengths and weaknesses of the proposed method. While the expectation is not for the method to outperform NeRD or NeRFactor, comparing it to these methods is important to identify what needs to be done to achieve similar results.\n\nAnother important and simple baseline is missing: how do the results compare to a simple 2D-based image editing method without using explicit 3D texture and illumination map optimization?  \n\n[1] Zhang, Kai, et al. \"Arf: Artistic radiance fields.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022. \\\n[2] Bao, Chong, et al. \"Sine: Semantic-driven image-based nerf editing with prior-guided editing field.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. \\\n[3] Srinivasan, Pratul P., et al. \"Nerv: Neural reflectance and visibility fields for relighting and view synthesis.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. \\\n[4] Boss, Mark, et al. \"Nerd: Neural reflectance decomposition from image collections.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.  \\\n[5] Zhang, Xiuming, et al. \"Nerfactor: Neural factorization of shape and reflectance under an unknown illumination.\" ACM Transactions on Graphics (ToG) 40.6 (2021): 1-18. \\\n[6] Liu, Steven, et al. \"Editing conditional radiance fields.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021. \\\n[7] Cheng, Yen-Chi, et al. \"Sdfusion: Multimodal 3d shape completion, reconstruction, and generation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\nQUESTIONS:\nIt would be helpful to compare the current methods with other recent works mentioned in the weakness to better understand their effectiveness. Additionally, it is unclear how text-guided lighting is mapped to the disentangled illumination representation. Is it solely through the use of diffusion priors or are new text embeddings or directions learned for specific lighting conditions? If diffusion models already include such priors, would optimization in 2D suffice without the need for explicit 3D texture and illumination map optimization?\n\nLIMITATIONS:\nThe paper lacks several key related works and comparisons to them, including simple 2D-based comparisons.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the task of texture editing for 3D models with the guidance of text. The task aims to manipulate the surface properties based on the text guidance to create corresponding visually realistic appearance. The prior works that this work targets to improve upon is the SDS loss related methods. \n\nIn high-level, this paper represents the scene with a 3D representation, differentiably render the representation into 2D images, and use a pretrained diffusion model to guide the editing through an optimization process. The proposed method contains two key components. First, the 3D scene is a decomposed representation, containing triangle meshes, texture map, and environment map (following the prior work nvdiffrec). Second, the paper explores for a better editing direction during the optimization process. Specifically, the relative direction and a gradual direction adjustment scheme are proposed to achieve better editing results. \n\nEvaluation is done on synthetic object data. The results are qualitatively visualized, and quantitatively evaluated with CLIP-based scores and user study.\n\nSTRENGTHS:\nThis paper contains a good amount of originality: \n\n- The task definition is well motivated. \n- The combination of the decomposed representation in nvdiffrec is novel and a well-grounded design choice. \n- The analysis and method design of edit direction are informative and sensible. \n\n\nQuality: \n\n- The qualitative performance of the current model still has a significant room for improvements. But it’s acceptable considering the challenges of the task. \n\nClarity: \n\n- This paper is well written and easy to follow. \n- The analysis is informative and convincing. \n\nSignificance: \n- The functionality this paper achieve is among the first works in the literature.\n\nWEAKNESSES:\n- The quantitative evaluation is relatively weak. There is no quantitative ablation for the proposed method designs. \n\n- The “illumination-aware” in the title is slightly misleading -- the major results and the demo video do not show editing capability for illumination. Fig.5 is one experiment to show simple editing of light intensity. But the quality is still preliminary, and it does not show evidence to edit lighting effects such as reflections and shadows. It fits the acronym quite well, but maybe not the most accurate in describing the method. Maybe is it more like “reflectance-aware”? \n\n- The method adopts the decomposed representation following nvdiffrec. Can the proposed method properly handle the ambiguity of material and light? To give a concrete example, if the original reconstructed object is under a flat and uniform lighting, but the intended edit result is with a directional lighting and thus contain strong shadow. In this case, will the proposed method still bake the shadow directly into the texture map? If this is the case, it’s best to also explicitly mention into main paper as a limitation. \n\n- The current editing is mainly done on synthetic objects. Can it handle real world reconstructed objects? \n\n- The editing is often quite simple, such as editing of color or brightness. How does the model work for more complex editing? E.g. add a mustache or a hat to the cow / fish?\n\nQUESTIONS:\nPlease see weaknesses section above.\n\nLIMITATIONS:\nThe authors discussed some limitations at the end of the paper. I would encourage the authors to also properly discuss the model's ability and limitation on editing of lighting effects.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ITEM3D, an illumination-aware model for automatic 3D texture editing based on text prompts. The authors address the challenges of complex 3D models and ambiguous text descriptions in texture editing. They propose leveraging the power of the diffusion model and optimizing disentangled texture and environment map representations using rendered images as an intermediary. The paper introduces a relative editing direction based on noise differences between source and target texts to improve appearance consistency. The authors also gradually adjust the editing direction to mitigate unexpected deviations caused by texture projection. The contributions include an efficient optimization pipeline for texture editing, the introduction of the relative editing direction, and the proposal of gradual adjustment to handle deviations.\n\nSTRENGTHS:\nThe paper presents a novel approach to automatic 3D texture editing based on text prompts. It combines the power of the diffusion model with the use of rendered images as intermediaries, introducing a relative editing direction and gradual adjustment techniques. This combination of methods and the focus on addressing challenges specific to 3D modeling make the approach original and distinct from previous works.\n\nThe paper is well-written and provides detailed explanations of the proposed method and its components. The authors offer clear insights into the challenges of texture editing in 3D models and provide a sound rationale for their approach. The experiments and evaluations are thorough, demonstrating the effectiveness of ITEM3D in generating visually natural textures and enabling relighting. The use of qualitative and quantitative analyses strengthens the quality of the results.\n\nThe paper addresses an important task in 3D modeling—automatic texture editing—and offers a practical solution with significant implications. By leveraging the diffusion model and incorporating text prompts, the proposed method enables users to manipulate the surface properties of 3D models in a realistic and visually appealing manner.\n\nWEAKNESSES:\nWhile the paper has several strengths, there are also a few areas where it could be improved:\n\nThe paper mainly focuses on the editing of textures in synthetic nerf dataset. However, it would be valuable to investigate the generalization of the proposed method to more complex 3D models and real world scenes, such as those with intricate geometry or high levels of detail. Assessing the performance and scalability of ITEM3D on such complex models would enhance the practicality and applicability of the proposed method.\n\nWhile the paper mentions the importance of efficiency in texture editing, it would be beneficial to provide more insights into the computational requirements and optimization strategies employed by ITEM3D. Specifically, discussing methods to improve the computational efficiency, reducing memory consumption, and addressing scalability issues would enhance the practical usability of the proposed method.\n\nComparison with Alternative Approaches: The paper compares the proposed ITEM3D with diffusion-based editing methods and mentions their limitations. However, it would be valuable to include a more comprehensive comparison with other state-of-the-art methods for texture editing in 3D models, such as Instruct-NeRF2NeRF. This would provide a clearer understanding of the advantages and limitations of ITEM3D in relation to existing alternatives.\n\nQUESTIONS:\nGeneralization and Real-World Application: Can the proposed ITEM3D method be applied to more complex and real-world 3D models beyond the synthetic NeRF datasets?\n\nThe paper briefly mentions the importance of efficiency in texture editing, but it would be beneficial to provide more insights into the computational requirements and optimization strategies employed by ITEM3D. \n\nWhile the paper compares ITEM3D with diffusion-based editing methods, it would be valuable to include a more comprehensive comparison with other state-of-the-art methods for texture editing in 3D models. How does ITEM3D fare against other approaches, such as Instruct-NeRF2NeRF or other relevant techniques?\n\nLIMITATIONS:\nThe limitation is discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}