Decorate3D: Text-Driven High-Quality Texture Generation for Mesh Decoration in the Wild
Authors: Yanhui Guo
Xinxin Zuo
Peng Dai
Juwei Lu
Xiaolin Wu
Li Cheng
Youliang Yan
Songcen Xu
Xiaofei Wu
Venue: NeurIPS 2023 poster (2023)
Keywords: Texture Generation
Text-Driven
3D-Consistent Editing
Neural Radiance Field
TL;DR: Text-driven high-quality  3D-consistent  texture generation for mesh retexturing

Abstract: This paper presents Decorate3D, a versatile and user-friendly method for the creation and editing of 3D objects using images. Decorate3D models a real-world object of interest by neural radiance field (NeRF) and decomposes the NeRF representation into an explicit mesh representation, a view-dependent texture, and a diffuse UV texture. Subsequently, users can either manually edit the UV or provide a prompt for the automatic generation of a new 3D-consistent texture.  To achieve high-quality 3D texture generation, we propose a structure-aware score distillation sampling method to optimize a neural UV texture based on user-defined text and empower an image diffusion model with 3D-consistent generation capability. Furthermore, we introduce a few-view resampling training method and utilize a super-resolution model to obtain refined high-resolution UV textures (2048$\times$2048) for 3D texturing. Extensive experiments collectively validate the superior performance of Decorate3D in retexturing real-world 3D objects. Project page: https://decorate3d.github.io/Decorate3D/.

Decorate3D: Text-Driven High-Quality Texture Generation for Mesh Decoration in the Wild
Yanhui Guo
Xinxin Zuo?
Peng Dai
Juwei Lu
Xiaolin Wu'
Xiaofei Wu² Noah's Ark Lab, Canada Uniiveriity of Alberta, Canada (guoy 143, xwu) @mcmaster. ca, wcw.ngguabbebbrrtt.ccc { (xinxin. zuol, peng. dai, juwei. wtwpweee.wjjuuuwi....cm @huawei. (yanyouliang, xusongcen, wuxiaofei2 www.weii..haaw..ooommmooo
Li Chengè
Youliang Yan Songeen Xu²
McMaster University, Canada
Abstract
This paper presents Decorate3D, a versatile and user-friendly method for the cre- ation and editing of 3D objects using images. Decorate3D models worldw object of interest by neural radiance field (NeRF) and decomposes the NeRF rep- resentation into an explicit mesh representation, view-dependent texture, and a diffuse UV texture. Subsequently, users can either manually edit the UV or provide prompt for the automatic generation new 3D-consistent texture. To achieve high-quality 3D texture generation, we propose a structure-awan score distillation sampling method to optimize neural UV texture based on user-defined text and empower image diffusion model with 3D-consistent generation ca- pability. Furthermore, we introduce few-view resampling training method and utilize super-resolution model to obtain refined high-resolution UV textures (2048x 2048) for 3D texturing. Extensive experiments collectively validate the superior performance of Decorate3D in retexturing real-world 3D objects. Project page: https: /ttctpp://ecoootteeeee githuu. thub. io /eccorateeD/
1 Introduction
The recent development of effective neural 3D reconstruction techniques such neural radiance field (NeRF) [12] has facilitated the creation realistic digital replicas for real-world 3D scenes. While significant progress has been made in realistic 3D reconstruction, to allow users to edit or retexture the acquired 3D objects or assets in the digital scene. To address this issue, this paper introduces Decorate 3D, user-friendly approach to editing 3D objects through either easy-to-use use manual editing or text prompt guided texture generation, as showcased in Fig. 1.
remaining challenge is how
Since the implicit representations of the NeRF model are tightly coupled, it is not trivial to achieve aforementioned operations for the 3D scene's 's decoration. To facilitate the modifiability of the representations, we bake the NeRF model into a triangle mesh representation with view-dependent texture in UV space, which is called Decomposition phase in this paper. The decoupled geometry and UV texture representation make the subsequent Decoration phase decoration phase, Decorate3D allows flexible texture editing and controllable generation of UV textures with the instruction of prompts.
manageable. In the
The success of the decoration phase demands solution the text-to-3D synthesis problem, which has been so far restricted by the lack of paired text and 3D assets. Recent advancements in text- to-image models [23, 19, 15] greatly facilitate text-driven 3D editing or generation [32, 1, 16, 11]. Researchers have proposed approaches using pre-trained text-to-2D diffusion models to optimize NeRF the shoo setting via the Score Distillation Sampling (SDS) [16] strategy. Although
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

impressive results have been achieved, the naive SDS optimization based on 2D diffusion models lacks 3D awareness, resulting in inharmonious textures that misalign with the geometry.
To address the above issues, during the decoration phase of Decorate3D, , we carry out a direct update of the global UV texture map via structure-aware SDS optimization. We employ a pre-trained depth-guided text-to-image latent diffusion model [29, 23] and incorporate the initial decoupled UV texture from the decomposition phase an additional form of structural guidance. We empirically find that rendering with the straightforwardly optimized UV texture from SDS tends to be noisy with color distortions. The reason is that the optimized UV texture stands for neural features in effect, which produce rendered neural images that necessitate neural interpreter. We incorporate the idea of deferred neural rendering [31] and synthesize images of different views by forwarding the rendered neural images to the encoder-decode of the latent diffusion model. However, the decoder's s -eewww synthesis causes jittering effects. To address this issue, few-view resampling training strategy is proposed to optimize global UV texture from sparsely sampled view directions. Finally, to further enhance the visual quality, we use a super-resolution diffusion model to improve the resolution, which is directly applied to the UV space.
Our key contributions can be summarized as follows. First, we propose a system pipeline to allow convenient 3D-consistent decoration of 3D objects captured in the wild; Second, with our structure- aware 3D texture generation, few-view resampling training. and super-resolution enhancement, we are able to synthesize high-qualip textures aligned well with the geometry; Finally, we demonstrate the effectiveness of Decorate3D on real-world datasets and have achieved superior performance over state-of-the-an approaches.
Do
NORF
"Fairy house with garden"
3D consistent image editing
Given captured images of an object, Decorate3D supports text-driven high- texture generation and user-friendly texture editing. Please zoom in for better visualization.
Figure I:
2 Related Work
Text-to-Image Diffusion Models The past two years have witnessed the success of multiple large diffusion models [20, 5, 23, 19, 15] that are able to generate impressive images with photo-realistic details conditioned on an input text prompt. The widely popular stable diffusion [23], was trained on rich paired text-image data and is conditioned on CLIP's [17] frozen text encoder. Beyond text-conditioning. ControlNet [36] extended the stable diffusion by training a parallel hyper-network that allows controllable generation with additional input modalities such as depth maps or edges.
Text-to-3D Generation Methods The development of 2D image generation also greatly facilitates the techniques of text-to-3D generation. CLIP-Mesh [7] proposed text-driven 3D content generation method using a pre-trained CLIP model. DreamFusion [16] first proposed a score distillation sampling (SDS) method achieve text-driven optimization for NeRF, relying on text-to-image diffusion model. Magic3D [9] improved DreamFusion 's resolution using a coarse-to-fine optimization strategy.
Latent-Paint [11] proposed to bring the NeRF the latent space and apply the SDS optimize a latent code for 3D scene generation. More recently, Fantasia3d [3] used hybrid scene representation of DMTET for SDS optimization, and 3DFuse [27] improved the 3D consistency by incorporating a coarse 3D prior into tne-ttunng the diffusion model. Shape-E E [6] trained a conditional 3D diffusion model using paired 3D and text data. Concurrent works, TEXTure [22] as well as Text2Tex [2], introduced optimization-fres iterative update schemes to paint 3D models from different viewpoints using septh-to-immge diffusion models. Although these optimization plausible results for simple geometries, they usually fail on complex surfaces, resulting in artifacts on the seamed areas. Decorate3D distinguishes itself from previous methods by providing 3D-consistent and high-quality structure-awer texture generation for diverse objects.
methods may generate

---

Decomposition Phase
Directons
View Deperadent
Spectler Color Color
Combine final curpue
Wild
OOMPPS
UV Couniiaazes
Diffes:
Render
Dittuse Color
Neural Renderer
UV Nounal Tester
Encodar
Render
Encoder
stent Diffission
Intiaine
Deoader
Bi
Transformer
SLLP
Rasdoon
Few view noampling training
Render
Depth
POOREED MLP
Strange sunglasses, fashion styl""
Velmld +
Super Resolution Diffusion
Backgmpapes unto global UV namn Iesture
Trainblee
3D-Consistent Decoration Phase
Figure 2: Overview of Decorate3D: In the decomposition phase, utilize NeuS [33] extract 3D mesh, and optimize two MI.Ps (i.e. MLPP and MT.Ps) s) to model view-dependent textures. In decoration phase, we initialize the UV neural texture v using MLPp,, and optimize the neural texture di via the depth-guided diffusion model with the score distillation sampling loss. The neural texture is rendered into sampled views and forwarded to the Neural Renderer synthesize color images. Afterward, few-view resampling training method to obtain a UV texture & in RGB space. Finally, we enhance the visual quality with super-resolution diffusion model.
optimize MLP through the
3 Approach
Decorate3D is two-phase framework enable 3D-consistent editing of real-world objects. The overall framework of Decorate3D is illustrated in Fig. 2. consists of a decomposition phase and decoration phase. Next, will introduce the decomposition phase in Sec 3.1 followed by the decoration phase. The decoration phase divided into three parts, including text-driven UV neural texture optimization (Sec 3.2), few-view resampling training (Sec 3.3), and super-resolution UV texture (Sec 3.4),
3.1 Decomposition Phase
Given the captured multi-view images of an object and their respective camera poses obtained by COLMAP [26],
easily train NeRF model for novel view synthesis. However, NeRF editing poses challenges due to its tightly coupled representations. To address these challenges and facilitate convenient 3D editing, well as seamless integration into downstream applications, we propose a solution: decomposing the NeRF representations into a 3D mesh and view-dependent texture. Specifically, employ NeuS [33] to reconstruct triangle mesh M of the object and calculate the UV atlas associated with the mesh using XAtlas [35]. To represent the real-world texture, we decompose the texture into a diffuse texture modeled by an MLP and diewddependent specular texture modeled by an MLPs s conditioned on given view direction.
Denoting the differentiable mesh rendering by R. formulate the view rendering process follows:
Ca, f. = Migmoid(MT.PP(v)). C, MLPs
(2)
c = R(ca + c. M).
(3)
3

---

where v refers to 2D UV coordinates after applying positional encoding, and f, are the intermediate features synthesize the specular color. The features f, are forwarded to the view-dependent network MLPs to generate view-dependent effects, conditioned on the view direction d. The final rendered image e obtained by summing up the diffuse color ca and the specular color C.--
To train the texture networks, we first adopt a typical reconstruction loss Ccotor to minimize the difference between the rendered image ē(x) and its corresponding captured ground truth image c(x) at each pixel x. The loss function is formulated as follows:
Leotor Σ |(ē(x) c(x)]l
x
In addition, we apply an L1 regularization to enforce sparsity on the specular color:
Especuter = Σ [x((x)]
(5)
The training of MLPp and MLPs can be achieved with a single-stage framework, as illustrated in the upper part of Fig. 2. More specifically, MLPP and MLPs are jointly optimized, where the gradient from the specular branch will only be used to update MLPs- and not be propagated to MLPp p. The MLPs is a by-product for decomposing the diffuse and specular textures, not used for the decoration phase. We follow instant-ngp [14] to accelerate the optimization to achieve convergence within 3 minutes on single NVIDIA V100 GPU. More details can be found in the supplementary.
3.2 Text-Driven Neural Texture Optimization
The text-driven texture generation the core element of the decoration phase of Decorate3D However, diversified 3D generation is often infeasible due to a lack of enough data pairs of text and 3D models. In this context, we utilize the SDS technique for the optimization of the neural texture.
Preliminary. Let us first introduce the SDS proposed by DreamFusion [16], which achieves text- to-3D generation based on a pre-trained text-to-image diffusion- based generative model [24]. They utilize NeRF representation to model the 3D scene. It is a parametric function x r(g). which can synthesize an image x at the desired camera pose. Here, MLP representing NeRF scene. The diffusion model é contains denoising function cs (xx;; y. t) that predicts the sampled noise given the noisy image x at timestep f, and text embedding y. The difference between the noisy image and the denoised image provides the gradient to update & such that the density regions with high probability are enforced to match the given text embedding. This gradient update method is named SDS, which is formulated follows:
a volumetric renderer, and & an
0x, a
V (E) Be, (x)
(6)
where the noise ~ (0, I) and w(t) is a weighting function. The neural rendering pipeline I'(E) and the diffusion model modular components of the framework, are amenable to selection. This offers a practical way for text-to-3D synthesis.
Neural Texture for 3D-Consistent Rendering. As compared with previous works [16, 11, 9] on optimizing a NeRF model, we directly optimize a neural texture over the UV space. Direct texture optimization is crucial to enforce 3D consistency and preserve texture details. In detail, first we exploit the pre-trained latent diffusion model as the optimization guidance, where has an encoder Ve, latent diffusion model's denoiser Cg. and decoder Va. In Decorate3D, the neural texture v6 E IR # x3 is set to 3 feature channels to fit the requirement of the diffusion model. The gradient calculated via SDS backpropagated through the diffusion model's encoder to the UV neural texture at high resolution of 512 x 512. Albeit yielding high-resolution images, the optimization computation reasonable since the optimization exerts an effect on the latent code z = Ve (R(1,, M,, P)) with a resolution of 64 x 64, where Ve is the diffusion model's 's encoder and P is the sampled camera pose. All network parameters are fixed, while only the neural texture &e is trainable.
Here, we call the optimized texture UV Neural Texture by following the deferred neural rendering [31]. The optimized texture does not act on RGB space, and they are neural features that need a Neural Renderer to interpret, which will be introduced shortly.

---

Structure-Aware SDS Optimization. Apart from addressing the 3D-consisteney issues, we also want maintain the coherence between 3D geometry and the generated texture. For example, for 3D human model, the generated facial texture should be attached to the 3D region corresponding to the face. This structure-aware requirement is overlooked by the naive SDS optimization leading to multi-face Janus problem. To moderate this issue, Decorate3D utilizes structure-aware SDS optimization by exploiting a depth-guided latent diffusion model [29, 23].
In detail, at each denoising step t, we compute the image latent code ž produced by the encoder V. and the depth latent code 2 downsampled from the estimated depth map. They are concatenated 2, and forwarded to the latent diffusion module (Eq. 8). Then, following the SDS technique, the gradient of neural texture is computed as follows (Eq. 9):
(7)
E (Z) y, cs (z(it) t) + Meo(Ze;y Ez(Z t)].
(8)
0z, 0V.
ve (b) 1/) = Ri, (())((
(9)
ov. Oeb
where e is the diffusion model's denoiser, y is the text embedding from the Transformer, and w(() is a weighting function. Decorate3D uses the classifier-fre free guidance scheme [15] stated in Eq. 8, and the guidance weight À for text conditioning set to 100. To match the depth-guided diffusion model, depth maps are predicted from the rendered views by using the depth estimator [21] of the depth-guided diffusion model rather than the depth buffer of the rendering pipeline. The overall SDS optimization framework is shown at the left bottom of Fig. 2.
In addition to structure-aware SDS optimization, Decorate3D adopts structure aware initialization. Thanks to the decomposition phase, Decorate3D has an inherent texture that well matches the geometry information extracted from the real-world object. Therefore, Decorate3D nerual texture & with the output of ML.pp. This initialization plays a vital role in the optimization process, which can accelerate the convergence as well as provide structure-awar information to help it converge to a structure-awar solution. We verify the effectiveness of the structure-aware techniques in Sec 4.4.
initialize the
Decorate3D's Neural Renderer. As aforementioned, directly optimizing the UV texture with SDS does not yield traditional RGB UV texture for the rendering pipeline. Fig. 3 shows an example of SDS optimization for text-to-image generation. The optimized texture, i.e. neural texture, requires a neural interpreter convert it back RGB space after the SDS optimization.
Previous work [11] applies a statistical linear transformation matrix to the optimized UV texture to interpret it in the RGB UV space. However this transformation matrix is suboptimal, and the misalignment will result in color shifts and artifacts.
In Decorate3D we introduce a neural renderer as the neural interpreter synthesize photo-realistic images from the opti- mized neural texture. This rendering process is formulated as follows:
w/O Interpreter
Interpreter
Figure 3: The text prompt is "An apple tree". The left shows the image from the direct SDS optimization on a 2D image. The right shows the image after applying interpreter.
Tp = (R
(10)
where vi is the neural texture, denotes the network of the neural renderer, and /p is the rendered image given the camera pose P. Here, Decorate3D 's neural renderer network is the encoder-decoder (i.e. VAE) module of the depth- gighhgguddeddd latent diffusion model [23, 29]. The network V and the traditional renderer R have amalgamated to form the neural renderer.
3.3 View Resampling Training
With the neural texture, Decorate3D can render 3D-consistent high-quality views Ip through the Neural Renderer with Eq. 10 by sampling different camera poses. But there still exists practical issue,
iee. jittering artifacts, that arises when rendering across different views. This is caused by the view-dependent neural renderer V. We empirically find that directly feeding the global UV neural texture vp to the diffusion model's VAE (i.e. ((V()), P) by swapping the network and renderer
5

---

in Eq. 10) can produce the RGB UV texture for the traditional rendering pipeline. But this naive approach usually yields blurry texturing results, as demonstrated Sec 4.4.
To solve the jittering problem, we devise a few-view resampling (FVR) training method, that transfers the synthesized view-dependent images into a global UV texture v. As shown in the right bottom of Fig. 2, we use a MLP;: & to represent the UV texture vb. In the FVR training, we sample N rendered views using the neural renderer with the neural texture tb. The sampled N views should overlay the mesh surface as much as possible. But setting big N may have negative effects, leading to over-smoothing textures on the overlapped areas that suffer jittering artifacts. The FVR training loss defined as follows:
()VR(²) = Σ
(11)
where v denotes the positional encoding of the UV coordinates of the RGB UV texture 12 and P. denotes the i-th sampled camera pose. The pose is sampled in spherical coordinates, with two elevation angles leam chosen from {-20", + ). four azimuth angles 3oom uniformly sampled between [0", 360%). and an appropriate viewing distance Team. In our experiments, N is set to 8.
3.4 Super-Resolutinn on UV Texture
From the SDS approach, we can only synthesize images at resolution of 512 x 512. To further improve the spatial resolution of the UV texture v Decorate3D applies super-resolution (SR) diffusion model with a x4 scale factor (25,30) on the UV texture to obtain a 2048 x 2048 UV texture. The rationale behind this successful application of UV texture upscaling is that the upsampling operation has spatial locality, concentrating on local textures such as edges. For this reason, SR models are usually trained on cropped image patches [28, 10] instead of the whole image to increase the training efficiency. This spatial locality of SR allows the SR model trained on natural images be directly applied to the UV texture. Directly applying the SR operation the UV texture is free from any jittering issues, and 3D consistency is well preserved.
Experiments
We compare Decorate3D to the state-of-the-ar (SOTA) techniques for text-to-3D texture generation and evaluate its performance from both qualitative and quantitative perspectives. The 3D assets and extended videos are presented in the supplementary material.
4.1 Implementation Details
To evaluate our approach, we collect real-world datasets from 14 different objects that vary in complexity, including boxes, monitors, shoes, statues, dolls, humans, and lighthouses. Ten objects are captured in the wild using a smartphone, and four objects are selected from some public real- world datasets [34, 8]. The size of each dataset ranges from 70-300 images, and the images are downsampled to a resolution width of 640. We use the Adam optimizer to optimize the MLP MLPs and MLP; & with learning rate of 1 10-3 and the và with a learning rate of x 10-2, The neural texture optimization in the decoration phase takes about 2 hours for 100K iterations, and the FVR training takes about 5 minutes for 30K iterations, which are measured on 8 NVIDIA V100 GPUs. Please refer to the supplementary material for more details.
4.2 Qualitative Evaluation and Comparison
3D-Consistent Text-Driven Texture Generation. Fig. 4, we show the textured mesh and its corresponding real-world image. It can be observed that Decorate3D is able to produce high-quality textured mesh. As the mesh geometry is fixed, we find only rexture-related keywords such as nouns or adjectives will affect the generated results, highlighted with a purple color in the prompt. For instance, in the prompt "An astronaut stands up in the milky way", "astronaut" and "milky way" dominate the generation, but the verb "stands up" and the quantifier "an" do not affect the results.

---

An astronaut stands up the milky way
A lotion case, painted with Starry Night
A white horse statue, with red hairs
STAR WARS
WO
A cosplay suit of armored medieval Japanese samurai
A detergent case, printed with movie poster Star Wars
doll model of the movic of Zootopia, wearing spiderman suit
Figure 4: Text-driven texture generation results Decorate3D.
Ineezon with lunds choped behiad tacks
Catamsnttth
man wean act medioaal
alocs,
wiard put traomare
Original
CLIP--Mesh
DrcamFusion
Patent-Paint
TEXTure
Decorate3D (Ours)
Figure 5: Qualitative comparison with other text-driven texture generation methods, including CLIP-Mesh [7], Latent-Paint [1]] and TEXTure [22]. Results from DreamFusion [16] are also shown here. Please zoom in for hetter visualization.

---

Fig. 5 shows the qualitative comparison with SOTA text-driven texture generation methods, including CLIP-Mesh [7], Latent-Paint [11] and TEXTure [22]. They all took the reconstructed 3D mesh as input, and the geometry was fixed during texture generation. We can observe that the generated textures from CLIP-MMehh and Latent- Patent-Paint lack texture details. The concurrent work TEXTure has some competitive results compared to Decorate3I) such as the results for shoes, because they use a depth-guidance strategy similar to ours, But TEXTure produces weird artifacts on the complex surfaces. For example, in the case of humans, the generated textures from TEXTure have obvious seams, and textures from different parts seem to be glued together and look messy. The artifacts are caused by the iterative update strategy across views. By contrast, the results of Decorate3D have fruitful and clear details. We also present the results from DreamFusion [16]. It tends to produce blurry results that do not actually match the text prompt.
Singlo-View Editing
Propagated to UV Texture
Textured Mesh
Adding Decomposed Specular Color (+MLPs)
A running horse having green stripes
Original
Text Single-View Editing
Propagated UV Texture
Textured Mesh
Figure 6: Two examples of applying Decorate3D for 3D-consistent texture editing. As showcased in the first example, scribble of the "NeurIPS 2023" pattern is painted onto 2D image (aka. single rendered view), which is propagated by Decorate3D to UV texture, updating the 3D-consistent rendered output.
3D-Consistent Texture Editing. Thanks to the decomposed 3D representations, Decorate3D opens up an easy way to edit textures. Fig. 6 depicts a practical and distinctive use case that can be accomplished by Decorate3D We edit one of the rendered images (not the UV texture), and propagate the editing to the UV texture to synthesize 3D-consistent edited textures from different views. The decomposed view-dependent specular texture can be added back to the textured model. Additionally, Decorate3D allows secondary editing of the text-driven texture generation.
4.3 Quantitative Evaluation and Comparison
CLIP R-Precision Metric. Following DreamFusion, we evaluate the CLIP R-Precision [18], an automated metric for the consistency of rendered multi-view images with respect to the input prompt. Here, calculate the average CLIPP score from the front-side, left-side, right-side, and back-side views. We use 70 prompts of 14 objects to generate the test results. For visual quality, we measure the NIQE [13] on rendered images, which is no-reference image quality assessment, Tab. I reports the CLIP R-Precision scores of the compared methods.
User Studies. We invite 44 volunteers to evaluate Decorate3D and its competitors using the mean- opinion-score (MOS) test. For each question, we prepare 5 video results, and the participants are asked to rate the results on scale from I (worst) to (best) based on the overall visual quality the results and the degree to which they match the text prompt. In the end, we receive 1 100 responses from the 44 volunteers. Fig. shows the average MOS scores of the compared methods. can be seen, Decorate3D is shown to be more favored by human users.
4.4 Ablation Studies
Initialization of UV Neural Texture. Fig. 8a shows an ablation study on the initialization of the UV neural texture and Fig. 11 shows more results. As observed, the optimization initialized with
8

---

Table I: Evaluating the correlation of text-driven gen- erated results with their text input using different CLIP models. The CLIP-Meste Pe-Mess 's scores may overfit, as uses the same CLIP model for training and eval. The NIQE is no-reference image quality evaluation metric.
2-3
R. Precision CLIP B/16
Method
CLIP B/32
CLIP L/14
NIQE
CLIP-Mesh [7]
31,95 2a.
30.16 am
23.99 2om
20
DreamFasion [16]
29.65 s-m
29.63 gx
24.38
24
16.68
TEXTure [22]
Satent-Paint [1]]
25.18
28.55
26.19 22:
23.15
21.22
17.71
Figure 7: User study results gathered from 44 partici- pants, The results are the average of all responses.
Decorate3 (Ours)
30.12 13.47
30.47 Loar
24.89
15.43
14.82
10.65
Insioitnatio
Hame
Hum
Ittm
12k
Firal
Iniialization with SILP
"Tello Iiiey del waara bandlt suif"
Oniine
Vaio dipeh
Nel depet
(a) An ablation study of neural texture's 's initialization.
(b) w/o depth vs. w/ depth.
"Woman marlin shoes,
with wizard style.
(1) VAE
(2) View Generation
(3) MLP.
(4) MLP, +R
(c) Ablation studies of FVR training and the super-resolution on UV texture.
Figure 8: Ablation studies of Decorate3D (a) Random initialization vs. initialization with MLPp. (b) Naive SDS without depth vs. structurc-awar aware SDS guided by depth. As observed, the naive SDS cannot generate the geometry- mommery-matching texture for the man't hands. (c) We compare the rendered results using (1) UV texture yielded by the diffusion model's VAE, (2) Per-view generation, (3) FVR trained MLP i. and (4) MLP; ; +SR.
MLPp converges much faster than the random initialization. Moreover, the MLPp provides very strong optimization prior to helping the SDS optimization converge to a better solution that well matches the mesh geometry. For example, initialization using MLP can generate the correct facial texture that fits the geometry, but on the contrary, the random initialization fails.
Effectiveness of Structure-Aware SDS. Fig. 8b presents the difference between the naive SDS without depth guidance and the structure-aware SDS with depth guidance, and Fig. shows more results. We can observe that the structure-aware SDS is able to produce geometry matthiinn textures. For example, the human's s arms should be clasped behind the back rather than being akimbo.
Is the Few-View Resampling Training Necessary? Fig. 9 compares the per-view gener- ated results of (R with the results rendered using the FVR-trained UV tex- ture. As observed, results of per-view generation have jittering effects between differ- ent views (look at the circle pattern), but results with MLP,, have achieved 3D consis- tency. Fig. 8c (1) presents the results using an alternative solution eliminate the jitter- ing, i.e. directly feeding the neural texture into the diffusion model's VAE by M(

---

Difference Map
Porviou Ceneusion
Synthesizing Galigned first view with depth)
Wifa FVR
MLP;;
Wi FVR
Figure 9: Comparison between the per-view generation (w/o FVR) and rendering with MLP3 (w/ FVR). The difference in pixels between neighboring frames is measured after aligning views to the first reference view.
As observed, following this way yields blurry textures. Given an RGB UV texture v Decorate3D can directly apply SR on U. By comparing Fig. 8c (3) and Fig. 8c (4)
observe more clear results after SR. Fig. 10 shows the effectiveness of FVR training's hyperparameter N. Setting big N for FVR will cause blurry results.
Effectiveness of Neural Renderer. As introduced in Sec .3.2, the op- timized UV texture by SDS a neural texture feature. A neural renderer is required convert the neural UV texture the RGB UV. The right of Fig. 11 shows the ablation study results on the neural renderer. As ob- served, the synthesized results without neural renderer over-saturated and noisy.
Figure 10: Results with dif- ferent N for FVR. A bigger N leads blurry textures.
3D Mash
Iaik & wio Depth
Inil wil Depth
wio Init
Depth
wi Tuit Depth
wio Neural Rendertr
Neural Bunderer
Figure Ablation studies on initialization, depth guidance, and neural renderer.
Conclusion
Decorate3D offers practical way of decorating real-world 3D models with a user-friendly approach through text-driven texture generation. Our Decorate3D 's techniques, combining directly optimizing UV neural texture, structure-aware optimization, FVR training, and SR enhancement on UV texture, collectively advance the line of 3D texture generation in pursuit of the best possible quality. Extensive experiments demonstrate superiority Decorate3D over SOTA methods.
Limitations First, the style and quality of the generated texture heavily depend on the pre-trained stable diffusion models. Second, even though Decorate3D adopts structure- optimization tech- niques, the multi-face Janus problem still remains flat geometry. For example, Decorate3D cannot distinguish the front and back sides of monitor object, as shown in our supplementary material. Lastly, Decorate3D does not support jointly optimizing both the geometry and texture, which may result in inconsistency with the mesh geometry. We remain these problems for future study.
10

---

Acknowledgments and Disclosure of Funding
We gratefully acknowledge the support MindSpore http::/www.mindspor.co CANN (Compute Architecture for Neural Networks) and Ascend Al Processor used for this research.
References
[1] Tim Brooks, Aleksander Holynski, and Alexci A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402, 2023.
[2] Dave Zhenyu Chen, Yawar Siddiqui, Hsin- Ling Lee, Sergey Tulyakov, and Matthias Nießner. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint arXiv:2303. 1396, 2023.
[3] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22246-22256, 2023.
[4] Yanhui Guo, Xiaolin Wu, and Xiao Shu. Data acquisition and preparation for dual-reference deep learning image super-resolution. IEEE Transactions on Image Processing, 31:4393- 4404, 2022.
[5] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47): 2022.
[6] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02246 02440. 2023.
[7] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. CLIP-mesh: Generating textured meshes from text using pretrained image-text models. SIGGRAPH Asia 2022 Conference Papers. ACM, nov 2022.
[8] Arno Knapitsch, Jaesik Park, Qian Yi Zhou, and Vladlen Koltun. Tanks and temples: Bench- marking large-scale scene reconstruction. ACM Transactions Graphics, 36(4), 2017.
[9] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309, 2023.
[10] Fangzhou Luo, Xiaolin Wu, and Yanhui Guo. Functional neural networks for parametric image restoration problems. Advances in Neural Information Processing Systems, 34:6762-6775. 2021.
[11] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or Latent-nerf for shape-guided generation of 3d shapes and textures. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12663-12677 12673, 2023.
[12] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: representing scenes as neural radiance fields for view synthesi. In European Conference on Computer Vision (ECCV), pages 405-421, 2020.
[13] Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters, 20(3):209-21,, 2013.
[14] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 11:4::12:1 July 2022.

---

[15] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image genera- tion and editing with text-guided diffusion models. In International Conference Machine Learning, pages 16784-16804. PMIR, 2022.
[16] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In International Conference on Learning Representations (ICLR), 2023.
[17] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 7748-8763. PMLR, 2021.
[18] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning, volume 139, pages 8748-8763. PMLR, 2021.
[19] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:200..06125, 2022.
[20] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[21] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Iransactions on pattern analysis and machine intelligence, 44(3): 2020.
[22] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text-guided texturing of 3d shapes. arXiv preprint arXiv:2302.0172) 2023.
[23] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, 2022.
[24] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, pages 36479-36494, 2022.
[25] Chitwan Saharia, Jonathan Ho. William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713 4726, 2022.
[26] Johannes Lutz Schönberger and Jan-Michael Frahm. Structure- feneture-from-ie motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104- 4113, 2016.
[27] Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin- Hwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. arXiv preprint arXiv:2303. 07937, 2023.
[28] Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pullern recognition, pages 1874-1883, 2016.
[29] Stability-AI.
Depth-guided stable diffusion model. ffusion, 2023.
https://github.
Subbility-AI/staane
[30] Stability-AI. Diffusion models for super-resolution. https://hugginee
co/ CompVis senovissldm-- http:mmmmmmmmmmmmeeeeeeee poteeerrrsolutiion 4x 4x-ooeximagess. 2023,
12

---

[31] Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. Acm Transactions on Graphies (TOG), 38, 2019.
[32] Can Wang, Ruixiang Jiang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Nerf-art: Text-driven neural radiance fields stylization. IEEE Trunsactions on Visualization and Computer Graphics, 2023.
[33] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems, $4:27171-27188 2021.
[34] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1190-1799, 2020.
[35] Jonathan Young. Xatlas: Mesh parameterization/ unwrapping library. https: https://github. com/ jpcy / xatlas, 2023.
[36] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3844 2023.
13


Reviews:
[{"review_id": "T831SeJduf", "reviewer": "Reviewer_MiDH", "summary": "This paper introduces Decorate3D, which enables text-guided 3D model editing by extracting and editing a learned UV texture. Specifically, Given multi-view images, Decorate3D first generates 3D mesh and UV textures based on NeuS. Then, it optimizes neural textures by the guidance of 3D structure (depth) and stable diffusion model. Finally,  an RGB UV texture is optimized and upsampled to generate the final result. Experiments demonstrate the state-of-the-art performance of Decorate3D.", "questions": "Some discussions about the above weaknesses and a quantitative ablation study would make the submission stronger.", "limitations": "Important limitations have been discussed in the paper.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The idea of this paper is well-motivated and presented.\n\n- A carefully designed pipeline (Nerf rendering, depth-aware texture optimization, few-shot texture re-optimization, and texture super-resolution) enables high-quality generation results. The 3D consistent decoration phase is novel and effective. \n\n- Best performance is achieved compared to SOTA. ", "weaknesses": "- The proposed method uses few-view resample training to obtain a UV texture that reduces the jittering effects. I am wondering if this step can be done in the decomposition phase or the text-driven neural texture optimization.\n\n- The super-resolution is applied to the UV texture. However, the UV texture is not a natural image, if the model is trained with natural images, will there be some domain gap leading to inferior results? Moreover, as super-resolution is only a post-processing step that enhances the results and is not one of the main contributions, I recommend spending fewer texts on this point.\n\n- I would like to see some quantitative ablation studies like Table 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThanks for your valuable comment and positive feedback. We have demonstrated extra ablations, including visual results and quantitative ablation study results. Please refer to the one-page response pdf.\n***\n* **Q1:** The proposed method uses few-view resample training to obtain a UV texture that reduces the jittering effects. I am wondering if this step can be done in the decomposition phase or the text-driven neural texture optimization.\n  \n  **A1:** The texture after decomposition is consistent, and there is no jittering problem. Therefore, it wouldn't make much difference if applying FVR in the decomposition phase. We use FVR after the text-driven neural texture optimization since the jittering problem is caused by the Neural Renderer when rendering the optimized texture across various camera views.\n***\n\n* **Q2:** The super-resolution is applied to the UV texture. However, the UV texture is not a natural image, if the model is trained with natural images, will there be some domain gap leading to inferior results?\n\n  **A2:** The rationale behind this successful application of UV texture upscaling is that 1) the upsampling operation has a spatial locality, concentrating on local textures such as edges. For this reason, SR models are usually trained on cropped image patches instead of the whole image to increase training efficiency. This spatial locality of SR allows the SR model trained on natural images to be directly applied to the UV texture. 2) We used the SR model that is fine-tuned over the pre-trained stable diffusion, which has a strong prior to boost its generalization ability. \n***\nAs suggested by the reviewer, we will move some details of the super-resolution part into the Implementation Details.\n"}, {"review_id": "bTpmrD3qd2", "reviewer": "Reviewer_xCPF", "summary": "This paper proposes a method to edit the textures for neural fields (NeRFs) using score distillation sampling and also export a mesh model with texture that can be used in traditional graphics pipelines (i.e. game engines, VFX). More specifically, the main contributions that I see from this work is the \"Few-view Resampling Training\" which can take an SDS-optimized RGB diffuse texture map (which is noisy due to the nature of SDS with LDMs), and refine it through LDM-driven re-rendering, which takes advantage of both \"LDM as a renderer\" and having a real 3D consistent 3D model that can be used. This is specifically a general technique that could be widely applicable in a variety of different tasks.  \n\nIn addition to this, they also create an entire pipeline to extract editable & high quality mesh representations from multi-view images (i.e. ones with good geometry, good UV parameterization, diffuse + specular separation, mostly based on existing tools) as well as another case study on SDS-driven texture generation.", "questions": "1. Why is the task at hand called 'decoration' as opposed to texturing or texture editing?\n\n2. At least to my eyes, it does not seem like the higher N for FVR makes a big difference. What would happen if we choose more extreme numbers for these, like N=1,2,4,8 and N=1024,2048? Is there a way to design a loss to make it robust to both extremes?\n\n3. Is there any cases where the multi view consistency fails for FVR? How does the number of views affect this?", "limitations": "The authors have adequately addressed the limitations. I think they could make a broader statement on the societal impacts, as content creation tools like this are something that could potentially impact labor markets (and displace artists) and is something that is based on diffusion models trained on large amounts of data (which often also means they are unattributed / has no provenance back to artists).", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "The main strength of this paper is in the \"few view resample training\", which takes as input a noisy 3D model, renders the 3D model, refines the rendered image using a \"neural renderer\" (which in this case is the VAE of an LDM), and back propagates the refinement back to the 3D model. This as far as I'm aware is an original idea that I have not seen at least in this specific context. The method also seems to be effective from the limited results I am able to see, and is something that can likely be incorporated into many different contexts. \n\nThe paper also proposes an end-to-end pipeline for doing NeRF -> mesh -> editing, and evaluates several different tricks to make this pipeline effective which they also evaluate in some limited ablation studies. This is significant as it provides a case study for implementation tricks in making this pipeline work (which in my experience tends to be a big part of SDS based pipelines).\n\nThe clarity of the paper could be improved, but is not something that significantly detriments the paper. This will be discussed further in the weaknesses.", "weaknesses": "The biggest weakness of this paper is in its clarity. With some restructuring and refinement, however, I think that this paper could be very convincing.\n\nFirst, the paper introduces the problem of 'mesh decoration'. The task really at hand is 'retexturing' or 'texture editing'. I'm not sure what the motivation for using the word 'decoration' is, but this is something that makes the paper unnecessarily confusing to grasp. \n\nSecond, the paper puts a lot of weight on discussing the end-to-end pipeline from reconstruction to retexturing. In reality, the meat of the contributions for this paper lies in the retexturing method (and specifically the few view refinement), and the rest feels like a distraction that is not core to the contribution. Making the writing and contribution statements more specific to the retexturing part of the pipeline, and treating the end-to-end pipeline as almost an 'implementation detail' would make the paper much more convincing. (i.e. the decomposition stage isn't really core to the method, since the same pipeline could be applicable for an existing 3D mesh). \n\nThird, the paper does not sufficiently compare and contrast their method with concurrent works like TEXTure which can be considered prior art given its more than 2 months before the deadline. The paper does compare against them in the evaluations, which is great, but it could use more discussion on _why_ these prior arts produce bad artifacts in their generation, and what fundamental differences makes this paper more advantageous. \n\nFourth, the results shown on the core contributions are rather light. It would be very illustrative to show (on multiple models) the rendered 3D models after the SDS optimization (with their artifacts & UV textures), after \"neural rendering\", and after refinement to really showcase the efficacy of the refinement method. \n\nFifth, the ablations are good but they could be on different figures with more examples. Some of the text space that is currently used for the description of 'prior things' like SDS and the decomposition stage could probably be placed in the supplemental or taken out to make more space for results. More results on especially the effects of higher viewpoints for the refinement step could be very useful.\n\nLastly, it would be useful potentially to write the approximate time for completion for each stage in Figure 2 to make the costs more clear.\n\nThese are not things that affect my rating, but nitpicks:\n\n22: \"Since the implicit representations of the NeRF model are tightly coupled\" this should be explained in more detail. I believe the authors are referring to the fact that it's difficult to disentangle geometry and texture from a typical NeRF model, but this phrase does not communicate this at all. \n\n41: \"The reason is that the optimized UV texture stands for neural features in effect, which produce rendered neural images that necessitate a neural interpreter\" I find the whole section starting with this sentence rather confusing and hard to interpret what it really means (without having read the rest of the paper at this point yet). Trying to describe this more precisely would help. For example, what does \"stands for ... in effect\" mean? What is a \"neural interpreter\"? I can make inferences but those are then inferences. At this point I'm also confused as a reader why the UV texture needs to be 'neural' or 'latent' at all.\n\n113: \"However, diversified 3D generation is often infeasible due to a lack of enough data pairs of text and 3D models\" I assume this sentence is in reference to an auto encoder sort of framework, but this is not explained anywhere. Also, I'm not really sure what \"diversified\" means. \n\n131: It's not made clear in this general section that the thing that is being passed to the encoder is a rendered image, not the texture map. Although this is clear from the figure, explicitly stating this here would be nice.\n\nThe writing could be improved stylistically in various places, like not starting sentences with \"And\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful suggestions that are helpful in improving our paper. We will carefully check and refine the descriptions that may lead to ambiguity. \n\nAs suggested by the reviewer, we demonstrate more ablations of each component of the proposed pipeline and show the results in the one-page response pdf. We will also reorganize the paper structure, emphasize more on the re-texturing part, and add those ablations into the final version. \n***\n* **Q1:** The motivation for using the word 'decoration'.\n\n  **A1:** We concur with your assessment that the essence of 'mesh decoration' lies in retexturing. However, a slight difference exists between them. Decorate3D's ultimate objective is to offer a solution for controlling or editing the texture of real-world 3D objects where a given 3D mesh is unavailable. \n\n   The proposed method successfully accomplishes the entire pipeline: 'real-world images $\\rightarrow$ user editing signal $\\rightarrow$ textured mesh', whereas most retexturing methods presume the availability of an ideal 3D mesh. In the real-world context, Decorate3D encounters an additional challenge and effectively overcomes it. \n\n   Additionally, from the standpoint of non-specialized users, the term 'decoration' carries greater expressiveness.  But, we will consider to revise the 'decoration phase' to 'retexturing phase' and change 'Decorate3D' to 'TextureGen3D' or 'DecoTexure3D' to improve the clarity. \n\n***\n\n* **Q2:** Analysis and comparison with the TEXTure paper. \n\n  **A2:**  In addition to the comparison cases presented in our main paper, we have provided more results in the supplementary material file (from Figure 18 to Figure 20) and the demo video (please refer to the demo video from 0:30 to 1:30). \n\n   TEXTure proposes a progressive UV texture generation approach utilizing 10 selected views. Specifically, it initiates texture generation by first creating a front-view image through a pre-trained stable diffusion model. Subsequently, this front view is propagated onto the UV texture, and its neighboring view is generated via an inpainting model conditioned on the priorly generated texture. Consequently, TEXTure excels at generating a high-quality texture for the object's front view. However, the final generated UV texture may exhibit seams across different sampled views, and the inconsistency could accumulate when progressively updating from the front view to the back view.\n\n   In contrast to the progressive updating strategy, our UV texture undergoes global optimization through a structure-aware SDS loss. As a result, the generated texture is seamless and consistent across both the front and back views. Additionally, our FVR training significantly enhances the quality of the generated UV texture.\n***\n* **Q3:** It would be very illustrative to show (on multiple models) the rendered 3D models after the SDS optimization (with their artifacts & UV textures), after \"neural rendering\", and after refinement to really showcase the efficacy of the refinement method.\n\n  **A3:** We add the results of extra quantitative and qualitative ablation studies in the one-page response pdf. In detail, Figure A demonstrates the ablations with and without initialization, depth condition as well as Neural Renderer. Figure B shows the difference map between neighboring frames before and after FVR to better visualize the jittering problem. We use ground truth depth to compute the perspective warping from neighboring frames to reference frame.\n\n***\n\n* **Q4:** It would be useful potentially to write the approximate time for completion for each stage in Figure 2 to make the costs more clear.\n\n  **A4:** The decomposition stage takes about 3 minutes. The neural texture optimization in the decoration phase takes about 2 hours for 100K iterations, and the FVR training takes about 5 minutes. \n\n***\n* **Q5:**  The effects of setting the number of views for FVR training. Is there a way to design a loss to make it robust to both extremes?\n  \n   **A5:**   If setting a rather small number of views, like N=1,2,4, the UV texture may not be fully covered by these limited views. As illustrated on the right side of Figure B in the one-page general response PDF, we demonstrated an extreme case with N=2048. The resulting generated texture appears slightly blurry, but the difference is not easily discernible when compared with N=256 or 512. However, it should be noted that increasing the number of views will also entail additional computation costs.\n\n    In our experiments, we set N=8 empirically, which includes 2 elevation angles {$\\{-20^\\circ, 20^\\circ\\}$ }and 4 azimuth angles uniformly sampled between [$0^\\circ$, $360^\\circ$]. This configuration adequately covers most of the mesh surface. \n\n    As suggested by the reviewer, we can consider an algorithm that maximizes the UV coverage while at the same time minimizing the overlapping pixels in the UV. \n\n    We can deduce a UV mask from the UV texture to identify the areas that were overlooked by the previous N views. This UV mask can be easily and accurately computed using the camera matrices of previously sampled N views and the UV atlas. Subsequently, we can take an additional step by sampling more views (M views) to cover the previously overlooked areas. Ultimately, we train the $\\text{MLP}_{\\tilde{\\phi}}$ using the FVR training with N+M views.\n***\n* **Q6:** Is there any cases where the multi-view consistency fails for FVR? How does the number of views affect this?\n\n  **A6:** Before applying FVR, those rendered images of different views are consistent except for the jittering effects. We haven't found such cases where multi-view consistency has failed. \n"}, {"review_id": "gMklaxaARe", "reviewer": "Reviewer_U4jz", "summary": "This paper proposes decorate3D, a method for re-texturing real-world 3D objects using text-conditioned image diffusion models. The proposed method can be split into a 3D reconstruction phase and a re-texturing phase. In the 3D reconstruction phase, a 3D mesh is reconstructed from set of multiview images via NeuS, and a view independent texture map is distilled via differentiable rendering. In the re-texturing phase, a depth-conditioned latent diffusion model is combined with SDS to optimize the texture map. The texture map is re-rendered by passing it through the encoder-decoder of stable diffusion to remove neural artifacts, but this step introduces jittering artifacts. Jittering artifacts are then removed via optimizing a MLP through few-view resample training to reconcile view-inconsistencies. Lastly, a super-resolution diffusion model is used to up-res the produced texture map.", "questions": "1. what is the rendering model used in the SDS optimization step? Is any lighting/view-dependent effects incorporated into the rendering equation (as used in magic3d and fantasia3d) or is it purely UV based retrieval from a neural texture? Do you have any ablations on this?\n2. could this method be adapted for texture synthesis by removing the initialization and using a textureless rendering of the geometry as input to the depth estimator (thereby preventing compounding artifacts between bad initial texture and bad depth estimation)?", "limitations": "I think the computational cost of this method is a limitation worth mentioning - it is by far the most expensive method to run versus its baselines, whose runtime ranges from seconds to tens of minutes on a single GPU, whereas Decorate3D requires hours on full 8 GPUs.\nPotential negative societal impacts such as identity theft, deep fakes, and manufacturing of disinformation should be mentioned.\n\n---------------------------------------------Post rebuttal:\nI think the changes to the manuscript promised by the authors will significantly improve the delivery and message of the paper by firmly substantiating their claims regarding the effectiveness of proposed techniques with more ablations. Therefore I'm changing my suggestion to acceptance.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. The method is technically impressive, utilizing many interesting tricks to overcome problems associated with latent diffusion models, and thereby obtaining visually impressive experimental results.\n2. The presentation of the method is precise and easy to follow. Despite the many moving parts, never once did I feel a need to backtrack due to inconsistent notations or frivolous math equations.\n3. The experimental procedure is detailed and well documented. One can be confident of the reproducibility of the results (as long as the authors release the real world data they've collected). The ablations are also fairely thorough, giving clear intuitions as to the effect of each component.", "weaknesses": "1. Novelty:\nMost components utilized in this method are either well known to the literature, or straight-forward extensions of existing workflows, such as NeuS for mesh reconstruction, disentangling view-dependency via differentiable rendering of two MLPs, using depth condition for text-to-3D, and appling super-resolution diffusion models on UV textures. Though the problem of SDS with LDMs as observed in figure 3 has not been formally studied in a research paper, knowledge of this problem is folklore within the community and the proposed neural renderer solution is rather simplistic. As such, it is not clear to me whether this paper contains enough technical novelty to be impactful in the text-to-3D field.\n2. Fairness of comparisons:\nThe experiments can be more convincing if other SDS based approaches (namely dreamfusion and latent paint) are also equipped with depth conditioned diffusion backbones instead of the vanilla backbone. I think these are sufficiently simple modification such that they can still be considered the same method, but adapted for the re-texturing task. By the same token, none of the included baselines was designed for the task of re-texturing, and the use of an initial texture provides a significant performance boost as illustrated in one of your ablations. It would be more fair if the view-independent MLP is provided as initialization for the baselines as was done for Decorate3D.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "* **Q1:** Novelty: Most components utilized in this method are either ..., such as NeuS for mesh reconstruction, disentangling view-dependency via differentiable rendering of two MLPs, using depth condition for text-to-3D, and applying super-resolution diffusion models on UV textures ... As such, it is not clear to me whether this paper contains enough technical novelty to be impactful in the text-to-3D field.\n\n  **A1:**  **We extend our gratitude to the reviewer for acknowledging the technical excellence of our proposed Decorate3D which achieves state-of-the-art results.** \n\n   We would like to re-emphasize here the major difference between our Decorate3D and existing retexturing techniques. Decorate3D is designed to handle noisy 3D objects derived from real-world images. In contrast, existing retexturing approaches usually work with a given ideal mesh, commonly collected from synthetic models. The real-world setting of Decorate3D gives rise to extra challenges, which cannot be addressed by a simple extension of existing workflows. For example, as shown by the ablation study, the performance of Decorate3D drops considerably without the proposed structure-aware initialization, structure-aware SDS optimization, or FVR training etc. To the best of the authors' knowledge, Decorate3D is the first method to provide a complete and effective text-driven texture generation solution, which simultaneously optimizes the high-quality and geometry-aware texture generation, for 3D objects obtained from real-world captured images.  Therefore, we have reason to believe that the work is novel and significant as appreciated at this point by other reviewers, and would like to humbly request the reviewer to reconsider the novelty issue.\n \n***\n* **Q2:** The experiments can be more convincing if other SDS-based approaches (namely dreamfusion and latent paint) are also equipped with depth-conditioned diffusion backbones instead of the vanilla backbone. ... It would be more fair if the view-independent MLP is provided as initialization for the baselines as was done for Decorate3D. \n\n  **A2:**  First of all, we want to point out that incorporating depth-conditioned backbone can mitigate the Multi-face Janus problem, but the the visually blurry results from DreamFusion and Latent Paint may not be attributed to the lack of depth guidance. DreamFusion is optimized over NeRF and Latent Paint is optimized over the latent space. None of them works on the UV RGB texture space, which is first verified by Decorate3D. This also prevents them from using the same initialization technique as Decorate3D to solve the challenges of retexturing real-world 3D objects. We also want to mention that the latest SOTA retexturing work, i.e. TEXTure adopted the depth guidance strategy but still performed much worse than Decorate3D. \n\n  Second, our superior performance does not just come from depth-conditioned diffusion backbone and initialization. **These two proposed techniques are used to guarantee that the generated texture can match the geometry.** Our neural renderer and Few-view Resampling Training play an important role in our high-fidelity texture. **We are the first to address this problem in UV texture generation, which the competitors cannot achieve.**\n\n***\n* **Q3:** What is the rendering model used in the SDS optimization step? \n\n  **A3:** We used the pure diffuse rendering model for UV texture without any lighting effects incorporated. This paper primarily focuses on improving the accuracy and quality of texture generation. While we agree that the utility of lighting decomposition and material modeling are useful (which will be our future work), our generated texture quality and fidelity are not affected by those factors.\n***\n* **Q4:** Could we remove the initialization and using a textureless rendering as input to the depth estimator?\n  \n  **A4:**  The depth estimator is trained over natural images, therefore using textureless rendering might not contribute to better depth estimation. If we remove the initialization, we could use the rendered depth from z-buffer which has to be normalized to feed into the depth-conditioned diffusion model. Please refer to Figure A for the results of the ablation study involving the removal of initialization. \n***\n* **Q5:**  Computational cost and  potential negative societal impacts such as identity theft, deep fakes, and manufacturing of disinformation should be mentioned, and a discussion on societal impacts. \n\n  **A5:**  The optimization-based generation naturally has a higher computational cost than the feed-forward methods. This shortage is a common and unsolved problem of current text-driven 3D generation techniques. Reducing the computational cost remains as our future work. For the discussion on societal impacts, we will add a broader impact statement. \n"}, {"review_id": "exmcO62uvs", "reviewer": "Reviewer_7jco", "summary": "The authors present Decorate3D, a technique for text-driven texturing of a 3D mesh given a NeRF representation of a given scene. To this end, the authors introduce a two-stage texturing scheme. First, the NeRF is decomposed into a 3D mesh and view-dependent texture map. Second, given the reconstructed diffuse UV texture map, the authors edit the mesh using a modified score-distillation objective that considers the structure, or depth, of the input. Finally, to mitigate some jittering artifacts of the resulting editing texture map, the authors propose a few-view resampling technique. The authors compare their texturing scheme to existing techniques and provide many visual examples to show the 3D consistency of the textured meshes. Additional ablation studies are provided to validate the core design choices of Decorate3D.", "questions": "- Regarding the FVR stage, the authors chose to sample 8 views, if I understand correctly. Couldn’t this miss texturing areas of objects with complex areas? Does the FVR assist mainly in simpler geometries? A discussion on where to use the FVR would be beneficial since I assume that the FVR can assist differently for different geometries.\n- For the 3D-consistent texture editing, wouldn’t editing the UV texture still maintain the 3D consistency? What is the intuition behind editing one of the rendered images?", "limitations": "The authors discuss the limitations of their method.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The visual results achieved by Decorate3D are impressive and appear to surpass existing texturing methods. Additional quantitative evaluations across numerous objects are used to further validate the effectiveness of the technique.\n- I am not familiar with existing works that operate over a real 3D scene and NeRF model. For example, to the best of my understanding, most works assume that a 3D mesh is provided. Here, the authors operate in a real-world setting, which adds an additional challenge that is overcome quite nicely.\n- Although the overall system is quite complex, the different components are presented quite nicely and can be understood after careful reading. The intuitions provided by the authors to motivate the different component help in understanding the design of Decorate3D.\n- Finally, many ablation studies are provided to validate the different components of Decorate3D. Although I would have liked to see more visual results, I believe this can easily be added to the revision.", "weaknesses": "**General Points:**   \n- The visual results of TEXTure raise some concern on whether the method was run correctly by the authors. From my experience with the official code base, the results should be of much higher quality. In the TEXTure paper itself, the authors show an Ironman texture of a mesh and the results look far better than those presented in the paper. Moreover, in DreamAvatar [Cao et al. 2023] the authors also compare to TEXTure and achieve much better results for TEXTure. These visual results also seem to contradict the quantitative results, which placed TEXTure quite closely to Decorate3D. I want to give the authors the benefit of the doubt here, but could the authors please clarify and verify how the results for TEXTure were obtained? \n- The method cannot edit the geometry, which is needed in real-world applications. Specifically, the authors do not explore the robustness of the quality of the resulting mesh. For example, does the method still work nicely if the mesh contains defects such as holes or a few faces? Moreover, the authors assume that there is some semantic relation between the prompts and the geometry in order to get reasonable results. \n    - This is discussed by the authors as a limitation. \n\n**Ablation Studies:** \n- After the decomposition stage, is the resulting diffuse texture map consistent across all views? From my understanding, already at this point, the texturing should be 3D-consistent. However, I could not find results obtained after the decomposition stage (e.g., reconstruction) to verify this. \n- It is difficult to assess the contribution of the structure-aware SDS from a single visual example. This also holds for the other ablations performed by the authors (e.g., the FVR training). Additional visual results, and ideally, more quantitative evaluations (e.g., as done in Table 1) would greatly assist in truly evaluating the contribution of each component. \n- I had a difficult time understanding the contribution of the few-view resampling training. If I understood correctly, Figure 8 is designed to show the improvements obtained using the FVR training. Could the authors provide some additional examples that illustrate this improvement? Could the super-resolution model be applied directly to the previous result? And if we do so, would this also help with the jittering effect? That is, I am wondering whether the improved results are from the FVR or from the super-resolution model. Based on the results provided, it appears that the FVR does provide some minor improvements, but additional examples and an ablation study on applying the super-resolution model directly to the previous step would be helpful to highlight this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed comment. We kindly remind you to check our supplymentary material that provides some video results. We think the video results are  helpful to dispel your concerns. \n***\n* **Q1:** Concern on results of TEXTure. Could the authors please clarify and verify how the results for TEXTure were obtained?  \n\n  **A1:** We use the officially released code provided by the authors of TEXTure. To make sure we have set everything up correctly, we validated the results by employing the same human mesh model shown in their paper.  Further details can be found in the accompanying one-page PDF document (see Figure C). \n\n  We also ran the demo on HuggingFace which produces similar results. The texture looks decent on the front view but the artifacts are quite obvious at the back view. This is caused by the error accumulation of their progressive texture updating strategy starting from the front view to the back. We found the same artifacts when using our experimental mesh models. \n\n  Our mesh models are even more challenging, because \n   * **(1)** it is captured from the real world, the mesh is not perfectly clean, and\n   * **(2)** the coordinates of the reconstructed object cannot be perfectly aligned with their settings of front/back view. \n\n   The supplementary material includes an anonymous project page, where we provide the human object file used in our experiments. If necessary, the reviewer could verify the TEXTure's results with our mesh file. In the anonymous page, you can click the 'code' button and the file path in the anonymous code link is 'docs/samples/qualitative/knightarmor/texture/mesh.obj'. \n\n***\n* **Q2:** Does the method still work nicely if the mesh contains defects such as holes or a few faces? \n\n  **A2:** Since the test cases of our mesh model are reconstructed from real-world captured images, they are not guaranteed to be watertight and there are some holes around the borders (see Figure C). \n\n   Since we use the UV parameterization to represent the texture, the generated texture quality is not affected by the number of faces. For example, the mesh of the Monitor object only has a few faces(1216 faces), while the generated texture is still decent. \n   \n   Please refer to Figure C in the one-page pdf document and the supplementary material.\n*** \n* **Q3:** After the decomposition stage, is the resulting diffuse texture map consistent across all views?\n\n  **A3:**  Yes, the diffuse texture is consistent across views. Please refer to the one-page pdf document and the demo video (from 3:00 to 3:20) in the supplementary material. \n \n***\n* **Q4:** Additional visual results, and ideally, more quantitative evaluations (e.g., as done in Table 1) would greatly assist in truly evaluating the contribution of each component.\n\n  **A4:** Please refer to Table A and Figure A in the one-page  pdf document. The supplementary material provides the vieo results. \n\n***\n* **Q5:**  Could the authors provide some additional examples that illustrate the improvement of using FVR? \n\n  **A5:**  In the one-page pdf document, we visualize the jittering problem on the left side of Figure B. The figure shows the error maps of neighboring views, where the views are aligned to a reference view by using the rendered depth. Please also refer to the supplementary video (starting from 1:50 to 2:04) to compare the results before and after FVR training. \n\n***\n* **Q6:** Could the super-resolution model be applied directly to the previous result to address the jittering effect? \n\n  **A6:**  FVR is designed to solve the jittering problem and super-resolution is applied directly on the UV Texture to enhance the resolution of the global texture. If we neglect the FVR while directly applying super-resolution on the rendered view after the Neural Renderer, i.e. $\\text{SR}(V(\\mathcal{R}(\\psi,\\mathcal{M},\\mathcal{P}_i)))$ , the jittering effects will still exist. \n\n***\n* **Q7:** In FVR stage, choosing 8 views will miss texturing areas of objects with complex areas?\n\n  **A7:**  Yes, there could be some missing areas if not covered by the sampled views. For the objects in our experiments, we assume N=8 views (2 elevation angles chosen from \\{$-20^\\circ,20^\\circ$\\} and 4 azimuth angles uniformly sampled between [$0^\\circ$, $360^\\circ$]) that can cover most of the mesh surface. \n\n   Technically, for more complex geometries, we can consider adopting a more general solution. We can infer a UV mask from the UV texture to indicate which areas are overlooked by previous N views. This UV mask can be easily computed by using the camera matrices of previously sampled N views and the UV atlas. Then we can take one more step to sample more views (M views)to cover the overlooked areas of previous N views. Finally, we can train the $\\text{MLP}_{\\tilde{\\phi}}$ with the N+M views using the FVR training. \n\n***\n* **Q8:** For the 3D-consistent texture editing, wouldn’t editing the UV texture still maintain the 3D consistency? What is the intuition behind editing one of the rendered images? \n\n  **A8:** Yes, directly editing the UV texture will surely maintain 3D consistency. However, editing UV texture is quite inconvenient for non-professional users, particularly when the targeted region for editing becomes fragmented across non-connected parts in the UV map. Hence, compared with editing UV texture, editing rendered images is much easier to operate for general users. 3D consistency can be maintained by propagating the editing onto UV map. \n\n***\nIf the reviewer has any further concerns, we are most willing to discuss them.\n"}]

Author rebuttal:
Thanks to all the reviewers for their constructive suggestions. Extra visual results and quantitative evaluations are included in our submitted one-page pdf document, as suggested by reviewers: 
* (1) Figure A shows more ablation study results on the proposed components of Decorate3D including initialization, depth guidance, and our neural renderer;
* (2) Table A provides the quantitative ablation study results;
* (3) Figure B visualizes the error map to demonstrate the jittering artifacts and provides more ablation study results using a bigger N for FVR training;
* (4) Figure C provides extra results of TEXTure, rendering results after our decomposition phase, and the real-world meshes.

**We would like to bring to your notice that  video results have been included in our supplementary material**. It is suggested to watch them for more visually appealing results from Decorate3D. 
Next, we will respond to each reviewer separately about their comments and questions. 

Meta-review: The paper proposed a framework to do text-guided texture generation in the wild, the idea that reviewers are excited about is mainly texture generation with a neural renderer, few-view resampling training, and a super-resolution diffusion model. These three components helped the paper to achieve better quality in texture generation (as shown in both qualitative results and the ablation studies). The rationale behind these three components can provide good insights for the community and thus, I recommend accepting the paper following the majority of the reviewers.

As promised in the rebuttal, please add more details for super-resolution modules, clarification on TEXTure comparisons, the fair comparisons with magic3d and fatasia3D, and more visualizations of the ablation studies (for neural renderer and few-view resampling training).
Decision: Accept (poster)