Title: Decorate3D: Text-Driven High-Quality Texture Generation for Mesh Decoration in the Wild
Venue: NeurIPS 2023
Accepted: yes
Keywords: Texture Generation

Text-Driven

3D-Consistent Editing

Neural Radiance Field
Abstract: This paper presents Decorate3D, a versatile and user-friendly method for the creation and editing of 3D objects using images. Decorate3D models a real-world object of interest by neural radiance field (NeRF) and decomposes the NeRF representation into an explicit mesh representation, a view-dependent texture, and a diffuse UV texture. Subsequently, users can either manually edit the UV or provide a prompt for the automatic generation of a new 3D-consistent texture.  To achieve high-quality 3D texture generation, we propose a structure-aware score distillation sampling method to optimize a neural UV texture based on user-defined text and empower an image diffusion model with 3D-consistent generation capability. Furthermore, we introduce a few-view resampling training method and utilize a super-resolution model to obtain refined high-resolution UV textures (2048$\times$2048) for 3D texturing. Extensive experiments collectively validate the superior performance of Decorate3D in retexturing real-world 3D objects. Project page: https://decorate3d.github.io/Decorate3D/.

Decorate3D: Text-Driven High-Quality Texture Generation for Mesh Decoration in the Wild

Abstract

This paper presents Decorate3D, a versatile and user-friendly method for the creation and editing of 3D objects using images. Decorate3D models a real-world object of interest by neural radiance field (NeRF) and decomposes the NeRF representation into an explicit mesh representation, a view-dependent texture, and a diffuse UV texture. Subsequently, users can either manually edit the UV or provide a prompt for the automatic generation of a new 3D-consistent texture. To achieve high-quality 3D texture generation, we propose a structure-aware score distillation sampling method to optimize a neural UV texture based on user-defined text and empower an image diffusion model with 3D-consistent generation capability. Furthermore, we introduce a few-view resampling training method and utilize a super-resolution model to obtain refined high-resolution UV textures (2048 × 2048) for 3D texturing. Extensive experiments collectively validate the superior performance of Decorate3D in retexturing real-world 3D objects. Project page: https://decorate3d.github.io/Decorate3D/ .

1 Introduction

The recent development of effective neural 3D reconstruction techniques such as neural radiance field (NeRF) [12] has facilitated the creation of realistic digital replicas for real-world 3D scenes. While significant progress has been made in realistic 3D reconstruction, one remaining challenge is how to allow users to edit or retexture the acquired 3D objects or assets in the digital scene. To address this issue, this paper introduces Decorate 3D, a user-friendly approach to editing 3D objects through either easy-to-use manual editing or text prompt guided texture generation, as showcased in Fig. 1.

Since the implicit representations of the NeRF model are tightly coupled, it is not trivial to achieve the aforementioned operations for the 3D scene's decoration. To facilitate the modifiability of the 3D representations, we bake the NeRF model into a triangle mesh representation with a view-dependent texture in UV space, which is called a Decomposition phase in this paper. The decoupled geometry and UV texture representation make the subsequent Decoration phase more manageable. In the decoration phase, Decorate3D allows flexible texture editing and controllable generation of UV textures with the instruction of prompts.

The success of the decoration phase demands a solution to the text-to-3D synthesis problem, which has been so far restricted by the lack of paired text and 3D assets. Recent advancements in textto-image models [23, 19, 15] greatly facilitate text-driven 3D editing or generation [32, 1, 16, 11]. Researchers have proposed approaches using pre-trained text-to-2D diffusion models to optimize NeRF in the zero-shot setting via the Score Distillation Sampling (SDS) [16] strategy. Although

impressive results have been achieved, the naive SDS optimization based on 2D diffusion models lacks 3D awareness, resulting in inharmonious textures that misalign with the geometry.

To address the above issues, during the decoration phase of Decorate3D , we carry out a direct update of the global UV texture map via a structure-aware SDS optimization. We employ a pre-trained depth-guided text-to-image latent diffusion model [29, 23] and incorporate the initial decoupled UV texture from the decomposition phase as an additional form of structural guidance. We empirically find that rendering with the straightforwardly optimized UV texture from SDS tends to be noisy with color distortions. The reason is that the optimized UV texture stands for neural features in effect, which produce rendered neural images that necessitate a neural interpreter. We incorporate the idea of deferred neural rendering [31] and synthesize images of different views by forwarding the rendered neural images to the encoder-decoder of the latent diffusion model. However, the decoder's view-by-view synthesis causes jittering effects. To address this issue, a few-view resampling training strategy is proposed to optimize a global UV texture from sparsely sampled view directions. Finally, to further enhance the visual quality, we use a super-resolution diffusion model to improve the resolution, which is directly applied to the UV space.

Our key contributions can be summarized as follows. First, we propose a system pipeline to allow convenient 3D-consistent decoration of 3D objects captured in the wild; Second, with our structureaware 3D texture generation, few-view resampling training, and super-resolution enhancement, we are able to synthesize high-quality textures aligned well with the geometry; Finally, we demonstrate the effectiveness of Decorate3D on real-world datasets and have achieved superior performance over state-of-the-art approaches.

"

Fairy house

with a

garden

"

NeRF

2 Related Work

Text-to-Image Diffusion Models The past two years have witnessed the success of multiple large diffusion models [20, 5, 23, 19, 15] that are able to generate impressive images with photo-realistic details conditioned on an input text prompt. The widely popular stable diffusion [23], was trained on rich paired text-image data and is conditioned on CLIP's [17] frozen text encoder. Beyond text-conditioning, ControlNet [36] extended the stable diffusion by training a parallel hyper-network that allows controllable generation with additional input modalities such as depth maps or edges.

Text-to-3D Generation Methods The development of 2D image generation also greatly facilitates the techniques of text-to-3D generation. CLIP-Mesh [7] proposed a text-driven 3D content generation method using a pre-trained CLIP model. DreamFusion [16] first proposed a score distillation sampling (SDS) method to achieve text-driven optimization for NeRF, relying on a text-to-image diffusion model. Magic3D [9] improved DreamFusion's resolution using a coarse-to-fine optimization strategy. Latent-Paint [11] proposed to bring the NeRF to the latent space and apply the SDS to optimize a latent code for 3D scene generation. More recently, Fantasia3d [3] used a hybrid scene representation of DMTET for SDS optimization, and 3DFuse [27] improved the 3D consistency by incorporating a coarse 3D prior into fine-tuning the diffusion model. Shape-E [6] trained a conditional 3D diffusion model using paired 3D and text data. Concurrent works, TEXTure [22] as well as Text2Tex [2], introduced optimization-free iterative update schemes to paint 3D models from different viewpoints using depth-to-image diffusion models. Although these optimization-free methods may generate plausible results for simple geometries, they usually fail on complex surfaces, resulting in artifacts on the seamed areas. Decorate3D distinguishes itself from previous methods by providing 3D-consistent and high-quality structure-aware texture generation for diverse objects.

Features

UV Neural Texture

Capturing Images in the Wild

View Dependent

Directions

Decomposition Phase

Initialize

Latent Diffusion

Encoder

"

Dr Strange

doll wears

a pair of

sunglasses

, in a

fashion style

"

Super-Resolution Diffusion

Specular Color

Diffuse UV Texture

Render

Render

Random Camera

Transformer

Render

Encoder

Decoder

Few-view

resampling training

Render

3D-Consistent Decoration Phase

Depth

Backpropagate onto global UV neural texture

Diffuse Color

exture

Combine to get

final output

Render

Mesh

Fixed Model

Trainable Model

Neural Renderer

NeRF

UV Coordinates

MLP

MLP

MLP

OMLPD

VLsds(Φ,)

&

3 Approach

Decorate3D is a two-phase framework to enable 3D-consistent editing of real-world objects. The overall framework of Decorate3D is illustrated in Fig. 2. It consists of a decomposition phase and a decoration phase. Next, we will introduce the decomposition phase in Sec 3.1 followed by the decoration phase. The decoration phase is divided into three parts, including text-driven UV neural texture optimization (Sec 3.2), few-view resampling training (Sec 3.3), and super-resolution on UV texture (Sec 3.4).

3.1 Decomposition Phase

Given the captured multi-view images of an object and their respective camera poses obtained by COLMAP [26], we can easily train a NeRF model for novel view synthesis. However, NeRF editing poses challenges due to its tightly coupled representations. To address these challenges and facilitate convenient 3D editing, as well as seamless integration into downstream applications, we propose a solution: decomposing the NeRF representations into a 3D mesh and a view-dependent texture. Specifically, we employ NeuS [33] to reconstruct a triangle mesh M of the object and calculate the UV atlas associated with the mesh using XAtlas [35]. To represent the real-world texture, we decompose the texture into a diffuse texture modeled by an MLP D and a view-dependent specular texture modeled by an MLP S conditioned on a given view direction.

Denoting the differentiable mesh rendering by R , we formulate the view rendering process as follows:

where v refers to 2D UV coordinates after applying positional encoding, and f s are the intermediate features to synthesize the specular color. The features f s are forwarded to the view-dependent network MLP S to generate view-dependent effects, conditioned on the view direction d . The final rendered image c is obtained by summing up the diffuse color c d and the specular color c s .

To train the texture networks, we first adopt a typical reconstruction loss L color to minimize the difference between the rendered image ˆ c ( x ) and its corresponding captured ground truth image c ( x ) at each pixel x . The loss function is formulated as follows:

In addition, we apply an L1 regularization to enforce sparsity on the specular color:

The training of MLP D and MLP S can be achieved with a single-stage framework, as illustrated in the upper part of Fig. 2. More specifically, MLP D and MLP S are jointly optimized, where the gradient from the specular branch will only be used to update MLP S , and not be propagated to MLP D . The MLP S is a by-product for decomposing the diffuse and specular textures, not used for the decoration phase. We follow instant-ngp [14] to accelerate the optimization to achieve convergence within 3 minutes on a single NVIDIA V100 GPU. More details can be found in the supplementary.

3.2 Text-Driven Neural Texture Optimization

The text-driven texture generation is the core element of the decoration phase of Decorate3D . However, diversified 3D generation is often infeasible due to a lack of enough data pairs of text and 3D models. In this context, we utilize the SDS technique for the optimization of the neural texture.

Preliminary. Let us first introduce the SDS proposed by DreamFusion [16], which achieves textto-3D generation based on a pre-trained text-to-image diffusion-based generative model [24]. They utilize a NeRF representation to model the 3D scene. It is a parametric function x = Γ( ξ ) , which can synthesize an image x at the desired camera pose. Here, Γ is a volumetric renderer, and ξ is an MLP representing a NeRF scene. The diffusion model ϕ contains a denoising function ϵ ϕ ( x t ; y , t ) that predicts the sampled noise given the noisy image x t at timestep t , and text embedding y . The difference between the noisy image and the denoised image provides the gradient to update ξ such that the density regions with high probability are enforced to match the given text embedding. This gradient update method is named SDS, which is formulated as follows:

where the noise ϵ ∼ N (0 , I ) and w ( t ) is a weighting function. The neural rendering pipeline Γ( ξ ) and the diffusion model ϕ as modular components of the framework, are amenable to selection. This offers a practical way for text-to-3D synthesis.

Neural Texture for 3D-Consistent Rendering. As compared with previous works [16, 11, 9] on optimizing a NeRF model, we directly optimize a neural texture over the UV space. Direct texture optimization is crucial to enforce 3D consistency and preserve texture details. In detail, first we exploit the pre-trained latent diffusion model as the optimization guidance, where it has an encoder V e , a latent diffusion model's denoiser ϵ ϕ , and a decoder V d . In Decorate3D , the neural texture ψ ∈ R H × W × 3 is set to 3 feature channels to fit the requirement of the diffusion model. The gradient calculated via SDS is backpropagated through the diffusion model's encoder to the UV neural texture at a high resolution of 512 × 512 . Albeit yielding high-resolution images, the optimization computation is reasonable since the optimization exerts an effect on the latent code z ψ t = V e ( R ( ψ, M , P )) with a resolution of 64 × 64 , where V e is the diffusion model's encoder and P is the sampled camera pose. All network parameters are fixed, while only the neural texture ψ is trainable.

Here, we call the optimized texture UV Neural Texture by following the deferred neural rendering [31]. The optimized texture does not act on RGB space, and they are neural features that need a Neural Renderer to interpret, which will be introduced shortly.

Structure-Aware SDS Optimization. Apart from addressing the 3D-consistency issues, we also want to maintain the coherence between 3D geometry and the generated texture. For example, for a 3D human model, the generated facial texture should be attached to the 3D region corresponding to the face. This structure-aware requirement is overlooked by the naive SDS optimization leading to a multi-face Janus problem. To moderate this issue, Decorate3D utilizes structure-aware SDS optimization by exploiting a depth-guided latent diffusion model [29, 23].

In detail, at each denoising step t , we compute the image latent code z ψ t produced by the encoder V e and the depth latent code z ψ d t downsampled from the estimated depth map. They are concatenated as z t and forwarded to the latent diffusion module (Eq. 8). Then, following the SDS technique, the gradient of neural texture is computed as follows (Eq. 9):

where ϵ ϕ is the diffusion model's denoiser, y is the text embedding from the Transformer, and w ( t ) is a weighting function. Decorate3D uses the classifier-free guidance scheme [15] as stated in Eq. 8, and the guidance weight λ for text conditioning is set to 100. To match the depth-guided diffusion model, depth maps are predicted from the rendered views by using the depth estimator [21] of the depth-guided diffusion model rather than the depth buffer of the rendering pipeline. The overall SDS optimization framework is shown at the left bottom of Fig. 2.

In addition to structure-aware SDS optimization, Decorate3D adopts structure-aware initialization. Thanks to the decomposition phase, Decorate3D has an inherent texture that well matches the geometry information extracted from the real-world object. Therefore, Decorate3D can initialize the nerual texture ψ with the output of MLP D . This initialization plays a vital role in the optimization process, which can accelerate the convergence as well as provide structure-aware information to help it converge to a structure-aware solution. We verify the effectiveness of the structure-aware techniques in Sec 4.4.

Decorate3D's Neural Renderer. As aforementioned, directly optimizing the UV texture with SDS does not yield a traditional RGB UV texture for the rendering pipeline. Fig. 3 shows an example of SDS optimization for text-to-image generation. The optimized texture, i.e . neural texture, requires a neural interpreter to convert it back to RGB space after the SDS optimization.

Previous work [11] applies a statistical linear transformation matrix to the optimized UV texture to interpret it in the RGB UV space. However this transformation matrix is suboptimal, and the misalignment will result in color shifts and artifacts.

In Decorate3D , we introduce a neural renderer as the neural interpreter to synthesize photo-realistic images from the optimized neural texture. This rendering process is formulated as follows:

where ψ is the neural texture, V denotes the network of the neural renderer, and I P is the rendered image given the camera

w/o Interpreter

w/ Interpreter

Figure 3: The text prompt is 'An apple tree'. The left shows the image from the direct SDS optimization on a 2D image. The right shows the image after applying interpreter.

pose P . Here, Decorate3D 's neural renderer network V is the encoder-decoder ( i.e . VAE) module of the depth-guided latent diffusion model [23, 29]. The network V and the traditional renderer R have amalgamated to form the neural renderer.

3.3 Few-View Resampling Training

With the neural texture, Decorate3D can render 3D-consistent high-quality views I P through the Neural Renderer with Eq. 10 by sampling different camera poses. But there still exists a practical issue, i.e . jittering artifacts, that arises when rendering across different views. This is caused by the view-dependent neural renderer V . We empirically find that directly feeding the global UV neural texture ψ to the diffusion model's V AE ( i.e . R ( V ( ψ ) , M , P ) by swapping the network and renderer

in Eq. 10) can produce the RGB UV texture for the traditional rendering pipeline. But this naive approach usually yields blurry texturing results, as demonstrated in Sec 4.4.

To solve the jittering problem, we devise a few-view resampling (FVR) training method, that transfers the synthesized view-dependent images into a global UV texture ˜ ψ . As shown in the right bottom of Fig. 2, we use a MLP ˜ ψ to represent the UV texture ˜ ψ . In the FVR training, we sample N rendered views using the neural renderer with the neural texture ψ . The sampled N views should overlay the mesh surface as much as possible. But setting a big N may have negative effects, leading to over-smoothing textures on the overlapped areas that suffer jittering artifacts. The FVR training loss is defined as follows:

where ˜ v denotes the positional encoding of the 2D UV coordinates of the RGB UV texture ˜ ψ , and P i denotes the i -th sampled camera pose. The pose is sampled in spherical coordinates, with two elevation angles θ cam chosen from {-20 ◦ , +20 ◦ } , four azimuth angles β cam uniformly sampled between [0 ◦ , 360 ◦ ] , and an appropriate viewing distance r cam . In our experiments, N is set to 8.

3.4 Super-Resolution on UV Texture

From the SDS approach, we can only synthesize images at a resolution of 512 × 512. To further improve the spatial resolution of the UV texture ˜ ψ , Decorate3D applies a super-resolution (SR) diffusion model with a × 4 scale factor [25, 30] on the UV texture to obtain a 2048 × 2048 UVtexture. The rationale behind this successful application of UV texture upscaling is that the upsampling operation has a spatial locality, concentrating on local textures such as edges. For this reason, SR models are usually trained on cropped image patches [28, 4, 10] instead of the whole image to increase the training efficiency. This spatial locality of SR allows the SR model trained on natural images to be directly applied to the UV texture. Directly applying the SR operation to the UV texture is free from any jittering issues, and 3D consistency is well preserved.

4 Experiments

We compare Decorate3D to the state-of-the-art (SOTA) techniques for text-to-3D texture generation and evaluate its performance from both qualitative and quantitative perspectives. The 3D assets and extended videos are presented in the supplementary material.

4.1 Implementation Details

To evaluate our approach, we collect real-world datasets from 14 different objects that vary in complexity, including boxes, monitors, shoes, statues, dolls, humans, and lighthouses. Ten objects are captured in the wild using a smartphone, and four objects are selected from some public realworld datasets [34, 8]. The size of each dataset ranges from 70-300 images, and the images are downsampled to a resolution width of 640. We use the Adam optimizer to optimize the MLP D , MLP S and MLP ˜ ψ with a learning rate of 1 × 10 -3 , and the ψ with a learning rate of 1 × 10 -2 . The neural texture optimization in the decoration phase takes about 2 hours for 100K iterations, and the FVR training takes about 5 minutes for 30K iterations, which are measured on 8 NVIDIA V100 GPUs. Please refer to the supplementary material for more details.

4.2 Qualitative Evaluation and Comparison

3D-Consistent Text-Driven Texture Generation. In Fig. 4, we show the textured mesh and its corresponding real-world image. It can be observed that Decorate3D is able to produce high-quality textured mesh. As the mesh geometry is fixed, we find only texture-related keywords such as nouns or adjectives will affect the generated results, highlighted with a purple color in the prompt. For instance, in the prompt 'An astronaut stands up in the milky way', 'astronaut' and 'milky way' dominate the generation, but the verb 'stands up' and the quantifier 'an' do not affect the results.

A

white horse

statue,

with red hairs

An

astronaut

stands up in

the

milky way

A

lotion case

, painted with

A man wears a cosplay suit of an

armored medieval Japanese samurai

A

detergent case

, printed with a

movie poster of Star Wars

A

doll

model of the movie of

Zootopia,

wearing a spiderman suit

stands up in

STAR

WARS

Ironman

, with hands

clasped behind back,

standing up

on the grass

An

ancient

Chinese-style

house

A

man

wears a set

of

medieval

knight armor

Woman martin

shoes

, with a

wizard style,

put

on a

treasure box

Original

CLIP-Mesh

DreamFusion

Latent-Paint

TEXTure

Fig. 5 shows the qualitative comparison with SOTA text-driven texture generation methods, including CLIP-Mesh [7], Latent-Paint [11] and TEXTure [22]. They all took the reconstructed 3D mesh as input, and the geometry was fixed during texture generation. We can observe that the generated textures from CLIP-Mesh and Latent-Paint lack texture details. The concurrent work TEXTure has some competitive results compared to Decorate3D , such as the results for shoes, because they use a depth-guidance strategy similar to ours. But TEXTure produces weird artifacts on the complex surfaces. For example, in the case of humans, the generated textures from TEXTure have obvious seams, and textures from different parts seem to be glued together and look messy. The artifacts are caused by the iterative update strategy across views. By contrast, the results of Decorate3D have fruitful and clear details. We also present the results from DreamFusion [16]. It tends to produce blurry results that do not actually match the text prompt.

Adding Decomposed Specular Color

(

)

Textured Mesh

Propagated to UV Texture

Single-View Editing

A

running horse

having

green stripes

like

zebra

Original

Text + Single-View Editing

Propagated to UV Texture

Textured Mesh

Adding Decomposed Specular Color

orate3l

WeuriPs

2023

3D-Consistent Texture Editing. Thanks to the decomposed 3D representations, Decorate3D opens up an easy way to edit textures. Fig. 6 depicts a practical and distinctive use case that can be accomplished by Decorate3D . We edit one of the rendered images (not the UV texture), and propagate the editing to the UV texture to synthesize 3D-consistent edited textures from different views. The decomposed view-dependent specular texture can be added back to the textured model. Additionally, Decorate3D allows secondary editing of the text-driven texture generation.

4.3 Quantitative Evaluation and Comparison

CLIP R-Precision Metric. Following DreamFusion, we evaluate the CLIP R-Precision [18], an automated metric for the consistency of rendered multi-view images with respect to the input prompt. Here, we calculate the average CLIP score from the front-side, left-side, right-side, and back-side views. We use 70 prompts of 14 objects to generate the test results. For visual quality, we measure the NIQE [13] on rendered images, which is a no-reference image quality assessment. Tab. 1 reports the CLIP R-Precision scores of the compared methods.

User Studies. We invite 44 volunteers to evaluate Decorate3D and its competitors using the meanopinion-score (MOS) test. For each question, we prepare 5 video results, and the participants are asked to rate the results on a scale from 1 (worst) to 5 (best) based on the overall visual quality of the results and the degree to which they match the text prompt. In the end, we receive 1100 responses from the 44 volunteers. Fig. 7 shows the average MOS scores of the compared methods. As can be seen, Decorate3D is shown to be more favored by human users.

4.4 Ablation Studies

Initialization of UV Neural Texture. Fig. 8a shows an ablation study on the initialization of the UV neural texture and Fig. 11 shows more results. As observed, the optimization initialized with

1K iters

5K iters

9K iters

12K iters

Final

Random Initialization

Initialization with

"

Hello Kitty

doll

wears a

bandit suit

"

w/o depth

w/ depth

Original

MLPp

(a) An ablation study of neural texture's initialization.

(b) w/o depth vs. w/ depth.

"Woman martin shoes

,

with a

wizard style

,

put on a

treasure box

"

(1) VAE

(2) Per-View Generation

(3)

(4)

+ SR

(c) Ablation studies of FVR training and the super-resolution on UV texture.

MLP D converges much faster than the random initialization. Moreover, the MLP D provides a very strong optimization prior to helping the SDS optimization converge to a better solution that well matches the mesh geometry. For example, initialization using MLP D can generate the correct facial texture that fits the geometry, but on the contrary, the random initialization fails.

Effectiveness of Structure-Aware SDS. Fig. 8b presents the difference between the naive SDS without depth guidance and the structure-aware SDS with depth guidance, and Fig. 11 shows more results. We can observe that the structure-aware SDS is able to produce geometry-matching textures. For example, the human's arms should be clasped behind the back rather than being akimbo.

Is the Few-View Resampling Training Necessary? Fig. 9 compares the per-view generated results of V ( R ( ψ, M , P )) with the results rendered using the FVR-trained UV texture. As observed, results of per-view generation have jittering effects between different views (look at the circle pattern), but results with MLP ˜ ψ have achieved 3D consistency. Fig. 8c (1) presents the results using an alternative solution to eliminate the jittering, i.e . directly feeding the neural texture into the diffusion model's VAE by R ( V ( ψ ) , M , P ) .

Synthesizing three views

(aligned to first view with depth)

1

2

3

1

2

1

3

Difference Map

w/o FVR

w/ FVR

1

2

1

3

View

Perview Generation

S

+

As observed, following this way yields blurry textures. Given an RGB UV texture ˜ ψ , Decorate3D can directly apply SR on ˜ ψ . By comparing Fig. 8c (3) and Fig. 8c (4) we can observe more clear results after SR. Fig. 10 shows the effectiveness of FVR training's hyperparameter N . Setting a big N for FVR will cause blurry results.

Effectiveness of Neural Renderer. As introduced in Sec .3.2, the optimized UV texture by SDS is a neural texture feature. A neural renderer is required to convert the neural UV texture to the RGB UV. The right of Fig. 11 shows the ablation study results on the neural renderer. As observed, the synthesized results without a neural renderer are over-saturated and noisy.

N = 16

N=256

3D Mesh

w/o Init & w/o Depth

w/ Init & w/o Depth

w/o Init & w/ Depth

w/ Init & w/ Depth

w/ Neural Renderer

w/o Neural Renderer

5 Conclusion

Decorate3D offers a practical way of decorating real-world 3D models with a user-friendly approach through text-driven texture generation. Our Decorate3D 's techniques, combining directly optimizing UV neural texture, structure-aware optimization, FVR training, and SR enhancement on UV texture, collectively advance the line of 3D texture generation in pursuit of the best possible quality. Extensive experiments demonstrate the superiority of Decorate3D over SOTA methods.

Limitations First, the style and quality of the generated texture heavily depend on the pre-trained stable diffusion models. Second, even though Decorate3D adopts structure-aware optimization techniques, the multi-face Janus problem still remains in a flat geometry. For example, Decorate3D cannot distinguish the front and back sides of a monitor object, as shown in our supplementary material. Lastly, Decorate3D does not support jointly optimizing both the geometry and texture, which may result in inconsistency with the mesh geometry. We remain these problems for future study.

Acknowledgments and Disclosure of Funding

We gratefully acknowledge the support of MindSpore (https://www.mindspore.cn/), CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.

References

Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18392-18402, 2023.

Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint arXiv:2303.11396 , 2023.

Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22246-22256, 2023.

Yanhui Guo, Xiaolin Wu, and Xiao Shu. Data acquisition and preparation for dual-reference deep learning of image super-resolution. IEEE Transactions on Image Processing , 31:43934404, 2022.

Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research , 23(47):1-33, 2022.

Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463 , 2023.

Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. CLIP-mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 Conference Papers . ACM, nov 2022.

Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics , 36(4), 2017.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 300-309, 2023.

Fangzhou Luo, Xiaolin Wu, and Yanhui Guo. Functional neural networks for parametric image restoration problems. Advances in Neural Information Processing Systems , 34:6762-6775, 2021.

Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12663-12673, 2023.

Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: representing scenes as neural radiance fields for view synthesi. In European Conference on Computer Vision (ECCV) , pages 405-421, 2020.

Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. Making a 'completely blind' image quality analyzer. IEEE Signal Processing Letters , 20(3):209-212, 2013.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. , 41(4):102:1-102:15, July 2022.

Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning , pages 16784-16804. PMLR, 2022.

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In International Conference on Learning Representations (ICLR) , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning , volume 139, pages 8748-8763. PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821-8831. PMLR, 2021.

René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence , 44(3):1623-1637, 2020.

Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text-guided texturing of 3d shapes. arXiv preprint arXiv:2302.01721 , 2023.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems , pages 36479-36494, 2022.

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(4):4713 - 4726, 2022.

Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 41044113, 2016.

Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, JinHwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. arXiv preprint arXiv:2303.07937 , 2023.

Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1874-1883, 2016.

Stability-AI. Depth-guided stable diffusion model. https://github.com/ Stability-AI/stablediffusion , 2023.

Stability-AI. Diffusion models for super-resolution. https://huggingface.co/ CompVis/ldm-super-resolution-4x-openimages , 2023.

Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. Acm Transactions on Graphics (TOG) , 38(4):1-12, 2019.

Can Wang, Ruixiang Jiang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Nerf-art: Text-driven neural radiance fields stylization. IEEE Transactions on Visualization and Computer Graphics , 2023.

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems , 34:27171-27183, 2021.

Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1790-1799, 2020.

Jonathan Young. Xatlas: Mesh parameterization / uv unwrapping library. https://github. com/jpcy/xatlas , 2023.

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3836-3847, 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present Decorate3D, a technique for text-driven texturing of a 3D mesh given a NeRF representation of a given scene. To this end, the authors introduce a two-stage texturing scheme. First, the NeRF is decomposed into a 3D mesh and view-dependent texture map. Second, given the reconstructed diffuse UV texture map, the authors edit the mesh using a modified score-distillation objective that considers the structure, or depth, of the input. Finally, to mitigate some jittering artifacts of the resulting editing texture map, the authors propose a few-view resampling technique. The authors compare their texturing scheme to existing techniques and provide many visual examples to show the 3D consistency of the textured meshes. Additional ablation studies are provided to validate the core design choices of Decorate3D.\n\nSTRENGTHS:\n- The visual results achieved by Decorate3D are impressive and appear to surpass existing texturing methods. Additional quantitative evaluations across numerous objects are used to further validate the effectiveness of the technique.\n- I am not familiar with existing works that operate over a real 3D scene and NeRF model. For example, to the best of my understanding, most works assume that a 3D mesh is provided. Here, the authors operate in a real-world setting, which adds an additional challenge that is overcome quite nicely.\n- Although the overall system is quite complex, the different components are presented quite nicely and can be understood after careful reading. The intuitions provided by the authors to motivate the different component help in understanding the design of Decorate3D.\n- Finally, many ablation studies are provided to validate the different components of Decorate3D. Although I would have liked to see more visual results, I believe this can easily be added to the revision.\n\nWEAKNESSES:\n**General Points:**   \n- The visual results of TEXTure raise some concern on whether the method was run correctly by the authors. From my experience with the official code base, the results should be of much higher quality. In the TEXTure paper itself, the authors show an Ironman texture of a mesh and the results look far better than those presented in the paper. Moreover, in DreamAvatar [Cao et al. 2023] the authors also compare to TEXTure and achieve much better results for TEXTure. These visual results also seem to contradict the quantitative results, which placed TEXTure quite closely to Decorate3D. I want to give the authors the benefit of the doubt here, but could the authors please clarify and verify how the results for TEXTure were obtained? \n- The method cannot edit the geometry, which is needed in real-world applications. Specifically, the authors do not explore the robustness of the quality of the resulting mesh. For example, does the method still work nicely if the mesh contains defects such as holes or a few faces? Moreover, the authors assume that there is some semantic relation between the prompts and the geometry in order to get reasonable results. \n    - This is discussed by the authors as a limitation. \n\n**Ablation Studies:** \n- After the decomposition stage, is the resulting diffuse texture map consistent across all views? From my understanding, already at this point, the texturing should be 3D-consistent. However, I could not find results obtained after the decomposition stage (e.g., reconstruction) to verify this. \n- It is difficult to assess the contribution of the structure-aware SDS from a single visual example. This also holds for the other ablations performed by the authors (e.g., the FVR training). Additional visual results, and ideally, more quantitative evaluations (e.g., as done in Table 1) would greatly assist in truly evaluating the contribution of each component. \n- I had a difficult time understanding the contribution of the few-view resampling training. If I understood correctly, Figure 8 is designed to show the improvements obtained using the FVR training. Could the authors provide some additional examples that illustrate this improvement? Could the super-resolution model be applied directly to the previous result? And if we do so, would this also help with the jittering effect? That is, I am wondering whether the improved results are from the FVR or from the super-resolution model. Based on the results provided, it appears that the FVR does provide some minor improvements, but additional examples and an ablation study on applying the super-resolution model directly to the previous step would be helpful to highlight this.\n\nQUESTIONS:\n- Regarding the FVR stage, the authors chose to sample 8 views, if I understand correctly. Couldn’t this miss texturing areas of objects with complex areas? Does the FVR assist mainly in simpler geometries? A discussion on where to use the FVR would be beneficial since I assume that the FVR can assist differently for different geometries.\n- For the 3D-consistent texture editing, wouldn’t editing the UV texture still maintain the 3D consistency? What is the intuition behind editing one of the rendered images?\n\nLIMITATIONS:\nThe authors discuss the limitations of their method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes decorate3D, a method for re-texturing real-world 3D objects using text-conditioned image diffusion models. The proposed method can be split into a 3D reconstruction phase and a re-texturing phase. In the 3D reconstruction phase, a 3D mesh is reconstructed from set of multiview images via NeuS, and a view independent texture map is distilled via differentiable rendering. In the re-texturing phase, a depth-conditioned latent diffusion model is combined with SDS to optimize the texture map. The texture map is re-rendered by passing it through the encoder-decoder of stable diffusion to remove neural artifacts, but this step introduces jittering artifacts. Jittering artifacts are then removed via optimizing a MLP through few-view resample training to reconcile view-inconsistencies. Lastly, a super-resolution diffusion model is used to up-res the produced texture map.\n\nSTRENGTHS:\n1. The method is technically impressive, utilizing many interesting tricks to overcome problems associated with latent diffusion models, and thereby obtaining visually impressive experimental results.\n2. The presentation of the method is precise and easy to follow. Despite the many moving parts, never once did I feel a need to backtrack due to inconsistent notations or frivolous math equations.\n3. The experimental procedure is detailed and well documented. One can be confident of the reproducibility of the results (as long as the authors release the real world data they've collected). The ablations are also fairely thorough, giving clear intuitions as to the effect of each component.\n\nWEAKNESSES:\n1. Novelty:\nMost components utilized in this method are either well known to the literature, or straight-forward extensions of existing workflows, such as NeuS for mesh reconstruction, disentangling view-dependency via differentiable rendering of two MLPs, using depth condition for text-to-3D, and appling super-resolution diffusion models on UV textures. Though the problem of SDS with LDMs as observed in figure 3 has not been formally studied in a research paper, knowledge of this problem is folklore within the community and the proposed neural renderer solution is rather simplistic. As such, it is not clear to me whether this paper contains enough technical novelty to be impactful in the text-to-3D field.\n2. Fairness of comparisons:\nThe experiments can be more convincing if other SDS based approaches (namely dreamfusion and latent paint) are also equipped with depth conditioned diffusion backbones instead of the vanilla backbone. I think these are sufficiently simple modification such that they can still be considered the same method, but adapted for the re-texturing task. By the same token, none of the included baselines was designed for the task of re-texturing, and the use of an initial texture provides a significant performance boost as illustrated in one of your ablations. It would be more fair if the view-independent MLP is provided as initialization for the baselines as was done for Decorate3D.\n\nQUESTIONS:\n1. what is the rendering model used in the SDS optimization step? Is any lighting/view-dependent effects incorporated into the rendering equation (as used in magic3d and fantasia3d) or is it purely UV based retrieval from a neural texture? Do you have any ablations on this?\n2. could this method be adapted for texture synthesis by removing the initialization and using a textureless rendering of the geometry as input to the depth estimator (thereby preventing compounding artifacts between bad initial texture and bad depth estimation)?\n\nLIMITATIONS:\nI think the computational cost of this method is a limitation worth mentioning - it is by far the most expensive method to run versus its baselines, whose runtime ranges from seconds to tens of minutes on a single GPU, whereas Decorate3D requires hours on full 8 GPUs.\nPotential negative societal impacts such as identity theft, deep fakes, and manufacturing of disinformation should be mentioned.\n\n---------------------------------------------Post rebuttal:\nI think the changes to the manuscript promised by the authors will significantly improve the delivery and message of the paper by firmly substantiating their claims regarding the effectiveness of proposed techniques with more ablations. Therefore I'm changing my suggestion to acceptance.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to edit the textures for neural fields (NeRFs) using score distillation sampling and also export a mesh model with texture that can be used in traditional graphics pipelines (i.e. game engines, VFX). More specifically, the main contributions that I see from this work is the \"Few-view Resampling Training\" which can take an SDS-optimized RGB diffuse texture map (which is noisy due to the nature of SDS with LDMs), and refine it through LDM-driven re-rendering, which takes advantage of both \"LDM as a renderer\" and having a real 3D consistent 3D model that can be used. This is specifically a general technique that could be widely applicable in a variety of different tasks.  \n\nIn addition to this, they also create an entire pipeline to extract editable & high quality mesh representations from multi-view images (i.e. ones with good geometry, good UV parameterization, diffuse + specular separation, mostly based on existing tools) as well as another case study on SDS-driven texture generation.\n\nSTRENGTHS:\nThe main strength of this paper is in the \"few view resample training\", which takes as input a noisy 3D model, renders the 3D model, refines the rendered image using a \"neural renderer\" (which in this case is the VAE of an LDM), and back propagates the refinement back to the 3D model. This as far as I'm aware is an original idea that I have not seen at least in this specific context. The method also seems to be effective from the limited results I am able to see, and is something that can likely be incorporated into many different contexts. \n\nThe paper also proposes an end-to-end pipeline for doing NeRF -> mesh -> editing, and evaluates several different tricks to make this pipeline effective which they also evaluate in some limited ablation studies. This is significant as it provides a case study for implementation tricks in making this pipeline work (which in my experience tends to be a big part of SDS based pipelines).\n\nThe clarity of the paper could be improved, but is not something that significantly detriments the paper. This will be discussed further in the weaknesses.\n\nWEAKNESSES:\nThe biggest weakness of this paper is in its clarity. With some restructuring and refinement, however, I think that this paper could be very convincing.\n\nFirst, the paper introduces the problem of 'mesh decoration'. The task really at hand is 'retexturing' or 'texture editing'. I'm not sure what the motivation for using the word 'decoration' is, but this is something that makes the paper unnecessarily confusing to grasp. \n\nSecond, the paper puts a lot of weight on discussing the end-to-end pipeline from reconstruction to retexturing. In reality, the meat of the contributions for this paper lies in the retexturing method (and specifically the few view refinement), and the rest feels like a distraction that is not core to the contribution. Making the writing and contribution statements more specific to the retexturing part of the pipeline, and treating the end-to-end pipeline as almost an 'implementation detail' would make the paper much more convincing. (i.e. the decomposition stage isn't really core to the method, since the same pipeline could be applicable for an existing 3D mesh). \n\nThird, the paper does not sufficiently compare and contrast their method with concurrent works like TEXTure which can be considered prior art given its more than 2 months before the deadline. The paper does compare against them in the evaluations, which is great, but it could use more discussion on _why_ these prior arts produce bad artifacts in their generation, and what fundamental differences makes this paper more advantageous. \n\nFourth, the results shown on the core contributions are rather light. It would be very illustrative to show (on multiple models) the rendered 3D models after the SDS optimization (with their artifacts & UV textures), after \"neural rendering\", and after refinement to really showcase the efficacy of the refinement method. \n\nFifth, the ablations are good but they could be on different figures with more examples. Some of the text space that is currently used for the description of 'prior things' like SDS and the decomposition stage could probably be placed in the supplemental or taken out to make more space for results. More results on especially the effects of higher viewpoints for the refinement step could be very useful.\n\nLastly, it would be useful potentially to write the approximate time for completion for each stage in Figure 2 to make the costs more clear.\n\nThese are not things that affect my rating, but nitpicks:\n\n22: \"Since the implicit representations of the NeRF model are tightly coupled\" this should be explained in more detail. I believe the authors are referring to the fact that it's difficult to disentangle geometry and texture from a typical NeRF model, but this phrase does not communicate this at all. \n\n41: \"The reason is that the optimized UV texture stands for neural features in effect, which produce rendered neural images that necessitate a neural interpreter\" I find the whole section starting with this sentence rather confusing and hard to interpret what it really means (without having read the rest of the paper at this point yet). Trying to describe this more precisely would help. For example, what does \"stands for ... in effect\" mean? What is a \"neural interpreter\"? I can make inferences but those are then inferences. At this point I'm also confused as a reader why the UV texture needs to be 'neural' or 'latent' at all.\n\n113: \"However, diversified 3D generation is often infeasible due to a lack of enough data pairs of text and 3D models\" I assume this sentence is in reference to an auto encoder sort of framework, but this is not explained anywhere. Also, I'm not really sure what \"diversified\" means. \n\n131: It's not made clear in this general section that the thing that is being passed to the encoder is a rendered image, not the texture map. Although this is clear from the figure, explicitly stating this here would be nice.\n\nThe writing could be improved stylistically in various places, like not starting sentences with \"And\".\n\nQUESTIONS:\n1. Why is the task at hand called 'decoration' as opposed to texturing or texture editing?\n\n2. At least to my eyes, it does not seem like the higher N for FVR makes a big difference. What would happen if we choose more extreme numbers for these, like N=1,2,4,8 and N=1024,2048? Is there a way to design a loss to make it robust to both extremes?\n\n3. Is there any cases where the multi view consistency fails for FVR? How does the number of views affect this?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations. I think they could make a broader statement on the societal impacts, as content creation tools like this are something that could potentially impact labor markets (and displace artists) and is something that is based on diffusion models trained on large amounts of data (which often also means they are unattributed / has no provenance back to artists).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper introduces Decorate3D, which enables text-guided 3D model editing by extracting and editing a learned UV texture. Specifically, Given multi-view images, Decorate3D first generates 3D mesh and UV textures based on NeuS. Then, it optimizes neural textures by the guidance of 3D structure (depth) and stable diffusion model. Finally,  an RGB UV texture is optimized and upsampled to generate the final result. Experiments demonstrate the state-of-the-art performance of Decorate3D.

STRENGTHS:
- The idea of this paper is well-motivated and presented.

- A carefully designed pipeline (Nerf rendering, depth-aware texture optimization, few-shot texture re-optimization, and texture super-resolution) enables high-quality generation results. The 3D consistent decoration phase is novel and effective. 

- Best performance is achieved compared to SOTA.

WEAKNESSES:
- The proposed method uses few-view resample training to obtain a UV texture that reduces the jittering effects. I am wondering if this step can be done in the decomposition phase or the text-driven neural texture optimization.

- The super-resolution is applied to the UV texture. However, the UV texture is not a natural image, if the model is trained with natural images, will there be some domain gap leading to inferior results? Moreover, as super-resolution is only a post-processing step that enhances the results and is not one of the main contributions, I recommend spending fewer texts on this point.

- I would like to see some quantitative ablation studies like Table 1.

QUESTIONS:
Some discussions about the above weaknesses and a quantitative ablation study would make the submission stronger.

LIMITATIONS:
Important limitations have been discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08