SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yanhui Guo $^{1}$  Xinxin Zuo $^{2}$  Peng Dai $^{2}$  Juwei Lu $^{2}$  Xiaolin Wu $^{1}$  Li Cheng $^{3}$  Youliang Yan $^{2}$  Songcen Xu $^{2}$  Xiaofei Wu $^{2}$
<sup>1</sup>McMaster University, Canada <sup>2</sup>Noah's Ark Lab, Canada <sup>3</sup>University of Alberta, Canada
{quoy143,xwu}@mcmaster.ca,lcheng5@ualberta.ca
{xinxin.zuol,peng.dai,juwei.lu}@huawei.com
{yanyouliang,xusongcen,wuxiaofei2}@huawei.com
This paper presents Decorate3D, a versatile and user-friendly method for the creation and editing of 3D objects using images. Decorate3D models a real-world object of interest by neural radiance field (NeRF) and decomposes the NeRF representation into an explicit mesh representation, a view-dependent texture, and a diffuse UV texture. Subsequently, users can either manually edit the UV or provide a prompt for the automatic generation of a new 3D-consistent texture. To achieve high-quality 3D texture generation, we propose a structure-aware score distillation sampling method to optimize a neural UV texture based on user-defined text and empower an image diffusion model with 3D-consistent generation capability. Furthermore, we introduce a few-view resampling training method and utilize a super-resolution model to obtain refined high-resolution UV textures  $(2048\times 2048)$  for 3D texturing. Extensive experiments collectively validate the superior performance of Decorate3D in retexturing real-world 3D objects. Project page: https://decorate3d.github.io/Decorate3D/.
The recent development of effective neural 3D reconstruction techniques such as neural radiance field (NeRF) [12] has facilitated the creation of realistic digital replicas for real-world 3D scenes. While significant progress has been made in realistic 3D reconstruction, one remaining challenge is how to allow users to edit or retexture the acquired 3D objects or assets in the digital scene. To address this issue, this paper introduces Decorate 3D, a user-friendly approach to editing 3D objects through either easy-to-use manual editing or text prompt guided texture generation, as showcased in Fig. 1.
Since the implicit representations of the NeRF model are tightly coupled, it is not trivial to achieve the aforementioned operations for the 3D scene's decoration. To facilitate the modifiability of the 3D representations, we bake the NeRF model into a triangle mesh representation with a view-dependent texture in UV space, which is called a Decomposition phase in this paper. The decoupled geometry and UV texture representation make the subsequent Decoration phase more manageable. In the decoration phase, Decorate3D allows flexible texture editing and controllable generation of UV textures with the instruction of prompts.
The success of the decoration phase demands a solution to the text-to-3D synthesis problem, which has been so far restricted by the lack of paired text and 3D assets. Recent advancements in text-to-image models [23, 19, 15] greatly facilitate text-driven 3D editing or generation [32, 1, 16, 11]. Researchers have proposed approaches using pre-trained text-to-2D diffusion models to optimize NeRF in the zero-shot setting via the Score Distillation Sampling (SDS) [16] strategy. Although
impressive results have been achieved, the naive SDS optimization based on 2D diffusion models lacks 3D awareness, resulting in inharmonious textures that misalign with the geometry.
To address the above issues, during the decoration phase of Decorate3D, we carry out a direct update of the global UV texture map via a structure-aware SDS optimization. We employ a pre-trained depth-guided text-to-image latent diffusion model [29, 23] and incorporate the initial decoupled UV texture from the decomposition phase as an additional form of structural guidance. We empirically find that rendering with the straightforwardly optimized UV texture from SDS tends to be noisy with color distortions. The reason is that the optimized UV texture stands for neural features in effect, which produce rendered neural images that necessitate a neural interpreter. We incorporate the idea of deferred neural rendering [31] and synthesize images of different views by forwarding the rendered neural images to the encoder-decoder of the latent diffusion model. However, the decoder's view-by-view synthesis causes jittering effects. To address this issue, a few-view resampling training strategy is proposed to optimize a global UV texture from sparsely sampled view directions. Finally, to further enhance the visual quality, we use a super-resolution diffusion model to improve the resolution, which is directly applied to the UV space.
Our key contributions can be summarized as follows. First, we propose a system pipeline to allow convenient 3D-consistent decoration of 3D objects captured in the wild; Second, with our structure-aware 3D texture generation, few-view resampling training, and super-resolution enhancement, we are able to synthesize high-quality textures aligned well with the geometry; Finally, we demonstrate the effectiveness of Decorate3D on real-world datasets and have achieved superior performance over state-of-the-art approaches.
Figure 1: Given captured images of an object, Decorate3D supports text-driven high-quality texture generation and user-friendly texture editing. Please zoom in for better visualization.
Text-to-Image Diffusion Models The past two years have witnessed the success of multiple large diffusion models [20, 5, 23, 19, 15] that are able to generate impressive images with photo-realistic details conditioned on an input text prompt. The widely popular stable diffusion [23], was trained on rich paired text-image data and is conditioned on CLIP's [17] frozen text encoder. Beyond text-conditioning, ControlNet [36] extended the stable diffusion by training a parallel hyper-network that allows controllable generation with additional input modalities such as depth maps or edges.
Text-to-3D Generation Methods The development of 2D image generation also greatly facilitates the techniques of text-to-3D generation. CLIP-Mesh [7] proposed a text-driven 3D content generation method using a pre-trained CLIP model. DreamFusion [16] first proposed a score distillation sampling (SDS) method to achieve text-driven optimization for NeRF, relying on a text-to-image diffusion model. Magic3D [9] improved DreamFusion's resolution using a coarse-to-fine optimization strategy. Latent-Paint [11] proposed to bring the NeRF to the latent space and apply the SDS to optimize a latent code for 3D scene generation. More recently, Fantasy3d [3] used a hybrid scene representation of DMTET for SDS optimization, and 3DFuse [27] improved the 3D consistency by incorporating a coarse 3D prior into fine-tuning the diffusion model. Shape-E [6] trained a conditional 3D diffusion model using paired 3D and text data. Concurrent works, TEXTure [22] as well as Text2Tex [2], introduced optimization-free iterative update schemes to paint 3D models from different viewpoints using depth-to-image diffusion models. Although these optimization-free methods may generate plausible results for simple geometries, they usually fail on complex surfaces, resulting in artifacts on the seamed areas. Decorate3D distinguishes itself from previous methods by providing 3D-consistent and high-quality structure-aware texture generation for diverse objects.
Figure 2: Overview of Decorate3D : In the decomposition phase, we utilize NeuS [33] to extract a 3D mesh, and optimize two MLPs (i.e.  $\mathrm{MLP}_{\mathcal{D}}$  and  $\mathrm{MLP}_S$ ) to model view-dependent textures. In the decoration phase, we initialize the UV neural texture  $\psi$  using  $\mathrm{MLP}_{\mathcal{D}}$ , and optimize the neural texture  $\psi$  via the depth-guided diffusion model with the score distillation sampling loss. The neural texture is rendered into sampled views and forwarded to the Neural Renderer to synthesize color images. Afterward, we optimize a MLP through the few-view resampling training method to obtain a UV texture  $\tilde{\psi}$  in RGB space. Finally, we enhance the visual quality of  $\tilde{\psi}$  with a super-resolution diffusion model.
Decorate3D is a two-phase framework to enable 3D-consistent editing of real-world objects. The overall framework of Decorate3D is illustrated in Fig. 2. It consists of a decomposition phase and a decoration phase. Next, we will introduce the decomposition phase in Sec 3.1 followed by the decoration phase. The decoration phase is divided into three parts, including text-driven UV neural texture optimization (Sec 3.2), few-view resampling training (Sec 3.3), and super-resolution on UV texture (Sec 3.4).
3.1 Decomposition Phase
Given the captured multi-view images of an object and their respective camera poses obtained by COLMAP [26], we can easily train a NeRF model for novel view synthesis. However, NeRF editing poses challenges due to its tightly coupled representations. To address these challenges and facilitate convenient 3D editing, as well as seamless integration into downstream applications, we propose a solution: decomposing the NeRF representations into a 3D mesh and a view-dependent texture. Specifically, we employ NeuS [33] to reconstruct a triangle mesh  $\mathcal{M}$  of the object and calculate the UV atlas associated with the mesh using XAtlas [35]. To represent the real-world texture, we decompose the texture into a diffuse texture modeled by an  $\mathrm{MLP}_{\mathcal{D}}$  and a view-dependent specular texture modeled by an  $\mathrm{MLP}_{\mathcal{S}}$  conditioned on a given view direction.
Denoting the differentiable mesh rendering by  $\mathcal{R}$ , we formulate the view rendering process as follows:
$$
\mathbf {c} _ {d}, \mathbf {f} _ {s} = \operatorname {s i g m o i d} \left(\mathrm {M L P} _ {\mathcal {D}} (\mathbf {v})\right), \tag {1}
$$
$$
\mathbf {c} _ {s} = \operatorname {M L P} _ {\mathcal {S}} \left(\mathbf {f} _ {s}, \mathbf {d}\right), \tag {2}
$$
$$
\mathbf {c} = \mathcal {R} \left(\mathbf {c} _ {d} + \mathbf {c} _ {s}, \mathcal {M}\right), \tag {3}
$$
where  $\mathbf{v}$  refers to 2D UV coordinates after applying positional encoding, and  $\mathbf{f}_s$  are the intermediate features to synthesize the specular color. The features  $\mathbf{f}_s$  are forwarded to the view-dependent network  $\mathrm{MLP}_S$  to generate view-dependent effects, conditioned on the view direction  $\mathbf{d}$ . The final rendered image  $\mathbf{c}$  is obtained by summing up the diffuse color  $\mathbf{c}_d$  and the specular color  $\mathbf{c}_s$ .
To train the texture networks, we first adopt a typical reconstruction loss  $\mathcal{L}_{color}$  to minimize the difference between the rendered image  $\hat{\mathbf{c}} (\mathbf{x})$  and its corresponding captured ground truth image  $\mathbf{c}(\mathbf{x})$  at each pixel  $\mathbf{x}$ . The loss function is formulated as follows:
$$
\mathcal {L} _ {\text {c o l o r}} = \sum_ {\mathbf {x}} \| \hat {\mathbf {c}} (\mathbf {x}) - \mathbf {c} (\mathbf {x}) \| ^ {2} \tag {4}
$$
In addition, we apply an L1 regularization to enforce sparsity on the specular color:
$$
\mathcal {L} _ {\text {s p e c u l a r}} = \sum_ {i} | \mathbf {c} _ {\mathbf {s}} (\mathbf {x} _ {i}) | \tag {5}
$$
The training of  $\mathrm{MLP}_{\mathcal{D}}$  and  $\mathrm{MLP}_S$  can be achieved with a single-stage framework, as illustrated in the upper part of Fig. 2. More specifically,  $\mathrm{MLP}_{\mathcal{D}}$  and  $\mathrm{MLP}_S$  are jointly optimized, where the gradient from the specular branch will only be used to update  $\mathrm{MLP}_S$ , and not be propagated to  $\mathrm{MLP}_{\mathcal{D}}$ . The  $\mathrm{MLP}_S$  is a by-product for decomposing the diffuse and specular textures, not used for the decoration phase. We follow instant-ngp [14] to accelerate the optimization to achieve convergence within 3 minutes on a single NVIDIA V100 GPU. More details can be found in the supplementary.
3.2 Text-Driven Neural Texture Optimization
The text-driven texture generation is the core element of the decoration phase of Decorate3D. However, diversified 3D generation is often infeasible due to a lack of enough data pairs of text and 3D models. In this context, we utilize the SDS technique for the optimization of the neural texture.
Preliminary. Let us first introduce the SDS proposed by DreamFusion [16], which achieves text-to-3D generation based on a pre-trained text-to-image diffusion-based generative model [24]. They utilize a NeRF representation to model the 3D scene. It is a parametric function  $\mathbf{x} = \Gamma (\xi)$ , which can synthesize an image  $\mathbf{x}$  at the desired camera pose. Here,  $\Gamma$  is a volumetric renderer, and  $\xi$  is an MLP representing a NeRF scene. The diffusion model  $\phi$  contains a denoising function  $\epsilon_{\phi}(\mathbf{x}_t;\mathbf{y},t)$  that predicts the sampled noise given the noisy image  $\mathbf{x}_t$  at timestep  $t$ , and text embedding  $\mathbf{y}$ . The difference between the noisy image and the denoised image provides the gradient to update  $\xi$  such that the density regions with high probability are enforced to match the given text embedding. This gradient update method is named SDS, which is formulated as follows:
$$
\nabla_ {\xi} \mathcal {L} _ {s d s} (\xi) = \mathbb {E} _ {t, \epsilon} \left[ w (t) \left(\epsilon_ {\phi} \left(\mathbf {x} _ {t}; \mathbf {y}, t\right) - \epsilon\right) \frac {\partial \mathbf {x} _ {t}}{\partial \xi} \right], \tag {6}
$$
where the noise  $\epsilon \sim \mathcal{N}(0, \mathbf{I})$  and  $w(t)$  is a weighting function. The neural rendering pipeline  $\Gamma(\xi)$  and the diffusion model  $\phi$  as modular components of the framework, are amenable to selection. This offers a practical way for text-to-3D synthesis.
Neural Texture for 3D-Consistent Rendering. As compared with previous works [16, 11, 9] on optimizing a NeRF model, we directly optimize a neural texture over the UV space. Direct texture optimization is crucial to enforce 3D consistency and preserve texture details. In detail, first we exploit the pre-trained latent diffusion model as the optimization guidance, where it has an encoder  $V_{e}$ , a latent diffusion model's denoiser  $\epsilon_{\phi}$ , and a decoder  $V_{d}$ . In Decorate3D, the neural texture  $\psi \in \mathbb{R}^{H\times W\times 3}$  is set to 3 feature channels to fit the requirement of the diffusion model. The gradient calculated via SDS is backpropagated through the diffusion model's encoder to the UV neural texture at a high resolution of  $512\times 512$ . Albeit yielding high-resolution images, the optimization computation is reasonable since the optimization exerts an effect on the latent code  $\mathbf{z}_t^{\psi} = V_e(\mathcal{R}(\psi ,\mathcal{M},\mathcal{P}))$  with a resolution of  $64\times 64$ , where  $V_{e}$  is the diffusion model's encoder and  $\mathcal{P}$  is the sampled camera pose. All network parameters are fixed, while only the neural texture  $\psi$  is trainable.
Here, we call the optimized texture UV Neural Texture by following the deferred neural rendering [31]. The optimized texture does not act on RGB space, and they are neural features that need a Neural Renderer to interpret, which will be introduced shortly.
Structure-Aware SDS Optimization. Apart from addressing the 3D-consistency issues, we also want to maintain the coherence between 3D geometry and the generated texture. For example, for a 3D human model, the generated facial texture should be attached to the 3D region corresponding to the face. This structure-aware requirement is overlooked by the naive SDS optimization leading to a multi-face Janus problem. To moderate this issue, Decorate3D utilizes structure-aware SDS optimization by exploiting a depth-guided latent diffusion model [29, 23].
In detail, at each denoising step  $t$ , we compute the image latent code  $\mathbf{z}_t^\psi$  produced by the encoder  $V_e$  and the depth latent code  $\mathbf{z}_t^{\psi_d}$  downsampled from the estimated depth map. They are concatenated as  $\mathbf{z}_t$  and forwarded to the latent diffusion module (Eq. 8). Then, following the SDS technique, the gradient of neural texture is computed as follows (Eq. 9):
$$
\mathbf {z} _ {t} = \left[ \mathbf {z} _ {t} ^ {\psi}, \mathbf {z} _ {t} ^ {\psi_ {d}} \right], \tag {7}
$$
$$
\tilde {\epsilon} _ {\phi} (\mathbf {z} _ {t}; \mathbf {y}, t) = \epsilon_ {\phi} (\mathbf {z} _ {t}; t) + \lambda \left[ \epsilon_ {\phi} (\mathbf {z} _ {t}; \mathbf {y}, t) - \epsilon_ {\phi} (\mathbf {z} _ {t}; t) \right], \tag {8}
$$
$$
\nabla_ {\psi} \mathcal {L} _ {s d s} (\phi , \psi) = \mathbb {E} _ {t, \epsilon} \left[ w (t) \left(\tilde {\epsilon} _ {\phi} \left(\mathbf {z} _ {t}; \mathbf {y}, t\right) - \epsilon\right) \frac {\partial \mathbf {z} _ {t}}{\partial V _ {e}} \frac {\partial V _ {e}}{\partial \psi} \right], \tag {9}
$$
where  $\epsilon_{\phi}$  is the diffusion model's denoiser,  $\mathbf{y}$  is the text embedding from the Transformer, and  $w(t)$  is a weighting function. Decorate3D uses the classifier-free guidance scheme [15] as stated in Eq. 8, and the guidance weight  $\lambda$  for text conditioning is set to 100. To match the depth-guided diffusion model, depth maps are predicted from the rendered views by using the depth estimator [21] of the depth-guided diffusion model rather than the depth buffer of the rendering pipeline. The overall SDS optimization framework is shown at the left bottom of Fig. 2.
In addition to structure-aware SDS optimization, Decorate3D adopts structure-aware initialization. Thanks to the decomposition phase, Decorate3D has an inherent texture that well matches the geometry information extracted from the real-world object. Therefore, Decorate3D can initialize the nerval texture  $\psi$  with the output of  $\mathrm{MLP}_{\mathcal{D}}$ . This initialization plays a vital role in the optimization process, which can accelerate the convergence as well as provide structure-aware information to help it converge to a structure-aware solution. We verify the effectiveness of the structure-aware techniques in Sec 4.4.
Decorate3D's Neural Renderer. As aforementioned, directly optimizing the UV texture with SDS does not yield a traditional RGB UV texture for the rendering pipeline. Fig. 3 shows an example of SDS optimization for text-to-image generation. The optimized texture, i.e. neural texture, requires a neural interpreter to convert it back to RGB space after the SDS optimization.
Previous work [11] applies a statistical linear transformation matrix to the optimized UV texture to interpret it in the RGB UV space. However this transformation matrix is suboptimal, and the misalignment will result in color shifts and artifacts.
In Decorate3D, we introduce a neural renderer as the neural interpreter to synthesize photo-realistic images from the optimized neural texture. This rendering process is formulated as follows:
$$
I _ {\mathcal {P}} = V \left(\mathcal {R} (\psi , \mathcal {M}, \mathcal {P})\right), \tag {10}
$$
where  $\psi$  is the neural texture,  $V$  denotes the network of the neural renderer, and  $I_{\mathcal{P}}$  is the rendered image given the camera
pose  $\mathcal{P}$ . Here, Decorate3D's neural renderer network  $V$  is the encoder-decoder (i.e. VAE) module of the depth-guided latent diffusion model [23, 29]. The network  $V$  and the traditional renderer  $\mathcal{R}$  have amalgamated to form the neural renderer.
w/o Interpreter
Figure 3: The text prompt is "An apple tree". The left shows the image from the direct SDS optimization on a 2D image. The right shows the image after applying interpreter.
w/ Interpreter
3.3 Few-View Resampling Training
With the neural texture, Decorate3D can render 3D-consistent high-quality views  $I_{\mathcal{P}}$  through the Neural Renderer with Eq. 10 by sampling different camera poses. But there still exists a practical issue, i.e. jittering artifacts, that arises when rendering across different views. This is caused by the view-dependent neural renderer  $V$ . We empirically find that directly feeding the global UV neural texture  $\psi$  to the diffusion model's VAE (i.e.  $\mathcal{R}(V(\psi),\mathcal{M},\mathcal{P})$  by swapping the network and renderer
in Eq. 10) can produce the RGB UV texture for the traditional rendering pipeline. But this naive approach usually yields blurry texturing results, as demonstrated in Sec 4.4.
To solve the jittering problem, we devise a few-view resampling (FVR) training method, that transfers the synthesized view-dependent images into a global UV texture  $\tilde{\psi}$ . As shown in the right bottom of Fig. 2, we use a MLP  $\tilde{\psi}$  to represent the UV texture  $\tilde{\psi}$ . In the FVR training, we sample  $N$  rendered views using the neural renderer with the neural texture  $\psi$ . The sampled  $N$  views should overlay the mesh surface as much as possible. But setting a big  $N$  may have negative effects, leading to over-smoothing textures on the overlapped areas that suffer jittering artifacts. The FVR training loss is defined as follows:
$$
\mathcal {L} _ {F V R} (\tilde {\psi}) = \frac {1}{N} \sum_ {i} ^ {N} \left\| \mathcal {R} \left(\operatorname {M L P} _ {\tilde {\psi}} (\tilde {\mathbf {v}}), \mathcal {M}, \mathcal {P} _ {i}\right) - V \left(\mathcal {R} \left(\psi , \mathcal {M}, \mathcal {P} _ {i}\right)\right) \right\| ^ {2}, \tag {11}
$$
where  $\tilde{\mathbf{v}}$  denotes the positional encoding of the 2D UV coordinates of the RGB UV texture  $\tilde{\psi}$ , and  $\mathcal{P}_i$  denotes the  $i$ -th sampled camera pose. The pose is sampled in spherical coordinates, with two elevation angles  $\theta_{cam}$  chosen from  $\{-20^{\circ}, +20^{\circ}\}$ , four azimuth angles  $\beta_{cam}$  uniformly sampled between  $[0^{\circ}, 360^{\circ}]$ , and an appropriate viewing distance  $r_{cam}$ . In our experiments,  $N$  is set to 8.
3.4 Super-Resolution on UV Texture
From the SDS approach, we can only synthesize images at a resolution of  $512 \times 512$ . To further improve the spatial resolution of the UV texture  $\tilde{\psi}$ , Decorate3D applies a super-resolution (SR) diffusion model with a  $\times 4$  scale factor [25, 30] on the UV texture to obtain a  $2048 \times 2048$  UV texture. The rationale behind this successful application of UV texture upscaling is that the upsampling operation has a spatial locality, concentrating on local textures such as edges. For this reason, SR models are usually trained on cropped image patches [28, 4, 10] instead of the whole image to increase the training efficiency. This spatial locality of SR allows the SR model trained on natural images to be directly applied to the UV texture. Directly applying the SR operation to the UV texture is free from any jittering issues, and 3D consistency is well preserved.
We compare Decorate3D to the state-of-the-art (SOTA) techniques for text-to-3D texture generation and evaluate its performance from both qualitative and quantitative perspectives. The 3D assets and extended videos are presented in the supplementary material.
To evaluate our approach, we collect real-world datasets from 14 different objects that vary in complexity, including boxes, monitors, shoes, statues, dolls, humans, and lighthouses. Ten objects are captured in the wild using a smartphone, and four objects are selected from some public real-world datasets [34, 8]. The size of each dataset ranges from 70-300 images, and the images are downsampled to a resolution width of 640. We use the Adam optimizer to optimize the  $\mathrm{MLP}_{\mathcal{D}}$ ,  $\mathrm{MLP}_{\mathcal{S}}$  and  $\mathrm{MLP}_{\tilde{\psi}}$  with a learning rate of  $1\times 10^{-3}$ , and the  $\psi$  with a learning rate of  $1\times 10^{-2}$ . The neural texture optimization in the decoration phase takes about 2 hours for 100K iterations, and the FVR training takes about 5 minutes for 30K iterations, which are measured on 8 NVIDIA V100 GPUs. Please refer to the supplementary material for more details.
4.2 Qualitative Evaluation and Comparison