SlowGuess/NeurIPS_2023_OCR huggingface dataset:
3D-Consistent Text-Driven Texture Generation. In Fig. 4, we show the textured mesh and its corresponding real-world image. It can be observed that Decorate3D is able to produce high-quality textured mesh. As the mesh geometry is fixed, we find only texture-related keywords such as nouns or adjectives will affect the generated results, highlighted with a purple color in the prompt. For instance, in the prompt "An astronaut stands up in the milky way", "astronaut" and "milky way" dominate the generation, but the verb "stands up" and the quantifier "an" do not affect the results.
An astronaut stands up in the milky way
A lotion case, painted with Starry Night
A white horse statue, with red hairs
A man wears a cosplay suit of an armored medieval Japanese samurai
A detergent case, printed with a movie poster of Star Wars
A doll model of the movie of Zootopia, wearing a spiderman suit
Ironman, with hands clasped behind back, standing up on the grass
Figure 4: Text-driven texture generation results of Decorate3D.
An ancient Chinese-style house
Latent-Paint
Figure 5: Qualitative comparison with other text-driven texture generation methods, including CLIP-Mesh [7], Latent-Paint [11] and TEXTure [22]. Results from DreamFusion [16] are also shown here. Please zoom in for better visualization.
TEXTure
Decorate3D (Ours)
A man wears a set of medieval knight armor
Original
Woman Martin shoes, with a wizardic style, put on a treasure box.
CLIP-Mesh
DreamFusion
In texture generation
Single-View Editing
Propagated to UV Texture
Fig. 5 shows the qualitative comparison with SOTA text-driven texture generation methods, including CLIP-Mesh [7], Latent-Paint [11] and TEXTure [22]. They all took the reconstructed 3D mesh as input, and the geometry was fixed during texture generation. We can observe that the generated textures from CLIP-Mesh and Latent-Paint lack texture details. The concurrent work TEXTure has some competitive results compared to Decorate3D, such as the results for shoes, because they use a depth-guidance strategy similar to ours. But TEXTure produces weird artifacts on the complex surfaces. For example, in the case of humans, the generated textures from TEXTure have obvious seams, and textures from different parts seem to be glued together and look messy. The artifacts are caused by the iterative update strategy across views. By contrast, the results of Decorate3D have fruitful and clear details. We also present the results from DreamFusion [16]. It tends to produce blurry results that do not actually match the text prompt.
Textured Mesh
Adding Decomposed Specular Color  $(+\mathrm{MLP}_{\mathcal{S}})$
Original
A running horse having green stripes like zebra
Text + Single-View Editing
Figure 6: Two examples of applying Decorate3D for 3D-consistent texture editing. As showcased in the first example, a scribble of the "NeurIPS 2023" pattern is painted onto a 2D image (aka. a single rendered view), which is propagated by Decorate3D to the UV texture, updating the 3D-consistent rendered output.
Textured Mesh
3D-Consistent Texture Editing. Thanks to the decomposed 3D representations, Decorate3D opens up an easy way to edit textures. Fig. 6 depicts a practical and distinctive use case that can be accomplished by Decorate3D. We edit one of the rendered images (not the UV texture), and propagate the editing to the UV texture to synthesize 3D-consistent edited textures from different views. The decomposed view-dependent specular texture can be added back to the textured model. Additionally, Decorate3D allows secondary editing of the text-driven texture generation.
4.3 Quantitative Evaluation and Comparison
CLIP R-Precision Metric. Following DreamFusion, we evaluate the CLIP R-Precision [18], an automated metric for the consistency of rendered multi-view images with respect to the input prompt. Here, we calculate the average CLIP score from the front-side, left-side, right-side, and back-side views. We use 70 prompts of 14 objects to generate the test results. For visual quality, we measure the NIQE [13] on rendered images, which is a no-reference image quality assessment. Tab. 1 reports the CLIP R-Precision scores of the compared methods.
User Studies. We invite 44 volunteers to evaluate Decorate3D and its competitors using the mean-opinion-score (MOS) test. For each question, we prepare 5 video results, and the participants are asked to rate the results on a scale from 1 (worst) to 5 (best) based on the overall visual quality of the results and the degree to which they match the text prompt. In the end, we receive 1100 responses from the 44 volunteers. Fig. 7 shows the average MOS scores of the compared methods. As can be seen, Decorate3D is shown to be more favored by human users.
Initialization of UV Neural Texture. Fig. 8a shows an ablation study on the initialization of the UV neural texture and Fig. 11 shows more results. As observed, the optimization initialized with
Table 1: Evaluating the correlation of text-driven generated results with their text input using different CLIP models. The CLIP-Mesh's scores may overfit, as it uses the same CLIP model for training and eval. The NIQE is a no-reference image quality evaluation metric.
Method	R-Precision ↑
CLIP B/32	CLIP B/16	CLIP L/14	NIQE ↓
CLIP-Mesh [7]	31.95 ±3.01	30.16 ±2.63	23.99 ±3.71	16.28 ±0.94
DreamFusion [16]	29.65 ±4.71	29.63 ±5.11	24.38 ±4.46	16.68 ±3.47
Latent-Paint [11]	25.18 ±4.30	26.19 ±2.81	21.22 ±3.03	17.71 ±1.39
TEXTure [22]	29.68 ±3.56	28.65 ±2.71	23.15 ±3.31	14.83 ±0.78
Decorate3D (Ours)	30.42 ±3.47	30.47 ±3.07	24.89 ±3.41	14.82 ±0.66
Figure 7: User study results gathered from 44 participants. The results are the average of all responses.
(a) An ablation study of neural texture's initialization.
(c) Ablation studies of FVR training and the super-resolution on UV texture.
Figure 8: Ablation studies of Decorate3D. (a) Random initialization vs. initialization with  $\mathrm{MLP}_{\mathcal{D}}$ . (b) Naive SDS without depth vs. structure-aware SDS guided by depth. As observed, the naive SDS cannot generate the geometry-matching texture for the man's hands. (c) We compare the rendered results using (1) UV texture yielded by the diffusion model's VAE, (2) Per-view generation, (3) FVR trained  $\mathrm{MLP}_{\tilde{\psi}}$ , and (4)  $\mathrm{MLP}_{\tilde{\psi}} + \mathrm{SR}$ .
$\mathrm{MLP}_{\mathcal{D}}$  converges much faster than the random initialization. Moreover, the  $\mathrm{MLP}_{\mathcal{D}}$  provides a very strong optimization prior to helping the SDS optimization converge to a better solution that well matches the mesh geometry. For example, initialization using  $\mathrm{MLP}_{\mathcal{D}}$  can generate the correct facial texture that fits the geometry, but on the contrary, the random initialization fails.
Effectiveness of Structure-Aware SDS. Fig. 8b presents the difference between the naive SDS without depth guidance and the structure-aware SDS with depth guidance, and Fig. 11 shows more results. We can observe that the structure-aware SDS is able to produce geometry-matching textures. For example, the human's arms should be clasped behind the back rather than being akimbo.
Is the Few-View Resampling Training Necessary? Fig. 9 compares the per-view generated results of  $V(\mathcal{R}(\psi, \mathcal{M}, \mathcal{P}))$  with the results rendered using the FVR-trained UV texture. As observed, results of per-view generation have jittering effects between different views (look at the circle pattern), but results with  $\mathrm{MLP}_{\tilde{\psi}}$  have achieved 3D consistency. Fig. 8c (1) presents the results using an alternative solution to eliminate the jittering, i.e. directly feeding the neural texture into the diffusion model's VAE by  $\mathcal{R}(V(\psi), \mathcal{M}, \mathcal{P})$ .
Figure 9: Comparison between the per-view generation (w/o FVR) and rendering with  $\mathrm{MLP}_{\tilde{\psi}}$  (w/FVR). The difference in pixels between neighboring frames is measured after aligning views to the first reference view.
As observed, following this way yields blurry textures. Given an RGB UV texture  $\tilde{\psi}$ , Decorate3D can directly apply SR on  $\tilde{\psi}$ . By comparing Fig. 8c (3) and Fig. 8c (4) we can observe more clear results after SR. Fig. 10 shows the effectiveness of FVR training's hyperparameter  $N$ . Setting a big  $N$  for FVR will cause blurry results.
Effectiveness of Neural Renderer. As introduced in Sec .3.2, the optimized UV texture by SDS is a neural texture feature. A neural renderer is required to convert the neural UV texture to the RGB UV. The right of Fig. 11 shows the ablation study results on the neural renderer. As observed, the synthesized results without a neural renderer are over-saturated and noisy.
Figure 10: Results with different  $N$  for FVR. A bigger  $N$  leads to blurry textures.
Figure 11: Ablation studies on initialization, depth guidance, and neural renderer.
Decorate3D offers a practical way of decorating real-world 3D models with a user-friendly approach through text-driven texture generation. Our Decorate3D's techniques, combining directly optimizing UV neural texture, structure-aware optimization, FVR training, and SR enhancement on UV texture, collectively advance the line of 3D texture generation in pursuit of the best possible quality. Extensive experiments demonstrate the superiority of Decorate3D over SOTA methods.
Limitations First, the style and quality of the generated texture heavily depend on the pre-trained stable diffusion models. Second, even though Decorate3D adopts structure-aware optimization techniques, the multi-face Janus problem still remains in a flat geometry. For example, Decorate3D cannot distinguish the front and back sides of a monitor object, as shown in our supplementary material. Lastly, Decorate3D does not support jointly optimizing both the geometry and texture, which may result in inconsistency with the mesh geometry. We remain these problems for future study.
We gratefully acknowledge the support of MindSpore (https://www.mindspore.cn/), CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
[1] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402, 2023.
[2] Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint arXiv:2303.11396, 2023.
[3] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. *Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation*. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 22246–22256, 2023.
[4] Yanhui Guo, Xiaolin Wu, and Xiao Shu. Data acquisition and preparation for dual-reference deep learning of image super-resolution. IEEE Transactions on Image Processing, 31:4393-4404, 2022.
[5] Jonathan Ho, Chitwan Sahara, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
[6] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463, 2023.
[7] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. CLIP-mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 Conference Papers. ACM, nov 2022.
[8] Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics, 36(4), 2017.
[9] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300–309, 2023.
[10] Fangzhou Luo, Xiaolin Wu, and Yanhui Guo. Functional neural networks for parametric image restoration problems. Advances in Neural Information Processing Systems, 34:6762-6775, 2021.
[11] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12663–12673, 2023.
[12] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: representing scenes as neural radiance fields for view syntheses. In European Conference on Computer Vision (ECCV), pages 405-421, 2020.
[13] Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters, 20(3):209-212, 2013.
[14] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102:1-102:15, July 2022.
[31] Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. Acm Transactions on Graphics (TOG), 38(4):1-12, 2019.
[32] Can Wang, Ruixiang Jiang, Mengei Chai, Mingming He, Dongdong Chen, and Jing Liao. Nerf-art: Text-driven neural radiance fields stylization. IEEE Transactions on Visualization and Computer Graphics, 2023.
[33] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems, 34:27171-27183, 2021.
[34] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1790-1799, 2020.
[35] Jonathan Young. Xatlas: Mesh parameterization / uv unwrapping library. https://github.com/jpcy/xatlas, 2023.
[36] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.