Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials
Authors: Ye Fang
Zeyi Sun
Tong Wu
Jiaqi Wang
Ziwei Liu
Gordon Wetzstein
Dahua Lin
Venue: NeurIPS 2024 poster (2024)
Keywords: Texture Synthesis
MLLMs
Material Refinement
3D Content Creation
Primary area: machine_vision
TL;DR: This paper proposes unleashing large multimodal model’s ability for painting 3D objects with realistic materials

Abstract: Physically realistic materials are pivotal in augmenting the realism of 3D assets across various applications and lighting conditions. However, existing 3D assets and generative models often lack authentic material properties. Manual assignment of materials using graphic software is a tedious and time-consuming task. In this paper, we exploit advancements in Multimodal Large Language Models (MLLMs), particularly GPT-4V, to present a novel approach, Make-it-Real: 1) We demonstrate that GPT-4V can effectively recognize and describe materials, allowing the construction of a detailed material library. 2) Utilizing a combination of visual cues and hierarchical text prompts, GPT-4V precisely identifies and aligns materials with the corresponding components of 3D objects. 3) The correctly matched materials are then meticulously applied as reference for the new SVBRDF material generation according to the original albedo map, significantly enhancing their visual authenticity. Make-it-Real offers a streamlined integration into the 3D content creation workflow, showcasing its utility as an essential tool for developers of 3D assets.

Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials
Ye Fang" Fudan University Shanghai AI Laboratory yefang236m. fudan. edu cn
Zeyi Sun"
Tong Wu
Shanghai Jiao Tong University
The Chinese University of Hong Kong Stanford University wutong (ernnng6@stttafoodd edu
Shanghai Al Laboratory
skyy2030sstu. edu.....uuuuuee..ee............. cn
Jiaqi Wang
Ziwei Liu
Gordon Wetzstein
Shanghai AI Laboratory wangjiage @pjlab. org. cn
S-Lab, NTU
Stanford University
ziwei. liuentu, edu, sg
gordon. www.wwewiwintttttforrd.oooo
Dahua Lin
The Chinese University of Hong Kong Shanghai AI Laboratory CPII under InnoHK dhlinRie. cuhk. edu. hk
Oregnn
Realistic
Postic
Rubber
Part-specific
PBR Compatible
Figure 1: Usage of Saake-it-Real. Our method can refine a wide range of albedo-map catdoommaponlly 3D objects from both CAD design and generative models. Our method enhances the realism of objects. enables part-specific material assignment to objects and generate PBR maps that are compatible with downstream engines.
Abstract
Physically realistic materials are pivotal in augmenting the realism of 3D assets across various applications and lighting conditions. However, existing 3D) assets and generative models often lack authentic material properties. Manual assign- ment of materials using graphic software is tedious and time-consuming task. In this paper, we exploit advancements in Multimodal Large Language Models (MLLMs), particularly GPT-4V, present a novel approach, Make-it-Real:
"Equal Contribution Cooresponning Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

We demonstrate that GPT-4V can effectively recognize and describe materials. 2) Utilizing a combination of visual cues and hierarchical text prompts, GPT-4V precisely identifies and aligns materials with the corresponding components of 3D objects. 3) The correctly matched materials are then meticulously applied as reference for the new SVBRDF material generation according to the original albedo map, significantly enhancing their visual authenticity. Make-it-Real offers a streamlined integration into the 3D content creation workflow, showcasing its utility as an essential tool for developers of 3D assets. Our project website is at: Petts::/Sunzar github. io/Make-it-Reall it-Real/.
Introduction
High-quality materials are important for the nuanced inclusion of view-dependent and lighting- dependent effects in 3D assets for traditional graphics pipelines, critical for achieving realism in gaming, online product showcasing, and virtual/augmented reality. However, many existing assets and generated objects often lack realistic material properties, limiting their application in downstream tasks. Furthermore, creating hand-designed realistic textures necessitates specialized graphic software and involves laborious and time-consuming process, compounded by significant creative challenges [32].
Traditional computer graphics methods have been either manually creating materials or reconstructing them from physical measurements. Emerging text-to-3D generative models [7, 34, 55, 10, 73, 50, 19, 51] and image-to-3D generative models [69, 58, 25, 63, 22, 62] are successful in creating complex geometries and detailed appearances, but they struggle generate physically realistic materials, hampering their practical applicability. Recent studies have also explored advanced aspects of appearance generation [11, 75, 70, 8, 40]. However, they often rely on simplified material models. For instance, [40] lacks the ability to produce metallic maps. All these approaches do not generate corresponding displacement and height maps, which restricts the diversity and realism of the generated materials, especially regarding depth and tactile qualities. Furthermore, these methods typically require relatively long training and inference time. Considering the abundance of high-quality 3D assets online [16, 17] that lack material attributes, and the maturity of 3D generative models in geometry and albedo modeling, we aim to recover materials from high-quality geometry and base-colored 3D meshes.
However, extracting and recovering material representations for 3D meshes is challenging. Unlike previous material recognition methods [40, 4, 20], this difficulty is heightened when identifying and separating different material regions within 3D objects in constricted albedo textures. These maps only reflect the base color and can be distorted, as shown by the globe in the upper right corner of Figure 1. Additionally, shadows and lighting can affect judgment. Thus, the model must have strong material recognition capabilities and prior knowledge of object types and materials.
The emergence of Multimodal Large Language Models (MLLMs) [5, 46, 22, 23, 59] provides novel approaches to problem-solving. These models have powerful visual understanding and recognition capabilities, along with vast repository of prior knowledge, which covers the task of material estimation. Specifically, we are using GPT-4V(ision) for matching of materials. We first create highly detailed descriptions for materials build comprehensive library. Next, we use GPT-4V to retrieve materials for each segmented part of the object, utilizing visual prompts [72] and hierarchical text prompts. Finally, we meticulously designed algorithms to generate SVBRDF maps with consistent albedo, achieving realistic visual quality.
Notably, our work differs from the aforementioned studies by leveraging prior knowledge from foundation models like GPT-4V to extract and infer materials in alhedo-only constrained scenarios. Additionally, we utilize existing material libraries as references to generate corresponding SVBRDF maps on a designed region-to-pixel algorithm. As illustrated in Figure 1, our approach features: 1) Enhanced 3D mesh realism: Leveraging GPT-4V's visual perception and external knowledge, our method improves the realism, depth, and visual quality of a wide range of mature 3D content genera- tion models. 2) Part- spre-spec material matching: Ensuring material consistency with a segmentation network and refinement process, enabling precise material property retrieval for each segment. 3) Rendering engine compatibility: Generating six comprehensive material maps(roughness, metallic, specular, normal, displacement, height), which are compatible with downstream rendering engines. Developers only need to paint albedo textures; material properties are then automatically generated, saving extensive time detailed ambient occlusion masks and material map creation.

---

In summary, our contributions are as follows:
We present the first exploration of leveraging multimodal large language models (MLLMs), GPT-4V for material recognition and unleashing their potential in applying real-world materials to extensive 3D objects with albedo-only.
We create a material library containing thousands of materials with highly detailed descrip- tions readily for MLLMs look up and assign.
We develope an effective pipeline for texture segmentation, material matching and SVBRDF maps generation, enabling the high-quality application of materials to 3D assets.
2 Related Work
3D Object Generation. The generation of 3D models using deep learning methods has experienced rapid development in recent times. The mainstream research can be primarily divided into two categories. The first category relies on techniques that optimize a Neural Radiance Field (NeRF) [43] or 3D Gaussian [30] guided by 2D diffusion model through score diste--dstillatiionn wwsiiillaiionsamminnnnggeeeeeeee
loss [48, 42, 39, 8, 64, 53, 56, 67]. The second category aims at obtaining 3D representation via direct inference model, c.g... Point-E [44], Shap-E [28], and LRM [25], proven fast with high quality through large-scale pretraining [24, 22, 57, 35, 55, 71, 58, 63, 65]. Although the capabilities of these methods are continuously improving, they still lack a high degree of realism in textures. More importantly, since the textures of 3D objects obtained by these methods are shaded, they cannot directly respond to lighting changes under different lighting conditions, leading to less realism. Although some works attempt to generate PBR textures, their results are considerably limited to generate physically realistic materials due to the low robustness of SDS [75, 8J. Our work is the first to introduce the prior knowledge of MLLMs into the texture synthesis process. We verifies that our method can be seamlessly and effectively applied to generated 3D objects, facilitating their downstream applications under different lighting condition.
Material Capture and Generation. Recent studies such as Material Palette [40], MatSim [20],
TwoshotBRDF [4] have made progress in the recognition and extraction of 3D object materials,
allowing for the retrieval of SVBRDF information from images of real materials [40] and combining object shape and illumination [4], but they fail to extract and infer the materials of 3D objects with only
albedo. On the other hand, works like Paint-it [75], Matlaber [70], Collaborative [60], Fantasia3D [8]. and TANGO [11] focus on generating text-controllable 3D meshes with physically-basee rendering material properties. However, these methods require either extensive training time on BRDF datasets or long inference time for generating materials. Additionally, they are limited to synthesizing only a subset of PBR textures and cannot generate the full range of material properties, such as height and displacement, which are essential for the fine tactile perception of object surfaces.
Multimodal Large Language Models. In the wake of the advancements achieved by large language models (LLMs) [5, 46, 12, 2, 23, 59], domain of research has increasingly turned attention towards multimodal large language models (MLLMs). Recent advances in this field focus on the integration of vision understanding capabilities with LLMs [76, 1, 37, 36, 26, 21, 3, 18, 54, 49, 6he advent of GPT-4V [45] has marked a significant milestone in the evolution of MI.LMs, demonstrating groundbreaking 2D comprehending capabilities and open-world knowledge. Although GPT-4V cannot directly process 3D data, pioneering work.. GPTEva13D [68]. first exploited GPT-4V's ability in evaluating the quality of generated 3D objects, and found that GPT-4V's judgement was in line with human evaluation. In this work, we delve into new application of GPT-4V for material assignment of 3D objects.
3 Methodology
3.1 Preliminary
Physically Based Rendering (PBR) materials are a compact representation of the bi-directional reflectance distribution function (BRDF), which describes how light is reflected from a surface. PBR material maps primarily encompass seven attributes: Albedo, Roughness, Metallic, Normal, Specular, Height, and Displacement. Based on the rendering equation [29], given location and the surface
3

---

(b) MLLM -besed Retrieval
80
Material Library
(+) Rendering and Segmentation
Reggaeseen Paarttiee
Shape
abbedo
PER
Maps
Marble
Caramic
Medal
hegh
Retrovaal Rosults
(c)
speccda
Ligitng
Figure 2: Overall pipeline. This pipeline of Make-it-Real is composed of image rendering and material segmentation, MLLM-based material retrieval, and SVBRDF Maps Generation. We finally use blender engine to conduct physically-base based rendering.
normal n, the incident light intensity at this point is denoted as Li x) along the direction wi: BRDF (wi, wi; æ) denotes the reflectance coefficient of the material viewing from direction wo- The observed light intensity Lo(wo; x) is calculated over the hemisphere Ω = {wi Wi >> 0}:
[o(wa: / Le(w;: x) (wos wis (w: n)dw;,
(1)
Given the advancements in generating high-quality 3D shapes with albedo maps, the restoration of realistic material properties remains a challenge. We highlight novel problem: given a 3D mesh (Ö) and known Albedo (A) map, which reflect the object's intrinsic appearance, the goal is to extract and restore all other SVBRDF attributes of the object, i.e.
Á) = {R, M, N, S, H, D). Our setup supports the popular Cook-Torrance analytical BRDF model [13]. In this parameterization, the BRDF includes components for albedo h2 E Re3, metallic bm E R, and roughness b, E R. For more complex surface simulations, such as displacement and height modeling, rendering engine to simulate the BRDF function fr (Wo,w;;z))
the Blender
3.2 Make-it-Real: A Framework for Material Matching and Generataion
In this section, outline material matching and generation pipeline, illustrated in Figure 2, which encompasses three stages: rendering and segmenting 3D meshes, retrieving matching materials using MI.LM, and generating spatially varying BRDF maps from coarse to fine.
3.2.1 Rendering and Material Segmentation
To accurately segment different material regions on 3D meshes with albedo maps, we propose an innovative segmentation strategy based on 2D image rendering in Figure 2 (a). Initially, we rasterization to render the input albedo mesh from various viewpoints to obtain series of images:
I(I,y) R (VV map [ratterize(O, Ve, x. v) .).
(2)
where I(x, y) is the pixel value at image coordinates (z,y), y). rasterize (Ō, U2 .) maps the 3D mesh from viewpoint 2 to 2D screen coordinates, (.) converts rasterized coordinates to UV coordinates, T is the albedo map, and R(. T) samples color from the T using the UV coordinates.
For the rendered images, we employ segmentation. Empirically, we select the main viewpoint with the largest projected area of the mesh, as is more likely to contain more details. To address potential over-segmentation, drawing inspiration from [74], extract non-overlapping segments from the masks to form distinct patches,
Semantic-SAM [33] to perform preliminary semantic

---

Prompt Template
Construct Library
Wood
Metal
Describe materioal
Material Description
appeorence features, inclading color, patterns, roughness...
Fabric
Records
Ceremic
Stone
Plastic
Retrieval Library
Stagel.
Identify the material of each (optional [Metol, Fabric, ... Wood]
Material
Type
Marble
Plaster
Library
Terracotta
similar sub material type of part (number id). (optional list
StageZ. Sub Type
[Foil, SheetMetol,
Concrete
Con you tell me which is the
Ground
Stage3, Finol Answert
best description match the
Leather
(number_ _dd_ (aptional (Descl, besc2, -))
Category Tree
PBR Texture Maps
Highly-detane Records
Ceranic_ Tic005: of
blue
smooth, motte ceronic finish and dight bevel edges.
MetealAAe polished breas like material with subtle veriations hinting gentle use on handling.
Terrocatho, Brick022; solished breas-lkke matirrial with voriations, hirting gentle handling
togghe 2014
specker
Telghh
The process of MILM retrieving materials from the Material Library. Utilizing GPT-4V model, we develop a material library, meticulously generating and cataloging comprehensive descriptions for each material. This structured repository facilitates hierarchical querying for material allocation in subsequent looking up processes.
Figure 3:
as detailed in our approach in Figure 4 (a). These patches are then merged based on similar colors to obtain the final material grouping. We incorporate Set-of-Mark [72] method to annotate each material segment with a unique identifier, sorted by area size from largest to smallest. This annotation acts as visual cue, enhancing the visual comprehension capabilities of large multimodal language models.
3.2.2 MLLM-based Material Retrieval
Material library with fine-grained annotations. To enable large multimodal language models to accurately retrieve and match materials, we construct finely annotated material library, as shown in Figure It is composed of three main components: comprehensive PBR texture maps, highly- detailed records, and category tree. It comprises 1,400 unique, tileable materials spanning 13 primary categories and 80 subtypes. The data primarily derives from the [61], which offers comprehensive PBR material textures under CCO license with 4K resolution. Each material is represented by seven maps: albedo, roughness, metallic, specular, normal, displacement and height. Accompanying each material are highly-detaildd annotations by GPT-4V, offering thorough descriptions of the material's visual characteristics and rich semantic information for the subsequent retrieval process. Created by crawling material sphere images and constructing prompts, these annotations capture subtle differences between materials, facilitating precise retrieval by GPT-4V, as detailed in Appendix B.4.
Hierarchical prompting for material retrieval. Due to the vast size of our material library, feeding all prompts to GPT-4V simultancously proves inefficient and challenging for memory retention. To ensure efficient and accurate material allocation in segmented areas of 3D meshes, we adopt hierarchical text prompting approach. The schematic of the designed prompt shown on the left side of Figure 3, and a complete querying process unfolds in Appendix B.4. This method starts by identifying the primary material types corresponding to each labeled region. Subsequently. hierarchical prompts guide GPT-4V to distinguish specific subclasses within the main material categories. We retrieve descriptions under these subclasses to ascertain for the segmented blocks. This hierarchical processing enables a more granular search of our material library, identifying the optimal description for each material segment. This approach aids assigning the most suitable materials to each region and reduces memory and time consumption.
most fitting descriptions
5

---

Segmenn
Rendered Image
Refined Masks
Original Masks
Mask Overlapping
Patches
Albedo-based Merging
(a) Mask Refinement 2D Image Space
Wordkes
W Woodees
Fubriigg
Texture Completion
Fabricaur
Fobvicaun
missing
Memory
Albedo Map
Visualization (w. missinng part)
Visualization (Refined Masks)
Feuture Centroid Clustering
froen Bace-projection
(b) Mask Refinement in UV Texture Space
Figure 4: Illustrations of mask refinement in 2D image space and UV texture space. (a) We effectively cluster concise mnterial-aware masks compared to original segmented parts from [33]. (b) We fix missing parts on the uv texture space to get complete texture partition map.
3.2.3 SVBRDF Maps Generation
We propose method to generate SVBRDF maps region-to-pinn tixel scale in Figure 2 (c). Initially, we segment texture map in uv space based on queried material regions on 2D image space. We then estimate BRDF values pixel space using the object's original albedo map for reference, ensuring consistency with the albedo map. This approach effectively enhances the realism of rendered surfaces.
Region-level texture map partitioning. Upon acquiring segmentation masks for material regions within the 2D rendered space of a 3D mesh, our objective transitions to transposing these segmen- tations from the 2D image space to the corresponding UV space. As described in Section 3.2. extract 2D image features from 3D mesh points via rasterization, and then we apply the material masks these features, facilitating the accurate transfer of segmentation UV space. To project the image feature I, back to the texture atlas Te with segmented image mask m, at the viewpoint us, we apply gradient-based optimization for T
the values of when rendered through the differential renderer R, as presented in Equation (3), In Equation (4), we then compute the difference between T.mask and the initialized T. transfer the mask image feature m, into the texture space, represented by mar- The term a denotes the difference coefficient. Due to the limited perspectives available in rendering, we avoid using naive median-filll filling approach to ensure that there are no missing areas on the texture map. Instead, we employ a blotk-centric clustering based on albedo, as illustrated in Figure (b), obtain cohesive and refined region masks. The process is shown in Appendix B.I
ØR
VT,Lc (R(mesh, Te, v2) -1)
[e) 2 mi OT.
(3)
mav
T mask
Tíl > -0
Pixel-level albedo-referenced estimation. To achieve precise estimation of spatially varying BRDF (SVBRDF) at the pixel level, we draw inspiration from techniques commonly utilized by artists in creating texture maps. Artists typically use albedo maps as a reference for constructing ambient occlusion masks and further generating SVBRDF maps for material properties, which occupies significant time portion of appearance modeling. Our method involves using the albedo map of the original object as reference to refine the retrieved materials. We enhance the querying process using KD-Tree algorithm, which searches for the nearest neighbor pixel index in the key albedo(retrieved map) for each RGB value the queried albedo(input map) pixel, detailed in Appendix B.2. This process ensures that areas with similar colors exhibit similar BRDF values, avoiding abrupt changes

---

material
P
!
Metal
Netaa
Monial
Cesemii
Macial
Plastic
J
Leather
Maroie
Wood
material
Ours
Plastic
w. materad
Mertal
Motal
Plsstii
Mebal
Ours
Plastic
Figure 5: Qualitative results of Make-it-Rest Real refining 3D asserts without PBR maps. Objects selected from Objaverse [15] with albedo maps only.
Geometry
Albedo
Roughness
Metallic
Height
Normal
w. Material
Figure 6: Visualization of generated texture maps. We visualize some SVBRDF maps, where the material maps are well aligned with the albedo maps.
in material properties; For regions with greater color differences, the distribution of differences aligns consistently with the input albedo, to simulate variations such as embossments or scratches. We retrieves SVBRDF values at pixel level, maintaining texture consistency with the albedo and producing appropriate concave or smooth surface effects. We further analyze the effects in Section 4.2.
7

---

Our
w/5.
Ours
Image-to-3D
Instant3D
Fantasia3D
MVDream
Ours
A pmn sitting alop manuscripts
old, solid, asymme- trical boorze hell
Mobile Suit Gundam
golden goblet on blue cushion
mug
placed on plate
Text-to-3D
Figure 7: Qualitative comparisons between Make-it- itakll refining results and 3D objects generated by edge-cutting 3D content creation models. The upper row depicts image-to-3i models (InstantMesh and Tri,oSR), and the lower row shows results of text-to-3I) models. Experiments
4.1 Experiment settings
To verify the effectiveness of Make-it-Real, we conduct refinement experiments mainly on two types of objects. The first type is artificial 3D assets, with the primary model from Objaverse [15] filtered by [55]; The second type is objects generated by tiite-of-theaar 3D generation methods. For existing 3D assets, pick 200 objects with diverse textures from Objaverse by human experts. For 3D generative models (InstantMesh [69]. TripoSR [58]. MVDream [53], Instant3D [34] and Fantasia3D [8]), we also generate 200 objects for each methods use prompts designed in GPTTEva31D [68]. We Make-it-Real to refine the objects and compare the texture quality before and after refinement. We perform both GPT-4V [45] based evaluation and user study the above objects (Detailed guidence and prompts for evaluation are available in Appendix B.5).
In addition to evaluation details, we also provide information about the rendering procedure in Appendix B.3 for reproducibility. This includes the rendering tools used the experiments, hyper parameters related to sack-projection. and the basic performance metries of the model during the experiments, such as time, memory, and number of queries.
4.2 Experiment results
Texture refinement for existing 3D assets. As shown in Figure 5, assets processed through Make- ital demonstrate the capability to accurately segment objects, assign various suitable materials, and synthesize high-fidelity. photorealistic textures. Some materials exhibit notable highlights, such as marble bathtub and globe made of gold. The interaction of these different materials with light varies significantly, leading to diverse reflective effects under the same environmental lighting, thus presenting a range of textures. Additionally, material properties vary across different regions; for instance, the globe's 's landmass and handle exhibit gold characteristics, while other parts identified

---

Table GPT evaluation and user preference. GPT's and user's preference comparison on Make-it- Real refined objects sourced from existing 3D assets and sfate-of-the- art 3D generation methods.
GPT Evaluation
User Preference
Domain
Method Source
Base object
+Make-it-Real
Base object +Make-it-Real
3D assets
Objaverse [15]
15.2%
84.8%
22.2%
77.8%
InstantMesh [69]
28.2%
71.8%
31.1%
68.9%
Image-to-3D
TripoSR [58]
36.4%
63.6%
33.0%
77.0%
Instant3D [35]
38.5%
61.5%
35.4%
64.6%
Text-to-3D
MVDream [53]
44.1%
55.9%
41.5%
58.5%
Fantasia3D
46.2%
53.8%
48.7%
51.3%
Image lapat
Senaato SAM
Ous
Figure 8: Ablation study of material segmentation refinement. Compared to direct usage of SemanticSAM [33], Our post-process, tailored for material segmentation on 3D object can produce more consistent results.
Figure 9: Ablation study of missing part refinement. Our method on the bottom row produces consistent texture maps and avoids the missing parts of material texture.
plastic. Furthermore, the texture and appearance of materials also vary, such the subtle wrinkles on the red box in the second column and the more pronounced color contrast at the base of the blue box, which enhances realism and reflects signs of use. Due to the albedd-reeerencee algorithm design in Section 3.2.3, regions with similar colors have similar BRDF values, avoiding abrupt changes in material properties, such the continuous gold surface of the globe and the silver body of the kettle. Regions with significant color differences display consistent differential distributions with the key map, such the embossed textures the lower left corner of the water bottle in Figure 5 and the subtle particle variations on the surface of the red oil drum. Additionally, visualization of texture maps demonstrates reasonably consistent textures with albedo, shown in Figure Quantitive results are shown in Table 1, and more qualitative results can be found in Appendix E.
Texture refinement for generated 3D objects. Figure 7 displays the qualitative results. By leveraging our model for enhancement, we observe that Make-it-Reen successfully generates appropriate material maps for both image-to-3D and text-to-3D models.
Similarly, our Make-it-Real successfully paint these models with materials. As reported in Table 1, human experts consistently favor the objects post-refinement across all evaluated 3D content creation methods. This preference aligns with the evaluations performed by GPT-4V, indicating a general consensus the enhancement in quality achieved through Makke-i--Real refinement process.

---

4.3 Ablation Study
Effects of mask refinement module. In Section 3.2.1, we performed additional material post- processing on the segmentation outcomes, as depicted in Figure 8. The refined results in the last row indicate that our module achieves precise material segmentation for most standard objects, thereby enabling more accurate queries by GPT-4V. In Section 3.2.3, we addressed the completion of missing regions in the texture map within the UV space the regional level, as illustrated in Figure This method not only increases texture coverage but also enhances the visual quality and consistency of the refined 3D model. More ablation studies can be found in Appendix D.
Effects of different texture maps. We validate the impact of various texture maps generated by Make-it-Real on the appearance of 3D objects, as illustrated in Figure 16 of Appendix D.1. We provide a detailed qualitative analysis of how different maps enhance the visual texture of materials. For example, increasing metallic results in dampening of the base albedo and increasing in the shine on the surface, while reducing roughness gives the surface a smoother appearance and enhances highlights. Meanwhile, displacement and height maps contribute to the fine-grained bump details on the object's surface.
Effects of different UV mappings. Since UV mapping is crucial step in the 2D-3D alignment tech- nique of our method, we conduct experiments to assess how its quality impacts model performance. The results, shown in Appendix D.2 and in the second column of Figure 17, indicate that UV map- pings with excessive fragmentation and color entanglement can cause issues, However, our method still performs well with artist-created UV mappings and Blender's 's built-in mapping techniques. This indicates our method still demonstrates good robustness with many mapping techniques.
5
Conclusion
In this paper, we present a novel framework leveraging MILMs prior of the world to build a material library and proposing an automatic pipeline to refine and synthesize new PBR maps for initial 3D) models, achieving highly photo-realistic PBR textures maps synthesis. Experimental results confirm that our approach can automatically refine both generated and CAD models achieve photo-realism under dynamic lighting conditions. We believe Make-it-Real is a new and promising solution in the last few procedures of AI based 3D content creation pipeline with the development of MLLMs like GPT-4V [45] as well as the roaring field of deep learning based 3D generation from scratch.
6
Acknowledgement
This project is funded in part by Shanghai Artificial Intelligence Laboratory, the National Key R&D Program of China (2022/D0160200 the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK. Dahua Lin is of CPII under the InnoHK.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. ArXiv, abs/2204. 14198, 2022. URL https: / //api. aapp..semaniiccclee
248476411. 3
[2] Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Tachard Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Z. Chen, Eric Chu, Clark, Laurent El Shafey, Yanping Huang, Kathleen S. Helis--ellsteen, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Michael Brooks, Michele Catasta, Yongzhou Cheng, Colin Cherry, Christopher A. Choquette-Choo Aakanksha Chowdhery, C Crépy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, M. C. D'iaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fan Feng, Vlad Fienber, Markus Freitag, Xavier García, Sebastian Gehrmann, Lucas
10

---

González, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, An Ren Hu, Jeffrey Hui, Jeremy Hurwitz. Michael Isard, Abe Ittycheriah, Matthew Jagielski. Wen Hao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Mu-Li Li, Wei Li, Yaguang Li, Jun Yu Li, Hyeontaek Lim, Han Lin, Zhong- Zhong Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alexandra Ros, Aurko Roy, Bremnan Saeta, Rajkumar Samuel, Renee Marie Shelby, Ambrose Slone, Daniel Smilkov, David R. So. Daniela Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang. Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Ke Xu, Yunhan Xu, Lin Wu Xue, Pengcheng Yin, Jiahui Yu, Qiaoling Zhang, Steven Zheng, Ce Zheng, Wei Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report. ArXiv, abs/2305. 2023. URL www.etiisccooiooooeeo.....ocoooo https//app..smmmiiiccclooaaa wwaecchhlarrrr/oroorooee org/CorpusID: : 258740735. 2,3
[3] Anas Awadalla, Irena Gao, Josh Gardner. Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Yitzhak Gadre, Shiori Sagawa. Jenia Jitsev, Simon Korbblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open- framework for training large autoregressive vision-language models. ArXiv, abs/2308.01390, 2023, URL https:s::// //api. sewanti www. ( 3
[4] Mark Boss, Varun Jampani, Kihwan Kim, Hendrik Lensch, and Jan Kautz. Two-shot spatially- varying brdf and shape estimation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3982-3991, 2020. 2,
[5] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vone
Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, Abs/2005.14165 2020. URL httpt://///pppt //ppi.pppp..// semanticscholar, org/ (Io.ppsII:21197177777 3.3
[6] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint 1..i:22311.12999 2023. 3
[7] Rui Chen, Y. Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appear- ance for high-quality text-to-3d content creation. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 22189-22199 2023. URL htwpt:t/t///aae /ppi...... semanti cscholar org/Corpus ID: 257757213
[8] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303. 2023. 2,3,8 8.9
[9] Xi Chen, Sida Peng, Dongchen Yang, Yuan Liu, Bowen Pan, Chengfei Lv, and Xiaowei Zhou. Intrinsicanything: Learning diffusion priors for inverse rendering under unknown illumination. arXiv preprint arXiv:2404.11599 1599,, 2024. 25
[10] Xu Chen, Tianjian Jiang. Jie Song, Jinlong Yang. Michael J Black, Andreas Geiger, and Otmar Hilliges. gdna: Towards generative detailed neural avatars. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2022. 2
[11] Yongwei Chen, Rui Chen, Jiabao Lei, Yabin Zhang, and Kui Jia. Tango: Text-driven photore- alistic and robust 3d stylization via lighting decomposition. Advances in Neural Information Processing Systems, 35:30923-30936, 2022. 2.3
[12] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh,
11

---

Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Gareía, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thamumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Hellssee,,,,,mm, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1 13, 2022. URL. https: / /api. semanti cscholar. ar
C.orpusID:247951000 247951931. 2,
[13] Robert L Cook and Kenneth E. Torrance. A reflectance model for computer graphics. ACM Transactions on Graphics (ToG), 1(1):7-24, 1982. 4
[14] Sri Aditya Deevi, Connor Lee, Lu Gan, Sushruth Nagesh, Gaurav Pandey, and Soon-Jo Chung. Rgb-x object detection via scene-specific fusion modules. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 7366-7375, 2024. 25
[15] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects, 2022. 7, 8,
[16] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Osear Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprint arXiv: 12.0805, 2022. 2, 25, 27, 28
[17] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte. Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl: A universe of 10m+ 3d objects. arXiv preprint arXiv:2307.05563. 2023. 2
[18] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang. Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internlm-xcomposer, Mastering free-form text-image composition and comprehension in vision-language large model, 2024. 3
[19] Zijian Dong, Xu Chen, Jinlong Yang, Michael Black, Otmar Hilliges, and Andreas Geiger. Ag3d: Learning to generate 3d avatars from 2d image collections. arXiv preprint 2355v2230..02312 2023. 2
[20] Manuel S Drehwald, Sagi Eppel, Jolina Li, Han Hao, and Alan Aspuru-Guzio One-shot recognition of any material anywhere using contrastive learning with physics-based rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23524- 23533, 2023. 3
[21] Danny Driess, F. Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Ho Vuong, Tianhe Yu, Wenlong Huang. Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Peter R. Florence. Palm-e: An embodied multimodal language model. In International Conference Machine Learning, 2023. URL https://app..smmntiii. semanti cacholla org/CorpusIIl 257364842. 3
[22] Zexin He and Tengfei Wang. Openlrm: Open-source large reconstruction models. https: 2023. 3,
[23] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and L. Sifre. Training compute-optimal large language models. ArXiv, abs/2203. 15556, 2022. URL https: // //api. semanticscholar. wtw.aicccoooaaooorrroououe ortp/ooppsID::24777eTT 2,
12

---

[24] Fangzhou Hong, Jiaxiang Tang, Ziang Cao, Min Shi, Tong Wu, Zhaoxi Chen, Tengfei Wang, Liang Pan, Dahua Lin, and Ziwei Liu. 3dtopia: Large text-to-3d generation model with hybrid diffusion priors, 2024. 3
[25] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi. Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d, 2023. 2
[26] Shaohan Huang. Li Dong, Wenhui Wang, Yaru Hao. Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is all you need: Aligning perception with language models. ArXiv, abs/2302. 14045, 2023. URL https:
//api.ssmanticscoo.... wttp://cw........ooooooaaaaaaa/a
[27] Krishna Murthy Jatavallabhula, Edward Smith,
Lafleche, Clement Fuji Tsang, Artem Rozantsev, Wenzheng Chen, Tommy Xiang, Rev Lebaredian, and Sanja Fidler. Kaolin: A pytorch library for accelerating 3d deep learning research, 2019. 19
[28] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.0244 024663...06 2023. 3
[29] James T Kajiya. The rendering equation. In Proceedings of the 13th annual conference Computer graphics and interactive techniques, pages 143-150, 1986.
[30] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL https: http///epooaaaaae inria. f. fr www.ffrraaprnaddde.mmmo http:////l-aggsssaaaaaaaaaiiaaea
[31] Peter Kocsis, Vincent Sitzmann, and Matthias Nießner. Intrinsic image diffusion for indoor single-view material estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5198-5 5208, 2024. 25
[32] Matthias Labschütz, Katharina Krösl, Mariebeth Aquino, Florian Grashäftl, and Stephanie Kohl. Content creation for 3d game with maya and unity 3d. Institute of Computer Graphics and Algorithms, Vienna University of Technology, 6:124, 2011.
[33] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-san Segment and recognize anything any granularity, 2023. 4. 6, 9
[34] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. htpps//aaxxvv.rrlabb/22 11.01144, 2023. 2,8
[35] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model, 2023. 3,
[36] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In Interna- tional Conference on Machine Learning, 2022. URL httttpps///ppp..eatticcccoooo DarpusID:24641400
www.nticschooll..t....oommom. org/
[37] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2:
Bootstrapping language-image pre-training with frozen image encoders and large language models. ArXiv, abs/2301.12597, 2023. hotlhttts://// //api. semanticschon owoo/oo///ooo.oooooooooooo 25636900509 3
[38] Zeyu Li, Ruitong Gan, Chuanchen Luo, Yuxi Wang, Jiaheng Liu, Ziwei Zhu Man Zhang, Qing Li, Xucheng Yin, Zhaoxiang Zhang, and Junran Peng. Materialseg3d: Segmenting dense materials from 2d priors for 3d assets. arXiv preprint arXiv:2404.138 13923, 2024. 26
13

---

[39] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 300-309, June 2023.
[40] Ivan Lopes, Fabio Pizzati, and Raoul de Charette. Material palette: Extraction of materials from a single image. arXiv preprint arXiv:2311.1706/ 2023. 2, 3,
[41] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with
pretrained models. Advances in Neural Information Processing Systems, 36, 2024. 26
[42] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprint arXiv:2211.06000, 2022.
[43] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[44] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022. 3
[45] OpenAI. Gpt-4v(ision) system card. OpenAl, 2023. URL /tt. sematicscchola.r org/Corpus ID: 263218003.. 3, 8, 10, 32, 33, 34
[46] OpenAI. Gpt-4 technical report. arXiv, pages 2303-08774, 2023. 2,
[47] Keunhong Park, Konstantinos Rematas, Ali Farhadi, and Steven M Seitz. Photoshape: Photo- realistic materials for large-scale shape collections. arXiv preprint arXiv:/809.099 09761, 2018.
22
[48] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022. 3
[49] Zhangyang Qi, Ye Fang, Zeyi Sun, Xiaoyang Wu, Tong Wu, Jiaqi Wang, Dahua Lin, and Hengshuang Zhao. Gpt4point: A unified framework for point-language understanding and generation. arxiv preprint arXiv:2312.02980 2023.
[50] Zhangyang Qi, Yunhan Yang, Mengchen Zhang, Long Xing, Xiaoyang Wu, Tong Wu, Dahua Lin, Xihui Liu, Jiaqi Wang, and Hengshuang Zhao. Tailor3d: Customized 3d assets editing and generation with dual-side images. arXiv preprint arXiv:2407.06191, 2024.
[51] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or Texture: Text-guided texturing of 3d shapes. arXiv preprint arXiv:2302.01721, 2023. 2, 18, 19
[52] Prafull Sharma, Julien Philip, Michael Gharbi, Bill Freeman, Fredo Durand, and Valentin Deschaintre. Materialistic: Selecting similar materials in images. ACM Transactions on Graphics (TOG), 42(4))1-14, 2023. 26
[53] Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi- view diffusion for 3d generation. 0rXiv:2308.165122 2023. 3, 8,
[54] Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Alpha-clip: A clip model focusing on wherever you want, 2023. 3
[55] Zeyi Sun, Tong Wu, Pan Zhang, Yuhang Zang, Xiaoyi Dong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Bootstrap3d: Improving 3d content creation with synthetic data. arXiv preprint arXiv:2406.00009. 2024. 2,
[56] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. ArXiv, ahs/2309. 16653, 2023. URL https: //api.pp../// semanticscholar htte/CoppuSI::2511311555
14

---

[57] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05005. 2024. 3
[58] Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from single image, 2024. 2, 3, 8,
[59] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, (bss2232..1997/ 2023. URL https: /api...... semanticscholar
org/Corpus ID: 2572152211 19404.44... 2,3
[60] Shimon Vainer, Mark Boss, Mathias Parger, Konstantin Kutsy, Dante De Nigris, Ciara Rowles, Nicolas Perony, and Simon Donné, Collaborative control for geometry-conditioned pbr image generation. arXiv preprint arXiv:2402.05591, 2024.
[61] Giuseppe Vecchio and Valentin Deschaintre. Matsynth: modern pbr materials dataset. arXiv
preprint arXiv:2401.06055. 2024. 5, 35
[62] Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation, 2023.2
[63] Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-lrm: Pose-free large reconstruction model for joint pose and shape prediction, 2023. 2,
[64] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305. 16213, 2023. 3
[65] Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshlrm: Large reconstruction model for high-quality mesh, 2024.
[66] Felix Wimbauer, Shangzhe Wu, and Christian Rupprecht. De-rendering 3d objects in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18490-18499, 2022. 25
[67] Tong Wu, Zhibing Li, Shuai Yang, Pan Zhang, Xinggang Pan, Jiaqi Wang. Dahua Lin, and Ziwei Liu. Hyperdreamer: Hyper-realistic 3d content generation and editing from single image, 2023.3 3
[68] Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein. Gpt-4v (ision) is human-aligned evaluator for text-to-3d generation. arXiv preprint arXiv:2401.04092. 2024. 3,8
[69] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. In- stantmesh: Efficient 3d mesh generation from single image with sparse-view large reconstruc- tion models. arXiv preprint arXiv:2404.07191, 2024. 2, 8,9
[70] Xudong Xu, Zhaoyang Lyu, Xingang Pan, and Bo Dai. Matlaber: Material-aware text-to-3d via latent brdf auto-encoder. arXiv preprint arXiv:2308. 09278, 2023. 2,
[71] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang. Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, and Kai Zhang. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model, 2023.
[72] Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023. 2,
[73] Shuai Yang. Jing Tan, Mengchen Zhang, Tong Wu, Yixuan Li, Gordon Wetzstein, Ziwei Liu, and Dahua Lin. Layerpano3d: Layered 3d panorama for hyper-immersive scene generation. arXiv preprint arXiv:2408.13252. 2024. 2
15

---

[74] Haiyang Ying, Yixuan Yin, Jinzhi Zhang, Fan Wang, Tao Yu. Ruqi Huang, and Lu Fang. Omniseg3d: Omniversal 3d segmentation hierarchical contrastive learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 20612-20622, 2024.
[75] Kim Youwang. Tae-Hyun Oh, and Gerard Pons-Moll. Paint-it: Text-to Text-to-texture synthesis via deep convolutional texture map optimization and thysically-basee rendering, arXiv preprint arXiv:2312.11360, 2023. 2,3
[76] Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, Wenwei Zhang, Hang Yan, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internin-xcomposer: A vision-language large model for advanced text-image comprehension and composition, 2023. 3
[77] Shangzhan Zhang, Sida Peng. Tao Xu, Yuanbo Yang, Tianrun Chen, Nan Xue, Yujun Shen, Hujun Bao, Ruizhen Hu, and Xiaowei Zhou. Mapa: Text-driven photorealistic material painting for 3d shapes. In ACM SIGGRAPH 2024 Conference Pupers, pages 1-12, 2024. 22
16

---

A Appendix Overview
In this appendix, we provide additional details and results that are not included in the main paper due the space limit. The attached video includes intuitive and interesting qualitative results of Make-it-Reat
B Details of Make-it-Real
In this section, we detail the pipeline that briefly outlined in the main paper. We commence by elaborating on the generation of SVBRDF Maps, incorporating illustrative figures detail the process involving operations in computer graphics. Appendix B.1 is dedicated to explaining the acquisition of region-level texture map partition. Following that, in Appendix B.2 we discuss the method of pixel-level albeddo-rferenced estimation. Then, in Appendix B.3 we declare some details of rendering procedure. Appendix B.4 details the prompt design material captioning and matching. Appendix B.5 reports the details of the GPT-4V based evaluation.
B.1 Texture Partition Module Design
As illustrated in Figure 10, our process initiates with the rendering of a 3D object incorporating the original albedo (i.e. query albedo) from multiple perspectives. Following this rendering phase, we employ GPT-4V alongside a segmentor to derive segmented masks for materials associated with each viewpoint. The subsequent step involves the extraction of regions masked in these images and their sack-projection onto the mesh of the object. By examining the object from all acquired viewpoints and applying UV unwrapping techniques, materials. Subsequently, each material's ' segment refined using a albedo-based mask refinement operation. Ultimately, by combining the segments of all materials, obtain region-level texture partition map, which serves to guide our subsequent, more detailed operations.
achieve preliminary segmentation of all
B.2 Albedo-referenced Module Design
As illustrated in Figure 11, developed pixel-level upon the foundations laid out in Appendix B.1. This module is inspired by a technique frequently employed by 3D artists, who often utilize albedo maps reference generate images of other
estimation module, building
render
bock papp
shape
query abedo
URWATD
ceremic
region-level parition map
C.
metal
texture mask
key albedo
Figure 10: Region-level texture partition module. This module extracts and back-project localized rendered images 3D mesh, using UV unwrapping for texture segmentation, thereby resulting in precise partition map of different materials.
17

---

part texture maps
final texture
metalness
matainess
query albedo
key
normal
equa zettist
Index
mase
(R))
match pioal
roughness
roughness
specular
(R,G,0) kg
piae!
neareast
nerighbour
height
height
Albedo RGB KD-Tree
[RGG
fast query
index merge
Figure 11: Pixel-level refereereeerenced estimation module. We generate spatially variant BRDF maps by referencing albedo maps, employing KD-Tree algorithm for efficient nearest neighbor searches, and normalizing colors via histogram equalization.
material properties. Accordingly, we designate the known albedo map the query albedo, and the albedo corresponding to the region of interest in the material as the key albedo.
The process precisely obtain the final SVBRDF maps is divided into four steps: 1) Initially, to address potential gaps in color intensity between the two albedo maps, histogram equalization is uniform color distribution across the image. 2) Subsequently, for each
employed achieve pixel on the query albedo-termed reference pixel-we seek the most similar neighboring color index on the key albedo. Given the high dimensionality of both maps (typically 1024x 1024 pixels), brute-force approach to this search would be computationally prohibitive. To this end, we accelerate the pixel query process using KD-Tre Tree algorithm, which organizes the RGB values of the key map's albedo into a KD- Tree for efficient nearest neighbor searches, reducing the computation time to under seconds. 3) The third step involves using the obtained indices to obtain corresponding values from the rest of the material maps. 4) Finally, by aggregating the query results for all material segments, we are able to generate the comprehensive spatially variant SVBRDF maps.
B.3 Rendering Procedure Details
In the context of computer graphics, "albedo" typically denotes the primary color of material, concept analogous to "base color" and "diffuse" in Physically-Bar -Based Rendering (PBR) paradigms, both representing the inherent color of the material under uniformly scattered illumination.
In accordance with workflow requirements, the inclusion of height maps, displacement maps, specular maps, and combinations of additional maps is optional. For objects demanding high surface details, such as the oil barrel Figure and the stone horse in Figure 16, displacement or height maps are incorporated and rendered using the mature 3D computer graphics software Blender engine. Regarding 2D-3D alignment techniques, including rasterization rendering, pook-projection, and UV unwrapping detailed in Section 3.2.1, we adhere the methodologies outlined in [51]. Specifically,
18

---

You showed with group of spherical PBR materials(all made tiles, which is kind ofCeramic, total 5), you generate caption for each(about 20-30 words), try to distinguishtheir difference. Only describe the appearance features (must including color" and detailedmaterial (such patterns, roughness, metainess, concave and convex patterns, condition))and don't give too much other information. Do not describe whether it is reflective not. Do notdescribe the shape overall object. Please represent the output result (1:Description 1, Description 2, 3: Description 4: Description Description 5)
dictionary to
gradient pastel purple to blue with smooth, matte ceramic finish slight bevel edges.", "Soft teal ceramic tiles with a satin finish, arranged in fish scale pattern with a slightly raised
texture."
"Creamy beige ceramic tiles with matte finish, laid in staggered brick pattern with flatsurface and sharp edges.".
"Pure white ceramic tiles with glossy finish, aligned in straight grid pattern with slightconvex curvature.", BBighh white ceramic tiles with matte finish, arranged an offset pattern with pronounced grout lines and a smooth texture.)
Figure 12: Material captioning. We demonstrate how to construct detailed material descriptions, registering material information in textual form, thus providing convenient bridge when querying with multi-modal large language models.
we utilize the Kaolin package (Kaolin [27]) and proj-projection with masks [51], with the model learning rate set to 0.02 in Equation (3) and the difference coefficient o set to 0.1 in Equation (4).
This approach facilitates the systematic generation of various material texture maps in the presence of only an albedo map, thereby ensuring consistency and realism within the Physically Based Rendering (PBR) workflow. Moreover, the flexible application of different map types and 2D-3D alignment techniques significantly enhances the detail and realism of rendered objects, effectively meeting the demands of diverse rendering scenarios.
We evaluate the basic performance the model by testing on a subset 70 objects. Specifically, we measure the average runtime, the number of material parts per object, the number of GPT-4 queries per object, and GPU memory usage, as shown Table 2. Additionally, we provide a detailed time consumption analysis for each step, presented in Table 3.
Table 2: Model performance metries. The table reports the model's runtime, average number of material parts per object, queries per object, and GPU memory usage.
Run Time (min)
# Material Parts/obj.
# Queries/obj.
Memory Cost (GB)
1.54
2.42
5.76
12.52
Table 3: Time usage statistics for each module.
Render & Seg
Mat Retrieval
Mat Generation
Total
Run Time (min)
0.26
0.49
0.79
1.54
B.4
Prompt Details for Multi-modal Large Language Models
Prompts design has emerged as a pivotal factor for eliciting desired outcomes from MILMMs. This section delineates the intricacies of our prompt design for both material captioning and matching.
19

---

g
This is a(or group rendered image(s) of an object using pure texture map without PBR material information. Please use its visual characteristics (including color, pattern, object type) along with your existing knowledge to INFER IMAGINE possible materials different parts.
...By choosing yellow metal for the continents and matte white plastic for the ocean areas, the globe maintains cleon, minimalist aesthetic while the contrasting materials bring the details and contours of the Earth's s surface. The combination of the shiny and matte finishes would make the globe on attractive decorative item, adding touch of modern sophistication to setting.
Identify material each part(marked with numerals), presented the exact form of (number "material"), Don't output other information. (aptional list of material is [Ceramic, Concrete, Fabric, Ground, Leather, Marble, Metal, Plaster, Plastíc, Stone, Terracotta, Wood, Misc]. The 'Misc' category is output when nothing else matches.)
(1: 'Mtaal, 2: "Plastic')
Select the most similar Metal material type of number part of the image, according to the analysis of corresponding part material(inlluding color, pattern, roughness, and so on...). If you find it difficult to subdivide, just output (). Don't output other information Only a single word representing the category from optional list needs ChristmasTreeeOn Treeernmmntt CorrugatedSteel, DiamondPlate, Foil, Metal, MetalPlates, MetalWalkway, PaintedMetal, Rust, SheetMetal, SolarPanell)
output. (optional list of material is (Chainmail, Chip.
Metal
Look at the material carefully of number 1 part of the image, here are some descriptions about metal materials, can you tell which Just tell me the final result in dict format with material name and descrption. Don't t output oth information.
best description match the part the image?
MMttl0000;; "A silver-colored metal with high-gloss finish and visible smudges and scratches." MMeaol_Metall02::: slightly tarnished silver metal with marks and few rust spots indicating weathering.",
Bestal_MeealO03"" "A clean silver metal surface with minimal weathering and few small scratches and dents."
material library look up...
(' Mietal_Metall07:: 'A golden color material with polished surface showing tarnish and subtle wear marks.".
'Metal "Metol_Metal355: Metal035': striations.")
burnished copper color with a weathered patina and faint swirling material
Metal Metal007
Material Generation
mask projection
post-process
Figure 13: Detailed prompts of GPT-4V based material matching. Prompts in blue changes according the current assigning part and GPT-4V's results.
20

---

Assuming you are an expert in determining whether the textures of models meet Photorealistic standards (mainly focus the quality of texture). have generated some 3D objects using different methods. These objects have the some shope but different textures. Above are the four-view images of a model generated by the two methods (the upper half is the four views of the object generated by method one, the lower half the four views the object generated by methad two). Please help me evaluate which of the Two objects is more Photorealistic terms of texture. Respond with if you think the upper half object is more realistic. Respond with 2 if you think the lower half object is more realistic. Also tell me the reason in new paragraph.
Based the provided images, my evaluation would favor object (the lower half object) as being more photorealistic in terms of texture.
The textures object 2 have more complex and nuanced appearance. The material of the uniform exhibits subtle variations in reflectivity and surface detail that suggest more realistic fabric texture. Additionally, the skin texture oppears more lifelike with variations in tone and reflectivity that are characteristic of human skin. The metal police badges object 2 has more nuanced reflection, which suggests more realistic interaction with environmental light. In contrast, the textures object while still detailed, lack the same. level of subtlety and variation. The uniform looks somewhat flat and less fabric-like, and the overall surface details are less convincing natural variations.
Figure 14: GPT-4V based evaluation prompts. We define a prompt for GPT-4V to generate human- aligned comparison over objects before and after our texture refinement.
Material eaptioning. Due to limit context length for image tokens, it is currently impossible for MLLMs to directly memories thousands of images of material balls. To address this, we use GPT- 4V generate detailed captions for each material ball. As demonstrated in Figure 12, we input concatenated images of material ball from same subcategory with prompt specifically tailored to highlight texture properties. This strategy guides GPT-4V generate detailed caption for each material ball, distinguishing the subtle differences between them. These detailed captions are then registered into material library.
Material matching. The MLLM-based material query process is exemplified through a simplified case, illustrated Figure 13. Initially, GPT-4V is queried to identify the basic material for matching. Following this preliminary matching, GPT-4V is queried for more specific materials using the names of subcategories filters, which narrows down the selection to few candidates within the same subcategory. For final material selection, GPT-4V is prompted with detailed captions from the library, directing to allocate the most suitable material. This is accomplished conjunction with meticulously engineered segmentation process in UV space, ultimately facilitating MLLM- M-bseedd material matching. is noteworthy that prompting GPT-4V to navigate through a three-level tree structure has been proven to enhance both the efficiency and accuracy of the matching process, as opposed to directly selecting from thousands of materials directly.
B.5 Evaluation Details of Quantitative Results
Evaluation details. For the purpose of evaluating objects pre- and post-refinement, we render four view images each object and concatenate them vertically to facilitate comprehensive assessment. We craft a specific prompt to guide GPT-4V in conducting an impartial comparison of the texture quality between two objects. As demonstrated in Figure 14. GPT-4V's advanced capabilities enable it to differentiate between the two objects, providing comparison results that closely align with assessments made by human experts. In the scenario evaluated, GPT-4V successfully identifies all enhancements (metal badges, human skin, and fabric textures).
21

---

generate
real image
regions
materials
only albodo
regions
materials
object textures
Material Palette
Make-it-Real
Figure 15: Comparison between previous method and Make-it-Rese We demonstrate the distinc- tions between Material Pa]ette[40] and our method terms of material identification and extraction. Our overall pipeline presents a more challenging task, where the input is a rendered image with only albedo information, and the output consists of textures for the entire object.
For human evaluators, we engage 16 volunteers with strong backgrounds in computer science, specif- ically in 3D modeling and generation. To ensure unbiased evaluations, we maintain gender balance through random selection. The assessment focuses on determining object material photorealism, as shown in Figure 14. Volunteers train on 40 Objaverse cases, with and without materials, to enhance their realism judgment for accurate test data evaluation.
C Additional Related Works
In the context of extracting real-world materials from single images, the methodologies most closely related to our work are Material Palette [40] and Photoshape [47]. Material Palette enables the extraction of materials at the region level from single image, generating tilable texture maps for corresponding areas. Photoshape, on the other hand, automates the assignment of real materials to different parts of 3D shape by training material classification network. However, this approach needs the training of a material classifier, which is constrained to a limited set of object categories (e.g., chairs) and material types. Besides, both methods require rendered images of objects with real materials as input.
Our problem setting poses more challenging task, illustrated in Figure 15, which involves identifying and recovering the original material properties from images objects that only have a albedo map, in addition to generating different material maps for the entire object. Humans have the capacity to intuitively infer the underlying material properties of 3D objects from single images containing only shape and basic colors. This capability is attributed
robust material recognition abilities and comprehensive prior knowledge of object categories, colors, and material attributes. Building on this concept, approach harnesses the potent image recognition capabilities and prior knowledge inherent large-scale multimodal language models to efficiently execute region-level material extraction and identification, which is further enhanced by subsequent 2D-3I) alignment and albedo-referenced techniques to generate and apply material texture maps for physically realistic rendering of 3D objects. Furthermore, [40] extracts the material takes an image containing three materials takes more than 10 minutes. In contrast, our method takes only about to 2 minutes match and generate all the materials.
minutes, and process
In more recent work, both MaPa [77] and our paper aim to enhance 3D objects with materials, but the key difference lies in material retrieval methods. MaPa [77] relies CLIP, while use MLLM, distinction that we believe significantly boosts retrieval effectiveness for these reasons: 1) Enhanced Global Context: CLIP focuses
at time, requiring other parts to be masked, which limits global semantic information. MILMM, however, retains this context by simply highlighting the relevant area. 2) Hierarchical Material Dictionary: We introduce hierarchical dictionary in
MLLM, providing descriptions and images of material spheres, enabling access to more extensive library of 1,394 materials compared MaPa's 1 118. 3) Advanced Explainability: With MLLM's rapid development, transitioning from CLIP to a multi-step hierarchical inference process with LLM offers improved performance and clearer decision makiin...
22

---

metainess
Wob, roughness
roughness
wub. roughness
roughness
displacement
dispiacement
wub. haight
haight
Figure 16: Effects of different texture maps. We evaluate the effects of metalness, roughness, and displacemen/heeight maps on the appearance of 3D objects.
D Additional Experiments
D.1 Effects of Different Texture Maps
We validate the impact of various texture maps generated by Make-it-Real on the appearance of 3D objects, as illustrated in Figure 16. For each example, the first column lacks the corresponding map enhanced by Make-it-Real, while the second column includes the same conditions with the specific material map. Initially, we compare the effect of metalness on object appearance. We observe that the objects the right, with a higher metalness map value, exhibit higher reflectivity, and the reflected light's 's color is similar to the material's 's own color, closely resembling worldwrrdd objects and appearing more aesthetically pleasing. In contrast, the objects on the left without the metalness map have lower reflectivity, and the reflected light tends to be white.
Next, we examine the impact of the roughness map, which controls the smoothness of the material's surface. We observe that the silver chess piece on the right, with a low roughness texture map, becomes smooth, and together with metalness, produces mirror-like reflection effect. On the other hand, the box on the left with a roughness map exhibits changes in light dispersion on its surface, with some areas showing highlights, while also adding and enriching scratch texture details on the surface.
Furthermore, we compare the effects of displacement and height on objects, both of which are usually optional and can also impact object appearance. Height maps typically use grayscale values to represent the surface's relative height, simulating relief effect through changes in lighting and shadows. As shown in the last row on the right, the object with a height map has an uneven surface, enhancing the sense of depth. Displacement mapping is more powerful technique that changes the
23

---

ori UV
light mv
smart_uP UV
sphere__v
unwrap_.u UV
albedo
Figure 17: Effects of different UV mappings of input mesh.
Geooeerry
Albedo
Make-it -Real
Artist
Figure 18: More Comparisons between Make-it-Rees and Artist-Create Materials.
vertex positions of the geometry based on the map's values, creating realistic relief effects. As shown on the left, the stone sculpture with displacement mapping exhibits very realistic details and a sense of relief. We acknowledge that the generated PBR maps are not true representations of physical maps, but our algorithm generates PBR maps that closely approximate the visual characteristics of true PBR properties, which significantly enhances the realism of 3D objects.
D.2 Effects of Different UV Mappings
We conduct experiments to evaluate the impact of UV mapping on our method. As shown in the second column of Figure 17, UV mappings with excessive fragmentation and color entanglement cause the 2D segmented images to mix with other regions when reprojected into the UV space, leading to material blending issues. However, mappings and Blender's 's built-in mapping methods in Figure 17, such as smart, sphere, and unwrap. This indicates that our method still demonstrates good robustness with many mapping techniques.
method shows good results with original artist-created UV
In practice, we observe that most objects Objaverse have UV mappings with good properties, meaning they are not excessively fragmented into small pieces. Additionally, can control the UV mapping process: For some low-quality UV maps and generative objects that originally lack UV maps, re-unwrap them to achieve higher quality.

---

objaverse
refined
Figure 19; Addressing Shading Issues with IntrinsicAnything [9]. The first row shows that the derendered albedo images from Objaverse are consistent with the original one. The last two rows demonstrate the successful derendering results for generative models, effectively reducing shading.
E
More Qualitive Results
E.1 Make-it-Real for Existing 3D Assets
In this section, show more qualitative results of Make-it-Real for existing 3D assets from [16]. We show more qualitative results of Make-it-Real for existing 3D assets from [16]. Results are shown Figures 20 and 21. The first row presents the original 3D object with only a albedo map, while the second row showcases the object enhanced by our method with various material maps.
Comparisions with Artist-Created Materials. We conduct user study comparing materials gener- ated by our Make-it-Real with those created by artists, visualizing some results in Figure 18 for close examination. For objects like pistol, wooden chair, speakers, and saxophone, Make-it-Real showed strong similarity to artist-made materials, maintaining consistent metallicity, roughness, highlights, and coloration. cases such shield, boots, landline phone, and oil drum, our method produced materials that, while different, are realistic and sometimes visually superior. Our pipeline approaches the level of refinement seen some artist-created materials, which often require significant time and effort, and clearly outperforms more basic and crude artist-generated materials. This highlights its effectiveness and potential for practical applications.
E.2 Visualization of generated texture maps.
In this section, we show visualization of generated texture maps in Figures 22 and 23. Our approach, guided by the query albedo reference, produces material maps with distinct material partitions and maintains a distribution consistent with the albedo map. As result, the enhanced object exhibits both realism and texture consistency.
F Limitations and future work
While promising results have been achieved by MLLMs for texture assignment through Make-it-Real Our work still faces several challenges.
Lighting Effect and Addressed Solutions. First, although our method can achieve appropriate texture assignment for albedo-only model, it does not support reverse transform from shaded texture map to albedo map for generated 3D objects. This causes problem of assigning different materials to the dark shadow and highlight area when generated object with mesh already shaded in different lighting conditions.
This issue less pronounced in artist-created models, where albedo maps have less shadows and lighting effects. Our tests show that the albedo obtained from inverse rendering methods [9] closely matches the original artist-created albedo maps from the Objaverse dataset, as illustrated the first row of Figure 19. For generative models, baked-in shading effects are more pronounced. Our method addresses this by integrating mature derendering algorithms [9, 14, 31, 66), specifically the Intrinsic Anything [9], into our pipeline with minimal complexity. This integration derenders from four
25

---

viewpoints and buck-projeet to obtain a better albedo map. then applies our Make-it- Make-it-Real method for material painting. As shown in the last row, this approach reduces lighting noise and supports wider range of inputs.
Impact of Model Quality and Segmentation. Second, find base model quality is essential for MLLM to assign correct materials. When base 3D object is of low quality(e.g. uneven surfaces or mixed colors across different parts), is difficult for MLLMs identify object properties when ground truth text prompt describing the object is not available. Additionally, there has been limited progress in material segmentation in 2D and 3D demoain [52, 38]. The material segmentation algorithm included in our proposed pipeline can serve as useful baseline and provide inspiration for future work. We also explore the integration of the pipeline with 3D segmentation networks, which shows promising results. For future work. we consider methods mitigating these challenges as well as adding user friendly control into our fully-automatic pipeline.
G Data Ethics
The datasets utilized in our work, specifically Objaverse, is composed entirely of inanimate objects. Importantly, the Objaverse dataset we used have undergone rigorous ethical filtering as part of the Cap3D [41]. Cap3D's ethical filtering process removed all potentially problematic data, including any identifiable human elements and NSFW content (detailed Sec 3.2 of Cap3D). This process included the removal of objects with licenses that do not permit commercial usage, the exclusion of objects lacking sufficient camera information for rendering. and the application of face detection and NSFW classifiers with high thresholds to ensure thorough filtering. The final dataset used in our work, therefore, does not contain any human-derived data or data related to human subjects.
We recognize the potential risk of misuse associated with this technology, particularly in the creation of realistic fake human representations. However, since our dataset fully excludes human data, this risk is substantially mitigated.
H Broader Impacts
Potential positive societal impacts. The proposed method facilitates more realistic and accurate representations of materials in 3D models, benefiting industries such gaming, virtual reality, and film, leading to more immersive and engaging experiences. By automating the material assignment process, Make-it-Real significantly reduces the time and effort required for 3D content creators, allowing for more efficient workflows and enabling creators to focus on more creative aspects of their work. This approach can democratize high -ualityal 3D content creation by making advanced material application techniques accessible to a broader range of users, including those without specialized skills in graphic software.
Potential negative societal impacts. The improved realism in 3D assets could be exploited for creat- ing highly convincing fake visuals deepfakes, which might be used in disinformation campaigns to mislead audiences. There a risk that the materials generated could inadvertently perpetuate biases or stereotypes if the training data for GPT-4V includes biased representations of certain materials or objects. As the method involves processing and recognizing visual data, there could be concerns regarding the privacy of any real-world images used inputs, particularly if they contain sensitive or personal information.
26

---

w/o material
Ours
Niea
Nedi
Frarte
Meta
Groend
Fratte
Word
Proskct
w/o material
Ours
w/o material
Ours
w/o material
Ours
Figure 20: More qualitative results of Make-it-Ress refining existing 3D assets without material. Objects are selected from Objaverse[16] with albedo only.
27

---

w/o material
Oars
Pluti
material
Ours
w/o materall
Ours
material
Ours
Musul
Figure 21: More qualitative results of Make-it-Ress refining existing 3D assets without material. Objects are selected from Objaverse[16] with albedo only.
28

---

query albedo
metalness
roughness
specular
normal
height
Figure 22: Visualization of generated texture maps. The first column represents the original query albedo map of 3D objects, while the subsequent columns showcase the corresponding material maps generated by Make-it-Real.
29

---

Metal
Metal
Wood
Motall
Motal
Woodd
Geometry
Albedo
Roughness
Metallic
Height
Normal
W. material
Figure 23: Visualization of generated texture maps. We show part of the SVBRDF material maps generated by Make-it-Real and the final rendering results. We displayed the texture maps and the corresponding 3D rendering effects. The albedo is the input, and the following four columns show the material effects in the UV space and the 3D object.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims the improvements are supported with extensive quantitative experiments with a great amount of qualitative cases. The proposed method is tested on many objects both sourced from Objaverse and generated by many edge-cutting 3D) generative models and proven be effective and robust.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide detailed discussion of limitations in Sec. F
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
31

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
For main results, we detail the full test settings in Sec. 4. For GPT-4V [45] based preference study, we provide detailed test prompts and test settings in Sup. B.5. Readers easily follow the same setting and reproduce all of our experiment results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
32

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips..
htw.mmmisssiooiiiooooooo.o for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.co.
wwwpc//gg//////eeeeeeeecce CoodeSubmiiieeee /ttec/gggiddeeoeeeubbbbii wwwssssssiiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Full experiment details is available in Sec.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The full test with is time consuming mainly due the rate limit of GPT- 4V [45] (each conversation takes around minute). We improve the experiment confidence by increasing the amount of test hundreds of objects.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
33

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Our paper mainly reliance on API access to GPT-4V [45] and Blender. The segmentation model can be deployed on any computing resources with most mainstream GPUs with CUDA and 2 8G memory.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics pettps://neurppe: cc cc/public//eeee Coublic/Ethiccoo istideeelllleeeess
Answer: [Yes]
Justification: The authors have read the NeurIPS Code Ethics carefully and find no confficts with it. Potential societal impacts answered in the next question.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Detailed discussions available in Sup. H
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
34

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Existing assets we use quality library of selectable real materials-wa build upon the MatSynth [61] dataset, which contains over 4,000 high resolution tileable materials under CCO license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
35

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer[[NA]
Justification: the paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
36

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "iduu4rf6A5", "reviewer": "Reviewer_s3Ca", "summary": "This paper tackles the problem of recovering materials for 3D meshes with known geometry and base color. The proposed solution consists of three steps:\n\n- Utilizing Semantic-SAM to segment the 3D objects, identifying and isolating various material regions.\n- Using hierarchical prompting to retrieve and match materials from a material library.\n- Generating the actual SVBRDF map based on the albedo and the retrieved materials.", "questions": "- How are viewpoints selected?\n\n- Does the method depend on good UV mapping?\n\n- How are the categories within the material library constructed?", "limitations": "The authors discuss the limitations of their method and broader impact in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The qualitative results look promising for both artist-created assets and objects generated by 3D generative models. \n\n- The paper explores and demonstrates the potential of leveraging large multimodal language models to describe material properties of the objects—an exciting finding that could inspire future research. \n\n- The design of hierarchical prompting for material library retrieval presents an intuitive and effective way to improve the efficiency of the query.", "weaknesses": "- In the user study, participants are asked to compare the base mesh and the refined mesh. However, users may naturally prefer objects with enhanced lighting effects, which does not effectively evaluate the quality of the generated materials. A more informative approach might be to compare an artist-created mesh with material to the same shape with a re-predicted material, which could provide more insights into material quality.\n\n- For the application presented in this paper, a common issue is that 3D models with a single albedo map often have baked-in shadows and other lighting effects. This is common in both artist-created models and especially in outputs from 3D generative models (e.g. is the shadow in Fig 7 bottom baked in albedo?). It's unclear how the proposed method addresses or might even be misled by these artifacts since it assumes that similar colors represent similar BRDF values in estimation. This could be a limitation when compared to alternative methods that directly optimize or predict the texture map. A discussion on this could be helpful to clarify.\n\n- The discussion on performance metrics is missing, such as runtime and the average number of parts and queries per object.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review and insightful comments! We will address your questions below and in the revised paper.\n***\n### W1: More Comparisons with Artist-Created Materials: \nThank you for the insightful suggestion! We conducted an additional user study comparing materials generated by our Make-it-Real method to those created by artists. We filtered the Objaverse dataset for objects with material maps and randomly sampled 200 objects for the experiment. \n\nOur results are shown in the following table. The second row highlights our method's superiority over albedo-only objects, with 73.1% of users recognizing enhanced material effects, reinforcing the paper's conclusions. The first row indicates that users rated the material effects generated by our method as superior to or on par with artist-created materials for 61.6% of the objects. This is a highly promising result.\n|                               | Win/Tie Combined% | A/B Win% | A/B Tie% | A/B Lose% |\n| :---------------------------: | :---------------: | :------: | :------: | :-------: |\n|    Make-it-Real v.s. Human    |       61.6        |   30.5   |   31.1   |   38.4    |\n| Make-it-Real v.s. Albedo only |         88.2         |   73.1   |   15.1   |   11.8    |\n\nWe examined these cases closely, as visualized in Figure C1. For objects like pistol, wooden chair, speakers, and saxophone, Make-it-Real showed strong similarity to artist-made materials, maintaining consistent metallicity, roughness, highlights, and coloration. In cases such as shield, boots, landline phone, and oil drum, our method produced materials that, while different, are realistic and sometimes visually superior. Our pipeline approaches the level of refinement seen in some artist-created materials, which often require significant time and effort, and clearly outperforms more basic and crude artist-generated materials. This highlights its effectiveness and potential for practical applications.\n\n***\n\n### W2: Addressing Baked-in Shadows and Lighting Effects \n\nThis issue is less pronounced in artist-created models, where albedo maps have less shadows and lighting effects. Our tests show that the albedo obtained from inverse rendering methods[1] closely matches the original artist-created albedo maps from the Objaverse dataset, as illustrated in the first row of Figure C3. For generative models, baked-in shading effects are more pronounced. Our method addresses this by integrating mature derendering algorithms, specifically the Intrinsic Anything[1], into our pipeline with minimal complexity. This integration derenders from four viewpoints and back-projects to obtain a better albedo map, then applies our Make-it-Real method for material painting. As shown in Figure C3's last row, this approach reduces lighting noise and supports a wider range of inputs. We will include more details about the derendering of inputs in the final version.\n\n[1] IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination.\n\n***\n\n### W3: Results of Performance Metrics\n\nThanks for your valuable feedback. We have added tests for average performance metrics, as shown in the table below. \n\n| Run Time(min) | # Material Parts/obj. | # Queris/obj. | Memory Cost(GB) |\n| :-----------: | :-------------------: | :-----------: | :-------------: |\n|     1.54      |         2.42          |     5.76      |      12.52      |\n\n\nAdditionally, we conducted further tests to measure the time required for each of the three stages drawn in our pipeline: (a) Rendering and Segmentation, (b) Material Retrieval, and (c) Material Generation.\n\n\n|                   | Render & Seg | Mat Retrieval | Mat Genertaion | Total |\n| :---------------: | :--------------------: | :---------------------: | :---------------------: | :---: |\n| **Run Time(min)** |          0.26          |          0.49           |          0.79           | 1.54  |\n\n***\n\n### Q1. The clarification of viewpoint selection\n\nWe explain some of the detalis in Sec 3.2.1, and here we provide further details. Initially, we perform rasterization for rendering, and the selection of viewpoints follows the approach used in TEXTure. We render a total of ten viewpoints: eight at 45-degree intervals around the horizontal axis and two additional views, one top-down and one bottom-up, forming our initial set of rendered images. \n\nAmong these rendered images, we select the viewpoint with the largest foreground area (the projected area in rasterization) as our primary viewpoint because it is more likely to contain more material information. This viewpoint is then used for subsequent material retrieval and map generation. We further refine the material generation using our proposed material inpainting method in Sec 3.2.3. \n\n***\n\n### Q2. Discussion of UV mapping of input mesh\n\nWe conducted experiments to evaluate the impact of UV mapping on our method. As shown in Figure C2, except for the second column where the lighting UV mapping method results in fragmented and interwoven colors, other unwrapping methods such as smart, sphere, and unwrap produce better results. Our method does not perform well with overly fragmented UV maps but demonstrates good robustness with other mapping methods.\n\nIn practice, we have observed that most objects in Objaverse have UV mappings with good properties, meaning they are not overly fragmented into small pieces. Additionally, for some low-quality UV maps and generative objects that originally lack UV maps, we can  control UV mapping, (re-)unwrapping them to achieve higher quality.\n\n***\n\n### Q3. Categories construction in material library\n\nThese materials originally come from AmbientCG, an open online repository, where each material has a fine-grained category name, such as \"Planks\". We follow the classification of material categories in MatSynth, resulting in 13 major categories and 80 subcategories. This forms our two-level hierarchical tree structure, which corresponds to our hierarchical prompt retrieval.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YBQz3on87w", "reviewer": "Reviewer_mcgT", "summary": "This paper introduces the large-scale multimodal language models to realistic material rendering of 3D objects. Specifically, this paper employs MLLM to retrieve materials from a material library for different parts of . By combining 2D-3D alignment and diffuse reflection reference techniques, it generates and applies material texture maps to objects, achieving realistic rendering of 3D objects.", "questions": "See the weakness.", "limitations": "The authors have not addressed the limitation in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces the MLLM into the texture inpainting pipeline, which is a interesting direction in the future.\n2. The experiment results demonstrates the effectiveness of this method.\n3. The paper is well-written and clearly articulated.", "weaknesses": "1. This paper shares some similarities with Mapa[1]. Could the authors further demonstrate the differences between Make-it-real and Mapa and the superiority of this method?\n2. The necessity of MLLM in the pipeline requires further validation. How does the performance of the MLLM-based retrieval method compare to other retrieval methods? More experiments on the retrieval results are needed to show the superiority of the MLLM\n-based methods.\n\n[1]Zhang, Shangzhan, et al. \"MaPa: Text-driven Photorealistic Material Painting for 3D Shapes.\" arXiv preprint arXiv:2404.17569 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and insightful comments. We will address your questions and concerns below and in the revised paper.\n\n***\n\n### W1. Differences between Make-it-Real and MaPa and the superiority of our method.\n\n- **MaPa**: Introduces a text-driven segment-based procedural material graph representation. It uses a pre-trained 2D diffusion model as an intermediary bridge to connect text and material graphs. The method includes segment-controlled image generation and material graph optimization. It simply use CLIP for material retrieval.\n- **Make-it-Real**: Utilizes Multimodal Large Language Models (MLLMs), specifically GPT-4V, to recognize and apply real-world materials to 3D objects. The method involves texture segmentation, material matching, and SVBRDF map generation. It creates a comprehensive material library with detailed descriptions, allowing MLLMs to search and assign materials automatically.\n\nThe core difference lies in how to perform the material retrieval. MaPa [1] still uses CLIP for the material retrieval process, while we rely directly on MLLM. We believe this core difference significantly improves the retrieval effectiveness for the following reasons:\n\n1. CLIP can only focus on one area at a time. When generating images that need retrieval, other parts need to be masked while retaining the local part. This causes the retrieval process to lose the global semantic information of the object. MLLM, on the other hand, can retain the global semantic information by only circling the specified area in the pixels [2].\n2. We creatively propose registering a hierarchical dictionary in MLLM by providing descriptions and images of material spheres, allowing MLLM to master a richer material library (The library is much larger and more practical with 1394 materials while MaPa only have 118).\n3. With the rapid development of MLLM, transitioning this process from CLIP retrieval to a multi-step hierarchical and logical inference decision process relying on LLM is more promising with better explainability.\n\nTo further validate the above points, we supplement the experiment detailed in W2 testing using MLLM for retrieval compared to using CLIP.\n\nThere are also other differences that improving the performance compared to MaPa:\n\n1. We propose a new method to achieve 3D segmentation leveraging Semantic-SAM and MLLM, which is more general than MaPa that adopt an out-of-date mesh segmentation method [3].\n***\n\n### W2. Experiments on the superiority of using MLLM for retrieval.\n\nwe conducted additional experiments comparing GPT-4V (our method) and CLIP (MaPa method) to verify the superiority of using MLLM for retrieval.\n\nSpecifically, we use the CLIP model as a baseline for comparison, which is the method adopted in PSDR-Scene [4] and MaPa [5]. We selected 100 objects from Objaverse with high object quality and annotated each object with different materials following the material classification of [4], resulting in 13 coarse types (Objaverse-C) and 80 subtypes (Objaverse-F). The retrieval results are reported in the table below. The results fully demonstrate the effectiveness of using MLLM for retrieval. We discover CLIP often fails in cases with metallic materials, largely due to the difficulty of extracting material regions without distortion. Our pipeline, using GPT-4V, queries four viewpoints for global shape and object semantics, can leveraging its strong open world prior knowledge to achieve promising results. Additionally, we observed that CLIP’s accuracy significantly drops when dealing with fine-grained categories compared to coarse categories, while GPT-4V maintains a high retrieval accuracy. This result also indicates that MLLM is more practical, as such systems often need to handle thousands of materials in a much finer granularity in real-world applications. \n\n|                | Objaverse-C | Objaverse-F |\n| :------------: | :---------: | :---------: |\n| GPT-4V / top-1 |  **70.59**  |  **64.71**  |\n| CLIP-L / top-1 |    28.53    |    10.89    |\n| CLIP-L / top-3 |    47.06    |    22.65    |\n\n***\n\n[1] Zhang S, Peng S, Xu T, et al. MaPa: Text-driven Photorealistic Material Painting for 3D Shapes\n\n[2] Yang J, Zhang H, Li F, et al. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v\n\n[3] Sagi Katz and Ayell, et al. Hierarchical mesh decomposition using fuzzy clustering and cuts\n\n[4] Yan K, Luan F, Hašan M, et al. Psdr-room: Single photo to scene using differentiable rendering\n\n[5] Vecchio G, Deschaintre V. MatSynth: A Modern PBR Materials Dataset", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "reYUhOJBBH", "reviewer": "Reviewer_LVe6", "summary": "The proposed work leverages GPT-4V to extract and infer materials in albedo-only scenarios, utilizing existing material libraries to generate SVBRDF maps with a region-to-pixel algorithm. This approach enhances 3D mesh realism, ensures precise part-specific material matching, and is compatible with rendering engines, generating six comprehensive material maps. Developers need only paint albedo textures, with other material properties automatically generated, saving significant time. The contributions include using multimodal large language models for material recognition, creating a detailed material library, and developing a pipeline for high-quality material application to 3D assets.", "questions": "- How are the PBR texture maps in the material library generated and how to ensure the accuracy of the material property?\n- How does the proposed method perform with low-quality input albedo maps, such as those with many small UV pieces?", "limitations": "- The proposed method relies on GPT-4V for material generation from a generated database. However, many offline tools like ShaderMap can accomplish the same task more cost-effectively. These offline tools require no training and can produce comprehensive, high-quality material decompositions. Given that the generated materials are not physically-based, what advantages does the proposed method offer over efficient offline software that requires no training or fine-tuning?", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The method greatly enhances the realism of 3D objects by utilizing GPT-4V's extensive visual perception, supported by experimental evidence.\n- It generates detailed PBR material maps (roughness, metallic, specular, normal, displacement, height) that are compatible with various rendering engines in a low-cost way.", "weaknesses": "- The method's effectiveness is largely dependent on the capabilities of MLLMs such as GPT-4V, which may not always produce accurate results. While the paper claims that the generated materials, represented as PBR maps, are compatible with downstream engines, it should be noted that this generation method does not rely on any physical principles of light transport and therefore cannot be genuinely considered as PBR maps.\n- The method's performance might be constrained by the quality and detail of the initial albedo maps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and insightful comments. We will address your questions and concerns below and in the revised paper.\n\n***\n\n### W1. Realism and Application of Generated PBR Maps\n\nThank you for pointing this out. We acknowledge that the generated PBR maps are not true representations of physical maps, and we will clarify this in the final version paper. Nevertheless, our algorithm generates PBR maps that closely approximate the visual characteristics of true PBR properties. As demonstrated in Figure 5, our results significantly enhance the realism of 3D objects. Additionally, the analysis in the appendix (Figure 16) further examines the effects of each individual map, which mimic real textures, such as metallic properties (e.g., the dampening of the base albedo and increased surface shine on parts with high metallicity like scales and axes), roughness (with low roughness objects showing specular highlights), and displacement (fine bump effects on stone horses and bathtubs). In all cases, we produce visually comparable effects to true PBR maps.\n\n***\n\n### W2. Impact of Albedo Map Quality on Performance\n\nWe agree that the final results can be affected by the quality and detail of the initial albedo maps. However, our method is designed to enhance the physical appearance even when the input albedo map varies in detail.\n\nOn one hand, for objects that require meticulous craftsmanship and significant attention to detail, such as the truck and shield in Figure 5, our method precisely segments and assigns appropriate materials, significantly enhancing realism and producing high-quality objects with highly refined materials.\n\nOn the other hand, for objects with relatively simple albedo maps that can be created quickly, like the saxophone shown in Figure C1, our method substantially improves the object's appearance under environmental lighting conditions, increasing realism and achieving a qualitative improvement.\n\n***\n\n### Q1. The clarification on PBR texture maps\n\nWe would like to provide some clarification about this. The PBR texture maps in the material library are sourced from MatSynth. This dataset compiles high-quality PBR materials from publicly available resources like AmbientCG, which are widely used by artists for creative work. The materials are typically created using methods such as photogrammetry, scanning, and high-precision software, ensuring both visual fidelity and accurate physical properties.\n\n***\n\n### Q2. Performance with Fragmented UV Mapping\n\nThanks for your insightful questions. Overly fragmented UV mapping degrades the performance of our method. As shown in the second column of  Figure C2, UV mappings with excessive fragmentation and color entanglement can cause the 2D segmented images to mix with other regions when reprojected into the UV space, leading to material blending issues. However, our method shows good results with original artist-created UV mappings and Blender's built-in mapping methods in Figure C2, such as `smart`, `sphere`, and `unwrap`. This indicates that our method still demonstrates good robustness with many mapping techniques.\n\nIn practice, we observe that most objects in Objaverse have UV mappings with good properties, meaning they are not excessively fragmented into small pieces. Additionally, we can control the UV mapping process: For some low-quality UV maps and generative objects that originally lack UV maps, we can re-unwrap them to achieve higher quality.\n\n***\n\n### L1. Advantages of the Proposed Method Over Offline Tools\n\nFirstly, I would like to acknowledge that ShaderMap is a valuable tool for material creation, and it's great to see a thoughtful comparison of different approaches.\n\nOur method offers several distinct advantages:\n\n1. **Focus on 3D Models**: Our approach excels in directly generating complete and fine-grained PBR maps for 3D models, offering more comprehensive integration with 3D assets compared to traditional tools. Tools like ShaderMap primarily focus on estimating higher-quality PBR texture maps from low-quality material images, such as photographs or instrument-captured material images, which may include lighting or shadows. These inputs are inherently material images and often do not focus on rendered views of 3D objects. In contrast, our method directly enhances 3D models, providing a more robust solution for integrating high-quality materials.\n2. **Complexity and Usability**: Our method stands out for its ease of use and ability to produce semantically meaningful material assignments without requiring professional knowledge of 3D software. While ShaderMap can generate basic PBR maps (such as normal and specular maps) from albedo maps, this functionality currently yields relatively simple material effects. The material effects may appear to have lighting effects but do not necessarily match true semantic meanings. Since the entire albedo map is used as input, the software lacks fine-tuned regional specificity. Further manual adjustments are typically needed for local map values, requiring user interaction and expertise in 3D software, as well as additional time to achieve realistic results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nnOxsKEoGa", "reviewer": "Reviewer_dDLL", "summary": "This paper presents a novel framework leveraging MLLM priors (GPT-4V) to build a material library and proposes an automatic pipeline to refine and synthesize new PBR maps for initial 3D models with diffuse albedo only. The pipeline integrates existing tools, such as GPT-4V and Semantic-SAM, while introducing novel techniques to refine and complete segmented masks, resulting in a set of full SVBRDF maps. Experimental results demonstrate that this approach can automatically refine both generated and CAD models to achieve photorealism under dynamic lighting conditions.", "questions": "Generally speaking, I have no further questions regarding this submission; my concerns are already listed above. While the novelty might be somewhat limited, the clear potential applications and its widespread applicability lead me to still stand on the positive side of this submission.", "limitations": "Limitations and potential negative social impact are well discussed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The provided results exhibit a decent quality, particularly for objects with metallic or specular materials\n2. The paper is well-written, covering all necessary details for reproduction.\n3. The potential application is clear: it can be used in a plug-and-play manner for 3D generative models to enhance their photorealism.", "weaknesses": "1. The novelty is kind of limited. The major components rely on powerful backbones such as GPT-4V and Semantic-SAM, with the novel contributions primarily found in the segmentation and material retrieval parts.\n2. The (quantitative) evaluation is sort of limited, with only GPT-4V-based and user-based study results reported in Table 1. Given that this is a novel application without standard metrics, I will not critique it too harshly.\n3. The quality of the results depends heavily on the initial albedo map. If the given albedo map is not clean (e.g., it contains baked-in lighting effects), the proposed method cannot correct it, resulting in weird SVBRDF maps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and insightful comments. We will address your questions and concerns below and in the revised paper.\n\n***\n\n### W1. A clarification of novelty of Make-it-Real\n\nThank you for the comments. Our key innovation lies not in the MLLM model technology itself, but in how we effectively leverage and fully explore the potential of MLLM to achieve innovative and practical applications. MLLM alone cannot perform material painting on objects, so we develop a practical and comprehensive pipeline to unlock this capability. \n\nOur core contributions include an advanced segmentation method in the UV space of materials, a rich set of visual and text cues for faster material retrieval, and an innovative algorithm that generates SVBRDF material maps using a region-to-pixel approach. This integration bridges 2D MLLM models and the 3D physical world, empowering large language models with the capability to accurately assign materials to objects. The generated materials are compatible with various rendering engines, enhancing the versatility and utility of our approach in creating realistic 3D assets.\n\n***\n\n### W2. About quantitative evaluation of painting materials\n\nThank you for your understanding regarding this novel application. Previous work primarily focused on 3D object generation without explicitly separating base color from different PBR properties. However, there is now a growing interest in enhancing the physical material properties of 3D objects, yet evaluation methods remain limited.\n\nObtaining ground truth for material properties is challenging mainly due to the ambiguity in material ground truth for certain objects; for example, a white albedo mug could be made of metal or ceramic, leading to different material maps for the same albedo input. Given this ambiguity, we use the evaluation methods described in GPT4Eval[1] and conduct user studies to directly assess improvements in visual effects. This approach allows for a clear and direct comparison of visual quality. We hope our exploration of material properties will inspire further interest and lead to the development of more comprehensive evaluation benchmarks.\n\n\n[1] GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation\n\n\n***\n\n### W3. Addressing Baked-in Shadows and Lighting Effects\n\nThis issue is less pronounced in artist-created models, where albedo maps have less shadows and lighting effects. Our tests show that the albedo obtained from inverse rendering methods[1] closely matches the original artist-created albedo maps from the Objaverse dataset, as illustrated in the first row of Figure C3. For generative models, baked-in shading effects are more pronounced. Our method addresses this by integrating mature derendering algorithms, specifically the Intrinsic Anything[1], into our pipeline with minimal complexity. This integration derenders from four viewpoints and back-projects to obtain a better albedo map, then applies our Make-it-Real method for material painting. As shown in Figure C3's last row, this approach reduces lighting noise and supports a wider range of inputs. We will include more details about the derendering of inputs in the final version.\n\n[1] IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L0KnXGeoYo", "reviewer": "Reviewer_jx7i", "summary": "This paper aims to assign material information to 3D objects that already have a base color. They do not generate material maps through training generative models but instead use a pre-established material library and build an automated recognition and indexing system to index from the material library. To use this framework, the authors pre-built a material library and proposed some prompt templates to prompt GPT-4V. In this process, the segmentation of parts is mainly performed using semantic-SAM. The authors iteratively segment different views and eventually fuse them into a single UV map.", "questions": "1. In line 39, the authors mention \"Additionally, shadows and lighting can affect judgment.\" I cannot understand how shadows and lighting come into play here. If the authors assume that only the base color is used, the view can be directly obtained through rasterization without the influence of shadows and lighting.\n\n2. For the section B.2, what is the key diffuse? Is it an existing diffuse map from your library? If this is the case, how to select the key diffuse based on the current query diffuse? Besides, what is \"3) The third step involves using the obtained indices to obtain the corresponding\n533 values from the rest of the material maps.\"? Please elaborate more about this part.", "limitations": "The authors point out the limitations of their work in the supplementary material and leave it for future work.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "1. The core of the entire system is retrieval rather than generation from a generative model. This approach generally offers better controllability, adjustability, and interpretability.\n\n2. Some techniques to accelerate querying are proposed, such as hierarchical prompting.", "weaknesses": "1. This task is very limited because it requires a 3D mesh with an already existing base color. The requirement for a base color is not very practical. In line 113, the authors mention, \"Given the advancements in generating high-quality 3D shapes with albedo maps, the restoration of realistic material properties remains a challenge.\" I disagree with this point. One of the main difficulties in current 3D generation or texture generation methods is generating a base color (lightning, shadow-free). Most methods, whether through 2D SDS or 3D rendering training, fail to generate the base color. The more pressing issue is actually estimating albedo and material properties from the final RGB image.\n\n2. In line 136, the authors mention, \"These patches are then merged based on similar colors to obtain the final material grouping.\" This is very limiting, as it implies that parts with the same base color are assumed to have the same material.\n\n3. The technical contributions of the paper are minimal, mainly leveraging the capabilities of GPT-4V for VQA. I do not believe designing prompt templates is a sufficient contribution. Additionally, it is hard to reproduce the authors' implementation since GPT-4V is close-sourced. I believe the authors should at least demonstrate the feasibility with another open-source VLM model.\n\n4. Most of the key implementation details are placed in the supplementary file. However, the step of retrieving the corresponding material based on the given albedo map and the material library established by the authors is crucial (i.e., section B.2). At the very least, this part needs to be included in the main text. Additionally, the procedure is hard to understand (please see my questions in the \"Questions\" section).\n\n5. I have some concerns about the evaluation of this task because it is difficult to assess the ability to assign materials. From the results provided by the authors, it is hard to see a significant difference between different parts. It seems that compared to the albedo-only baseline, the main difference is the more pronounced lighting effect caused by the higher metallicity in the authors' results. Besides, I feel the quality is poor from the video demo at 00:56.\n\n6. This work is more related to texture and material generation, rather than performing inverse rendering from existing photo-realistic rendered images. In the related work section, studies on texture generation need to be mentioned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and insightful comments. We will address your questions and concerns below and in the revised paper.\n\n***\n\n### W1. Importance of Addressed Task for Material Painting\n\nWhile the derendering task mentioned by the reviewer is important, we believe the task of adding materials to albedo-only 3D objects is also essential for reducing the extensive time 3D artists spend creating realistic texture maps. Although graphics tools like ShaderMap integrate this functionality, they are limited to producing simple lighting effects. Our method fully explores the potential of MLLMs to accurately capture complex materials and provide a solution for realistic, part-level material generation.\n\nAdditionally, we believe that derendering and material restoration are not mutually exclusive fields but rather complement each other. Research in the latter area can serve as a tool for automating the generation of 3D material data and help researchers gain a deeper understanding of material properties and their realistic effects in the physical world.\n\n***\n\n### W2. Discussion of Material Grouping Method\n\nThank you for pointing out. We acknowledge that, in theory, the assumption that parts with the same base color have the same material can be limiting. However, in our experiments, we found that in most cases in the dataset, semantic patches obtained through SemanticSAM tend to have similar colors when they represent the same material. Based on this observation, we implemented a merging process, carefully controlling the merging threshold based on experimental results. This approach reduces the number of GPT-4V queries without compromising effectiveness. We will include more details about this in the final version.\n\n***\n\n### W3. Unleashing GPT-4V's Ability for Material Painting Beyond VQA\n\nOur core contribution is not in applying multimodal methods for VQA but in effectively leveraging and fully exploring the potential of MLLM to achieve an innovative and practical application: material painting. GPT-4V, even with well-designed VQA tricks, cannot perform material painting on objects because it is confined to the 2D space. By designing a comprehensive pipeline, we aim to unlock MLLM's ability to understand the 3D physical world and generate realistic material representations.\n\nOur pipeline include an advanced segmentation method in the UV space of materials, a rich set of visual and text cues for faster material retrieval, and an innovative algorithm that generates SVBRDF material maps using a region-to-pixel approach. This integration bridges 2D MLLM models and the 3D physical world, empowering large language models with the capability to paint materials to objects at the part level.\n\n***\n\n### W4. Key Implementation Details\n\nYou are correct that Appendix B.2 contains the key implementation details of our method. In Section 3.2.3, we describe the essential aspects of SVBRDF map generation in text form and refer to figures in the appendix to aid understanding. However, we acknowledge that the main text contains limited illustrations for this section. We will enhance the discussion and include more illustrative figures in the final version to improve clarity.\n\n***\n\n### W5. Material Effect of Presented Results\n\nMaterials are inherently complex physical properties, and differences can be seen by zooming in on different regions, such as the globe, bathtub, and shield in Figure 5. I would like to clarify that the material effect is not solely due to higher metallicity producing a brighter lighting effect. Instead, it results from the interaction of multiple PBR maps: increased metallicity results in dampening of the base albedo and increased surface shine, increased roughness reduces specular highlights, and displacement and height control fine surface details. These values are assigned based on the albedo, type, and local region of the object, so not all parts have the same higher metallic settings. (More infomation in Figure 16)\n\nRegarding the video demo, we're sorry for the quality of the rendered scene at the end frame. Due to significant memory requirements for rendering scenes, this picture may not fully reflect the clarity and precision of the objects. We recommend referring to the clearer examples in the main part of the video and paper.\n\n***\n\n### W6. The Clarification on Related Works\n\nWe have cited and discussed several texture generation works in the *Material Capture and Generation* subsection in Sec 2. Related Work. We introduce recent works on texture generation, such as Paint-it and Collaborative, which generate various material texture maps based on shape. We also reference earlier works like Fantasia3D and Matlaber. We discuss the approaches and limitations of these methods, and we will update the final version with more related works in this area.\n\n***\n\n### Q1. The significance of identifying shadows and lighting\n\nSorry for any confusion. We would like to clarify that besides retrieving materials for albedo-only 3D objects, our approach involves describing rendered thumbnails of real materials in the Material library to construct the dataset annotations. These thumbnails may include lighting and shadows. The task in our whole setting requires two capabilities of recognizing materials: 1. Accurately describing real materials under ambient lighting, and 2. Inferring local materials from distorted albedo. We will add more explanations about this point in the final version.\n\n***\n\n### Q2. Method of Key Diffuse and Indexing\n\nThe key diffuse refers to the diffuse map of the material retrieved by the MLLM. This can be understood through the multiple arrows stemming from `index` in Figure 11. After obtaining the nearest neighbor coordinates in step 2), we use the coordinates to index the values of the same position in other material maps paired with the current diffuse. We will draw more illustration(like serial number) on our figure to make it more clear.\n\n***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers and appreciate the constructive comments and the recognition of novelty, and we are grateful for that most reviewers score our work with positive initial ratings (two accept, and two borderline accept). Our work introduces the MLLM into the texture inpainting pipeline **[mcgT]**, which is an exciting finding **[s3Ca]** and a interesting direction in the future **[s3Ca, mcgT]**. The plug-and-play **[dDLL]** design with accelerated hierarchical prompting **[jx7i]** can achieve SVBRDF map generation in a low-cost way **[LVe6]**.

Regarding the weaknesses and questions proposed by the reviewers, we have provided detailed responses to each reviewer separately. Here, we summarize the key questions and general points that we believe will interest all reviewers.

***

### Summary of Key Questions/Comments

**S1.** Reviewer s3Ca, dDLL, jx7i point out that Baked lighting effects on albedo map do harm to quality.

**S2.** Reviewer LVe6, s3Ca about Performance with Fragmented UV Mapping.

**S3.** Reviewer s3Ca about Comparison with artists created materials.

***

We will then address the comments/questions summarized above by referring to the new experimental results attached in the single-page PDF:

**GR1. (for S1) Baked lighting effects [Figure C3]**

This issue is less pronounced in artist-created models, where albedo maps have less shadows and lighting effects. Our tests show that the albedo obtained from inverse rendering methods[1] closely matches the original artist-created albedo maps from the Objaverse dataset, as illustrated in the first row of Figure C3. For generative models, baked-in shading effects are more pronounced. Our method addresses this by integrating mature derendering algorithms, specifically the Intrinsic Anything[1], into our pipeline with minimal complexity. This integration derenders from four viewpoints and back-projects to obtain a better albedo map, then applies our Make-it-Real method for material painting. As shown in Figure C3's last row, this approach reduces lighting noise and supports a wider range of inputs. We will include more details about the derendering of inputs in the final version.


***

**GR2. (for S2) Performance with different UV mapping [Figure C2]**

Overly fragmented UV mapping degrades the performance of our method. As shown in the second column of  Figure C2, UV mappings with excessive fragmentation and color entanglement can cause the 2D segmented images to mix with other regions when reprojected into the UV space, leading to material blending issues. However, our method shows good results with original artist-created UV mappings and Blender's built-in mapping methods in Figure C2, such as smart (default setting in blender), sphere, and unwrap. This indicates that our method still demonstrates good robustness with many mapping techniques.

In practice, we observe that most objects in Objaverse have UV mappings with good properties, meaning they are not excessively fragmented into small pieces. Additionally, we can control the UV mapping process: For some low-quality UV maps and generative objects that originally lack UV maps, we can re-unwrap them to achieve higher quality.

***

**GR3. (for S3) More Comparisons with Artist-Created Materials [Figure C1]**

We visualize some of the results comparing our Make-it-real generated materials with artists. Quantative results are detailed in rebuttal response to Reviewer s3Ca.

***

We sincerely thank the reviewers for their insightful comments and suggestions on our work. We will incorporate all of the rebuttal experiments and discussions into the revised version of our work.

***

[1] Chen, Xi, et al. IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination.

[2] Boss M, Huang Z, Vasishta A, et al. SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglement.

Meta-review: After reading the rebuttal, four out of five reviewers supported acceptance. One reviewer remained negative insisting that the assumptions about the input (meshes with given albedo maps) are simplistic and that most of the contribution comes from the use of the GPT-4V. The AC does not agree that the assumption is a problem -- there are already methods that can populate meshes with initial albedo maps (e.g., Text2Tex) and basically they solve an orthogonal problem. The technical novelty is indeed not that impressive wrt material retrieval (i.e., prompting GPT-4V for material retrieval), yet: (a) the idea of leveraging large multimodal language models to describe material properties is still underexplored and can inspire future research (as mentioned by s3Ca), (b) the segmentation module of Fig. 2 and optimization of Eq. 3 are novel enough components. The AC agrees with the rest of the reviewers that the contribution of the paper is above the bar, and the results deserve recognition. Thus, the AC recommends acceptance and strongly encourages the authors to include material from the rebuttal in the final version of the paper.
Decision: Accept (poster)