Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials

Authors: ye  fang
zeyi  sun
tong  wu
jiaqi  wang
ziwei  liu
gordon  wetzstein
dahua  lin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 18:06:08

Abstract: Physically realistic materials are pivotal in augmenting the realism of 3D assets across various applications and lighting conditions. However, existing 3D assets and generative models often lack authentic material properties. Manual assignment of materials using graphic software is a tedious and time-consuming task. In this paper, we exploit advancements in Multimodal Large Language Models (MLLMs), particularly GPT-4V, to present a novel approach, Make-it-Real: 1) We demonstrate that GPT-4V can effectively recognize and describe materials, allowing the construction of a detailed material library. 2) Utilizing a combination of visual cues and hierarchical text prompts, GPT-4V precisely identifies and aligns materials with the corresponding components of 3D objects. 3) The correctly matched materials are then meticulously applied as reference for the new SVBRDF material generation according to the original albedo map, significantly enhancing their visual authenticity. Make-it-Real offers a streamlined integration into the 3D content creation workflow, showcasing its utility as an essential tool for developers of 3D assets.

Main paper: 
 introduction: 
 High-quality materials are important for the nuanced inclusion of view-dependent and lightingdependent effects in 3D assets for traditional graphics pipelines, critical for achieving realism in gaming, online product showcasing, and virtual/augmented reality. However, many existing assets and generated objects often lack realistic material properties, limiting their application in downstream tasks. Furthermore, creating hand-designed realistic textures necessitates specialized graphic software and involves a laborious and time-consuming process, compounded by significant creative challenges [32] .Traditional computer graphics methods have been either manually creating materials or reconstructing them from physical measurements. Emerging text-to-3D generative models [7, 34, 55, 10, 73, 50, 19, 51] and image-to-3D generative models [69, 58, 25, 63, 22, 62] are successful in creating complex geometries and detailed appearances, but they struggle to generate physically realistic materials, hampering their practical applicability. Recent studies have also explored advanced aspects of appearance generation [11, 75, 70, 8, 40] . However, they often rely on simplified material models. For instance, [40] lacks the ability to produce metallic maps. All these approaches do not generate corresponding displacement and height maps, which restricts the diversity and realism of the generated materials, especially regarding depth and tactile qualities. Furthermore, these methods typically require relatively long training and inference time. Considering the abundance of high-quality 3D assets online [16, 17] that lack material attributes, and the maturity of 3D generative models in geometry and albedo modeling, we aim to recover materials from high-quality geometry and base-colored 3D meshes.However, extracting and recovering material representations for 3D meshes is challenging. Unlike previous material recognition methods [40, 4, 20] , this difficulty is heightened when identifying and separating different material regions within 3D objects in constricted albedo textures. These maps only reflect the base color and can be distorted, as shown by the globe in the upper right corner of Figure 1 . Additionally, shadows and lighting can affect judgment. Thus, the model must have strong material recognition capabilities and prior knowledge of object types and materials.The emergence of Multimodal Large Language Models (MLLMs) [5, 46, 12, 2, 23, 59] provides novel approaches to problem-solving. These models have powerful visual understanding and recognition capabilities, along with a vast repository of prior knowledge, which covers the task of material estimation. Specifically, we are using GPT-4V(ision) for matching of materials. We first create highly detailed descriptions for materials to build a comprehensive library. Next, we use GPT-4V to retrieve materials for each segmented part of the object, utilizing visual prompts [72] and hierarchical text prompts. Finally, we meticulously designed algorithms to generate SVBRDF maps with consistent albedo, achieving realistic visual quality. Notably, our work differs from the aforementioned studies by leveraging prior knowledge from foundation models like GPT-4V to extract and infer materials in albedo-only constrained scenarios. Additionally, we utilize existing material libraries as references to generate corresponding SVBRDF maps on a designed region-to-pixel algorithm. As illustrated in Figure 1 , our approach features: 1) Enhanced 3D mesh realism: Leveraging GPT-4V's visual perception and external knowledge, our method improves the realism, depth, and visual quality of a wide range of mature 3D content generation models. 2) Part-specific material matching: Ensuring material consistency with a segmentation network and refinement process, enabling precise material property retrieval for each segment. 3) Rendering engine compatibility: Generating six comprehensive material maps(roughness, metallic, specular, normal, displacement, height), which are compatible with downstream rendering engines. Developers only need to paint albedo textures; material properties are then automatically generated, saving extensive time on detailed ambient occlusion masks and material map creation.Material Capture and Generation. Recent studies such as Material Palette [40] , MatSim [20] , TwoshotBRDF [4] have made progress in the recognition and extraction of 3D object materials, allowing for the retrieval of SVBRDF information from images of real materials [40] and combining object shape and illumination [4] , but they fail to extract and infer the materials of 3D objects with only albedo. On the other hand, works like Paint-it [75] , Matlaber [70] , Collaborative [60] , Fantasia3D [8] , and TANGO [11] focus on generating text-controllable 3D meshes with physically-based rendering material properties. However, these methods require either extensive training time on BRDF datasets or long inference time for generating materials. Additionally, they are limited to synthesizing only a subset of PBR textures and cannot generate the full range of material properties, such as height and displacement, which are essential for the fine tactile perception of object surfaces.Multimodal Large Language Models. In the wake of the advancements achieved by large language models (LLMs) [5, 46, 12, 2, 23, 59] , domain of research has increasingly turned its attention towards multimodal large language models (MLLMs). Recent advances in this field focus on the integration of vision understanding capabilities with LLMs [76, 1, 37, 36, 26, 21, 3, 18, 54, 49, 6] .The advent of GPT-4V [45] has marked a significant milestone in the evolution of MLLMs, demonstrating groundbreaking 2D comprehending capabilities and open-world knowledge. Although GPT-4V cannot directly process 3D data, a pioneering work , GPTEval3D [68] , first exploited GPT-4V's ability in evaluating the quality of generated 3D objects, and found that GPT-4V's judgement was in line with human evaluation. In this work, we delve into a new application of GPT-4V for material assignment of 3D objects. 
preliminary: 
 Physically Based Rendering (PBR) materials are a compact representation of the bi-directional reflectance distribution function (BRDF), which describes how light is reflected from a surface. PBR material maps primarily encompass seven attributes: Albedo, Roughness, Metallic, Normal, Specular, Height, and Displacement. Based on the rendering equation [29] , given a location x and the surface normal n, the incident light intensity at this point is denoted as L i (ω i ; x) along the direction ω i ; BRDF f r (ω o , ω i ; x) denotes the reflectance coefficient of the material viewing from direction ω o .The observed light intensity L o (ω o ; x) is calculated over the hemisphere Ω = {ω i :EQUATIONGiven the advancements in generating high-quality 3D shapes with albedo maps, the restoration of realistic material properties remains a challenge. We highlight a novel problem: given a 3D mesh ( Õ) and known Albedo ( Ã) map, which reflect the object's intrinsic appearance, the goal is to extract and restore all other SVBRDF attributes of the object, i.e. Make-it-Real( Õ, Ã) = {R, M, N, S, H, D}.Our setup supports the popular Cook-Torrance analytical BRDF model [13] . In this parameterization, the BRDF includes components for albedo b a ∈ R 3 , metallic b m ∈ R, and roughness b r ∈ R. For more complex surface simulations, such as displacement and height modeling, we use the Blender rendering engine to simulate the BRDF function f r (ω o , ω i ; x).In this section, we outline our material matching and generation pipeline, illustrated in Figure 2 , which encompasses three stages: rendering and segmenting 3D meshes, retrieving matching materials using MLLM, and generating spatially varying BRDF maps from coarse to fine.To accurately segment different material regions on 3D meshes with albedo maps, we propose an innovative segmentation strategy based on 2D image rendering in Figure 2 (a). Initially, we use rasterization to render the input albedo mesh from various viewpoints to obtain a series of images:EQUATIONwhere I(x, y) is the pixel value at image coordinates (x, y), rasterize( Õ, v t , •) maps the 3D mesh Õ from viewpoint v t to 2D screen coordinates, UV map (•) converts rasterized coordinates to UV coordinates, T is the albedo map, and R(•, T ) samples color from the T using the UV coordinates.For the rendered images, we employ the Semantic-SAM [33] to perform preliminary semantic segmentation. Empirically, we select the main viewpoint with the largest projected area of the mesh, as it is more likely to contain more details. To address potential over-segmentation, drawing inspiration from [74] , we extract non-overlapping segments from the masks to form distinct patches, Ceramic_Tile005：A gradient of pastel purple to blue tiles with a smooth, matte ceramic finish and slight bevel edges.Metal_Metal042A：A polished brass-like material with subtle variations, hinting at gentle use or handling. as detailed in our approach in Figure 4 (a). These patches are then merged based on similar colors to obtain the final material grouping. We incorporate Set-of-Mark [72] method to annotate each material segment with a unique identifier, sorted by area size from largest to smallest. This annotation acts as a visual cue, enhancing the visual comprehension capabilities of large multimodal language models.Material library with fine-grained annotations. To enable large multimodal language models to accurately retrieve and match materials, we construct a finely annotated material library, as shown in Figure 3 . It is composed of three main components: comprehensive PBR texture maps, highlydetailed records, and a category tree. It comprises 1,400 unique, tileable materials spanning 13 primary categories and 80 subtypes. The data primarily derives from the [61] , which offers comprehensive PBR material textures under a CC0 license with 4K resolution. Each material is represented by seven maps: albedo, roughness, metallic, specular, normal, displacement and height. Accompanying each material are highly-detailed annotations by GPT-4V, offering thorough descriptions of the material's visual characteristics and rich semantic information for the subsequent retrieval process. Created by crawling material sphere images and constructing prompts, these annotations capture subtle differences between materials, facilitating precise retrieval by GPT-4V, as detailed in Appendix B.4.Hierarchical prompting for material retrieval. Due to the vast size of our material library, feeding all prompts to GPT-4V simultaneously proves inefficient and challenging for memory retention. To ensure efficient and accurate material allocation in segmented areas of 3D meshes, we adopt a hierarchical text prompting approach. The schematic of the designed prompt is shown on the left side of Figure 3 , and a complete querying process unfolds in Appendix B.4. This method starts by identifying the primary material types corresponding to each labeled region. Subsequently, hierarchical prompts guide GPT-4V to distinguish specific subclasses within the main material categories. We retrieve all descriptions under these subclasses to ascertain the most fitting descriptions for the segmented blocks. This hierarchical processing enables a more granular search of our material library, identifying the optimal description for each material segment. This approach aids in assigning the most suitable materials to each region and reduces memory and time consumption. We fix missing parts on the uv texture space to get a complete texture partition map.We propose a method to generate SVBRDF maps on a region-to-pixel scale in Figure 2 (c ). Initially, we segment texture map in uv space based on queried material regions on 2D image space. We then estimate BRDF values in pixel space using the object's original albedo map for reference, ensuring consistency with the albedo map. This approach effectively enhances the realism of rendered surfaces.Region-level texture map partitioning. Upon acquiring segmentation masks for material regions within the 2D rendered space of a 3D mesh, our objective transitions to transposing these segmentations from the 2D image space to the corresponding UV space. As described in Section 3.2.1,we extract 2D image features from 3D mesh points via rasterization, and then we apply the material masks to these features, facilitating the accurate transfer of segmentation to UV space. To project the image feature I t back to the texture atlas T t with segmented image mask m t at the viewpoint v t , we apply gradient-based optimization for L t over the values of T t when rendered through the differential renderer R, as presented in Equation (3). In Equation (4), we then compute the difference between T mask t and the initialized T t to transfer the mask image feature m t into the texture space, represented by m uv . The term σ denotes the difference coefficient. Due to the limited perspectives available in rendering, we avoid using a naive median-filling approach to ensure that there are no missing areas on the texture map. Instead, we employ a block-centric clustering based on albedo, as illustrated in Figure 4 (b), to obtain cohesive and refined region masks. The process is shown in Appendix B.1EQUATIONEQUATIONPixel-level albedo-referenced estimation. To achieve precise estimation of spatially varying BRDF (SVBRDF) at the pixel level, we draw inspiration from techniques commonly utilized by artists in creating texture maps. Artists typically use albedo maps as a reference for constructing ambient occlusion masks and further generating SVBRDF maps for material properties, which occupies a significant time portion of appearance modeling. Our method involves using the albedo map of the original object as a reference to refine the retrieved materials. We enhance the querying process using a KD-Tree algorithm, which searches for the nearest neighbor pixel index in the key albedo(retrieved map) for each RGB value of the queried albedo(input map) pixel, detailed in Appendix B.2. This process ensures that areas with similar colors exhibit similar BRDF values, avoiding abrupt changes in material properties; For regions with greater color differences, the distribution of differences aligns consistently with the input albedo, to simulate variations such as embossments or scratches.We retrieves SVBRDF values at pixel level, maintaining texture consistency with the albedo and producing appropriate concave or smooth surface effects. We further analyze the effects in Section 4.2. 4 Experiments 
experiment settings: 
 To verify the effectiveness of Make-it-Real, we conduct refinement experiments mainly on two types of objects. The first type is artificial 3D assets, with the primary model from Objaverse [15] filtered by [55] ; The second type is objects generated by state-of-the-art 3D generation methods. For existing 3D assets, we pick 200 objects with diverse textures from Objaverse by human experts. For 3D generative models (InstantMesh [69] , TripoSR [58] , MVDream [53] , Instant3D [34] and Fantasia3D [8] ), we also generate 200 objects for each methods use prompts designed in GPTEval3D [68] . We use Make-it-Real to refine the objects and compare the texture quality before and after refinement. We perform both GPT-4V [45] based evaluation and user study on the above objects (Detailed guidence and prompts for evaluation are available in Appendix B.5).In addition to evaluation details, we also provide information about the rendering procedure in Appendix B.3 for reproducibility. This includes the rendering tools used in the experiments, hyper parameters related to back-projection, and the basic performance metrics of the model during the experiments, such as time, memory, and number of queries.Texture refinement for existing 3D assets. As shown in Figure 5 , assets processed through Makeit-Real demonstrate the capability to accurately segment objects, assign various suitable materials, and synthesize high-fidelity, photorealistic textures. Some materials exhibit notable highlights, such as a marble bathtub and a globe made of gold. The interaction of these different materials with light varies significantly, leading to diverse reflective effects under the same environmental lighting, thus presenting a range of textures. Additionally, material properties vary across different regions; for instance, the globe's landmass and handle exhibit gold characteristics, while other parts are identified Regions with significant color differences display consistent differential distributions with the key map, such as the embossed textures on the lower left corner of the water bottle in Figure 5 and the subtle particle variations on the surface of the red oil drum. Additionally, visualization of texture maps demonstrates reasonably consistent textures with albedo, shown in Figure 6 . Quantitive results are shown in Table 1 , and more qualitative results can be found in Appendix E.Texture refinement for generated 3D objects. Figure 7 displays the qualitative results. By leveraging our model for enhancement, we observe that Make-it-Real successfully generates appropriate material maps for both image-to-3D and text-to-3D models.Similarly, our Make-it-Real successfully paint these models with materials. As reported in Table 1 , human experts consistently favor the objects post-refinement across all evaluated 3D content creation methods. This preference aligns with the evaluations performed by GPT-4V, indicating a general consensus on the enhancement in quality achieved through Make-it-Real refinement process.Effects of mask refinement module. In Section 3.2.1, we performed additional material postprocessing on the segmentation outcomes, as depicted in Figure 8 . The refined results in the last row indicate that our module achieves precise material segmentation for most standard objects, thereby enabling more accurate queries by GPT-4V. In Section 3.2.3, we addressed the completion of missing regions in the texture map within the UV space at the regional level, as illustrated in Figure 9 . This method not only increases texture coverage but also enhances the visual quality and consistency of the refined 3D model. More ablation studies can be found in Appendix D.Effects of different texture maps. We validate the impact of various texture maps generated by Make-it-Real on the appearance of 3D objects, as illustrated in Figure 16 of Appendix D.1. We provide a detailed qualitative analysis of how different maps enhance the visual texture of materials. For example, increasing metallic results in dampening of the base albedo and increasing in the shine on the surface, while reducing roughness gives the surface a smoother appearance and enhances highlights. Meanwhile, displacement and height maps contribute to the fine-grained bump details on the object's surface.Effects of different UV mappings. Since UV mapping is a crucial step in the 2D-3D alignment technique of our method, we conduct experiments to assess how its quality impacts model performance.The results, shown in Appendix D.2 and in the second column of Figure 17 , indicate that UV mappings with excessive fragmentation and color entanglement can cause issues. However, our method still performs well with artist-created UV mappings and Blender's built-in mapping techniques. This indicates our method still demonstrates good robustness with many mapping techniques. 
conclusion: 
 In this paper, we present a novel framework leveraging MLLMs prior of the world to build a material library and proposing an automatic pipeline to refine and synthesize new PBR maps for initial 3D models, achieving highly photo-realistic PBR textures maps synthesis. Experimental results confirm that our approach can automatically refine both generated and CAD models to achieve photo-realism under dynamic lighting conditions. We believe Make-it-Real is a new and promising solution in the last few procedures of AI based 3D content creation pipeline with the development of MLLMs like GPT-4V [45] as well as the roaring field of deep learning based 3D generation from scratch. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Full experiment details is available in Sec. 4Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The full test with is time consuming mainly due to the rate limit of GPT-4V [45] (each conversation takes around 1 minute). We improve the experiment confidence by increasing the amount of test to hundreds of objects.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: Existing assets we use is a quality library of selectable real materials-we build upon the MatSynth [61] dataset, which contains over 4,000 high resolution tileable materials under CC0 license.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer:[NA]Justification: the paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this appendix, we provide additional details and results that are not included in the main paper due to the space limit. The attached video includes intuitive and interesting qualitative results of Make-it-Real.In this section, we detail the pipeline that briefly outlined in the main paper. We commence by elaborating on the generation of SVBRDF Maps, incorporating illustrative figures to detail the process involving operations in computer graphics. Appendix B.1 is dedicated to explaining the acquisition of region-level texture map partition. Following that, in Appendix B.2 we discuss the method of pixel-level albedo-referenced estimation. Then, in Appendix B.3 we declare some details of rendering procedure. Appendix B.4 details the prompt design for material captioning and matching. Appendix B.5 reports the details of the GPT-4V based evaluation.As illustrated in Figure 10 , our process initiates with the rendering of a 3D object incorporating the original albedo (i.e. query albedo) from multiple perspectives. Following this rendering phase, we employ GPT-4V alongside a segmentor to derive segmented masks for the materials associated with each viewpoint. The subsequent step involves the extraction of regions masked in these images and their back-projection onto the mesh of the object. By examining the object from all acquired viewpoints and applying UV unwrapping techniques, we achieve preliminary segmentation of all materials. Subsequently, each material's segment is refined using a albedo-based mask refinement operation. Ultimately, by combining the segments of all materials, we obtain a region-level texture partition map, which serves to guide our subsequent, more detailed operations.As illustrated in Figure 11 , we developed a pixel-level albedo-referenced estimation module, building upon the foundations laid out in Appendix B.1. This module is inspired by a technique frequently employed by 3D artists, who often utilize albedo maps as a reference to generate images of other The process to precisely obtain the final SVBRDF maps is divided into four steps: 1) Initially, to address potential gaps in color intensity between the two albedo maps, histogram equalization is employed to achieve a more uniform color distribution across the image. 2) Subsequently, for each pixel on the query albedo-termed a reference pixel-we seek the most similar neighboring color index on the key albedo. Given the high dimensionality of both maps (typically 1024x1024 pixels), a brute-force approach to this search would be computationally prohibitive. To this end, we accelerate the pixel query process using a KD-Tree algorithm, which organizes the RGB values of the key map's albedo into a KD-Tree for efficient nearest neighbor searches, reducing the computation time to under ten seconds. 3) The third step involves using the obtained indices to obtain corresponding values from the rest of the material maps. 4) Finally, by aggregating the query results for all material segments, we are able to generate the comprehensive spatially variant SVBRDF maps.In the context of computer graphics, "albedo" typically denotes the primary color of a material, a concept analogous to "base color" and "diffuse" in Physically-Based Rendering (PBR) paradigms, both representing the inherent color of the material under uniformly scattered illumination.In accordance with workflow requirements, the inclusion of height maps, displacement maps, specular maps, and combinations of additional maps is optional. For objects demanding high surface details, such as the oil barrel in Figure 5 and the stone horse in Figure 16 , displacement or height maps are incorporated and rendered using the mature 3D computer graphics software Blender engine. Regarding 2D-3D alignment techniques, including rasterization rendering, back-projection, and UV unwrapping detailed in Section 3.2.1, we adhere to the methodologies outlined in [51] . Specifically, You are showed with a group of spherical PBR materials(all made of tiles, which is a kind ofCeramic, in total 5), can you generate a caption for each(about 20-30 words), try to distinguishtheir difference. Only describe the appearance features (must including color" and detailedmaterial'(such as patterns, roughness, metalness, concave and convex patterns, condition))and don't give too much other information. Do not describe whether it is reflective or not. Do notdescribe the shape of overall object. Please use a dictionary to represent the output result {1:Description 1, 2: Description 2, 3: Description 3, 4: Description 4, 5: Description 5) {1: "A gradient of pastel purple to blue tiles with a smooth, matte ceramic finish and slight bevel edges.", 2: "Soft teal ceramic tiles with a satin finish, arranged in a fish scale pattern with a slightly raised texture.". 3: "Creamy beige ceramic tiles with a matte finish, laid in a staggered brick pattern with flatsurface and sharp edges.", 4: "Pure white ceramic tiles with a glossy finish, aligned in a straight grid pattern with a slightconvex curvature.", 5: "Bright white ceramic tiles with a matte finish, arranged in an offset pattern with pronounced grout lines and a smooth texture.} Figure 12 : Material captioning. We demonstrate how to construct detailed material descriptions, registering material information in textual form, thus providing a convenient bridge when querying with multi-modal large language models. we utilize the Kaolin package (Kaolin [27] ) and back-projection with masks [51] , with the model learning rate set to 0.02 in Equation (3) and the difference coefficient σ set to 0.1 in Equation ( 4). This approach facilitates the systematic generation of various material texture maps in the presence of only an albedo map, thereby ensuring consistency and realism within the Physically Based Rendering (PBR) workflow. Moreover, the flexible application of different map types and 2D-3D alignment techniques significantly enhances the detail and realism of rendered objects, effectively meeting the demands of diverse rendering scenarios.We evaluate the basic performance of the model by testing on a subset of 70 objects. Specifically, we measure the average runtime, the number of material parts per object, the number of GPT-4 queries per object, and GPU memory usage, as shown in Table 2 . Additionally, we provide a detailed time consumption analysis for each step, presented in Table 3 . Prompts design has emerged as a pivotal factor for eliciting desired outcomes from MLLMs. This section delineates the intricacies of our prompt design for both material captioning and matching. "Metal_Metal001": "A silver-colored metal with a high-gloss finish and visible smudges and scratches.", "Metal_Metal002": "A slightly tarnished silver metal with marks and a few rust spots indicating weathering.", "Metal_Metal003": "A clean silver metal surface with minimal weathering and a few small scratches and dents."... material library look up... This is a(or a group of) 3D rendered image(s) of an object using a pure texture map without PBR material information. Please use its visual characteristics (including color, pattern, object type) along with your existing knowledge to INFER and IMAGINE the possible materials of different parts....By choosing a yellow metal for the continents and a matte white plastic for the ocean areas, the globe maintains a clean, minimalist aesthetic while the contrasting materials bring out the details and contours of the Earth's surface. The combination of the shiny and matte finishes would make the globe an attractive decorative item, adding a touch of modern sophistication to its setting. Assuming you are an expert in determining whether the textures of 3D models meet Photorealistic standards (mainly focus on the quality of texture), we have generated some 3D objects using two different methods. These objects have the same shape but different textures. Above are the four-view images of a model generated by the two methods (the upper half is the four views of the object generated by method one, and the lower half is the four views of the object generated by method two). Please help me evaluate which of the two objects is more Photorealistic in terms of texture. Respond with 1 if you think the upper half object is more realistic. Respond with 2 if you think the lower half object is more realistic. Also tell me the reason in a new paragraph.Based on the provided images, my evaluation would favor object 2 (the lower half object) as being more photorealistic in terms of texture.The textures on object 2 have a more complex and nuanced appearance. The material of the uniform exhibits subtle variations in reflectivity and surface detail that suggest a more realistic fabric texture. Additionally, the skin texture appears more lifelike with variations in tone and reflectivity that are characteristic of human skin. The metal police badges on object 2 has a more nuanced reflection, which suggests a more realistic interaction with environmental light. In contrast, the textures on object 1, while still detailed, lack the same level of subtlety and variation. The uniform looks somewhat flat and less fabric-like, and the overall surface details are less convincing as natural variations.Figure 14 : GPT-4V based evaluation prompts. We define a prompt for GPT-4V to generate humanaligned comparison over objects before and after our texture refinement.Material captioning. Due to limit context length for image tokens, it is currently impossible for MLLMs to directly memories thousands of images of material balls. To address this, we use GPT-4V to generate detailed captions for each material ball. As demonstrated in Figure 12 , we input concatenated images of material ball from same subcategory with prompt specifically tailored to highlight texture properties. This strategy guides GPT-4V to generate detailed caption for each material ball, distinguishing the subtle differences between them. These detailed captions are then registered into material library.Material matching. The MLLM-based material query process is exemplified through a simplified case, as illustrated in Figure 13 . Initially, GPT-4V is queried to identify the basic material for matching. Following this preliminary matching, GPT-4V is queried for more specific materials using the names of subcategories as filters, which narrows down the selection to a few candidates within the same subcategory. For final material selection, GPT-4V is prompted with detailed captions from the library, directing it to allocate the most suitable material. This is accomplished in conjunction with a meticulously engineered segmentation process in UV space, ultimately facilitating MLLM-based material matching. It is noteworthy that prompting GPT-4V to navigate through a three-level tree structure has been proven to enhance both the efficiency and accuracy of the matching process, as opposed to directly selecting from thousands of materials directly.Evaluation details. For the purpose of evaluating objects pre-and post-refinement, we render four view images of each object and concatenate them vertically to facilitate a comprehensive assessment. We craft a specific prompt to guide GPT-4V in conducting an impartial comparison of the texture quality between two objects. As demonstrated in Figure 14 . GPT-4V's advanced capabilities enable it to differentiate between the two objects, providing comparison results that closely align with assessments made by human experts. In the scenario evaluated, GPT-4V successfully identifies all enhancements (metal badges, human skin, and fabric textures). Comparison between previous method and Make-it-Real. We demonstrate the distinctions between Material Palette [40] and our method in terms of material identification and extraction.Our overall pipeline presents a more challenging task, where the input is a rendered image with only albedo information, and the output consists of textures for the entire object.For human evaluators, we engage 16 volunteers with strong backgrounds in computer science, specifically in 3D modeling and generation. To ensure unbiased evaluations, we maintain gender balance through random selection. The assessment focuses on determining object material photorealism, as shown in Figure 14 . Volunteers train on 40 Objaverse cases, with and without materials, to enhance their realism judgment for accurate test data evaluation.In the context of extracting real-world materials from single images, the methodologies most closely related to our work are Material Palette [40] and Photoshape [47] . Material Palette enables the extraction of materials at the region level from a single image, generating tilable texture maps for corresponding areas. Photoshape, on the other hand, automates the assignment of real materials to different parts of a 3D shape by training a material classification network. However, this approach needs the training of a material classifier, which is constrained to a limited set of object categories (e.g., chairs) and material types. Besides, both methods require rendered images of objects with real materials as input.Our problem setting poses a more challenging task, as illustrated in Figure 15 , which involves identifying and recovering the original material properties from images of objects that only have a albedo map, in addition to generating different material maps for the entire object. Humans have the capacity to intuitively infer the underlying material properties of 3D objects from single images containing only shape and basic colors. This capability is attributed to our robust material recognition abilities and comprehensive prior knowledge of object categories, colors, and material attributes.Building on this concept, our approach harnesses the potent image recognition capabilities and prior knowledge inherent in large-scale multimodal language models to efficiently execute region-level material extraction and identification, which is further enhanced by subsequent 2D-3D alignment and albedo-referenced techniques to generate and apply material texture maps for physically realistic rendering of 3D objects. Furthermore, [40] extracts the material takes 3 to 4 minutes, and process an image containing three materials takes more than 10 minutes. In contrast, our method takes only about 1 to 2 minutes to match and generate all the materials.In more recent work, both MaPa [77] and our paper aim to enhance 3D objects with materials, but the key difference lies in material retrieval methods. MaPa [77] relies on CLIP, while we use MLLM, a distinction that we believe significantly boosts retrieval effectiveness for these reasons: 1) Enhanced Global Context: CLIP focuses on one area at a time, requiring other parts to be masked, which limits global semantic information. MLLM, however, retains this context by simply highlighting the relevant area. 2) Hierarchical Material Dictionary: We introduce a hierarchical dictionary in MLLM, providing descriptions and images of material spheres, enabling access to a more extensive library of 1,394 materials compared to MaPa's 118. We validate the impact of various texture maps generated by Make-it-Real on the appearance of 3D objects, as illustrated in Figure 16 . For each example, the first column lacks the corresponding map enhanced by Make-it-Real, while the second column includes the same conditions with the specific material map. Initially, we compare the effect of metalness on object appearance. We observe that the objects on the right, with a higher metalness map value, exhibit higher reflectivity, and the reflected light's color is similar to the material's own color, closely resembling real-world objects and appearing more aesthetically pleasing. In contrast, the objects on the left without the metalness map have lower reflectivity, and the reflected light tends to be white.Next, we examine the impact of the roughness map, which controls the smoothness of the material's surface. We observe that the silver chess piece on the right, with a low roughness texture map, becomes smooth, and together with metalness, produces a mirror-like reflection effect. On the other hand, the box on the left with a roughness map exhibits changes in light dispersion on its surface, with some areas showing highlights, while also adding and enriching scratch texture details on the surface.Furthermore, we compare the effects of displacement and height on objects, both of which are usually optional and can also impact object appearance. Height maps typically use grayscale values to represent the surface's relative height, simulating a relief effect through changes in lighting and shadows. As shown in the last row on the right, the object with a height map has an uneven surface, enhancing the sense of depth. Displacement mapping is a more powerful technique that changes the vertex positions of the geometry based on the map's values, creating realistic relief effects. As shown on the left, the stone sculpture with displacement mapping exhibits very realistic details and a sense of relief. We acknowledge that the generated PBR maps are not true representations of physical maps, but our algorithm generates PBR maps that closely approximate the visual characteristics of true PBR properties, which significantly enhances the realism of 3D objects.We conduct experiments to evaluate the impact of UV mapping on our method. As shown in the second column of Figure 17 , UV mappings with excessive fragmentation and color entanglement can cause the 2D segmented images to mix with other regions when reprojected into the UV space, leading to material blending issues. However, our method shows good results with original artist-created UV mappings and Blender's built-in mapping methods in Figure 17 , such as smart, sphere, and unwrap. This indicates that our method still demonstrates good robustness with many mapping techniques.In practice, we observe that most objects in Objaverse have UV mappings with good properties, meaning they are not excessively fragmented into small pieces. Additionally, we can control the UV mapping process: For some low-quality UV maps and generative objects that originally lack UV maps, we can re-unwrap them to achieve higher quality. In this section, we show more qualitative results of Make-it-Real for existing 3D assets from [16] . We show more qualitative results of Make-it-Real for existing 3D assets from [16] . Results are shown in Figures 20 and 21 . The first row presents the original 3D object with only a albedo map, while the second row showcases the object enhanced by our method with various material maps.Comparisions with Artist-Created Materials. We conduct a user study comparing materials generated by our Make-it-Real with those created by artists, visualizing some results in Figure 18 for close examination. For objects like pistol, wooden chair, speakers, and saxophone, Make-it-Real showed strong similarity to artist-made materials, maintaining consistent metallicity, roughness, highlights, and coloration. In cases such as shield, boots, landline phone, and oil drum, our method produced materials that, while different, are realistic and sometimes visually superior. Our pipeline approaches the level of refinement seen in some artist-created materials, which often require significant time and effort, and clearly outperforms more basic and crude artist-generated materials. This highlights its effectiveness and potential for practical applications.In this section, we show visualization of generated texture maps in Figures 22 and 23 . Our approach, guided by the query albedo reference, produces material maps with distinct material partitions and maintains a distribution consistent with the albedo map. As a result, the enhanced object exhibits both realism and texture consistency.While promising results have been achieved by MLLMs for texture assignment through Make-it-Real. Our work still faces several challenges.Lighting Effect and Addressed Solutions. First, although our method can achieve appropriate texture assignment for albedo-only model, it does not support reverse transform from shaded texture map to albedo map for generated 3D objects. This causes problem of assigning different materials to the dark shadow and highlight area when generated object with mesh already shaded in different lighting conditions. This issue is less pronounced in artist-created models, where albedo maps have less shadows and lighting effects. Our tests show that the albedo obtained from inverse rendering methods [9] closely matches the original artist-created albedo maps from the Objaverse dataset, as illustrated in the first row of Figure 19 . For generative models, baked-in shading effects are more pronounced. Our method addresses this by integrating mature derendering algorithms [9, 14, 31, 66] , specifically the Intrinsic Anything [9] , into our pipeline with minimal complexity. This integration derenders from four viewpoints and back-projects to obtain a better albedo map, then applies our Make-it-Real method for material painting. As shown in the last row, this approach reduces lighting noise and supports a wider range of inputs.Impact of Model Quality and Segmentation. Second, we find base model quality is essential for MLLM to assign correct materials. When base 3D object is of low quality(e.g., uneven surfaces or mixed colors across different parts), it is difficult for MLLMs to identify object properties when ground truth text prompt describing the object is not available. Additionally, there has been limited progress in material segmentation in 2D and 3D demoain [52, 38] . The material segmentation algorithm included in our proposed pipeline can serve as a useful baseline and provide inspiration for future work. We also explore the integration of the pipeline with 3D segmentation networks, which shows promising results. For future work, we consider methods mitigating these challenges as well as adding user friendly control into our fully-automatic pipeline.The datasets utilized in our work, specifically Objaverse, is composed entirely of inanimate objects. Importantly, the Objaverse dataset we used have undergone rigorous ethical filtering as part of the Cap3D [41] . Cap3D's ethical filtering process removed all potentially problematic data, including any identifiable human elements and NSFW content (detailed in Sec 3.2 of Cap3D). This process included the removal of objects with licenses that do not permit commercial usage, the exclusion of objects lacking sufficient camera information for rendering, and the application of face detection and NSFW classifiers with high thresholds to ensure thorough filtering. The final dataset used in our work, therefore, does not contain any human-derived data or data related to human subjects.We recognize the potential risk of misuse associated with this technology, particularly in the creation of realistic fake human representations. However, since our dataset fully excludes human data, this risk is substantially mitigated.Potential positive societal impacts. The proposed method facilitates more realistic and accurate representations of materials in 3D models, benefiting industries such as gaming, virtual reality, and film, leading to more immersive and engaging experiences. By automating the material assignment process, Make-it-Real significantly reduces the time and effort required for 3D content creators, allowing for more efficient workflows and enabling creators to focus on more creative aspects of their work. This approach can democratize high-quality 3D content creation by making advanced material application techniques accessible to a broader range of users, including those without specialized skills in graphic software.Potential negative societal impacts. The improved realism in 3D assets could be exploited for creating highly convincing fake visuals or deepfakes, which might be used in disinformation campaigns or to mislead audiences. There is a risk that the materials generated could inadvertently perpetuate biases or stereotypes if the training data for GPT-4V includes biased representations of certain materials or objects. As the method involves processing and recognizing visual data, there could be concerns regarding the privacy of any real-world images used as inputs, particularly if they contain sensitive or personal information. Objects are selected from Objaverse [16] with albedo only. Objects are selected from Objaverse [16] with albedo only. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: After reading the rebuttal, four out of five reviewers supported acceptance. One reviewer remained negative insisting that the assumptions about the input (meshes with given albedo maps) are simplistic and that most of the contribution comes from the use of the GPT-4V. The AC does not agree that the assumption is a problem -- there are already methods that can populate meshes with initial albedo maps (e.g., Text2Tex) and basically they solve an orthogonal problem. The technical novelty is indeed not that impressive wrt material retrieval (i.e., prompting GPT-4V for material retrieval), yet: (a) the idea of leveraging large multimodal language models to describe material properties is still underexplored and can inspire future research (as mentioned by s3Ca), (b) the segmentation module of Fig. 2 and optimization of Eq. 3 are novel enough components. The AC agrees with the rest of the reviewers that the contribution of the paper is above the bar, and the results deserve recognition. Thus, the AC recommends acceptance and strongly encourages the authors to include material from the rebuttal in the final version of the paper.

Reviews: review: paper_summary: paper_summary: This paper tackles the problem of recovering materials for 3D meshes with known geometry and base color. The proposed solution consists of three steps:

- Utilizing Semantic-SAM to segment the 3D objects, identifying and isolating various material regions.
- Using hierarchical prompting to retrieve and match materials from a material library.
- Generating the actual SVBRDF map based on the albedo and the retrieved materials.
strength_weakness: strength_weakness: - In the user study, participants are asked to compare the base mesh and the refined mesh. However, users may naturally prefer objects with enhanced lighting effects, which does not effectively evaluate the quality of the generated materials. A more informative approach might be to compare an artist-created mesh with material to the same shape with a re-predicted material, which could provide more insights into material quality.

- For the application presented in this paper, a common issue is that 3D models with a single albedo map often have baked-in shadows and other lighting effects. This is common in both artist-created models and especially in outputs from 3D generative models (e.g. is the shadow in Fig 7 bottom baked in albedo?). It's unclear how the proposed method addresses or might even be misled by these artifacts since it assumes that similar colors represent similar BRDF values in estimation. This could be a limitation when compared to alternative methods that directly optimize or predict the texture map. A discussion on this could be helpful to clarify.

- The discussion on performance metrics is missing, such as runtime and the average number of parts and queries per object.
questions: questions: - How are viewpoints selected?

- Does the method depend on good UV mapping?

- How are the categories within the material library constructed?
limitations: limitations: The authors discuss the limitations of their method and broader impact in the paper.
paper_summary: paper_summary: This paper introduces the large-scale multimodal language models to realistic material rendering of 3D objects. Specifically, this paper employs MLLM to retrieve materials from a material library for different parts of . By combining 2D-3D alignment and diffuse reflection reference techniques, it generates and applies material texture maps to objects, achieving realistic rendering of 3D objects.
strength_weakness: strength_weakness: 1. This paper shares some similarities with Mapa[1]. Could the authors further demonstrate the differences between Make-it-real and Mapa and the superiority of this method?
2. The necessity of MLLM in the pipeline requires further validation. How does the performance of the MLLM-based retrieval method compare to other retrieval methods? More experiments on the retrieval results are needed to show the superiority of the MLLM
-based methods.

[1]Zhang, Shangzhan, et al. "MaPa: Text-driven Photorealistic Material Painting for 3D Shapes." arXiv preprint arXiv:2404.17569 (2024).
questions: questions: See the weakness.
limitations: limitations: The authors have not addressed the limitation in the paper.
paper_summary: paper_summary: The proposed work leverages GPT-4V to extract and infer materials in albedo-only scenarios, utilizing existing material libraries to generate SVBRDF maps with a region-to-pixel algorithm. This approach enhances 3D mesh realism, ensures precise part-specific material matching, and is compatible with rendering engines, generating six comprehensive material maps. Developers need only paint albedo textures, with other material properties automatically generated, saving significant time. The contributions include using multimodal large language models for material recognition, creating a detailed material library, and developing a pipeline for high-quality material application to 3D assets.
strength_weakness: strength_weakness: - The method's effectiveness is largely dependent on the capabilities of MLLMs such as GPT-4V, which may not always produce accurate results. While the paper claims that the generated materials, represented as PBR maps, are compatible with downstream engines, it should be noted that this generation method does not rely on any physical principles of light transport and therefore cannot be genuinely considered as PBR maps.
- The method's performance might be constrained by the quality and detail of the initial albedo maps.
questions: questions: - How are the PBR texture maps in the material library generated and how to ensure the accuracy of the material property?
- How does the proposed method perform with low-quality input albedo maps, such as those with many small UV pieces?
limitations: limitations: - The proposed method relies on GPT-4V for material generation from a generated database. However, many offline tools like ShaderMap can accomplish the same task more cost-effectively. These offline tools require no training and can produce comprehensive, high-quality material decompositions. Given that the generated materials are not physically-based, what advantages does the proposed method offer over efficient offline software that requires no training or fine-tuning?
paper_summary: paper_summary: This paper presents a novel framework leveraging MLLM priors (GPT-4V) to build a material library and proposes an automatic pipeline to refine and synthesize new PBR maps for initial 3D models with diffuse albedo only. The pipeline integrates existing tools, such as GPT-4V and Semantic-SAM, while introducing novel techniques to refine and complete segmented masks, resulting in a set of full SVBRDF maps. Experimental results demonstrate that this approach can automatically refine both generated and CAD models to achieve photorealism under dynamic lighting conditions.
strength_weakness: strength_weakness: 1. The novelty is kind of limited. The major components rely on powerful backbones such as GPT-4V and Semantic-SAM, with the novel contributions primarily found in the segmentation and material retrieval parts.
2. The (quantitative) evaluation is sort of limited, with only GPT-4V-based and user-based study results reported in Table 1. Given that this is a novel application without standard metrics, I will not critique it too harshly.
3. The quality of the results depends heavily on the initial albedo map. If the given albedo map is not clean (e.g., it contains baked-in lighting effects), the proposed method cannot correct it, resulting in weird SVBRDF maps.
questions: questions: Generally speaking, I have no further questions regarding this submission; my concerns are already listed above. While the novelty might be somewhat limited, the clear potential applications and its widespread applicability lead me to still stand on the positive side of this submission.
limitations: limitations: Limitations and potential negative social impact are well discussed.
paper_summary: paper_summary: This paper aims to assign material information to 3D objects that already have a base color. They do not generate material maps through training generative models but instead use a pre-established material library and build an automated recognition and indexing system to index from the material library. To use this framework, the authors pre-built a material library and proposed some prompt templates to prompt GPT-4V. In this process, the segmentation of parts is mainly performed using semantic-SAM. The authors iteratively segment different views and eventually fuse them into a single UV map.
strength_weakness: strength_weakness: 1. This task is very limited because it requires a 3D mesh with an already existing base color. The requirement for a base color is not very practical. In line 113, the authors mention, "Given the advancements in generating high-quality 3D shapes with albedo maps, the restoration of realistic material properties remains a challenge." I disagree with this point. One of the main difficulties in current 3D generation or texture generation methods is generating a base color (lightning, shadow-free). Most methods, whether through 2D SDS or 3D rendering training, fail to generate the base color. The more pressing issue is actually estimating albedo and material properties from the final RGB image.

2. In line 136, the authors mention, "These patches are then merged based on similar colors to obtain the final material grouping." This is very limiting, as it implies that parts with the same base color are assumed to have the same material.

3. The technical contributions of the paper are minimal, mainly leveraging the capabilities of GPT-4V for VQA. I do not believe designing prompt templates is a sufficient contribution. Additionally, it is hard to reproduce the authors' implementation since GPT-4V is close-sourced. I believe the authors should at least demonstrate the feasibility with another open-source VLM model.

4. Most of the key implementation details are placed in the supplementary file. However, the step of retrieving the corresponding material based on the given albedo map and the material library established by the authors is crucial (i.e., section B.2). At the very least, this part needs to be included in the main text. Additionally, the procedure is hard to understand (please see my questions in the "Questions" section).

5. I have some concerns about the evaluation of this task because it is difficult to assess the ability to assign materials. From the results provided by the authors, it is hard to see a significant difference between different parts. It seems that compared to the albedo-only baseline, the main difference is the more pronounced lighting effect caused by the higher metallicity in the authors' results. Besides, I feel the quality is poor from the video demo at 00:56.

6. This work is more related to texture and material generation, rather than performing inverse rendering from existing photo-realistic rendered images. In the related work section, studies on texture generation need to be mentioned.
questions: questions: 1. In line 39, the authors mention "Additionally, shadows and lighting can affect judgment." I cannot understand how shadows and lighting come into play here. If the authors assume that only the base color is used, the view can be directly obtained through rasterization without the influence of shadows and lighting.

2. For the section B.2, what is the key diffuse? Is it an existing diffuse map from your library? If this is the case, how to select the key diffuse based on the current query diffuse? Besides, what is "3) The third step involves using the obtained indices to obtain the corresponding
533 values from the rest of the material maps."? Please elaborate more about this part.
limitations: limitations: The authors point out the limitations of their work in the supplementary material and leave it for future work.

Comments: title: Thanks for the comment
Thanks for the comment
Thanks for the response
comment: Thank you for your thoughtful feedback. We acknowledge that the current version of our paper lacks detailed information regarding the human experts who participate in the evaluation of our results.  To address this, we provide additional details below and in the revised paper:

1. **Number and Selection of Experts**:  We engage 16 volunteers, each with a solid background in computer science, specifically related to 3D modeling and generation. These experts have at least one year of research or work experience in the 3D field and hold advanced educational backgrounds. We ensure gender balance through random selection to avoid any bias in the evaluations.

   To protect their privacy, no personally identifiable information is disclosed. Each expert evaluates 400 data points(detailed in Sec 4.1), and we obtain their explicit consent to use the study data and publish the evaluation results.

2. **Procedures and Instructions**: We base our evaluation criteria primarily on whether the material appearance of the objects is photorealistic, with a focus on accurately replicating the attributes of real physical materials, similar to the instructions and examples shown in Figure 14. Each expert undergoes thorough training, starting with an analysis of 40 ground truth cases from Objaverse, with and without materials for comparison. During this training phase, we provide the correct answers, indicating which objects are closer to reality. This training helps experts determine which object's appearance is closer to real materials, enabling more accurate assessment of realism, which further prepares them to evaluate the subsequent test data.

3. **Ethics of Test Data**: As detailed in Sec 4.1, our data sources adhere strictly to NeurIPS guidelines and ethical standards for data usage. Notably, our study involves no human subjects; all test data consists solely of inanimate 3D mesh representations.

We hope these additions clarify the procedure of the our evaluations and suggest that the results align with the standards of rigorous scientific research. We also ensure that these updates fully comply with the NeurIPS ethical guidelines regarding research involving human evaluators. We will include this information in the revised version of the paper.

Thank you again for your valuable suggestions. We remain committed to improving the transparency and reproducibility of our work.
Thank you for your thoughtful feedback. We acknowledge that the current version of our paper lacks detailed information regarding the human data and transparency, we address you concerns below and in the revised paper.

**Human Derived Data and Human Subjects:** The datasets utilized in our work, specifically Objaverse, is composed entirely of inanimate objects. Importantly, the Objaverse dataset we used have undergone rigorous ethical filtering as part of the Cap3D [1]. Cap3D’s ethical filtering process removed all potentially problematic data, including any identifiable human elements and NSFW content (detailed in Sec 3.2 of Cap3D). This process included the removal of objects with licenses that do not permit commercial usage, the exclusion of objects lacking sufficient camera information for rendering, and the application of face detection and NSFW classifiers with high thresholds to ensure thorough filtering. The final dataset used in our work, therefore, does not contain any human-derived data or data related to human subjects.

**Deception and Fraud:** We acknowledge the reviewers' concern regarding the potential misuse of the technology to create realistic fake human representations. However, given that our dataset does not include any human data, this risk is substantially mitigated. 

**Lack of Discussion on Ethical Concerns:** While our current work strictly involves inanimate objects that have been ethically filtered to exclude any human-derived data, we recognize the importance of addressing the broader implications of this technology. We have discussed some ethical issues in Appendix F. Moreover, we will further enhance the discussion in this section by addressing the risks of deception and fraud associated with the creation of realistic human representations. This section will explore potential misuse scenarios and outline strategies to mitigate these risks, underscoring our commitment to responsible AI development.

**Transparency and Reproducibility:** We will expand this discussion in the revised version of our data handling processes, including detailed information on the ethical filtering procedures and any potential biases. This will also include demographic details of the human experts involved, where relevant, to ensure a thorough and responsible approach to AI development.

[1] Luo T, Rockwell C, Lee H, et al. Scalable 3d captioning with pretrained models[J]. Advances in Neural Information Processing Systems, 2024, 36.
Thank you for your continued engagement with our paper. We appreciate your comments and would like to respond to the concerns you have raised. 

We understand your concern regarding the assumption of an existing diffuse map. Diffuse-only objects are widely found in datasets like Objaverse-XL[1], Objaverse[2], and Shapenet[3] because creating them is relatively easy. However, generating additional material maps from a diffuse map remains a time-consuming step. Our approach significantly accelerates the materially realistic 3D content creation process. Moreover, our method can be easily complemented with existing derendering techniques when shadows exist in the diffuse maps (e.g., generated objects), as shown in Figure C3 of our rebuttal PDF. These techniques are orthogonal and can work well together to meet different requirements and scenarios, as mentioned in our response to W1.

Concerning the contribution besides utilizing GPT-4v, we acknowledge the significant potential and prior knowledge that GPT-4v brings. However, it alone is insufficient to accomplish our goals. We develop an entire pipeline, comprising texture segmentation, material matching, and SVBRDF map generation, that introduce domain knowledge and abilities far beyond GPT-4v.

Regarding visual quality, we recognize that there is room for improvement. In the meantime, few existing methods can automatically generate comprehensive material maps, particularly for displacement and height maps. We wish our Make-it-Real can offer new insights to accelerate artistic creation.

Thank you once again for your time and valuable feedback. Your insights provide us with important considerations for improving our work, and we look forward to incorporating these enhancements as we continue our research.

---

[1] Deitke M, Liu R, Wallingford M, et al. Objaverse-xl: A universe of 10m+ 3d objects.

[2] Deitke M, Schwenk D, Salvador J, et al. Objaverse: A universe of annotated 3d objects.

[3] Chang A, Funkhouser T, Guibas L, et al. Shapenet: An information-rich 3d model repository.
Thank authors for the rebuttal. Although some technical details were clarified, my main concern remains unresolved and still exists. That is, the assumptions about the material are overly simplistic, and most of the contributions come from the use of the GPT-4v, with not much application of knowledge related to materials. Besides, the visual quality is not good enough, especially in the case that an existing diffuse map is assumed to be provided.

Currently, there has been an abundance of work that leverages multimodal models to assist in addressing specific domain issues. However, this paper’s design in the area of domain knowledge is weak (i.e., the assumptions are too simplistic). This leads me to believe that the technical contributions of this paper are insufficient for acceptance.

I respect the opinions of the other reviewers, but I still maintain my opinion.