Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials

Authors: yawar  siddiqui
tom  monnier
filippos  kokkinos
mahendra  kariya
yanir  kleiman
emilien  garreau
oran  gafni
natalia  neverova
andrea  vedaldi
roman  shapovalov
david  novotny
meta  genai
t u munich
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 12:33:57

Abstract: We present Meta 3D AssetGen (AssetGen), a significant advancement in text-to-3D generation which produces faithful, high-quality meshes with texture and material control. Compared to works that bake shading in the 3D object’s appearance, AssetGen outputs physically-based rendering (PBR) materials, supporting realistic relighting. AssetGen generates first several views of the object with separate shaded and albedo appearance channels, and then reconstructs colours, metalness and roughness in 3D, using a deferred shading loss for efficient supervision. It also uses a sign-distance function to represent 3D shape more reliably and introduces a
corresponding loss for direct shape supervision. This is implemented using fused kernels for high memory efficiency. After mesh extraction, a texture refinement transformer operating in UV space significantly improves sharpness and details. AssetGen achieves 17% improvement in Chamfer Distance and 40% in LPIPS over the best concurrent work for few-view reconstruction, and a human preference of 72% over the best industry competitors of comparable speed, including those that support PBR. Project page with generated assets: https://assetgen.github.io

Main paper: 
 introduction: 
 Generating 3D objects from text or image prompts has enormous potential for 3D graphics, with applications in animation, gaming and virtual reality. However, while image and video generators have improved dramatically [68, 44, 55, 42, 97, 103] , 3D generators are not ready yet for professional use. In fact, 3D generators are often slow and produce artifacts in the generated 3D meshes and textures. Many 3D generators, furthermore, "bake" appearance as albedo, ignoring how materials respond to variable environmental illumination. This results in visually unattractive outputs, especially for reflective materials, which look out of place when put in novel environments.In this paper, we introduce Meta 3D AssetGen, a significant step-up in text-conditioned 3D generation. AssetGen generates assets in under 30 seconds while outperforming prior methods of comparable speed in faithfulness, in quality of the generated 3D meshes and, especially, in quality and control of materials, by supporting Physically-Based Rendering (PBR) [87] . The model generates albedo, metalness, and roughness so that rendered scenes can accurately reflect environmental illumination.In addition, we focus on meshes as the output representation due to their prevalence in applications and compatibility with PBR.AssetGen uses the two-stage design epitomized by [42] . The first stage stochastically generates four images of the object from four canonical viewpoints, and the second stage deterministically reconstructs the 3D shape, appearance and materials of the object from these views (Fig. 1 ). The two-stage approach is faster and more robust than SDS-based techniques that perform test-time optimization [68] and, so far, produces better results than single-stage 3D generators [37, 61, 94, 79] .The first question we ask is how this design should be extended to support PBR. We show that it is difficult for the image-to-3D stage to predict PBR channels from an image as this problem is ambiguous and the model is deterministic. However, we also show that it is difficult offload PBR prediction to the text-to-image model; while this is stochastic, which handles ambiguity, the PBR channels are statistically different from the natural images used for pre-training, which makes fine-tuning difficult. Our solution is to give the text-to-image model the simpler task of outputting shaded appearance and albedo only, and task the image-to-3D stage with inferring the PBR channels from these. This reduces the statistical gap for the text-to-image model and still removes most of the ambiguity for the image-to-3D model.We also note that the quality of 3D shapes and meshes is crucial for PBR modelling. Hence, the second question we study is how to improve 3D quality. We do so by learning a reconstruction network, MetaILRM, which outputs directly a signed-distance field (SDF). SDFs are better than opacity fields for meshing, as the zero level set of an SDF traces the object's surface more reliably. Furthermore, the SDF can be directly supervised using ground-truth depth maps, which is not immediately possible for opacity. The crucial contribution here is to add SDF support, including the VolSDF [108] formulation for differentiable rendering, to the memory-efficient Lightplane kernels [6] . In this way, we can use the stronger SDF representation together with larger batches and photometric loss supervision on high-resolution renders, improving both shapes and textures.Finally, we note that much of the quality of the final asset depends on texture quality. MetaILRM's textures can still be slightly blurrier than the input image due to the limited resolution of the volumetric representation. The third question we investigate is how to maximize the texture quality. To this end, we introduce a new texture refiner network which upgrades the extracted albedo and materials by fusing information extracted from the original views, resolving possible conflicts between them.We demonstrate the effectiveness of AssetGen on the image-to-3D and text-to-3D tasks. For image-to-3D, we attain state-of-the-art performances among existing few-view mesh-reconstruction methods when measuring the accuracy of the recover shaded and PBR texture maps. For text-to-3D, we conduct extensive user studies to compare the best methods from academia and industry that have comparable inference time, and outperform them in terms of visual quality and text alignment. such generators on text-to-image models that are trained on billions of captioned images. Among these, works like [75, 56] finetune 2D diffusion models to output 3D representations, but the quality is limited due to the large 2D-3D domain gap. Other approaches can be dived into two groups.The first group contains methods that build on DreamFusion, a seminal work by [68] , and distill 3D objects by optimizing NeRF via the SDS loss, matching its renders to the belief of a pre-trained textto-image model. Extensions have considered: (i) other 3D representations like hash grids [44, 69] , meshes [44] and 3D Gaussians (3DGS) [81, 111, 12] ; (ii) improved SDS [91, 95, 119, 30] ; (iii) monocular conditioning [69, 82, 113, 78] ; (iv) predicting additional normals or depth for better geometry [70, 78] . Yet, distillation methods are prone to issues such as the Janus effect (duplicating object parts) and content drift [74] . A common solution is to incorporate view-consistency priors into the diffusion model, by either conditioning on cameras [47, 73, 32, 11, 69] or by generating multiple object views jointly [74, 98, 93, 40, 118] . Additionally, SDS optimization is slow and requires minutes to hours per assets; this issue is partly addressed in [52, 101] with amortized SDS.The second group of methods includes faster two-stage approaches [46, 51, 49, 107, 106, 8, 83, 28, 23] that start by generating multiple views of the object using a text-to-image or text-to-video model [54, 13] tuned to output multiple views of the object followed by per-scene optimization using NeRF [58] or 3DGS [38] . However, per-scene optimization requires several highly-consistent views which are difficult to generate reliably. Instant3D [42] improves speed and robustness by generating a grid of just four views followed by a feed-forward network (LRM [29] ) that reconstructs the object from these. One-2-3-45++ [45] replaces the LRM with a 3D diffusion model. Our AssetGen builds on the Instant3D paradigm and upgrades the LRM to output PBR materials and an to use a SDF-based representation of 3D shape. Furthermore, it starts from grids of four views with shaded and albedo channels, key to predicting accurate 3D shape and materials from images.3D reconstruction from images. 3D scene reconstruction, in its traditional multi-view stereo (MVS) sense, assumes access to a dense set of scene views. Recent reconstruction methods such as NeRF [58] optimize a 3D representation by minimizing multi-view rendering losses. There are two popular classes of 3D representation: (i) explicit representations like meshes [22, 114, 24, 63, 59, 76] or 3D points/Gaussians [38, 25] , and (ii) implicit representations like occupancy fields [65] , radiance fields [58, 62] and signed distance functions (SDF) [109] . Compared to occupancy fields, SDF [66, 108, 92, 17, 21] simplifies surface constraints integration, improving scene geometry. For this reason, we also use SDFs and demonstrate that they outperform occupancy fields.Sparse-view reconstruction instead assumes few input views (usually 1 to 8). An approach to mitigate the lack of dense multiple views is to leverage 2D diffusion priors in optimization [55, 99] , but this is often slow and fragile. More recently, authors have focused on training feed-forward reconstructors on large datasets [14, 36, 57, 48, 100, 60, 94] . In particular, [29] trains a large Transformer [89] to predict NeRF using a triplane representation [7, 9] . Followups study 3D representations like meshes [103, 97] and 3DGS [120, 105, 80, 115] , improved backbones [96, 97] and training protocols [86, 35] . We introduce three extensions to LRM: (i) an SDF formulation for improved geometry, (ii) PBR material prediction for relighting, and (iii) a texture refiner for better texture details.3D modeling with PBR materials. Most 3D generators output 3D objects with baked illumination, either view-dependent [58, 38] or view-independent [29] . Since baked lighting ignores the model's response to environmental illumination, it is unsuitable for graphics pipelines that simulate lighting. Physically-based rendering (PBR) defines material properties so that a suitable shader can account for illumination realistically. Several MVS works have considered estimating PBR materials using NeRF [4, 3, 102] , SDF [116] , differentiable meshes [63, 27] or 3DGS [33, 43] . In generative modelling, [10, 70, 50, 104] augment the text-to-3D SDS optimization [68] with a PBR model. Differently from them, we integrate PBR modeling in our feed-forward text-to-3D network, unlocking for the first time fast text-based generation of 3D assets with controllable PBR materials.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: See Sec. 1 and the abstract.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See App. A.2Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 AssetGen is a two-stage pipeline (Fig. 2 ). The first stage, text-to-image (Sec. 3.1) , maps text to an image grid containing four object views with material information. The second stage, image-to-3D, comprises a novel PBR-based sparse-view reconstruction model (Sec. 3.2) and a new texture refiner (Sec. 3.3). As such, AssetGen is applicable to two tasks: text-to-3D (stage 1+2) and image-to-3D (stage 2 only). The goal of the text-to-image module is to generate several views of the generated 3D object. To this end, we employ an internal text-to-image diffusion model pre-trained on billions of text-annotated images, with an architecture similar to Emu [16] . Similar to [74, 42] , we finetune the model to predict a grid of four images I i , i = 1, . . . , 4, each depicting the object from canonical viewpoints π i . Note that I i are RGB images of the shaded object. We tried deferring the PBR parameter extraction to the image-to-3D stage, but this led to suboptimal results. This is due to the determinism of the image-to-3D stage, which fails to model ambiguities when assigning materials to surfaces.A natural solution, then, is to predict the PBR parameters directly in the text-to-image stage. These consists of the albedo ρ 0 (by which we mean the base color, which is the same as albedo only for zero metalness), the metalness γ, and the roughness α. However, we found this to be ineffective too because the metalness and roughness maps deviate from the distribution of natural images making them a hard target for finetuning. Our novel solution is to train the model to generate instead a 4-view grid with 6 channels, 3 for the shaded appearance I and 3 more for the albedo ρ 0 . This reduces the finetuning gap, and removes enough ambiguity for accurate PBR prediction in the image-to-3D stage.We now describe the image-to-3D stage, which solves the reconstruction tasks given either a small number of views I i (few-view reconstruction), or the 4-view 6-channel grid of Sec. 3.1.At the core of our method is a new PBR-aware reconstruction model, MetaILRM, that reconstructs the object given N ≥ 1 posed images (I i , π i ) N i=1 , where I i ∈ R H×W ×D and π i ∈ Π is the camera viewpoint. As noted in Sec. 3.1, we consider N = 4 canonical viewpoints π 1 , . . . , π 4 (fixed to 20 • elevation and 0 • , 90 • , 180 • , 270 • azimuths) and D = 6 input channels. The output is a 3D field representing the shape and PBR materials of the object as an SDF s : R 3 → R, where s(x) is the signed distance from the 3D point x to the nearest object surface point, and a PBR function k : R 3 → R 5 , where k(x) = (ρ 0 , γ, α) are the albedo, metalness and roughness.The key to learning the model is the differentiable rendering operator R. This takes as input a field ℓ : R 3 → R D , the SDF s, the viewpoint π, and a pixel u ∈ U = [0, W ) × [0, H), and outputs the projection of the field on the pixel according to the rendering equation [58] , which has the same number of channels D as the rendered field ℓ:EQUATIONHerex t = x 0 -tω o , t ∈ [0, ∞)is the ray that goes from the camera center x 0 through the pixel u along direction -ω o ∈ S 2 . The function σ(x | s) is the opacity of the 3D point x and is obtained from the SDF value s(x) using the VolSDF [108] formulaEQUATIONwhere a, b are the hyper-parameters. We use Eq. ( 1) to render several different types of fields ℓ, the most important of which is the radiance field, introduced next along with the material model.Reflectance model. The appearance of the object Î(u) = R(u | L, s, π) in a shaded RGB image Î is obtained by rendering its radiance field ℓ(x) = L(x, ω o | k, n), where n is the field of unit normals. The radiance is the light reflected by the object in the direction ω o of the observer (see App. A.8 for details), which in PBR is given by:EQUATIONwhereω o , ω i ∈ H(n) = {ω ∈ S 2 : n • ω ≥ 0}are two unit vectors pointing outside the object and L(x, -ω i ) is the radiance incoming from the environment at x from direction ω i in the solid angle dΩ i . The Bidirectional Reflectance Distribution Function (BRDF) f tells how light received from direction -ω i (incoming) is scattered into different directions ω o (outgoing) by the object [20] .In PBR, we consider a physically-inspired model for the BRDF, striking a balance between realism and complexity [2, 87, 77, 15, 90] ; specifically, we use the Disney GGX model [90, 5] , which depends on parameters ρ 0 , γ, and α only (see App. A.12.1 for the parametric form of f ). Hence, the MetaI LRM predicts the triplet k(x) = (ρ 0 , γ, α) at each 3D point x.Deferred shading. In practice, instead of computing Î(u) = R(u | L, s, π) using Eqs. ( 1) and (3), we use the process of deferred shading [20] :EQUATIONwhere L env is the environment radiance (assumed to be the same for all x), k = R(u | k, s, π) and n = R(u | n, s, π) are rendered versions of the material and normal fields. The advantage of Eq. ( 4) is that the BRDF f is evaluated only once per pixel, which is much faster and less memory intensive than doing so for each 3D point during the evaluation of Eq. ( 1), particularly for training/backpropagation. During training, furthermore, the environment light is assumed to be a single light source at infinity, so the integral (4) reduces to evaluating a single term.Training formulation and losses. MetaILRM is thus a neural network that takes as input a set of images (I i , π i ) N i=1 and outputs estimates ŝ and k for the SDF and PBR fields. We train it from a dataset of mesh surfaces M ⊂ R 3 with ground truth PBR materials k : M → R 5 .Reconstruction models are typically trained via supervision on renders [29, 97] . However, physically accurate rendering via Eq. ( 1) is very expensive. We overcome this hurdle in two ways. First, we render the raw ground-truth PBR fields k and use them to supervise their predicted counterparts with the MSE loss, skipping Eq. (1). For the rendered albedo ρ 0 -which is similar enough to natural images -we also use the LPIPS [117] loss:L pbr = LPIPS R(• | ρ0 , ŝ, π), R(• | ρ 0 , M, π) + R(• | k, ŝ, π) -R(• | k, M, π) 2 . (5)We further supervise the PBR field by adding a computationally-efficient deferred shading loss:EQUATION)The weight w(u) = n(u) • n(u) is the dot product of the predicted and ground-truth normals at pixel u. It discounts the loss where the predicted geometry is not yet learnt. Fig. 14 (b ) visualizes deferred shading and the rendering loss.Finally, we also supervise the SDF field with a direct loss L sdf (implemented as in [1] ), a depth-MSE loss L depth between the depth renders and the ground truth, and with a binary cross-entropy L mask between the alpha-mask renders and the ground-truth masks. Refer to App. A.6.2 for more details.LightPlane implementation. We base MetaILRM on LightplaneLRM [6] , a variant of LRM [29] exploiting memory and compute-efficient Lightplane splatting and rendering kernels, offering better quality reconstructions. However, since LightplaneLRM uses density fields, which are suboptimal for mesh conversion [92, 66, 1] , we extend the Lightplane rendering GPU kernel with a VolSDF [108] renderer using Eq. ( 2). Additionally, we also fuse into the kernel the direct SDF loss L sdf since a naive autograd implementation is too memory-heavy.The MetaILRM module of Sec. 3.2 outputs a sign distance function s, implicitly defining the object surface A = {x ∈ R 3 | s(x) = 0} as a level set of s. We use the Marching Tetrahedra algorithm [18] to trace the level set and output a mesh M ≈ A. Then, xAtlas [112] extracts a UV map ϕ : [0, V ] 2 → M , mapping each 2D UV-space point v = ϕ(x) to a point x ∈ M on the mesh.Next, the goal is to extract a high-quality 5-channel PBR texture image K ∈ R V ×V ×5 capturing the albedo, metalness, and roughness of each mesh point. The texture image K can be defined directly by sampling the predicted PBR field k as K(v) ← k(ϕ(v)), but this often yields blurry results due to the limited resolution of MetaILRM. Instead, we design a texture refiner module which takes as input the coarse PBR-sampled texture image as well as the N views representing the object and outputs a much sharper texture K. In essence, this modules leverages the information from the different views to refine the coarse texture image. The right part of Fig. 2 illustrates this module.More specifically, it relies on a network Φ which is fedN + 1 texture images {K i } N i=0 . First, each pixel v ∈ [0, V ] 2 of K 0 ∈ R V ×V ×11is annotated with the concatenation of the normal, the 3D location, and the output of MetaILRM's PBR field k(ϕ(v)) evaluated at v's 3D point ϕ(v). The remaining K 1 , . . . , K N correspond to partial texture images with 6 channels (for the base and shaded colors) which are obtained by back-projecting the object views to the mesh surface. The network Φ utilises two U-Nets to fuse {K i } N i=0 into the enhanced texture K. Φ's goal is to select, for each UV point v, which of the N input views provides the best information. Specifically, each partial texture image K i is processed in parallel by a first U-Net, and the resulting information is communicated via cross attention to a second U-Net whose goal is to refine K 0 into the enhanced texture K. Please refer to App. A.7 for further details.Such a network is trained on the same dataset and supervised with the PBR and albedo rendering losses as MetaILRM. The only difference is meshes (whose geometry is fixed) are rendered differentiably using PyTorch3D's [71] mesh rasterizer instead of the Lightplane SDF renderer.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiments: 
 Our training data consists of 140,000 meshes of diverse semantic categories created by 3D artists. For each asset, we render 36 views at random elevations within the range of [-30 • , 50 • ] at uniform intervals of 30 • around the object, lit with a randomly selected environment map. We render the shaded images, albedo, metalness, roughness, depth maps, and foreground masks from each viewpoint. The text-to-image stage is based on an internal text-to-image model architecturally similar to Emu [16] , fine-tuned on a subset of 10,000 high-quality 3D samples, captioned by a Cap3D-like pipeline [53] that uses Llama3 [88] . The other stage utilizes the entire 3D dataset instead.For evaluation, following [105, 103, 42] , we assess visual quality using PSNR and LPIPS [117] between the rendered and ground-truth images. PSNR is computed in the foreground region to avoid metric inflation due to the empty background. Geometric quality is measured by the L1 error between the rendered and ground-truth depth maps (of the foreground pixels), as well as the IoU of the object silhouette. We further report Chamfer Distance (CD) and Normal Correctness (NC) for 20,000 points uniformly sampled on both the predicted and ground-truth shapes. Material decomposition is evaluated with LPIPS and PSNR on the albedo image, and PSNR alone for the metalness and roughness channels. All metrics are calculated on meshified outputs rather than on neural renders. We tackle the sparse-view reconstruction task of predicting a 3D mesh from 4 posed images of an object on a subset of 332 meshes from Google Scanned Objects (GSO) [19] . We compare against state-of-the-art Instant3D-LRM [42] , GRM [105] , In-stantMesh [103] , and MeshLRM [97] . We also include LightplaneLRM [6] , an improved version of Instant3D-LRM, which serves as our base model. MeshLRM [97] has not been opensourced so we compare only qualitatively to meshes from their webpage. All methods are evaluated using the same input views at 512 2 resolution. Since none of the latter predict PBR materials and since GSO lacks ground-truth PBR materials, for fairness, we use a variant of our model that predicts shaded object textures.As shown in Figs. 4 and 9 and Tab. 3, our method outperforms all baselines across all metrics. GRM captures texture detail well but struggles with fine geometric structures when meshified. InstantMesh and LightplaneLRM improve geometry but fall short on finer details and texture quality. Our approach excels in reconstructing shapes with detailed geometry and high-fidelity textures. Next, we consider the task of sparse-view reconstruction with PBR materials, where the goal is to reconstruct the 3D geometry and texture properties (albedo, metalness, and roughness) from four posed shaded 2D views of an object. This is done on an internal dataset of 256 artist-created 3D meshes, curated for high-quality materials. Since there are no existing few-view feed-forward PBR reconstructors, we conduct an ablation study in Tab. 1 and Figs. 3 and 13 .While adding material prediction with additional MLP heads provides some improvements, we observe that incorporating the deferred shading loss and texture refinement is essential for high-quality PBR decomposition. Example PBR predictions are shown in Fig. 8 .Finally, we evaluate text-to-3D with PBR materials. We compare against state-of-the-art feed-forward methods that generate assets at comparable speed (≈ 10 to 30 s per asset). This includes text-to-3D variants of GRM [105] , InstantMesh [103] , and LightplaneLRM [6] . GRM uses Instant3D's 4-view grid generator, InstantMesh receives the first view from our 2D diffusion model and subsequently generates 6 views, while LightplaneLRM accepts 4 views from our grid generator. Since these methods bake lighting instead of generating PBR materials, for evaluation we apply flat texture 5 : Qualitative comparison for text-to-3D. We compare 3D meshes generated by Meta 3D AssetGen and state-of-the-art baselines. We include material decomposition for methods producing PBR materials (Luma Genie and our Meta 3D AssetGen). Our approach produces higher quality materials with better-defined metalness and roughness, and a more accurate decoupling of lighting effects in the albedo. shading to our outputs. Additionally, we compare with the preview stage of Meshy v3 [85] and LumaAI Genie 1.0 [84] , proprietary text-to-3D methods with PBR workflow capable of creating assets within 30 and 15 s respectively. A comparison with the significantly longer refinement stages for Luma and Meshy is provided in the appendix. Fig. 5 shows that AssetGen meshes are visually more appealing and have meaningful materials Figs. 6 and 12 provide more examples and comparisons and showcase fine-grained material control.For quantitative evaluation, we conducted an extensive user study in Tab. 2 using the 404 deduplicated text prompts from DreamFusion [68] . Users were shown 360 • videos of the generated and baseline meshes and were asked to rate them based on 3D shape quality and alignment with the text prompt. A total of 11,080 responses were collected, with significant preference for AssetGen's meshes.Finally, we ablate the effect of generating dual-channel albedo+shaded grids compared to albedo-only input in Fig. 3 revealing significant PBR decomposition superiority of the former. Additionally, Fig. 13 illustrates the effect of our deferred shading loss. 
conclusions: 
 We have introduced Meta 3D AssetGen, a significant advancement in sparse-view reconstruction and text-to-3D. Meta 3D AssetGen can generate 3D meshes with high-quality textures and PBR materials faithful to the input text. This uses several key innovations: generating multi-view grids with both shaded and albedo channels, introducing a new reconstruction network that predicts PBR materials from this information, using deferred shading to train this network, improving geometry via a new scalable SDF-based renderer and SDF loss, and introducing a new texture refinement network. Comprehensive evaluations and ablations demonstrate the effectiveness of these design choices and state-of-the-art performance.  
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Reported metrics have negligible or zero variance.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The following is our statement addressing concerns about training data raised during the review process. The data used in this paper was not obtained by scraping the internet. The data was purchased from a well-respected and widely-known vendor of 3D graphic assets. We acquired a data license that explicitly allows use of the models in machine-learning applications, including all applications in this paper. We follow the terms and conditions of this license scrupulously, also based on internal legal advice.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] ,Justification: See App. A.1Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Data or models are not released Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We use an internal dataset of artist-created meshes whose use has been approved by a professional legal team.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes] Justification: See App. A.4.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [Yes] Justification: The internal legal and privacy audit has approved our user study.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Safeguards should be implemented to prevent abuse, such as filtering input text prompts and detecting unsafe content in generated 3D models. Additionally, our generation process may be vulnerable to biases present in the data and 3D models it relies on, potentially perpetuating these biases in the generated content. Despite these risks, our method can augment the work of artists and creative professionals by serving as a complementary tool to boost productivity. It also holds the potential to democratize 3D content creation, making it accessible to those without specialized knowledge or expensive proprietary software.Meta 3D AssetGen significantly advances shape generation but faces several limitations. Despite the fine-tuning of the multiview image grid generator for view consistency, it is not guaranteed, potentially impacting 3D reconstruction quality. Since we use an SDF as an underlying representation, the reconstructor may incorrectly model translucent objects or thin structures like hair or fur. Additionally, while our scalable Triton [67] implementation supports a triplane representation at a resolution of 128 × 128, this representation is inefficient, as much of its capacity is used for empty regions. Future work could explore scalable representations such as octrees, sparse voxel grids, and hash-based methods, which may remove the need for a separate texture enhancement model. We also only predict albedo, metalness and roughness, and not emissivity or ambient occlusions. Finally, our method has only been tested on object-level reconstructions, leaving scene-scale 3D generation for future research.This section describes additional qualitative comparisons that, due to limited space, could not be included in the main paper. Firstly, please refer to the video attached in the supplementary material which provides a holistic presentation of Meta 3D AssetGen's qualitative results. In Fig. 7 , we highlight the contributions of MetaILRM in geometry, texture and material reconstruction. In Fig. 12 , we visualize the control of materials provided by Meta 3D AssetGen, i.e., metalness and roughness, by changing the text prompt for the same concept. Fig. 8 visualizes the renders of the material maps extracted with MetaILRM given four input test views. In Fig. 9 , we provide a more extensive qualitative comparison to MeshLRM, the strongest few-view reconstruction baseline. Finally, Fig. 6 provides a gallery of text-conditioned generations depicting Blender-shaded renders together with the rendered PBR maps.As described in Sec. 4.2, we conducted an user study on 404 meshes generated using the DreamFusion [68] prompt-set on a standard crowdsourcing marketplace. In the study, users were shown 360 • Figure 7 : MetaILRM builds upon LightplaneLRM [6] , providing improved geometry by employing SDF as a representation, along with direct scalable losses in 3D, improved texture using a UV space texture refiner, and material decomposition by predicting material properties regularized through a novel deferred shading loss. videos of the generated and baseline meshes and were asked to rate them based on 3D shape quality and alignment with the text prompt as shown in Fig. 11 . They were asked to consider various factors like identity (whether the object matches what is described in the prompt), texture, existence of Janus problems, and bad geometry (like floaters, disconnected components, etc). 11,080 responses were collected in total, with 5 responses per pair of videos, to eliminate variance in user preference. spectively. Tab. 4 contains the results of our user-study. Meta 3D AssetGen significantly outperforms Meshy in both text fidelity and visual quality while being 10× faster. Surprisingly, Meta 3D AssetGen is on par with Luma Genie in text fidelity and wins in 40% of cases in visual quality. This is a remarkable result considering Meta 3D AssetGen's 20× better generation time. We show further qualitative comparisons in Fig. 10 .A.6.1 Grid GeneratorWe employ a text-to-image diffusion model pre-trained on billions of images annotated with text [16] and expand its input and output channels by a factor of 2 to support simultaneous generation of shaded appearance and albedo. We finetune the model to predict a grid of four images I i , i = 1, . . . , 4, in similar fashion to [74, 42] via minimization of the standard diffusion loss. Training spans a total of 2 days, employing 32 A100 GPUs with a total batch size of 128 and a learning rate of 10 -5 .As mentioned in the main paper, MetaILRM is optimized using the direct SDF loss L sdf , PBR loss L pbr , deferred shading loss L def , the binary cross-entropy mask loss L mask , and the depth-MSE loss L depth so the global objective is:L = 0.5L sdf + L pbr + 0.5L def + 0.1L mask + 0.1L depth .The texture refiner uses only PBR loss and the deferred shading loss: L pbr + 0.5L def .In each training batch, we randomly sample 4 views per scene as source input views I i , and another 4 target views I tgt , into which we render the sdf-field predicted by MetaILRM, or the mesh predicted by the texture refiner. We then evaluate the aforementioned losses in the target views. 3 scenes per GPU are sampled randomly, and we train on 64 GPUS NVIDIA A100 gpus, yielding an effective batch size of 3 × 4 × 64 = 768 images. The total loss has been optimized using Adam [41] with learning rate 10 -4 for 13K steps.Figure 12 : Generated assets for the prompt: "A cat made of <MATERIAL>". Meta 3D Asset-Gen predicts various plausible PBR material maps leading to realistic interaction with the environment light (sphere-mapped in the center)Figure 13 : Using a deferred shading loss on rendered channels enhances PBR quality, resulting in more defined metalness and roughness, such as increased metalness in the lantern's metal parts and decreased roughness in its glass parts.Besides verifying quantitatively the benefits of the deferred shading loss L def in Tab. 1, we also provide a qualitative proof in Fig. 13 . Specifically, the PBR materials predicted from albedo&shaded channels exhibit better metalness map on the actual metallic parts of the 3D lantern asset.We follow Azinovic et al.'s [1] direct SDF supervision for the SDF field. Given a pixel p in an image and the sampled points S p on the ray corresponding to the pixel, the direct SDF loss is computed asEQUATIONL fr sdf is a 'free-space' objective, which forces the MLP to predict a value of 1 for samples s ∈ S fs p which lie between the camera origin and the truncation region of a surface:EQUATIONwhere D s is the predicted SDF from the MLP. For samples within the truncation region (s ∈ S tr p ), we apply L tr sdf , the signed distance objective of samples close to the surface.EQUATIONA naïve PyTorch implementation of this is memory intensive, because of the evaluation of B × H × W × N ray points, where B, H, W, N ray are the number of target images in a batch, the height, width, and the number of points per ray respectively. Therefore, to support large batch sizes, image resolution, and denser point sampling on rays, we implement the direct SDF loss using custom Triton [67] kernels.The depth loss L depth minimizes the mean-squared error between the rendered depth prediction R depth (• | ŝ, π) the ground-truth depth R depth (• | M, π)L depth = R depth (• | ŝ, π) -R depth (• | M, π) 2 ,where R depth (s, π) is an operator rendering the depth-map of the shape representation s (mesh or an SDF) from the viewpoint π.Having described a high-level overview of our texture refiner in Sec. 3.3, here we provide more details.As mentioned, the texture refiner network Φ accepts N + 1 texture images K i in total. The first input to the network is the augmented texture image K 0 ∈ R V ×V ×11 given by:∀v ∈ [0, V ] 2 : K 0 (v) = k(x v ) ⊕ n(x v ) ⊕ x v , if v ∈ Im(ϕ), 0, otherwise, where x v = ϕ(v).The condition v ∈ Im(ϕ) selects 'valid' UV points that correspond to mesh points; and ⊕ denotes channel-wise concatenation, so that K 0 (v) is the stack of the 5 PBR parameters k(x v ) from MetaI LRM, normal n(x v ), and the 3D point x v .In addition to K 0 , we input to the network Φ texture images K i , each extracted by looking up information from the corresponding input view I i directly (thus sidestepping MetaILRM). As noted above, each valid texture point v corresponds to a unique 3D point x v = ϕ(v) ∈ M on the mesh, which in turn projects to a pixel u = π i (x v ) in the image I i . Let χ i (v) ∈ {0, 1} be the flag that tells if point x v is visible in image I i or not. When point x v is visible in several views, it is best measured in the most frontal one, which is captured by the cosine ω o • n(x v ) between the normal n at x v We apply it to both the ground truth channels (top) and the predicted channels (middle). The error is calculated as the difference between the two, weighted by the similarity between ground truth normals and predicted normals, to avoid penalizing shading errors due to incorrect normals.and the ray direction ω v ∝ x 0 -x v . All this information is packed into additional texture imagesK i ∈ R V ×V ×(D+1) by setting: ∀v ∈ [0, V ] 2 : K i (v) = I i (π i (x v )) ⊕ (ω v • n(x v )), if v ∈ Im(ϕ) and χ i (v) = 1, 0, otherwise.The texture network Φ is a U-Net that takes as input the texture augmented texture image K 0 and outputs the final enhanced texture K ∈ R V ×V ×5 . This network also fuses information from the view-specific texture images K i . The goal is to select, for each UV point v, which of the N input views provides the best information. This is achieved via cross-attention. Specifically, each K i is processed in parallel by another U-Net, and the first queries information across all the others via multi-head cross attention. In Fig. 14 (a) , we provide an illustration of the latter cross-view attention layer.A.8 Physically-Based Rendering: Radiance, BRDFs, and modelsWe briefly summarise key notion of radiometry and standard BRDF models, and then provide a precise expression of the BRDF model used in Meta 3D AssetGen.The radiant flux Φ is the electromagnetic power flowing through a particular surface A ⊂ R 3 oriented by the unit normal n. The radiance L(p, ω) is the radiant flux density at p towards a particular direction ω per unit orthogonal area dA ⊥ and per unit solid angle dΩ. The unit vector ω points at the direction of propagation of the flux.The flux density is measured with respect to an area which is orthogonal to the direction of propagation ω. In fact, the energy flow is the same through all areas that cut the same 'tube of flux'; the specific area is irrelevant and not a property of the radiation. This dependency is removed by considering the normalized area dA ⊥ , which is orthogonal to the direction of the flux.Because the radiance is expressed in units of orthogonal area dA ⊥ , in order to compute the flux through the surface patch dA, which may not be orthogonal, we must account for the foreshortening factor, which relates the areas dA and dA ⊥ :dA ⊥ = |⟨n, ω⟩| dA.With this, the flux that passes through dA towards direction ω in the solid angle dΩ is dΦ = L(p, ω) |⟨n, ω⟩| dA dΩ.Note that dΦ depends on both ω and n whereas L only depends on ω. This reinforces the notion that n is a property of the surface, not of the radiation.Let p be a point on a surface A that separates two media and let dA be a surface patch sitting at p.Let n be the normal at this point. We now consider the case where A separates air or empty space from an opaque object, with n pointing towards the outside of this object.Let ω be an orientation on the same side of the surface as n, i.e., such that ⟨n, ω⟩ ≥ 0. From the viewpoint of the object, we interpret L(p, ω) as outgoing radiant flux and L(p, -ω) as incoming radiant flux. These two quantities are related by the Bidirectional Reflectance Distribution Function (BRDF) f , defined such that:EQUATIONIn this definition, for convenience both ω i and ω o are taken on the same side as n (hence the negative sign in front of ω i ). A useful consequence is that we do not need to take the absolute value of the inner product ⟨n, ω i ⟩ as this is positive by definition.The BRDF thus takes the radiation receives from direction -ω i and distributes it along various outgoing directions ω o . Integrating over all incoming directions, gives use the overall radiation reflected towards ω o :EQUATIONwhere H(n) = {ω : ⟨n, ω⟩ ≥ 0} is the hemisphere. Next, we provide common basic models for the BRDF function in PBR.In diffuse reflectance, the radiation is absorbed by the material, internally scattered in random directions, and output again to give rise to a uniform distribution. Namely, the diffuse BRDF is:f (p, ω i , ω o ) = R πwhere 0 ≤ R ≤ 1 is the fraction of power reflected by the diffusion process. The 1/π factor ensures that the total energy is conserved when R = 1.The reflection for a perfectly flat interface between two media at p is specular: the incoming light radiation -ω i is partially reflected in the specular direction ω o = r(n, ω i ) = 2n⟨n, ω i ⟩ -ω i , and partially transmitted. This phenomena is characterised by Fresnel's equations, which are derived from Maxwell's equations, utilising continuity conditions for the electromagnetic field at the interface between the two media. Fresnel's equations describe the planar radiation in full, including its polarisation (in the most general case, using phasors, and thus complex numbers). For graphics, we assume that light is unpolarized, so we only calculate the power. The fraction of power reflected is given by Fresnel's coefficient (using Schlick's approximation [72] ):F (⟨n, ω i ⟩) = F 0 + (1 -F 0 ) |⟨n, ω i ⟩| 5 , F 0 = n1 -n2 n1 + n2 2 .Here n1 and n2 are the indices of reflectivity of the two media, respectively. This equation is valid for dielectrics (non-metallic objects), but also used as an approximation for metals by tweaking F 0 .In order to write this relation as a BRDF, we write:L(p, ω o ) = H(n) f (p, ω i , ω o ) ⟨n, ω i ⟩L(p, -ω i ) dΩ i = R(⟨n, r(n, ω o )⟩)L(p, -r(n, ω o )).Hence, the BRDF must be a delta function centered at ω * i = r(n, ω o ):EQUATIONwhere we used the fact that ⟨n, r(n, ω o )⟩ = ⟨n, ω o ⟩ and where δ r(n,ωo) is the delta distribution centered at r(n, ω o ).Rough surfaces can be though of as a collection of randomly-oriented flat microfacets, each reflecting light in a specular fashion. Consider a point p on a surface and incoming and outgoing radiation directions ω i and ω o . If the point contains a microfacet that enables light to reflect from ω i to ω o , then the normal of the microfacet must be m ∝ ω i + ω o . If the microfacet is oriented elsewhere, then no light flows in the direction ω o . Hence, we can writem = h(ω i , ω o ) = ω i + ω o |ω i + ω o |as a function of the incoming and outgoing radiation. This is called half vector as it sits in between the two vectors ω i and ω o .Now we wish to derive the macro BRDF f (p, ω i , ω o ) from the micro BRDF F (p, ω i , ω o |m), where we have emphasized the fact that the BRDF is oriented relative to the microfacet normal m. A short calculation [90] shows that:f (p, ω i , ω o ) = f m (p, ω i , ω o |m) ⟨m, ω i ⟩ ⟨n, ω i ⟩ ⟨m, ω o ⟩ ⟨n, ω o ⟩ 1 ⟨m, n⟩ .In practice, there is a distribution over possible surface normals m, characterised by the microfacet distribution function D(m|n). The latter is defined such that D(m|n) dA dΩ m is the total area of the microfacets within patch dA of the macrosurface with orientation in dΩ m . In practice, only part of the microfacet is visible and illuminated, depending on the interaction with other facets. This is accounted for by the shadowing-masking functionG(ω i , ω o , m, n) ∈ [0, 1].The expected reflectance is thus:f (p, ω i , ω o ) = H(n) ⟨m, ω i ⟩ ⟨n, ω i ⟩ ⟨m, ω o ⟩ ⟨n, ω o ⟩ f m (p, ω i , ω o |m) D(m|n) G(ω i , ω o , m, n) dΩ m .Plugging Eq. ( 12) in the value for the mirror-like reflectance f m for each microfacet, we get [90] :f (p, ω i , ω o ) = F (⟨h, ω o ⟩)D(h|n) G(ω i , ω o , h, n) 4⟨n, ω i ⟩ ⟨n, ω o ⟩ where h = h(ω i , ω o ).A.12 Standard microfacet models Here, we discuss common choices for the functions D and G in standard PBR models.The Torrance-Sparrow model. The oldest such model is due to Torrance and Sparrow [87] . They simply assume a Gaussian model for the microfacet distribution function:D(m|n) = b exp -α 2 θ , where cos θ = ⟨m, n⟩,and b is a suitable normalization constant. For the shadowing-masking function they pick:G(ω i , ω o , m, n) = min 1, 2⟨m, n⟩⟨n, ω i ⟩ ⟨m, ω i ⟩ , 2⟨m, n⟩⟨n, ω o ⟩ ⟨m, ω o ⟩ .This function has a simple geometric derivation under a basic geometric model of the microfacets.The Beckmann-Spizzichino-Smith model. Beckmann and Spizzichino [2] suggested the model:D(m|n) = 1 πα 2 cos 4 θ e -tan 2 θ α 2where cos θ = ⟨m, n⟩.Smith [77] noted that the shadowing-masking function should be derived from the microfacet distribution function, which describes the micro-geometry of the surface. They also proposed a factorized modelG(ω i , ω o , m, n) = G 1 (ω i , n)G 1 (ω o , n). For Beckmann's distribution, the Smith shadowing-masking function is given by:G 1 (ω, n) = 2 1 + erf(a) + 1 a √ π e -a 2 where a = 1 α tan θ ωand cos θ ω = ⟨n, ω⟩.The GGX model. The GGX model by [90] is a variant of the Beckmann model, with slightly different microfacet distribution and shadowing-masking function:EQUATIONwhere cos θ = ⟨m, n⟩ and cos θ ω = ⟨n, ω⟩.A.12.1 The BRDF model used in Meta 3D AssetGenThe BRDF model used in our paper combines diffuse and GGX BRDFs:f (ω i , ω o |k(x), n) = R π + F (h|n)D(h|n)G 1 (n, ω i )G 1 (n, ω o ) 4(n • ω i )(n • ω o ) .The first term is the diffuse component (Lambertian reflection), where R ∈ [0, 1] is the fraction of light power reflected by diffusion. The second term in the specular component, where F is Schlick's approximation [72] 5 of Fresnel's reflectance and where F 0 ∈ [0, 1] is the Fresnel coefficient at normal incidence. The unit vector h ∝ ω i + ω o is the half vector, which is the orientation needed to reflect ω i into ω o by the rough material's microfacets (generally different from but averaging to n). The function D and G 1 are the microfacet distribution function and the shadowing-masking function given by:F (ω i |n) = F 0 + (1 -F 0 )(1 -n • ω i )D(m|n) = α 2 π((m • n) 2 (α 2 -1) + 1) 2 , G 1 (n, ω) = 2(n • ω) n • ω + (n • ω) 2 (α 2 -1) + α 2 .These are the same as Eq. ( 13) with the trigonometric functions expanded in terms of dot products. In this model, the reflectance R and reflectance at normal incidence F 0 are RGB triplets. The specular highlight color F 0 is approximately white (equal to 1) for dielectrics, and colored for metals; furthermore, metals have no diffuse component (R = 0). Thus, we introduce the parameter metalness γ ∈ [0, 1] and the base color ρ 0 ∈ [0, 1] 3 and define:R = ρ 0 (1 -γ), F 0 = 1(1 -γ) + ρ 0 γ.In this manner, the albedo is used as diffuse color or specular color depending on whether the material is a dielectric or a metal. In the paper, we call the parameter ρ 0 albedo as it is a better known concept; however, in this model ρ 0 is the albedo only when γ = 0, i.e., when the object is a perfect dielectric.The BRDF is thus fully described by the albedo ρ 0 , the roughness α, and the metalness γ, for a total of five scalar parameters. Hence, the LRM predicts the triplet k(x) = (ρ 0 , γ, α) at each 3D point x.NeurIPS Paper Checklist 


Decision: This paper proposes a 2-stage feed-forward framework, named EmuLRM, for text-to-3D that predicts PBR materials to support realistic relighting. In the 1st stage, 4 views with both shaded and albedo channels are generated by a text-to-image diffusion model to asist the prediction of PBR materials in the 2nd stage. In the 2nd stage, the authors extend LightplaneLRM to predict SDF together with PBR materials. The authors also introduce deferred shading, a new scalable SDF-based render, and a direct SDF loss to improve training efficiency and scalability. Finally, they propose a texture refinement step operating in the UV space to improve the sharpness and details of the textures by leveraging the reference views through cross-attention. 

This paper outperforms other feed-forward methods and produces sota results. Its results are also comparable to sota optimization based methods. Sufficient quantitative and qualitative results are presented to support and validate various design choices.

Overall, all reviewers are positive about this paper (accept, weak accept, weak accept, weak accept). The authors have addressed all the concerns rasied by the reviewers in their rebuttal. It is recommended to accept this paper.

Reviews: review: paper_summary: paper_summary: The paper mainly works on text-to-3D with PBR materials. It follows the diffusion-based multiview generation and LRM-based reconstruction paradigm. It is a fast feed-forward solution for PBR generation: The diffusion model predicts both shaded and albedo, and the LRM predicts PBR via differentiable rendering with an efficient pixel-wise deferred shading loss. Moreover, it designs a UNet-based texture refiner to project input images to mesh, bringing sharp textures. Also, it adapts LightplaneLRM with the VolSDF renderer and SDF loss to enhance geometry. The generation only takes 30 seconds. It outperforms state-of-the-art feed-forward baselines and performs comparably to top optimization-based industry solutions.
strength_weakness: strength_weakness: 1. Image-to-3D may have a degraded material decomposition. Since the proposed approach does not predict albedo for input images, PBR prediction correctness will drop significantly as pointed out in the ablation study (Table 1, Fig. 3).
2. The diffusion model is inherently stochastic, which leads to variance in albedo prediction. How is this variance? How will this variance impact the material decomposition afterward? This point is not quantitatively evaluated.
3. The paper evaluates PSNR for metallic. It is great. But metallic is usually binary. A simple 2x2 confusion matrix on metallic prediction on single-material objects can show the performance more intuitively.
4. If input images are taken under non-uniform lighting conditions, I doubt if albedo can still be correctly decomposed.
5. Lack of failure case analysis. Following the previous point, I wonder if there are some typical failure cases for PBR prediction.
questions: questions: - Fewer than 10% meshes are selected as high-quality samples to finetune Emu (L226). What is the filtering standard? In terms of appearance quality, material correctness, and composition complexity? Please elaborate more.
- L221: What dataset is used? Is it publicly available, like Objaverse? Is the dataset used under proper license?
- How is the text-to-image model (3-channel output) adapted to output 6 channels? Please provide more details on that.
- For results in Figure 7., if I understand correctly, they are deterministic? What is their PBR PSNR? (How cherry-picked are they?)
- May consider adding some qualitative comparisons to optimization-based PBR generation baselines like Richdreamer. Although they are time-costly and may not be as competitive as commercial solutions, such a comparison can strengthen the claim.
- In the Image-to-3D part (excluding texture refinement), Are albedo predictions only used as inputs to MLP?
- Some qualitative examples of Luma Genie 1.0, stage 2 vs the proposed approach (A.5). It would help to show where optimization helps to get an edge.
- 30-sec Runtime breakdown to each step.
limitations: limitations: Some suggestions are included in the Questions above.
paper_summary: paper_summary: This paper introduces a two-stage 3D asset generation pipeline that outputs high-quality 3D meshes with PBR materials in approximately 30 seconds. The technical novelties include:
- In text-to-image stage, generating multiple views of both shaded and unshaded images, and predicting PBR materials during the image-to-3D stage.
- Using SDF instead of opacity field which yields higher-quality meshes.
- Introducing a new texture refiner designed to recover and enhance details in texture maps.
strength_weakness: strength_weakness: I found certain technical novelties challenging to assess:
- The use of SDF instead of opacity fields in the reconstruction pipeline is not a novel idea but more of an extension to Instant3D. This approach has also been explored in 3D generation works such as MeshLRM and Latte3D.
- The texture refiner, which I found the most interesting and novel part of the pipeline, is difficult to evaluate from the provided qualitative examples. Looking at the texture maps in Figures 2 and 13, which contain many small components and seams from xAtlas outputs, one concern is whether these artifacts might affect the UNet's predictions in UV space, e.g. causing color inconsistencies. The ablation in Figure 14 only shows textures improved in flat regions, without the corresponding UV maps. Showing texture maps before and after refinement around seams would help readers better understand the model's effectiveness, and justify refinement in UV space as opposed to image space.
questions: questions: Regarding the blurriness of the pre-refinement texture map: Is this a result of the image-to-3D reconstruction process or an issue with sampling from the texture field? If it is due to the sampling process, why not optimizing with respect to volSDF renderings to recover the details?
limitations: limitations: The authors discuss the limitations of their method and broader impact in the paper.
paper_summary: paper_summary: The paper adopts a two-stage PBR information generation method. In the first stage, the text-to-image model is used to predict the PBR channel. In the second stage, it uses SDF to reconstruct the geometry and PBR information. Then, a cross-view attention texture reflector network is used to improve the coarse UV of the input.
strength_weakness: strength_weakness: The technical novelty is weak.

I noticed that neither the dataset nor the code are intended to be open sourced. I understand that this technology involves commercial use, but I believe that at least all the test sets used in the experiments and the 3D (PBR) assets generated on the test sets should be open sourced to facilitate comparison in future work.

The PBR resolution of 512x512 with 2x2 grid for 4-view is still relative low.
questions: questions: Recently, Unique3d [1] generated mesh with higher-resolution texture. As Emu3d and Unique3d [1] are concurrent works, it is not necessary to directly compare with it. However, theoretically speaking, compared to Unique3d, the texture resolution of Emu3d is still relatively low. It is suggested to add a paragraph in the paper to discuss the advantages and disadvantages of the two methods relative to each other.

[1] Wu, Kailu, et al. "Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image." arXiv preprint arXiv:2405.20343 (2024).
limitations: limitations: No. But they have discussed in the limitation part. They use a triplane representation, which cannot represent large-scale scenes.
paper_summary: paper_summary: Emu3D is a novel approach in text-conditioned 3D generation, producing high-quality 3D meshes and materials using Physically-Based Rendering (PBR). It uses a two-stage process: first generating images from standard viewpoints, then reconstructing the 3D shape and appearance. This approach is faster and more reliable than earlier methods and creates realistic 3D graphics by modeling light interaction with surfaces. Emu3D also improves mesh generation by predicting a signed-distance field (SDF) for better quality and refining textures for greater detail. It surpasses existing methods in both image-to-3D and text-to-3D tasks, offering superior visual quality and text alignment.
strength_weakness: strength_weakness: - The generated PBR material includes only limited components: albedo, metallicity, and roughness. PBR materials typically also consist of other components, such as index of reflection, scattering, coat, sheen etc. Therefore, the generated material can only represent surface reflectance from diffuse to specular.
questions: questions: - What’s the performance of the proposed method over anisotropy materials?
- What's the performance of the extracted mesh over transparent object like a class bottle?
limitations: limitations: - The generated material has significant representational limitations, as it primarily models surface reflection using only albedo, metalness, and roughness.

Comments: comment: Thank the authors for the detailed rebuttal and the additional results provided. It’s surprising to see that the texture refinement is effective, even with UV texture maps that contain numerous isolated pieces. 

Regarding the third point, I meant to ask about optimizing with respect to the renderings of volSDF, rather than the input images. Nevertheless, I agree with the authors’ point regarding the time cost. 

The rebuttal has addressed my major concerns, and I have adjusted my scores accordingly.
We thank the reviewer for raising this important point. We understand these concerns and take ethics and IP rights extremely seriously. 


The data used in this paper was **not** obtained by scraping the internet. The data was purchased from a well-respected and widely-known vendor of 3D graphic assets. We acquired a data license that **explicitly allows** use of the models in machine-learning applications, including all applications in this paper. We follow the terms and conditions of this license scrupulously, also based on internal legal advice. The exact commercial details of the license are, as it may be expected, confidential.
With the reviewer-author discussion period nearing its end, we want to ensure that our responses have thoroughly addressed your primary concerns. We recognize the importance of this dialogue in resolving the issues raised by the reviewers and are committed to providing any further clarifications you might need. Please let us know if there are any additional questions or if more explanation is required.

Warm regards,
Emu3D Authors
With the reviewer-author discussion period nearing its end, we want to ensure that our responses have thoroughly addressed your primary concerns. We recognize the importance of this dialogue in resolving the issues raised by the reviewers and are committed to providing any further clarifications you might need. Please let us know if there are any additional questions or if more explanation is required.

Warm regards,
Emu3D Authors
With the reviewer-author discussion period nearing its end, we want to ensure that our responses have thoroughly addressed your primary concerns. We recognize the importance of this dialogue in resolving the issues raised by the reviewers and are committed to providing any further clarifications you might need. Please let us know if there are any additional questions or if more explanation is required.

Warm regards,
Emu3D Authors
With the reviewer-author discussion period nearing its end, we want to ensure that our responses have thoroughly addressed your primary concerns. We recognize the importance of this dialogue in resolving the issues raised by the reviewers and are committed to providing any further clarifications you might need. Please let us know if there are any additional questions or if more explanation is required.

Warm regards,
Emu3D Authors