Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials
Authors: Yawar Siddiqui
Tom Monnier
Filippos Kokkinos
Mahendra Kariya
Yanir Kleiman
Emilien Garreau
Oran Gafni
Natalia Neverova
Andrea Vedaldi
Roman Shapovalov
David Novotny
Venue: NeurIPS 2024 poster (2024)
Keywords: text to 3d
3d generative models
sparse view reconstruction
3d shape generation
Primary area: generative_models
TL;DR: A fast text to 3D generative model with high quality geometry, textures which supports PBR decomposition of materials

Abstract: We present Meta 3D AssetGen (AssetGen), a significant advancement in text-to-3D generation which produces faithful, high-quality meshes with texture and material control. Compared to works that bake shading in the 3D object’s appearance, AssetGen outputs physically-based rendering (PBR) materials, supporting realistic relighting. AssetGen generates first several views of the object with separate shaded and albedo appearance channels, and then reconstructs colours, metalness and roughness in 3D, using a deferred shading loss for efficient supervision. It also uses a sign-distance function to represent 3D shape more reliably and introduces a
corresponding loss for direct shape supervision. This is implemented using fused kernels for high memory efficiency. After mesh extraction, a texture refinement transformer operating in UV space significantly improves sharpness and details. AssetGen achieves 17% improvement in Chamfer Distance and 40% in LPIPS over the best concurrent work for few-view reconstruction, and a human preference of 72% over the best industry competitors of comparable speed, including those that support PBR. Project page with generated assets: https://assetgen.github.io

Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials
Yawar Siddiqui'
Tom Monnier*
Filippos Kokkinos*
Mahendra Kariya
Yanir Kleiman
Emilien Garreau
Oran Gafni
Natalia Neverova
Andrea Vedaldi
Roman Shapovalov* David Novotny
GenAl, Meta
'TU Munich; intern with Meta *core technical contributors
Text-to-3D generation
Image-to-3l generation
text promot
> a train
engine made out clay
source views
Material decomposition
Relighting different
cat made silver
albedo
metalness
raughness
cat made of zock
Figure 1: We present Meta 3D AssetGen, a novel text- or image-conditiones generator 3D meshes with chysically-based rendering materials (top). Meta 3D AssetGen produces meshes with detailed geometry and high-quality textures, and decomposes materials into albedo, metalness, and roughness (bottom left), which allows realistically relight objects in new environments (bottom right).
Abstract
We present Meta 3D AssetGen (AssetGen), a significant advancement in text-to-3I generation which produces faithful, high-quality meshes with texture and material control. Compared works that bake shading the 3D object's appearance, AssetGen outputs physically-base rendering (PBR) materials, supporting realistic relighting. AssetGen generates first several views of the object with separate shaded and albedo appearance channels, and then reconstructs colours, metalness and roughness in 3D, using a deferred shading loss for efficient supervision. It also uses a sign-distance distance function to represent 3D shape more reliably and introduces a corresponding loss for direct shape supervision. This implemented using fused kernels for high memory efficiency. After mesh extraction, texture refinement transformer operating in UV space significantly improves sharpness and details. AssetGen achieves 17% improvement in Chamfer Distance and 40% in L.IPS over the best concurrent work for few-view reconstruction, and human preference 72% over the best industry competitors of comparable speed, including those that support PBR. Project page with generated assets: https://asset ievoon..gtti github.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Generating 3D objects from text or image prompts has enormous potential for 3D graphics, with applications animation, gaming and virtual reality. However, while image and video generators have improved dramatically [68 55 97 103 3D generators are not ready yet for professional use. In fact, 3D generators are often slow and produce artifacts in the generated 3D meshes and textures. Many 3D generators, furthermore, "bake" appearance albedo, ignoring how materials respond to variable environmental illumination. This results in visually unattractive outputs, especially for reflective materials, which look out of place when put in novel environments.
In this paper, we introduce Meta 3D AssetGen, significant step-up in text-conditioned 3D generation. AssetGen generates assets in under 30 seconds while outperforming prior methods of comparable speed in faithfulness, in quality of the generated 3D meshes and, especially, in quality and control of materials, by supporting Physicully-Baxer Rendering (PBR) [87] The model generates albedo, metalness, and roughness so that rendered scenes can accurately reflect environmental illumination. In addition, we focus on meshes as the output representation due to their prevalence in applications and compatibility with PBR.
AssetGen uses the two-stage design epitomized by 42 The first stage stochastically generates four images of the object from four canonical viewpoints, and the second stage deterministically reconstructs the 3D shape, appearance and materials of the object from these views (Fig. 1 The two-stage approach faster and more robust than SDS- techniques that perform test-time optimization [68] and, so far, produces better results than single-stage 3D generators [37 6 94 79)
The first question we ask is how this design should be extended support PBR. We show that is difficult for the image-to-3D stage to predict PBR channels from an image as this problem ambiguous and the model deterministic. However, we also show that difficult offload PBR prediction to the text-to-image model; while this is stochastic, which handles ambiguity, the PBR channels are statistically different from the natural images used for pre-training, which makes fine-tuning difficult. Our solution is to give the rext-to-image model the simpler task of outputting shaded appearance and albedo only, and task the image-to-3 stage with inferring the PBR channels from these. This reduces the statistical gap for the text-to-imen model and still removes most of the ambiguity for the image-to-3] model.
We also note that the quality of 3D shapes and meshes is crucial for PBR modelling. Hence, the second question study is how to improve 3D quality. We do so by learning a reconstruction network, MetalLRM, which outputs directly a signed-distanee field (SDF). SDFs are better than opacity fields for meshing, as the zero level set of an SDF traces the object's surface more reliably. Furthermore, the SDF can be directly supervised using ground-truth depth maps, which is not immediately possible for opacity. The crucial contribution here to add SDF support, including the VoISDF [T08] formulation for differentiable rendering, to the memory- efficieet Lightplaane kernels 66 In this way, we can use the stronger SDF representation together with larger batches and photometric loss supervision on high-resolution renders, improving both shapes and textures.
Finally, we note that much of the quality of the final asset depends on texture quality. MetalL_MM textures can still he slightly blurrier than the input image due to the limited resolution of the volumetric representation. The third question we investigate how to maximize the texture quality. To this end, we introduce a new texture refiner network which upgrades the extracted albedo and materials by fusing information extracted from the original views, resolving possible conflicts between them.
We demonstrate the effectiveness of AssetGen on the image-to-31 and text-to-3D tasks. For image-to- 3D, we attain state-of-the-ee art performances among existing few-view mesh-reconstruutton methods when measuring the accuracy of the recover shaded and PBR texture maps. For text-to-3D, we conduct extensive user studies to compare the best methods from academia and industry that have comparable inference time, and outperform them in terms of visual quality and text alignment.
2 Related Work
Text-to-3D. Inspired by text-to-image models, early text-to-3D approaches 33
64
34
26
110
train 3D diffusion models on datasets of captioned 3D assets. Yet, the limited size and diversity of 3D data prevents generalization to open-vocabulaty prompts. Recent works thus pivoted into basing

---

such generators on text-to- imaag models that are trained on billions of captioned images. Among these, works like 77 [66 finetune 2D diffusion models to oulput 3D representations, but the quality is limited due to the large 2D-3D domain gap. Other approaches can be dived into two groups.
The first group contains methods that build on DreamFusion, seminal work by [68], and distill 31> objects by optimizing NeRF via the SDS loss, matching its renders to the belief of a pre-trained text-
to-image model. Extensions have considered: (i) other 3D representations like hash grids [44 69)
meshes 44 and 3D Gaussians (3DGS) 88
12;; (ii) improved SDS [9T] 95 199
30]; ; (iii)
monocular conditioning 69 82 113 78 : (iv) predicting additional normals or depth for better
geometry [70 [78]. Yet, distillation methods are prone to issues such as the Janus effect (duplicating
object parts) and content drift (74). A common solution is to incorporate
cew-consitt priors
into the diffusion model, by either conditioning on cameras [47 73 32 [T 69] or by generating 118 Additionally, SDS optimization is slow and
multiple object views jointly 74 98 93
requires minutes to hours per assets; this issue is partly addressed in 522 101 with amortized SDS.
The second group of methods includes faster two-stage approaches [46 51
49
107
106
g 83
28 233 that start by generating multiple views of the object using a text-to-image or text-to-video
13 tuned to output multiple views of the object followed by per-scene optimization using NeRF 58 or 3DGS (38]. However, per-scene optimization requires several highly-consistent views which are difficult to generate reliably. Instant3D
model [54
42 improves speed and robustness by generating a grid of just four views followed by feed ferwaf network (LRM (291) that reconstructs the object from these. One-2-3-45++4 (45] replaces the LRM with 3D diffusion model. Our AssetGen builds on the Instant31) paradigm and upgrades the LRM to output PBR materials and an to use a SDDFbasedd representation of 3D shape. Furthermore, it starts from grids of four views with shaded and albedo channels, key to predicting accurate 3D shape and materials from images.
3D reconstruction from images. 3D scene reconstruction, in its traditional multi-view stereo (MVS) sense, assumes access to dense set of scene views. Recent reconstruction methods such as NeRF [58] optimize 3D representation by minimizing multi-view rendering losses. There are two popular
classes of 3D representation: (i) explicit representations like meshes [22 114 24 63 59 76 or
3D points/Gaussiaa
[38] 25 and (ii) implicit representations like occupancy fields [65] radiance
fields (88 62 and signed distance functions (SDF) [109]. Compared to occupancy fields, SDF [66]
108 92 17 21 simplifies surface constraints integration, improving scene geometry. For this reason,
we also use SDFs and demonstrate that they outperform occupancy fields.
Sparse-view reconstruction instead assumes few input views (usually 1 8). An approach to mitigate the lack of dense multiple views is to leverage 2D diffusion priors in optimization [55] [9], but this is often slow and fragile. More recently, authors have focused on training feed-forward reconstructors on
large datasets [1 36 57
48 100 60) 194). In particular, [29] trains a large Transformer (89 to predict
NeRF using a triplane representation 7 Followups study 3D representations like meshes [03 97
and 3DGS [120] 105 80 (15). improved backbones [96] 97 and training protocols [86] 35 We
introduce three extensions LRM: (i) an SDF formulation for improved geometry, (ii) PBR material
prediction for relighting, and (iii) a texture refiner for better texture details.
3D modeling with PBR materials. Most 3D generators output 3D objects with baked illumination, 38) or view-independent (29). Since baked lighting ignores the model's response to environmental illumination, is unsuitable for graphics pipelines that simulate lighting. Physically-based rendering (PBR) defines material properties so that a suitable shader can account for illumination realistically. Several MVS works have considered estimating PBR materials using
either view-dependent [58]38]
NeRF 4 3 modelling, 100
[02].
SDF [16). differentiable meshes 63 27 or 3DGS (3 43
In generative 50 104) augment the text-to-3D SDS optimization [68] with a PBR model.
70
Differently from them, we integrate PBR modeling in our feed-forward text-to-3D network, unlocking for the first time fast text-based generation of 3D assets with controllable PBR materials.
3 Method
AssetGen is a two-stage pipeline (Fig. 2 The first stage, text-to-image (Sec. 3.II. maps text to an image grid containing four object views with material information. The second stage, image-to-3D, comprises a novel PBR- sparse-view reconstruction model (Sec. 3.2) and a new texture refiner (Sec. 3.3). (Sec. As such, AssetGin is applicable to two tasks: text-to-3D (stage 1+2) and image-to-3D (stage 2 only).
3

---

imngge
< PromNy
Multte Mo Muuiiiiio
Diffusion Model
Shaded
MLP vas
Alaodo Meudie Rzugh
Voi50F Renderer
Def Shadee
Project to UV Space
Differentiable
Rendering
Def. Shader
Fexture
+
Partial views UV specy
Rendering:
MLP spr
Rendering losses
View
losses using
Albedk
MLP oxeeese
Direct SDF loss
Attention
Mesh Extractor
Presdcnaal Teatree
Beffeed Testur
Triplane
Refiner
Image Triplane Network
Exumacaal Moh
Outgun Mesh
Figure 2: Overview. Given text prompt, AssetGen generates 3D mesh with PBR materials in two stages. The first text-to- -ontageeema stage (blue) predicts 6-channel image depicting 4 views of the object with shaded and albedo colors. The second image-to-3I) stage includes steps. First, 3D) reconstructor (dubbed Metall RM) outputs a triplane-supporte SDF field converted into a mesh with textured PBR materials (orange). Then, PBR materials are enhanced with texture refiner which recovers missing details from the input views (green).
3.1 Text-to-image: Generating shaded and albedo images from text
The goal of the text-to- text-to-image module is to generate several views of the generated 3D object. To this end, we employ an internal text-to-ime to-image diffusion model pre-trained on billions of text-annotated images, with an architecture similar to Emu [16]. Similar to 774
we finetune the model to predict grid of four images 4,, i ..... each depicting the object from canonical viewpoints n, Note that 7; are RGB images of the shaded object. We tried deferring the PBR parameter extraction to the image-to-31 stage, but this led suboptimal results. This due the determinism of the image-to-33 stage, which fails to model ambiguities when assigning materials to surfaces.
natural solution, then, predict the PBR parameters directly in the text-to-image stage. These consists of the albedo po (by which we mean the base color, which is the same as albedo only for zero metalness). the metalness y, and the roughness a. However, we found this to be ineffective too because the metalness and roughness maps deviate from the distribution of natural images making them hard target for finetuning. Our novel solution is train the model generate instead 4-view grid with 6 channels, for the shaded appearance I and 3 more for the albedo Po- This reduces the finetuning gap, and removes enough ambiguity for accurate PBR prediction in the image-to-3D stage.
3.2 Image-to-3D: A PBR-based large reconstruction model
We now describe the image-to-3l stage, which solves the reconstruction tasks given either a small number of views I; (few-view reconstruction), the 4-view 6-channel grid of Sec. 3.1
At the core of our method is new PBR-aware reconstruction model, MetalLRM, RM, that reconstructs the object given N > 1 posed images (4, n. T where 1, E I xW x and E II is the camera viewpoint. As noted in Sec. 3.1 we consider N = canonical viewpoints ,....., T4 (fixed to 20" elevation and 0% 90% 180 270 azimuths) and D 6 input channels. The output is 3D field representing the shape and PBR materials of the object as an SDF IR3 - IR where s(x: is the signed distance from the 3D point to the nearest object surface point, and PBR function k: R + R$, where k(x)= (Po. a)) are the albedo, metalness and roughness.
The key to learning the model is the differentiable rendering operator R. This takes as input a field l: R$ + RD. the SDF 8, the viewpoint म, and pixel u E U [o, W) [0, H), and outputs projection of the field the pixel according to the rendering equation [8), which has the same number channels D as the rendered field E:
S ((x))o(x s)e 1o) dt.
R(u (alle,s (,8,T)
Here æ, = .r 1wo. E [0, oo) the ray that goes from the camera center ao through the pixel along direction -wo E S?. The function a[ is the opacity of the 3D point and is obtained

---

from the SDF value s(x) using the VoISDF [08] formula
0 2
1 sign s(x) (1-e-la(e)i
(2)
where a, 6 are the hyper-parameters. We use Eq. ① to render several different types of fields e, the most important of which is the radiance field, introduced next along with the material model.
Reflectance model. The appearance of the object f(u)) R(u L, 8, R)) in shaded RGB image I is obtained by rendering its radiance field f(x) L(x, w. k, n), where is the field of unit normals. The radiance is the light reflected by the object in the direction wo of the observer (see App. A.8 for details), which in PBR given by:
{ f(wi, Wa k(x)) n(=)) (()))))((,,..)(nnin (n))
L(x, Wo h. n))
wi) dS2,
H(m)
where wo: wi E ((n)==(( E n w....... 2 0} are two unit vectors pointing outside the object and ww) is the radiance incoming from the environment at æ from direction wi in the solid angle dS2, The Bidirectional Reflectance Distribution Function (BRDF) f tells how light received from direction -w, (incoming) is scattered into different directions wo (outgoing) by the object [20).
In PBR, we consider a physically- inspiiee model for the BRDF, striking a balance between realism and complexity 177 90]: specifically, we use the Disney GGX model [], which depends on parameters Pu. 7, and only (see App. 12. for the parametric form of f). Hence, the Metal LRM predicts the triplet k(xx)= = (po, 7, a)) at each 3D point æ.
Deferred shading, In practice, instead of computing I(u)) -Ruu L, S, x) using Eqs. and 3).
we use the process of deferred shading (20):
f(w)= = Raee(u k,
(H(()) f(w,, w. ke (mm (-w;) wi) d52,
where Lew is the environment radiance (assumed to be the same for all x), Te = R(u s, and n = R(u s, T) are rendered versions of the material and normal fields. The advantage of Eq. (4) is that the BRDF f is evaluated only once per pixel, which is much faster and less memory intensive than doing for each 3D point during the evaluation of Eq. ( particularly for shining/backpropagationn During training. furthermore, the environment light is assumed to be a single light source at infinity, so the integral (4) reduces to evaluating a single term.
Training formulation and losses. MetalLRM is thus a neural network that takes as input a set of images (I,, m. ) and outputs estimates and & for the SDF and PBR fields. We train it from a dataset of mesh surfaces M c 3 with ground truth PBR materials k M + RS.
Reconstruction models are typically trained via supervision on renders 29 977. However, physically accurate rendering via Eq. (1) is very expensive. We overcome this hurdle in two ways. First, we render the raw ground-truth PBR fields & and use them to supervise their predicted counterparts with the MSE loss, skipping Eq. (I. For the rendered albedo po which is similar enough to natural images- -
we also use the LPIPS [17 loss:
Sphr = LPIPS R(- po. 5, T), R(- po. M,
M,,)), m)) + R(. 8, R(- k. M,
(5)
We further supervise the PBR field by adding a compuuationalll--efiiiienn deferred shading loss:
Laef = Il C (Rael-- k, 6, T) - R. | k, 1T)
(6)
The weight v(()= nuu)) ru(u) is the dot product of the predicted and ground-truth normals at pixel u. It discounts the loss where the predicted geometry is not yet learnt. Fig. 14 (b) visualizes deferred shading and the rendering loss.
Finally, we also supervise the SDF field with direct loss Lear (implemented as in D., depth-MSE loss Ldepth between the depth renders and the ground truth, and with a binary cross-entropy Lmask between the alpha-mask renders and the ground-truth masks. Refer to App. A.6.2 for more details.
LightPlane implementation. We base MetalL on LightplaneLIM 6 variant of LRM 22]
exploiting memory and compute-efficient Lightppnane splatting and rendering kernels, offering better quality reconstructions. However, since LightplaneL_M uses density fields, which are suboptimal for mesh conversion [92] 666| 1. we extend the Lightplane rendering GPU kernel with VoISDF [108
renderer using Eq. ( Additionally, we also fuse into the kernel the direct SDF loss L since a
naive autograd implementation is too memory- heavy..
5

---

3.3 Mesh extraction and texture refiner
The MetalLRM module of Sec. 3.2 outputs a sign distance function s, implicitly defining the object surface A = {ær E R s a = 0} as level set of s. We use the Marching Tetrahedra algorithm [8] trace the level set and output a mesh M = A. Then, xAtlas [12 extracts a UV map o [0, V > M, mapping each 2D UV-space point
((x) to a point D: E M on the mesh.
Next, the goal is extract high-quality 5-channel PBR texture image K E RVXVx capturing the albedo, metalness, and roughness of each mesh point. The texture image K can be defined directly by sampling the predicted PBR field k as ((v) + k(((v)) but this often yields blurry results due to the limited resolution of MetalLRM. Instead, we design texture refiner module which takes as imput the coarse 7BR-sampled texture image as well as the N views representing the object and outputs a much sharper texture K. In essence, this modules leverages the information from the different views to refine the coarse texture image. The right part of Fig. 2 iillustrates this module.
More specifically, it relies pixel v E [0. 2 of Ko RVXVXII is annotated with the concatenation of the normal, the 3D location, and the output of MetalLRM's PBR field (()()) evaluated at v's 3D point Φ(v). The remaining K1, .1..... KN correspond to partial texture images with 6 channels (for the base and shaded colors) which are obtained by back-projecting the object views to the mesh surface. The network t utilises two U-Nets fuse {K,),² into the enhanced texture K. 4's goal is select, for each UV point v, which of the N input views provides the best information. Specifically, each partial texture image K2 is processed in parallel by a first U-Net, and the resulting information is communicated via cross attention to second U-Net whose goal is to refine Ko into the enhanced texture K. Please refer App. A.7 for further details.
network T which is fed N texture images (K:) First, each
Such a network is trained on the same dataset and supervised with the PBR and albedo rendering losses as M.MalLRM.. The only difference is meshes (whose geometry is fixed) are rendered differentiably using PyTorch3'ss [T] mesh rasterizer instead of the Lightplane SDF renderer.
4 Experiments
Our training data consists of 140,000 meshes of diverse semantic categories created by 3D artists. For each asset, we render 36 views at random elevations within the range of [-30", [.0) at uniform intervals of 30 around the object, lit with randomly selected environment map. We render the shaded images, albedo, metalness, roughness, depth maps, and foreground masks from each viewpoint. The text-to-image stage based on an internal text-to-image model architecturally similar to Emu [16), fine-tuned on subset of 10,000 high-quality 3D samples, captioned by Cap3ID-like The other stage utilizes the entire 3D dataset instead.
pipeline 53 that uses Llama3 [8]
For evaluation, following 105
103 42 we assess visual quality using PSNR and LPIPS 117
between the rendered and ground images. PSNR is computed in the foreground region avoid metric inflation due to the empty background. Geometric quality is measured by the LI error between the rendered and ground truth depth maps (of the foreground pixels), as well as the IoU of the object silhouette. We further report Chamfer Distance (CD) and Normal Correctness (NC) for 20,000 points uniformly sampled on both the predicted and ground- ground-truth shapes. Material decomposition
Table 1: Four-view reconstruction with PBR eval- uating the accuracy of the PBR renders for Metal LRM and ablations. Methods in top / bottom accept 4 views with shaded/ shaded&allbeoo color channels.
Table 2: Win-rate of AssetGen in text-to- 3D user study evaluating visual quality and the alignment between the prompt and the generated meshes. AssetGen beats all base- lines 30 see budget (on an A100 GPU).
LPIPSI albedo
PSNR
Method
albedo
metal rough
Visual quality
Text fidelity
Method
PBR
Lightplane..MM w/ SDF L C=
0.117
17.14
12.39
15.25
F C Material prediction
0.097
20.66
15.99
20.25
GRM 1OS
96.7 %
93.3 %
F == E Deferred shading loss
0.093
21.12
18.64
20.66
Instantessh
103
99.3 %
97.3 %
G F Texture refinement
0.087
21.97
22.19
20.85
LightplaneL RM 6
66.6 %
N/A
F Albedo & shaded input
0.084
23.02
20.43
21.18
Meshy v3 G
94.6 %
91.3 %
H Texture refinement
0.069
24.39
27.28
20.63
Luma Genie 1.0
84
72.3 %
72.8 %

---

is evaluated with LPIPS and PSNR on the albedo image, and PSNR alone for the metalness and roughness channels. All metries are calculated on meshified outputs rather than on neural renders.
4.1 Sparse-view reconstruction
We tackle the sparse-view reconstruc- tion task of predicting a 3D mesh from posed images of an object on subset of 332 meshes from Google Scanned Objects (GSO) [19]. We compare against state-of of-the-art
Table 3: Four-view reconstruction on GSO comparing the appearance and geometry of MetalLRM (outputting baked- light texture) to baselines (top) and ablations (bottom). CD values multiplied by 10-2
Method
LPIPS
PSNRT
Depth.
loUt
CDJ
NC1
Instant3D-LRM [42], GRM [105), In-
Instam3D-LRM 29
0.124
18.54
0.325
0.930
1.630
0.844
stantMesh [03. and MeshLRM [97]
GRM [O05]
0.100
19.87
0.364
0,949
1.490
0.873
We also include LightplaneLRM (6]
InstantMesh 103
0.113
20.63
0.334
0,937
1.364
0.848
an improved version of Instant3D-
MetalI (ours)
0.057
22.49
0.173
0.968
1.137
0.885
LRM. which serves as our base model. MeshLRM (97] has not been open- sourced so we compare only qualita- tively to meshes from their webpage. All methods are evaluated using the same input views at 5122 resolution. Since of the latter predict PBR materials and since GSO lacks ground-truth PBR materials, for fairness, we use a variant of our model that predicts shaded object textures.
Lightplane [6]
0.095
18.60
0.456
0.953
1.313
0.872
B= VolSDF sendering
0.094
20.91
0.201
0.957
1.212
0.875
C= B+ Direct SDF loss
0.083
21.75
0.173
0.968
1.137
0.885
D= C+ Texture refinement
0.057
22.49
0.173
0.968
1.137
0.885
As shown in Figs. and and Tab. our method outperforms all baselines across all metrics. GRM captures texture detail well but struggles with fine geometric structures when meshified. InstantMesh and Lightplane improve geometry but fall short on finer details and texture quality. Our approach excels in reconstructing shapes with detailed geometry and high-fidelity textures.
Ablations Tab. and Fig. [sso that incorporating our scalable SDF-based rendering and direct SDF loss into the base LightplaneLRN model enhances geometric quality. Adding texture refinement further brings fine texture details.
Ouijnats using our LRM using only slanked
dreszed So arroi
Next, we consider the task of sparse-view recon- struction with PBR materials, where the goal is to reconstruct the 3D geometry and texture properties (albedo, metalness, and roughness) from four posed shaded 2D views of an object. This is done on an internal dataset of 256 artist-created 3D meshes, cu- rated for high-quality materials. Since there are no existing few-view feed-forward PBR reconstructors. we conduct an ablation study in Tab. T and Figs. and
Shaded Input
Staded
Albalo
Metallic
Rough
Allbedo
Outpuss using our 1 RR using hoth shudell
albodo imput
Figure 3: Qualitative ablation on albedo generation. In text-to-3D, generating 4 views representing albedo colors alongside shaded RGB colors improves material estimation for our 3D reconstructor. With both inputs, the model accurately predicts the armor metal- lic and smooth, while the bear's fur is rough.
While adding material prediction with additional MLP heads provides some improvements, we ob- serve that incorporating the deferred shading loss and texture refinement is essential for high-qualis PBR decomposition. Example PBR predictions are shown
Fig.
4.2 Text-to-3D generation
Finally, we evaluate text-to-3D with PBR materials. We compare against state-of-the-ae feed-forward methods that generate assets at comparable speed (> 10 to 30 per asset). This includes text-to-3D variants of GRM [OS], InstantMesh [103), and LightplaneI 6). GRM uses Instant3D's 4-view grid generator, InstantMesh receives the first view from our 2D diffusion model and subsequently generates views, while Lightplanel. RRM accepts views from our grid generator. Since these methods bake lighting instead of generating PBR materials, for evaluation apply flat texture
7

---

Inpur Viewa
GRA
InstureMeshh
Lightplanet.no RMM
Ours unseetee
Ours
Ground Truth
hE AOONS
Figure 4: Qualitative comparison for sparse-view reconstruction. AssetGen gives better geometry (shown in orange) and higher fidelity texture (inset) compared state of the art. SDF representation along with the direct SDF loss gives a better geometry compared to the base LightplaneLRN model which occupancy (row and 5). Furthermore, our texture refiner greatly enhances texture fidelity (row 5 and 6).

---

"A pug made of metal"
Albedo
Metallic
Albedo
Metallic
Rough
Rough
GRM
InstantMesh Lightplancl
Luma Genie
Ours
"An old vintage car"
GRM
InstantMesh LightplaneLRM
Luma Genie
Ours
"A fresh einnamon roll covered in glaze"
GRM
InstantMesh
LightplaneLRM
Luma Genie
Ours
Figure 5: Qualitative comparison for text-to-3D. We compare 3D meshes generated by Meta 3D AssetGen and state-of-the-art baselines. We include material decomposition for methods producing PBR materials (Luma Genie and our Meta 3D AssetGen). Our approach produces higher quality materials with better-defined metalness and roughness, and a more accurate decoupling of lighting effects in the albedo.
shading to our outputs. Additionally, we compare with the preview stage of Meshy LumaAl Genie 1.0 (84. proprictary text-to-3D methods with PBR workflow capable of creating assets within 30 and 15 s respectively. A comparison with the significantly longer refinement stages for Luma and Meshy is provided in the appendix. Fig. 5 shows that AssetGen meshes are visually more
855 and
appealing and have meaningful materials Figs. 6 and 12 provide more examples and comparisons and showcase fine-grained material control.
For quantitative evaluation, we conducted an extensive user study in Tab. 2 Zluiing the 404 deduplicated text prompts from DreamFusion (68). Users were shown 360" videos of the generated and baseline meshes and were asked to rate them based on 3D shape quality and alignment with the text prompt. A total of 1,080 responses were collected, with significant preference for AssetGen's 's meshes.
Finally, we ablate the effect of generating dual-channel albedo+shaded grids compared albedo-only input in Fig. 3 revealing significant PBR decomposition superiority of the former. Additionally, Fig. 13 illustrates the effect of our deferred shading loss.
5
Conclusions
We have introduced Meta 3D AssetGen, significant advancement in sparse-view reconstruction and text-to-3D, Meta 3D AssetGen can generate 3D meshes with high-quality textures and PBR materials faithful to the input text. This uses several key innovations: generating multi-view grids with both shaded and albedo channels, introducing new reconstruction network that predicts PBR materials from this information, using deferred shading to train this network, improving geometry via a new scalable SDF-based renderer and SDF loss, and introducing new texture refinement network. Comprehensive evaluations and ablations demonstrate the effectiveness of these design choices and ofate-of-the-ar performance.

---

"a red-eyed el-ee tree frog"
"a wedge of cheese on silver platter" "
Geometry
Albedo
Geoneity
Albedo
Metalness
Roughness
Metalmess
Roughness
"an amigurumi bulldozer"
"a bichon frise wearing academic regalia"
Gcommtry
Alleddo
Geometry
Alhedd
Metalness
Roughness
Metalness
Roughness
"a train engine made of clay"
"a marble bust of mouse" "
Geometry
Alhedo
Genmetry
A Abedlo
Metalness
Roughness
Metalness
Roughness
"a nest with white eggs and one golden egg"
"a toilet made out of gold"
Geometry
Albedo
Geornetry
Albedo
Metalness
Roughness
Metalness
Roughness
Figure 6: Text-to-3D meshes generated by Meta 3D AssetGin along with their PBR decomposition. Note that Meta 3D) AssetGen provides detailed albedo and material properties, as highlighted by the metalness of the platter (top right) and the golden objects (last row).
10

---

References
[1] Dejan Azinović, Ricardo Martin- Brrtin- Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 6290-630) June 2022.
[2] P. Beckmann and A. Spizzichino. The Scattering of Electromagnetic Waves from Rough Surfaces. Pergamon Press, 1963.
[3] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch. NeRD: Neural Reflectance Decomposition from Image Collections. In 202 IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[4] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan T. Barron, and Hendrik P. A. Lensch. Neural-PIL: Neural Pre- Pre-Integrated Lighting for Reflectance Decomposition. arXiv preprint, 2021.
[5] Brent Burley. Physically-basee based shading at disney. Technical report, Disney, 2012.
[6] Ang Cao, Justin Johnson, Andrea Vedaldi, and David Novotny. Lightplane: Highly-scalable components for neural 3d fields. arXiv, 2024.
[7] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J. Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-awan 3D generative adversarial networks. In Proc. CVPR, 2022.
[8] Eric Chan, Koki Nagano, Matthew Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3D-aware diffusion models. Proc. ICCV, 2023.
[9] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial radiance fields. In arXiv,, 2022.
[10] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3D: Disentangling geometry and appearance for high-quality text-to-3D content creation: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv.es, abs/2303. 13873, 2023.
[11] Yabo Chen, Jiemin Fang, Yuyang Huang, Taoran Yi, Xiaopeng Zhang, Lingxi Xie, Xinggang Wang. Wenrui Dai, Hongkai Xiong, and Qi Tian. Cascade- Zero 123: One image to highly consistent 3D with self self-prompted nearby views. arXiv.cs htd//211..04244 2023.
[12] Zilong Chen, Feng Wang, and Huaping Liu. Text-to-3D using Gaussian splatting. arXiv, 2309.16585, 2023.
[13] Zilong Chen, Yikai Wang, Feng Wang, Zhengyi Wang, and Huaping Liu. V3D: Video diffusion models are effective 3D generators. arXiv, 2403.06738. 2024.
[14] Christopher B. Choy, Danfei Xu, Young Gwak, Kevin Chen, and Silvio Savarese. 3D-R2N2: A unified approach for single and multi-view 3D object reconstruction. In Proc. ECCV, 2016.
[15] Robert L. Cook and Kenneth E. Torrance. reflectance model for computer graphics. In Doug Green, Tony Lucido, and Henry Fuchs, editors, Proc. SIGGRAPH, 1981.
[16] Xiaoliang Dai, Ji Hou, Chih-Yao Ma. Sam S. Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, Matthew Yu, Abhishek Kadian, Filip Radenovic, Dhruv Mahajan, Kunpeng Li, Yue Zhao, Vladan Petrovic, Mitesh Kumar Singh, Simran Motwani, Yi Wen. Yiwen Song, Roshan Sumbaly, Vignesh Ramanathan, Zijian He, Peter Vajda, and Devi Parikh. Emu: Enhancing image generation models using photogenic needles in haystack. CoRR, abs/2309. 2023.
[17] François Darmon, Bénédicte Bascle, Jean-Clément Devaux, Pascal Monasse, and Mathieu Aubry. Improving neural implicit surfaces geometry with patch warping. Proc. CVPR, 2022.

---

[18] Akio Doi and Akio Koide. An efficient method of triangulating equi-valued surfaces by using tetrahedral cells. IEICE TRANSACTIONS on Information and Systems, 74(1):214- 1991.
[19] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B. McHugh, and Vincent Vanhoucke. Google Scanned Objects: A high- quality dataset of 3D scanned household items. In Proc. ICRA, 2022.
[20] James D. Foley, Andries van Dam, Steven Feiner, and John F. Hughes. Computer graphics principles and practice, 3nd Edition. Addison- Wesley, 2013.
[21] Qiancheng Fu, Qingshan Xu, Yew-Soon Ong, and Wenbing Tao. Geo-Neus: Geometry- Consistent Neural Implicit Surfaces Learning for Multi-view Reconstruction. In NeurIPS, 2022.
[22] Jun Gao, Wenzheng Chen, Tommy Xiang, Clement Fuji Tsang, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Learning deformable tetrahedral meshes for 3D reconstruction, In Proc. NeurIPS, 2020.
[23] Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Be Martin-Brualla, Pratul Srinivasan, Jonathan T. Barron, and Ben Poole. CAT3D: Create Anything in 3D with Multi-View Diffusion Models. arXiv.es, 2024.
[24] Shubham Goel, Georgia Gkioxari, and Jitendra Malik. Differentiable Stereopsis: Meshes from multiple views using differentiable rendering. In CVPR, 2022.
[25] Antoine Guédon and Vincent Lepetit. SuGaR: Surface-aligned Gaussian splatting for efficient
3D mesh reconstruction and high-quality mesh rendering. arXivves, abs/2311.12775, 2023.
[26] Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz. 3DGen: Triplane latent diffusion for textured mesh generation. corr, 0as/2333.053777 2023.
[27] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, Light, and Material De- composition from Images using Monte Carlo Rendering and Denoising. arXiv preprint, 2022.
[28] Lukas Höllein, Aljaž Božič, Norman Müller, David Novotny, Hung- Yu Tseng, Christian Richardt, Michael Zollhöfer, and Matthias Nießner. ViewDiff: 3D-Consistent Image Genera- tion with Text-to-lmage Models. arXiv preprint, 2024.
[29] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: Large reconstruction model for single image to 3D. In Proc. ICLR, 2024.
[30] Yukun Huang, Jianan Wang, Yukai Shi, Xianbiao Qi, Zheng-Jun Zha, and Lei Zhang. Dreamtime: An improved optimization strategy for text-to-3D content creation. CoRR, abs/2306.12444 2023.
[31] Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Picter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. CVPR, 2022.
[32] Yifan Jiang, Hao Tang, Jen-Hao Rick Chang, Liangchen Song, Zhangyang Wang, and Lian- gliang Cao. Efficient-3Dim: Learning a generalizable single-image novel-view synthesizer in one day. arXiv, 2023.
[33] Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao. Xiaoxiao Long. Wenping Wang, and Yuexin Ma. GaussianShader: 3D Gaussian splatting with shading functions for reflective surfaces. arXiv.es, abs/2311.1797., 2023.
[34] Heewoo Jun and Alex Nichol. Shape-R: Generating conditional 3D implicit functions. arXiv, 2023.
[35] Philip Torr Junlin Han, Filippos Kokkinos. Vfusion3d: Learning scalable 3d generative models from vidco diffusion models. arXiv preprint, 2024.
12

---

[36] Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik. Learning category- specific mesh reconstruction from image collections. Proc. ECCV, 2018.
[37] Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy Mitra. HoloDiffusion: training 3D diffusion model using 2D images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[38] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian Splatting for real-time radiance field rendering. Proc. SIGGRAPH, 42(4), 2023.
[39] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Popa Tiberiu. Clip-mesh: Generating textured meshes from text using pretrained image-text models. SIGGRAPH Asia 2022 Conference Papers, December 2022.
[40] Seungwook Kim, Yichun Shi, Kejie Li, Minsu Cho, and Peng Wang. Multi-view image prompted mueti-view diffusion for improved 3D generation. arXiv, 2404.17419, 2024.
[41] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proc. ICLR, 2015.
[42] Jiahao Li, Hao Tan, Kai Zhang. Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3D: Fast text-to-3D with sparse-view generation and large reconstruction model. Proc. ICLR, 2024.
[43] Zhihao Liang, Qi Zhang, Ying Feng. Ying Shan, and Kui Jia. GS-IR: 3D Gaussian splatting for inverse rendering. arXiv.cs, abs/2311.164444 2023.
[44] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung-Yi Yi Lin. Magic3D: High-resolution text-to-3D content creation. arXives, abs/2211. 2022.
[45] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45++: Fast single image 3D objects with consistent multi-view generation and 3D diffusion. arXiv.cs, abs/2311 1. 1.07885, 2023.
[46] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3D mesh in 45 seconds without per-shape optimization. In Proc. NeurIPS, 2023.
[47] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: Zero-shot one image to 3D object. In Proc. ICCV, 2023.
[48] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3D reasoning. arXivess, abs/ 904.01786, 2019.
[49] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. SyneDremmer: Generating multiview-consistem images from a single-view image. arXiv, 2309.03453, 2023.
[50] Zexiang Liu, Yangguang Li, Youtian Lin, Xin Yu. Sida Peng, Yan-Pei Cao, Xiaojuan Qi, Xiaoshui Huang, Ding Liang, and Wanli Ouyang. UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generation. arXiv preprint, 2023.
[51] Xiaoxiao Long, Yuanchen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Hai Zhang, Mare Habermann, Christian Theobalt, and Wenping Wang. Wonder3D: Single image to 3D using cross-domain difffusio.. arXiv.cs,
Ma, Song-Hai
2023.
[52] Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Sharp, Tsung-Yi Yi Lin, Ming-Yo Yu Liu, Sanja Fidler, and James Lucas. ATT3D: amortized text-to-3I object synthesis. In Proc. ICCV, 2023.
Lin, Towaki Takikawa, Nicholas
[53] Tiange Luo, Chris Rockwell, Honglak I Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. arXiv preprint, 2023.
13

---

[54] Luke Mlelas-Krrian Iro Laina, Christian Rupprecht, Natalia Neverova, Andrea Vedaldi, Oran Gafni, and Filippos Kokkinos. IM-3D: Iterative multiview diffusion and reconstruction for high-quality 3D generation. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
[55] Luke Melas--Kyrizz Christian Rupprecht, Iro Laina, and Andrea Vedaldi. RealFusion: 360 reconstruction of any object from single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ( (CVPR), 2023.
[56] Antoine Mercier, Ramin Nakhli, Mahesh Reddy, and Rajeev Yasarla. HexaGen3D: Stabledif- fusion is just one step away from fast and diverse text-to-3D generation. arXiv, 2024.
[57] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy Networks: Learning 3D Reconstruction in Function Space. In CVPR. 2019.
[58] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proc. ECCV, 2020.
[59] Tom Monnier, Jake Austin, Angjoo Kanazawa, Alexei A. Efros, and Mathieu Aubry. Dif- ferentiable blocks world: Qualitative 3d decomposition by rendering primitives. arXiv, abs/2307.05473. 2023.
[60] Tom Monnier, Matthew Fisher, Alexei A. Efros, and Mathieu Aubry. Share With Thy Neigh- bors: Single-Vier View Reconstruction by
Consistency. In ECCV, 2022.
[61] Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, and Matthias Nießner. DiffRF: Rendering-guidee 3D radiance field diffusion. In Proc. CVPR, 2023.
[62] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. In Proc. SIGGRAPH, 2022.
[63] Jacob Munkberg, Wenzheng Chen, Jon Hasselgren, Alex Evans, Tianchang Shen, Thomas Muller, Jun Gao, and Sanja Fidler. Extracting Triangular 3D Models, Materials, and Lighting From Images. In CVPR, 2022.
[64] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-E: A system for generating 3D point clouds from complex prompts. arXiv.cs, abs/2212.08751,
2022.
[65] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision. In CVPR, 2020.
[66] Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: unifying neural implicit surfaces and radiance fields for multi-view reconstruction. arXiv.es, abs/2104.10078, 2021.
[67] OpenAl. Triton: Open-source gpu programming for neural networks. https://githu: com/triton-lange Laang/trtoonn
[68] Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D diffusion. In Proc. ICLR, 2023.
[69] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem. Magic 123: One image high-quality 3D object generation using both 2D and 3D diffusion priors. arXiv.cs, abs/2306. 17843, 2023.
[70] Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo. Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3D, arXiv.es, abs/2311.16918,, 2023.
14

---

[71]
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan- Yen Lo. Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv, 2020.
[72] Christophe Schlick. An inexpensive BRDF model for physically-based rendering. Comput. Graph. Forum, 13(3), 1994.
[73] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. 22ro123++: a single image to consistent multi-view diffusion base model. arXiv.cs, abs/2310.15110, 2023.
[74] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3D generation. In Proc. ICLR, 2024.
[75] J. Ryan Shue, Fric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3D neural field generation using triplane diffusion. arXiv.es, abs/2211 11122211.1666 1.16677, 2022.
[76] Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, and Matthias Nießner. MeshGPT: Generating triangle meshes with decoder-only transformers. arXiv.cs, abs/2311. 2023.
[77] B. Smith. Geometrical shadowing of a random rough surface. IEEE Trans. on Antennas and Propagation, 15(5), 1967.
[78] Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. DreamCraft3D: Hierarchical 3D generation with bootstrapped diffusion prior. arXiv.es, abs/2310.1681 2023.
[79] Stanislaw Szymanowicz. Christian Rupprecht, and Andrea Vedaldi. Viewset diffusion: (0- jimage-conditioned 3D generative models from 2D data. In Proceedings of the International Conference on Computer Vision (ICCV), 2023.
[80] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. LGM: Large multi-view Gaussian model for high-resolution 3D) content creation. arXiv, 2402.05054, 2024.
[81] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. DreamGaussian: Genera- tive gaussian splatting for efficient 3D content creation. arXiv, 2309. 16653, 2023.
[82] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-It-3D: High-fidelity 3d creation from A single image with diffusion prior. arXiv.cs, abs/2303.14188 14184, 2023.
[83] Shitao Tang, Jiacheng Chen, Dilin Wang, Chengzhou Tang, Fuyang Zhang, Yuchen Fan, Vikas Chandra, Yasutaka Furukawa, and Rakesh Ranjan. MVDiffusion++: A dense high- resolution multi-view diffusion model for single or sparse-view 3d object reconstruction. arXiv, 2402. 12712, 2024.
[84] Luma Team. Luma genie 1.0. Inttps://www luma ai. www.commmmaaa. -awuuaa-geiee-0///
[85] Meshy Team. Meshy AT 3D Model Generator with pbr materials meshy.ai.
https
/ wwww.meshy.co
[86] Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. TripoSR: fast 3DD object reconstruction from single image. arXiv, 2403.02151, 2024.
[87] K. E. Torrance and E. M. Sparrow. Theory for off-specular reflection from roughened surfaces.
J. Opt. Soc. Am., 57(9), 1967.
[88] Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, D. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, A. Hartshorm, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez,
15

---

Madian Khabsa, Isabel M. Kloumann, A. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, liyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv, 00//2307.09288 7 2023.
[89] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. NeurIPS, 2017.
[90] Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. Microfacet models for refraction through rough surfaces. In Proc. Eurographics, 2007.
[91] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich. Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation. In CVPR, 2023.
[92] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv.cs, abs/2106. 10689, 2021.
[93] Peng Wang and Yichun Shi. ImageDream: Image-prompt multi-view diffusion for 3D genera- tion. In Proc. ICLR, 2024.
[94] Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, and Baining Guo. Rodin: A generative model for sculpting 3D digital avatars using diffusion. In Proc. CVPR, 2023.
[95] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lificDreamer: High-fidelity and diverse text-to-3D generation with variational score distillation. arXivaes, abs/2305.162. 2023.
[96] Zhengyi Wang. Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. CRM: Single image 3D textured mesh with convolutional reconstruction model. arXiv, 2403.05034, 2024.
[97] Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. MeshI.M: large reconstruction model for high-quality mesh. arXiv, 2404. 12385, 2024.
[98] Haohan Weng, Tianyu Yang. Jianan Wang, Yu Li, Tong Zhang, C. L. Philip Chen, and Lei Zhang. Consistent Improve consistency for one image to 3D object synthesis. arXiv, 2023.
[99] Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park. Ruiqi Gao, Daniel Watson, Pratul P. Srinivasan, Dor Verbin, Jonathan T. Barron, Ben Poole, and Aleksander Holynski. ReconFusion: 3D Reconstruction with Diffusion Priors. arXiv preprint, 2023.
[100] Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi. Unsupervised learning of probably symmetric deformable 3D objects from images in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[101] Kevin Xie, Jonathan Lorraine, Tianshi Cao, Jun Gao, James Lucas, Antonio Torralba, Sanja Fidler, and Xiaohui Zeng. LATTE3D: Large-scale amortized texr-to-enhanced3D synthesis. In arXiv, 2024.
[102] Zhang Xiuming, Srinivasan Pratul P., Deng Boyang, Debevec Paul, Freeman William T., and Barron Jonathan NeRFactor: neural factorization of shape and reflectance under an unknown illumination. In Proc. SIGGRAPH, 2021.
16

---

[103] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. In- stantMesh: efficient 3D mesh generation from a single image with sparse-view large recon- struction models. arXiv, 2404.07191, 2024.
[104] Xudong Xu, Zhaoyang Lyu, Xingang Pan, and Bo Dai. MATLABER: Material-Aware Text-to- 3D via LAtent BRDF EncodeR. arXiv preprint, 2023.
[105] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang. Sida Peng, Yujun Shen, and Gordon Wetzstein. GRM: Large gaussian reconstruction model for efficient 3D reconstruction and generation. arXiv, 2403.14621 2024.
[106] Jiayu Yang, Ziang Cheng, Yunfei Duan, Pan Ji, and Hongdong Li. ConsistNet: Enforcing 3D consistency for multi-view images diffusion. arXiv.cs, abs/2310.10343, 2023.
[107] Yunhan Yang, Yukun Huang, Xiaoyang Wu, Cuan-Chen Guo, Song-Hai Zhang, Hengshuang Zhao, Tong He, and Xihui Liu. DreamComposer: Controllable 3D object generation via multi-view conditions. arXiv.es, abs/2312.03611 2023.
[108] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. arXiv.cs, abs/2106. 12052, 2021.
[109] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In Proc. NewrIPS, 2020.
[110] Lior Yariv, Omri Puny, Natalia Neverova, Oran Gafni, and Yaron Lipman. Mosaic-SDD for 3D generative models. arXiv.cs, abs/2312.092222, 2023.
[111] Taoran Yi, Jiemin Fang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. GrussianDreamer: Fast generation from text to 3D gaussian splatting with point cloud priors. arXiv.cs, abs/23 2023.
[112] Jonathan Young. Xatlas: Mesh parameterization / uv unwrapping library, 2022. GitHub repository.
[113] Wangbo Yu, Li Yuan, Yan-Pei Cao, Xiangjun Gao, Xiaoyu Li, Long Quan, Ying Shan, and Yonghong Tian. HiFi-123: Towards high-fidelity image to 3D content generation. arXiv.cs, abs/2310.06744, 2023.
[114] Jason Y. Zhang, Gengshan Yang, Shubham Tulsiani, and Deva Ramanan. NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild. NeurIPS, 2021.
[115] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. GS-LRM: large reconstruction model for 3D Gaussian splatting, arXiv, 19702, 2024.
[116] Kai Zhang, Fujun Luan, Qianqian Wang. Kavita Bala, and Noah Snavely. PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting. arXiv preprint, 2021.
[117] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proc. CVPR, pages 586-595, 2018.
[118] Xiaoyu Zhou, Xingjian Ran, Yajiao Xiong, Jinlin He, Zhiwei Lin, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang. GALA3D: Towards text-to-3I complex scene generation via layout- generative gaussian splatting. arXivecs abs/2402.002077. 2024.
[119] Junzhe Zhu and Peiye Zhuang. HiFA: High-fidelity text-to-3D with advanced diffusion guidance. CoRR, abs/2305. 18766, 2023.
[120] Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane meets Gaussian splatting: Fast and generalizable single-view 3D reconstruction with transformers. arXivees abs/2312.09147, 2023.
17

---

A Appendix
A.1 Societal Impact
Safeguards should be implemented to prevent abuse, such as filtering input text prompts and detecting unsafe content in generated 3D models. Additionally, our generation process may be vulnerable to biases present in the data and 3D models it relies on, potentially perpetuating these biases in the generated content. Despite these risks, our method can augment the work of artists and creative complementary tool to boost productivity. It also holds the potential
professionals by serving to democratize 3D content creation, making it accessible to those without specialized knowledge or expensive proprietary software.
A.2 Limitations
Meta 3D AssetGen significantly advances shape generation but faces several limitations. Despite the fine-tuning of the multiview image grid generator for view consistency, it is not guaranteed, potentially impacting 3D reconstruction quality. Since we use an SDF as an underlying representation, the reconstructor may incorrectly model translucent objects thin structures like hair fur. Additionally, while our scalable Triton [67] implementation supports triplane representation at resolution of 128 x 128, this representation inefficient, as much of its capacity used for empty regions. Future work could explore scalable representations such as octrees, sparse voxel grids, and hash-base methods, which may remove the need for separate texture enhancement model. We also only predict albedo, metalness and roughness, and not emissivity or ambient occlusions. Finally, our method has only been tested on object-level reconstructions, leaving scene-scale 3D generation for future research.
A.3 Additional qualitative comparisons
This section describes additional qualitative comparisons that, due to limited space, could not be included in the main paper. Firstly, please refer to the video attached in the supplementary material which provides holistic presentation highlight the contributions of MetalLRM in geometry, texture and material reconstruction. In Fig. 12] we visualize the control of materials provided by Meta 3D AssetGen, i.e, metalness and roughness, by changing the text prompt for the same concept. Fig. & visualizes the renders of the material maps extracted with Metall given four input test views. In Fig. 9 we provide more extensive qualitative comparison MeshL.M,, the strongest few-view reconstruction baseline. Finally, Fig. 6 provides gallery of text-conditioned generations depicting Blender-shaded renders together with the rendered PBR maps.
Meta 3D AssetGen's 's qualitative results. In Fig. 7 we
A.4 User-study details
As described in Sec. 4.2 we conducted an user study on 404 meshes generated using the DreamFu- sion (68] prompt-set on a standard crowdsoureing marketplace. the study, users were shown 360
Ligherberi
Alboda
Metaliic
Rough
Improved Geometry
Enhanced & Relightable Texture
Material Decomposition
MetalLRM builds upon LightplaneL.RM 6 providing improved geometry by employing SDF representation, along with direct scalable losses in 3D, improved texture using UV space texture refiner, and material decomposition by predicting material properties regularized through novel deferred shading loss.
Figure 7:
18

---

Shaded
Albodo Metallic Rough
Shaded Albedo Metallic Rough
GT
Input Views
Input Views
I
Recons.
Recons.
Shaded
Albedo Metallic Rough
Shaded
Albedo Metallic Rough
Input Views
GT
Input Vieus
GT
Recons.
Shaded Albodo Metallic Rough
Shaded
Albedo Metallic Rough
Input Views
Joput
Recons.
Recons.
Figure 8: Sparse view reconstruction with intrinsic decomposition. Here the MetalLRM takes
shaded views as input and reconstructs the 3D object along with it's albedo, metallic and roughness properties.
Input Views
MeshLRM
Ours
Ground Truth
Figure 9: Qualitative comparison on the task of sparse view reconstruction against MeshLRM [97]. Note the higher quality texture detail in results. Since open-source implementation of MeshLRM has not yet been released, we compare against the meshes provided on their webpage.
videos of the generated and baseline meshes and were asked to rate them based on 3D shape quality and alignment with the text prompt shown in Fig. [I] They were asked to consider various factors like identity (whether the object matches what is described in the prompt), texture, existence of Janus problems, and bad geometry (like floaters, disconnected components, etc). 11,080 responses were collected in total, with 5 responses per pair of videos, to eliminate variance in user preference.
A.5 Additional text-to-3D comparisons
While Tab. 2 compared Meta 3D As- setGen's 's text-to-3D generations to sev- eral fast baselines, for completeness, this section includes additional comparisons to significantly slower methods. More specifically, we conduct the same user study in Tab. but we compare to "refinement" stages of the industry base- lines Meshy v3 and Luma Genie whose asset generation time is 5 and 10 min re-
Table 4: Win-rate of Meta 3D AssetGen in text-to-3D study evaluating visual quality and text alignment of the generated meshes. In addition to Tab, 2 here we compare slower baselines. Our Meta 3D AssetGen generates a 3D asset within 30 sec.
Method
Visual quality
lidelity
Meshy v3, refined
77.5
80.9
300 sec
Luma Genic 1.0, refined (hi-res) DEL
51.2
46.3
600
19

---

RichDreamer
Unique3D
Stable Fast 3D
Luma Stage
Luma Stage II
Ours
'A pug madde of mestal"
RichDreamer
Unique3D
Stable Fast 3D
Luma Stage
Luma Stage II
Ours
*An olel vintage car"
*A fresh chayanion aonoon rpll
RichDreamer
Unique3D
Stable Fast 3D
Luma Stage
Luma Stage II
Ours
coverea La gLaze"
Harbe
Maiee
Madie
Walle
Figure 10: Comparison against RichDreamer, Unique3D, the very recently released Stable Fast 3D Mesh, and Luma Stage 2 (Refinement). RichDreamer takes around hour per mesh, whereas we generate mesh in less than seconds with significantly better PBR materials. RichDer struggles to separate lighting effects from albedo and generates suboptimal geometry. Unique3D also produces inferior geometry compared to ours and cannot generate PBR materials. Stable Fast 3D Mesh predicts only a single value for metallicity and roughness instead of generating map. It tends produce suboptimal geometry and flat objects, as with the and the pug respectively. Luma Stage 2 takes around minutes, generates much better textures than Luma Stage 1, but still struggles to separate illumination from albedo, as evidenced by the car hood.
PLEASE ANSWER BOTH QUESTIONS BELOW.
PLEASE ANSWER THE QUESTION BELOW.
Mur twrf ALIONS more with the olien textorenct cocrdin to fector foxt to thhunessl:
With
reddened
that
accondnng so the 3 factor (ciaal charpeocs and siual concapect:
elephant
Video 1:
Video
Video
Video
Queation video pTate??
Vew
Wew
Figure 11: User study interface submitted to standard crowdsoureing marketplace. Participants are shown videos corresponding to Meta 3D AssetGen and baseline in a random order, and asked their preference in terms either quality (left) or faithfulness to the text prompt (right).
20

---

spectively. Tab, Lab..ooo contains the results of our user-study. Meta 3D AssetGen significantly outperforms Meshy in both fidelity and visual quality while being 10x faster. Surprisingly, Meta 3D AssetGen is on par with Luma Genie in text fidelity and wins in 40% of cases in visual quality. This is remarkable result considering Meta 3D AssetGen's x better generation time. We show further qualitative comparisons in Fig. TO
A.6
Additional technical details
A.6.1 Grid Generator
We employ a text-to-image diffusion model pre-trained on billions of images annotated with text [16 and expand its input and output channels by factor of 2 to support simultaneous generation of shaded appearance and albedo. We finetune the model to predict grid of four images 4, i 1..... 4, in similar fashion [74] 42 via minimization of the standard diffusion loss. Training spans total of days, employing 32 A 100 GPUs with a total batch size of 128 and a learning rate of 10-5.
A.6.2 MetalLRM
As mentioned the main paper, MetalLRM is optimized using the direct SDF loss Cuti, PBR loss phr. deferred shading loss Laet, the binary cross-entropy mask loss Lmask. and the depth-MSE loss Leeph so the global objective is:
L = 0.5L,ar + Lphr + 0.5 + 0.1.mask + 0.1 Ldepth-
The texture refiner uses only PBR loss and the deferred shading loss: Cphr 0.5
In each training batch, we randomly sample 4 views per scene as source input views I, and another target views n into which render the sdf-field predicted by MetalLRM, the mesh predicted by the texture refiner. We then evaluate the aforementioned losses in the target views. scenes per GPU are sampled randomly, and we train 64 GPUS NVIDIA A100 gpus, yielding effective batch size of 3 x x 64 = 768 images. The total loss has been optimized using Adam [41] with learning rate 10-4 for 13K steps.
A.6.3 Deferred shading loss ablation
Messile
deferred w/ deferred
w/o deferred deferred
Rough
Metainess
Shaded
"shiny plastis"
"silver
Metallic
Roughness
Albedo
"roole"
"rusted irew"
Figure 12: Generated assets for the prompt: "A cat made of <MATERIAL>". Meta 3D Asset- Gen predicts various plausible PBR material maps leading to realistic interaction with the environment light (sphere-mapped in the cen- ter)
Figure 13: Using a deferred shading loss on ren- dered channels enhances PBR quality, resulting in more defined metalness and roughness, such as in- creased metalness in the lantem's 's metal parts and decreased roughness in glass parts.
21

---

Besides verifying quantitatively the benefits of the deferred shading loss Lat in Tab. [ we also provide qualitative proof in Fig. [3] Specifically, the PBR materials predicted from albedo&shadeed channels exhibit better metalness map on the actual metallic parts of the 3D lantern asset.
A.6.4 Direct SDF loss Ladr
We follow Azinovic et al.'s [[] direct SDF supervision for the SDF field. Given a pixel p in an image and the sampled points Sp on the ray corresponding to the pixel, the direct SDF loss is computed as
(p) = Cr (p) + (p)
(7)
C is a 'free-space" objective, which forces the MLP to predict value of for samples E Sp which lie between the camera origin and the truncation region of surface:
C (p)
(D.-1) SESS
(8)
where D, the predicted SDF from the MLP. For samples within the truncation region (s Sp), we apply - the signed distance objective of samples close the surface.
L (p)
Σ (D2 - D2)² #ESY
naive PyTorch implementation of this is memory intensive, because of the evaluation of B x H x W x Newy points, where B, H, W, Nray are the number of target images in batch, the height, width, and the number of points per ray respectively. Therefore, to support large batch sizes, image resolution, and denser point sampling on rays, we implement the direct SDF loss using custom Triton 67 kernels.
A.6.5 Depth loss Ldepth
The depth loss Ldepth minimizes the mean- mean-squared error between the rendered depth prediction Rdcptl 8, IT) the ground-truth depth Rdspth (- M, .)
|Rdepts (- | x) Recpi(( (- M, nT)
where Respeh (s, X) is operator rendering the depth-map of the shape representation (mesh or an SDF) from the viewpoint π.
A.7 Texture refiner
Having described a high-level overview of our texture refiner in Sec. 3.3 details.
here we provide more
As mentioned, the texture refiner network & accepts N + texture images K; in total. The first input to the network is the augmented texture image Ko E RVVVx11 given by:
Ve E [0, v3: Ko(v)= = {6, 0,
(k(a.))) @ nxx))) @ Bay if e E Im((),
where x, = @
otherwise.
The condition E Im()) selects 'valid' UV points that correspond to mesh points; and @ denotes channel-wise concatenation, so that Ko(v) is the stack of the 5 PBR parameters k(x,) from Metal LRM, normal n(xx)) and the 3D point T...
In addition Kú, we input to the network texture images K,, each extracted by looking up information from the corresponding input view I, directly (thus sidestepping MetalLRM), As noted above, each valid texture point corresponds to a unique 3D point C. Φ(v)
on the mesh (z(2,) in the image 7, Let (v) E {0, 1) be the flag that tells
which in turn projects to a pixel point a is vixible in image 7 or not. When point a, is visible several views, it best measured in the most frontal one, which is captured by the cosine w. n²xx,) between the normal n at æ
22

---

Deferred Shaded GT Chunnels
Query
Key & Value
MultiHead
Attention
Deferee Shaded Predicted Chanmels
Cosine Similarity of Normal Maps
Features from partial views in UV
(a) Cross Attention Layer
(b) Deferred Shading Loss Example
Figure 14: (a) Illustration of Cross-Vice Attention. Cross-vites view attention facilitates communication between the UNet branches processing the predicted texture features and the UV space projected input views. This layer blends the predicted texture features with the UV projected input view features based their match using multiheaded attention mechanism. (b) Example of Deferred Shading Loss Calculation. Deferred shading computes pixel shading using albedo, metalness, roughness, normals, object position, and light source position. We apply it to both the ground truth channels (top) and the predicted channels (middle). The error is calculated as the difference between the two, weighted by the similarity between ground truth normals and predicted normals, avoid penalizing shading errors due to incorrect normals.
and the ray direction w. æo Z.. All this information is packed into additional texture images K; E (V++) by setting:
Ee 0, vJ:: K,(o) = [((T(()))) @ (w. n(xx)), if E Im( and X²(u) = 1,
0,
otherwise.
The texture network T is U-Net that takes as input the texture augmented texture image K2 and outputs the final enhanced texture K IR VxVx This network also fuses information from the view-specific texture images K,. The goal is to select, for each UV point which of the N input views provides the best information. This is achieved via cross-attention. Specifically, each K; is processed in parallel by another U-Net, and the first queries information across all the others via multi-head cross attention. In Fig. 14 (a), we provide an illustration the latter cross-view attention layer.
A.8
Physically-Baser -Based Rendering: Radiance, BRDFs, and models
We briefly summarise key notion of radiometry and standard BRDF models, and then provide precise expression of the BRDF model used in Meta 3D AssetGen.
A.8.1 Radiance
The radiant flux the electromagnetic power flowing through a particular surface A R oriented by the unit normal n. The radiance L(p, w) is the radiant flux density at y towards a particular direction w per unit orthogonal area dA, and per unit solid angle dl,, The unit vector w points at the direction of propagation of the flux.
23

---

The flux density is measured with respect to an area which is orthogonal to the direction of propagation w. In fact, the energy flow is the same through all areas that cut the same 'tube of flux'; the specific area is irrelevant and not property of the radiation. This dependency is removed by considering the normalized area dA1, which orthogonal to the direction of the flux.
Because the radiance is expressed in units of orthogonal area dA1, in order to compute the flux through the surface patch d'A, which may not be orthogonal, we must account for the foreshortening factor, which relates the areas dA and dA1:
dA_ (n, w))l dA.
With this, the flux that passes through dA towards direction w in the solid angle di is
d$ L(p, w)) (n,, w)| dA dQ.
Note that d& depends on both w and n. whereas L only depends on w. This reinforces the notion that n a property of the surface, not of the radiation.
A.9 Reflectance models
Let be point on a surface A that separates two media and let dA be surface patch sitting at Let n be the normal at this point. We now consider the case where separates air or empty space from an opaque object, with n pointing towards the outside of this object.
Let be orientation the side of the surface as n. i.e. such that (n, 0. From the
viewpoint of the object, we interpret L(p, w) as outgoing radiant flux and L(p, -w) as incoming radiant flux. These two quantities related by the Bidirectional Reflectance Distribution Function (BRDF) f, defined such that:
dL(p,wo) f(p. wi, wo) (n,, Lp) w.).
(10)
dΩ
In this definition, for convenience both w and wo are taken on the same side as n (hence the negative sign in front of wi). A useful consequence is that we do not need to take the absolute value of the inner product (n, wi) as this is positive by definition.
The BRDF thus takes the radiation receives from direction -w; and distributes it along various outgoing directions wo- Integrating over all incoming directions, gives use the overall radiation reflected towards wa:
dl(pp)o) dwi = SHS [(, wa) (n, L.) ww; dZ4.
Ip.wa) = (H(())) dΩg
(11)
where H(n) = (w: (n, w) 2 0} is the hemisphere. Next, we provide common basic models for the BRDF function in PBR.
A.9.1 Diffuse reflectance
In diffuse reflectance, the radiation is absorbed by the material, internally scattered in random directions, and output again to give rise to uniform distribution. Namely, the diffuse BRDF is:
f(p w.) R
where 0 < R < is the fraction of power reflected by the diffusion process. The 1/x factor ensures that the total energy is conserved when R = 1.
A.10 Specular reflectance
The reflection for a perfectly flat interface between two media at is specular: the incoming light radiation -w; is partially reflected in the specular direction w. = r(n, w,) 2n wi) Wj and partially transmitted. This phenomena is characterised by Fresnel's equations, which are derived from Maxwell's equations, utilising continuity conditions for the electromagnetic field at the interface between the two media. Fresnel's equations describe the planar radiation full, including its polarisation (in the most general case, using phasors, and thus complex numbers). For graphics,
24

---

assume that light is unpolarized, so we only calculate the power. The fraction of power reflected is given by Fresnel'ss coefficient (using Schlick's approximation [72
2
[1 012
F((n, wi)) = Fo + (1 Fb) (n, wi)l Fo
î1 + f22
Here îu and ft2 the indices of reflectivity of the two media, respectively. This equation is valid for dielectrics (non-metallic objects), but also used as an approximation for metals by tweaking Po-
In order to write this relation BRDF, we write:
wp) JH(m) [(, wi, w.) (n, Lp, ww,) ds2 R((n, r(n, w.))) -r(n) -r(n, w.)).
Hence, the BRDF must be a delta function centered at w = r(nz, wo):
F((n, Sy (ri
f(, wi, w))
(12)
(n,
where used the fact that (n, r (n, wo) and where )y(n is the delta distribution centered at r(n, w.).
A.11 Microfacet models
Rough surfaces can be though of as collection of randomly-oriented flat microfacets, each reflecting light in specular fashion. Consider point on a surface and incoming and outgoing radiation directions w; and wa- If the point contains microfacet that enables light reflect from w; w.. then the normal of the microfacet must be m x wi + W.. If the microfacet is oriented elsewhere, then no light flows in the direction w. Hence, we can write
m h hiw,,,
Wi J wi.
as a function of the incoming and outgoing radiation. This is called half vector as sits between the two vectors wi and w
Now we wish derive the macro BRDF f(p, wi, w.) from the micro BRDF F(p, wi, w. m)), where we have emphasized the fact that the BRDF is oriented relative to the microfacet normal m. A short calculation 90 shows that:
(m, wi) (m, wa) (n2 wi) (n, w.) (m, n)
f(p Wi, w.) = fm(p, wi, w./m)
In practice, there is distribution over possible surface normals m. characterised by the microfacet distribution function D(m(n)) The latter is defined such that D(m n) dA dSam is the total area of the microfacets within patch d.A of the macrosurface with orientation in dΩm In practice, only part of the microfacet is visible and illuminated, depending on the interaction with other facets. This is accounted for by the shadowing thadowing-maskingg reflectance is thus:
masking function G(w,, Wo m, n) [0, 1]. The expected
(m, wi) (m, wa) f(p, Wi, w.) H(n) (n.ww) (n, w.) fm wi, w. w./m) D(m(n) G(wi, Wa, m, n2) ddm-
Plugging Eq. 12 in the value for the mirror-like reflectance fm for each microfacet, we get 90
[p, wi, w.) F((h, wo) (n) G(wi, wo: n2) where h = -li(wr, wo). 4(n, wi) (n, w.)
A.12 Standard microfacet models
Here, we discuss common choices for the functions D and G in standard PBR models.
25

---

The Torrance-Sparrow model. The oldest such model is due to Torrance and Sparrow [87] They simply assume a Gaussian model for the microfacet distribution function:
Dmm(n) bexp (-a²0), where cos 0 (m, n)),
and b is a suitable normalization constant. For the shadowing-masking function they pick:
G(wi, Wo, m, n) min 2 n) n) wi) 2(m, n) (2) wo?
(m,
(m, wo)
This function has simple geometric derivation under basic geometric model of the microfacets.
The Beckmam-SpiizccinnSSSiii model. Beckmann and Spizzichino [2 suggested the model:
1 Lan3 44
D(m/n) TO cos 0
where cos 0 (m, n)
function should be derived from the microfacet distribution function, which describes the micro-geometry of the surface. They also proposed factorized model G(w,, Wo. m, n) = G1(w), G) n2). For Beckmann's 's distribution, the Smith shadowing maadowing-masking function given by:
Smith 77
noted that the shadowing-maskin
Giwwn = 1 erf(a)
av= ceem where a = lan Ow
and cos Ow = (n,, w).
The GGX model. The GGX model by 90 variant of the Beckmann model, with slightly different microfacet distribution and shadowing sladowing-masking function:
a²
D(m(n) cos 0(a2 + tan 0)22
Gi(w,
(13)
1 V1 + a tan Aw
where cos (m, n. and cos Ow (n,
A.12.1 The BRDF model used in Meta 3D AssetGen
The BRDF model used in our paper combines diffuse and GGX BRDFs:
R ((h)))) (n, (i))ii(a (n.
f(wi, wa (k)))) n)
4(n-wi)(m (n w.)
The first term is the diffuse component (Lambertian reflection), where [0, is the fraction of light power reflected by diffusion. The second term in the specular component, where F is Schlick's approximation 72 (n) Fu (1 (1) n w;) of Fresnel's reflectance and where Fo E [0, 1] is the Fresnel coefficient at normal incidence. The unit vector h xc wi + w, is the half vector, which the orientation needed to reflect wi into wo by the rough material's microfacets (generally different from but averaging to n2). The function D and Gi are the microfacet distribution function and seadowing-maskinne function given by:
a²
2(n
D(m|n)= 7(mm n) 1) (n, n- V(n w) a
These are the same as Eq. [3) with the trigonometric functions expanded in terms of dot products. In this model, the reflectance and reflectance at normal incidence Fo RGB triplets. The specular highlight color Fo is approximately white (equal to 1) for dielectrics, and colored for metals: furthermore, metals have no diffuse component (R 0). Thus, we introduce the parameter metalness E [0, 1] and the base color E [0, 1)s and define:
(1 7),
Fg 101 7) + PoT-
In this manner, the albedo is used diffuse color specular color depending whether the material a dielectric or a metal. the paper, we call the parameter Po albedo as it is a better known concept; however, in this model Po is the albedo only when 7 0, i.e., when the object a perfect dielectric.
The BRDF thus fully described by the albedo Po. the roughness a, and the metalness 7, for a total of five scalar parameters. Hence, the LRM predicts the triplet k(x) (po, 7, at each 3D point
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Sec. and the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See App. A.2
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "L "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
No theoretical results present.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please see Secs. and and Apps. A.6.2 and A.7 for all technical details required for reproducing the paper results.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer:
[No]
Justification:
We do not currently plan to open-source the method or training data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee //nips.com /ttc: ssiniiiiccypyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please see Secs. 3 and and Apps. A.6.2 and A.7 for all technical details required for reproducing the paper results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification: Reported metrics have negligible or zero variance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
29

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g,, negative emror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Sec.
4 and App. A.6.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:ppps.//aoossscooooooo////aa
Answer: [Yes]
The following our statement addressing concerns about training data raised during the review process. The data used in this paper was not obtained by scraping the internet. The data was purchased from a well-respected and widely-known vendor of 3D graphic assets. We acquired data license that explicitly allows use of the models in machine-learning applications, including applications in this paper. We follow the terms and conditions of this license scrupulously, also based on internal legal advice.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes],
Justification: See App. A.I
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Data models not released
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We internal dataset of artist-created meshes whose use has been approved by professional legal team.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: See App. A.4
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [Yes]
Justification: The internal legal and privacy audit has approved our user study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "AlN6CxL9UG", "reviewer": "Reviewer_z8qF", "summary": "The paper mainly works on text-to-3D with PBR materials. It follows the diffusion-based multiview generation and LRM-based reconstruction paradigm. It is a fast feed-forward solution for PBR generation: The diffusion model predicts both shaded and albedo, and the LRM predicts PBR via differentiable rendering with an efficient pixel-wise deferred shading loss. Moreover, it designs a UNet-based texture refiner to project input images to mesh, bringing sharp textures. Also, it adapts LightplaneLRM with the VolSDF renderer and SDF loss to enhance geometry. The generation only takes 30 seconds. It outperforms state-of-the-art feed-forward baselines and performs comparably to top optimization-based industry solutions.", "questions": "- Fewer than 10% meshes are selected as high-quality samples to finetune Emu (L226). What is the filtering standard? In terms of appearance quality, material correctness, and composition complexity? Please elaborate more.\n- L221: What dataset is used? Is it publicly available, like Objaverse? Is the dataset used under proper license?\n- How is the text-to-image model (3-channel output) adapted to output 6 channels? Please provide more details on that.\n- For results in Figure 7., if I understand correctly, they are deterministic? What is their PBR PSNR? (How cherry-picked are they?)\n- May consider adding some qualitative comparisons to optimization-based PBR generation baselines like Richdreamer. Although they are time-costly and may not be as competitive as commercial solutions, such a comparison can strengthen the claim.\n- In the Image-to-3D part (excluding texture refinement), Are albedo predictions only used as inputs to MLP?\n- Some qualitative examples of Luma Genie 1.0, stage 2 vs the proposed approach (A.5). It would help to show where optimization helps to get an edge.\n- 30-sec Runtime breakdown to each step.", "limitations": "Some suggestions are included in the Questions above.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Good performance.\n\n    a. It compares with and outperforms some very recent SOTA approaches like InstantMesh and MeshLRM as well as some commercial products like Meshy and Luma Genie.\n\n    b. It showcases that it can predict well on objects with mixed materials, which is very impressive. Its PBR prediction is SOTA performance. Also, it is not optimization-based and thus fast.\n\n    c. Its texture refinement module is robust and preserves many details (Figure 4, 8), which helps win out in terms of visual appearance.\n2. It conducts many quantitative and qualitative ablation studies to validate each component’s effectiveness.\n3. The writing is very clear and easy to follow. Notations are used properly and are well explained. Each design choice is well-motivated and elaborated. It also includes a nice preliminary reading of the used BRDF model.", "weaknesses": "1. Image-to-3D may have a degraded material decomposition. Since the proposed approach does not predict albedo for input images, PBR prediction correctness will drop significantly as pointed out in the ablation study (Table 1, Fig. 3).\n2. The diffusion model is inherently stochastic, which leads to variance in albedo prediction. How is this variance? How will this variance impact the material decomposition afterward? This point is not quantitatively evaluated.\n3. The paper evaluates PSNR for metallic. It is great. But metallic is usually binary. A simple 2x2 confusion matrix on metallic prediction on single-material objects can show the performance more intuitively.\n4. If input images are taken under non-uniform lighting conditions, I doubt if albedo can still be correctly decomposed.\n5. Lack of failure case analysis. Following the previous point, I wonder if there are some typical failure cases for PBR prediction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for detailed review and the helpful comments!\n\n1. **Degraded PBR quality with only RGB shaded inputs** (weakness #1):\n\nYes, indeed the PBR quality suffers when only RGB shaded inputs are provided (upper half Table 1) since the material decomposition is ambiguous. For this reason, the complete proposed approach for text-to-3D partially mitigates this problem by generating both the shaded and the albedo grid, which enhances the PBR quality (Fig. 4). \n\n2. **Evaluation of variance in albedo and shaded RGB prediction** (weakness #2):\n\nYes, for a given text prompt, the albedo and shaded RGB produced by the Emu model can vary due to the stochastic nature of diffusion. However, evaluating the effects quantitatively is challenging in the absence of the absolute ground truth. For evaluating the quality of outputs in the text-to-3D setup, we conducted a user study for user preferences using the fixed model seed, but evaluating the quality of material might be a bit more challenging for a non-professional user study participant.\n\n3. **Confusion matrix for metalness** (weakness #3):\n\nIt is true that, physically, materials are either metallic or dielectric, but computer graphics artists often use non-binary values of metalness, for example to represent materials that are a mixture of dielectrics and metals (e.g., certain rock conglomerates, transitions between oxidized/rusted and polished metal); in our own dataset, a non-trivial proportion of metallic assets have non-binary metalness. This justifies using PSNR or similar regression metric for metalness too.\n\n4. **Non uniform lighting** (weakness #4):\n\nIf the input views are captured with non-uniform lighting, it will indeed hamper PBR reconstruction performance. However, in our application, i.e., text-to-3D, lighting is controlled by fine-tuning the Emu grid generator on 3D models rendered with consistent lighting.This makes it biased toward generating images with consistent across views and with uniform lighting.\n\n5. **Failure cases** (weakness #5):\n\nWe will add a discussion and visualizations corresponding to failure cases in the final draft.\n\n\n---\n\n\n### **Answers to questions**\n\n\n1. **Filtering standard for the 10K objects used for fine tuning Emu:**\n\nTo filter the dataset to obtain the top 10K objects, we first remove those with too low aesthetic score. We then select the top 10K objects with the highest CLIP score (dot product between CLIP of the object render and CLIP of the caption). This number was empirically determined, as including more assets, i.e. including assets with a lower CLIP score, results in decreased text-image alignment. To control material quality, we only select assets with valid metallicness and roughness maps. Compositional quality is generally low, as a vast majority of assets in the dataset are single objects rather than scenes.\n \n2. **Details about the dataset:**\n\nWe use an internal proprietary dataset which is similar to Objaverse. The usage has been scrutinized by the professional legal counsel, which raised no concerns about copyrights. Copyright issues are why we did not use Objaverse.\n\n3. **Adapting 3-channel Emu with 6-channel outputs:**\n\nWe employ a text-to-image diffusion model pre-trained on billions of images annotated with text and expand its input and output channels by a factor of 2 to support simultaneous generation of the shaded appearance and albedo (Section A.6.1). This requires only the input and output layers to be adapted.\n\n4. **Figure 7 results:**\n\nYes, correct, the results are deterministic. Here, the task is sparse view reconstruction from 4 posed RGB images (without the albedo input), resulting in 3D meshes and their PBR materials. The results correspond to config G in Table 1, which shows the mean PBR PSNR on the whole test set. For the 6 images shown in the figure, the mean PSNR is 23.49 for albedo, 24.91 for metallicity, and 21.02 for roughness, with the standard deviation of (2.72, 3.86, 4.17) respectively. The object selection for the figure was intended to target objects with a non-uniform metalness or roughness (not just a single value) to make for an interesting visualization and highlight the model’s effectiveness.\n\n5. **Comparison to Richdreamer:**\n\nPlease check the PDF in the common response.\n\n6. **How are albedo predictions used by the LRM:**\n\nIn the vanilla Instant3D LRM, an image encoder encodes patchwise feature tokens of the image using a pre-trained DINO network (Caron et al. 2021). LightplaneLRM further extends this by adding Splatter layers on the DINO outputs in the subsequent image-to-triplane transformer. For accommodating additional albedo predictions, we add the encoded albedo patchwise features to the DINO network. This enables access to albedo as well as shaded image information to DiNO, and therefore subsequently to the image-to-triplane transformer and the splatting layers.\n\nWe will add these details to the final draft.\n\n7. **Qualitative Luma Genie 2.0 results:**\n\nPlease check the PDF in the common response.\n\n8. **Runtime breakdown:**\n\nEmu image generation is 20s, LRM reconstruction is 2s, meshification is 8s, and texture refinement < 1s.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6rIK0njJfz", "reviewer": "Reviewer_Zgd2", "summary": "This paper introduces a two-stage 3D asset generation pipeline that outputs high-quality 3D meshes with PBR materials in approximately 30 seconds. The technical novelties include:\n- In text-to-image stage, generating multiple views of both shaded and unshaded images, and predicting PBR materials during the image-to-3D stage.\n- Using SDF instead of opacity field which yields higher-quality meshes.\n- Introducing a new texture refiner designed to recover and enhance details in texture maps.", "questions": "Regarding the blurriness of the pre-refinement texture map: Is this a result of the image-to-3D reconstruction process or an issue with sampling from the texture field? If it is due to the sampling process, why not optimizing with respect to volSDF renderings to recover the details?", "limitations": "The authors discuss the limitations of their method and broader impact in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed pipeline introduces several components to enhance and extend the existing pipeline [39]. Most design choices are intuitive and well-justified through ablation studies, which include both qualitative and quantitative analyses. \n\nAdditionally, the paper compares the proposed pipeline with SOTA methods, demonstrating more favorable results through a user study.", "weaknesses": "I found certain technical novelties challenging to assess:\n- The use of SDF instead of opacity fields in the reconstruction pipeline is not a novel idea but more of an extension to Instant3D. This approach has also been explored in 3D generation works such as MeshLRM and Latte3D.\n- The texture refiner, which I found the most interesting and novel part of the pipeline, is difficult to evaluate from the provided qualitative examples. Looking at the texture maps in Figures 2 and 13, which contain many small components and seams from xAtlas outputs, one concern is whether these artifacts might affect the UNet's predictions in UV space, e.g. causing color inconsistencies. The ablation in Figure 14 only shows textures improved in flat regions, without the corresponding UV maps. Showing texture maps before and after refinement around seams would help readers better understand the model's effectiveness, and justify refinement in UV space as opposed to image space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review and the feedback on our work!\n\n1. **Use of SDF instead of opacity fields:**\n\nWe do not claim that using SDF in 3D reconstruction or generation is novel per se (e.g., StyleSDF, SDFusion, GET3D, Latte3D, etc. uses it).\n\nWe do note that we are among the first to use SDF in LRMs specifically. Other works that do the same include MeshLRM, InstantMesh, Large Tensorial Model (LDM) and Direct3D, but these are concurrent (released either after or just before the submission deadline).\n\nWhat makes our approach stand out, even among these more recent papers, is the fact that we use the SDF formulation directly, in a single “phase”. Other papers usually require a first stage training with opacity fields, followed by a second stage where the opacity field is replaced with SDFs / meshes. In other words, we show that two stages are unnecessary. \n\nFurthermore, we do not simply exchange NeRF opacity fields with a VolSDF formulation. Instead, we implement the SDF renderer by extending the open source Lightplane Triton kernels supporting large batch sizes, image resolution, and denser point sampling on rays. Additionally, we enable a direct loss on the SDF values (A.6.4), also implemented as Triton kernels for scalability (note that a naive pytorch implementation of such a loss only allows a batch size of 1 image on an a100 GPU at our base resolution due to increase memory requirements). While these are “engineering” contributions, they are key to making this a practical method.\n\nNote that we compare against the concurrent works MeshLRM and InstantMesh, which were released before our submission.\n\n\n2. **Texture refinement evaluation, before and after refinement outputs:**\n\nOur texture refiner is evaluated extensively both quantitatively and qualitatively:\n\n(i) *Table 1*: Qualitative evaluation of 4-view PBR reconstruction on the internal dataset. The effectiveness of the texture refiner is evident from improvements in LPIPS and PBR PSNR from config F to config G, where the only change is the addition of the texture refiner. Further improvement from config H to I, again solely due to the texture refiner, underscores its efficacy.\n\n(ii) *Table 3*: Qualitative evaluation of 4-view non-PBR reconstruction on the GSO dataset. The significant improvement in texture quality, particularly in PSNR and LPIPS, between config C and D, demonstrates the impact of adding the texture refiner.\n\n(iii) *Fig. 4, row 5 vs. row 6 (ours without tex-refiner and ours with tex-refiner)*: Qualitative evaluation shows the effectiveness of the texture refiner, with clear improvements in texture fidelity. \n\n(iv) *Fig. 14*: Qualitative evaluation further demonstrating the impact of texture refinement.\n\n(v) *Supplementary Video*: 360 degree view texture refinement ablation for select assets.\n\nFor before and after refinement visualizations of uv texture maps, please check the common PDF doc. We will further add UV space improvements to the final draft for better understanding as suggested.\n\n\n3. **Blurriness in Image-to-3D and why not optimization:**\n\nWe suspect that the degradation in textures is due to the volumetric 3D representation of colors. Since texture is inherently a surface property, extending it into 3D space beyond the surface is inefficient, leading to blurriness.\n\nFor instance, this degradation occurs in all volumetric LRM-based methods like Instant3D, Instant Mesh, TripoSR, and NeRF stage of MeshLRM, SparseNeuS, etc. In contrast, approaches like GRM or the mesh stage of MeshLRM, which use pixel-aligned Gaussians and rasterization-based rendering respectively, capture textures much better due to their near-surface or surface representation. Similarly, our texture refinement uses UV representation with rasterization for rendering textures, ensuring textures exist only on the surface.\n\nWhile optimizing with respect to input images is a viable approach, it has issues:\n(a) Since the entire shape might not be visible in the input views, seams can appear between optimized visible and unoptimized non-visible areas, leading to incoherence and contrast mismatch.\n(b) Reconstructed objects are not perfect, leading to projection mismatches between the rendered images of the reconstructed object and the input images. This mismatch can provide incorrect supervision to the optimization objective, as the projections of imperfect 3D reconstructions may not align with the input images.\n(c) Time cost: Optimization takes a longer time.\n\nA learned approach like our texture refiner can combat both (a) and (b) since the training process learns to overcome these issues in a data-driven manner and can do so with a single feedforward pass, addressing the time cost (c).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6tbYOMRaGJ", "reviewer": "Reviewer_cPQZ", "summary": "The paper adopts a two-stage PBR information generation method. In the first stage, the text-to-image model is used to predict the PBR channel. In the second stage, it uses SDF to reconstruct the geometry and PBR information. Then, a cross-view attention texture reflector network is used to improve the coarse UV of the input.", "questions": "Recently, Unique3d [1] generated mesh with higher-resolution texture. As Emu3d and Unique3d [1] are concurrent works, it is not necessary to directly compare with it. However, theoretically speaking, compared to Unique3d, the texture resolution of Emu3d is still relatively low. It is suggested to add a paragraph in the paper to discuss the advantages and disadvantages of the two methods relative to each other.\n\n[1] Wu, Kailu, et al. \"Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image.\" arXiv preprint arXiv:2405.20343 (2024).", "limitations": "No. But they have discussed in the limitation part. They use a triplane representation, which cannot represent large-scale scenes.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Methods used are quite effective. The writing is relatively easy to understand.\nThe experiments are comprehensive. The performance is indeed quite good.", "weaknesses": "The technical novelty is weak.\n\nI noticed that neither the dataset nor the code are intended to be open sourced. I understand that this technology involves commercial use, but I believe that at least all the test sets used in the experiments and the 3D (PBR) assets generated on the test sets should be open sourced to facilitate comparison in future work.\n\nThe PBR resolution of 512x512 with 2x2 grid for 4-view is still relative low.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and the helpful comments!\n\n1. **Technical Novelty**:\n\nTo our knowledge, Emu3D is one of the first academic work that performs text-to-3D with PBR in a feed-forward way. We introduce several novel aspects that come together to achieve high quality and efficient text-to-3D reconstruction with PBR materials. These include:\n\n(i) A novel albedo + shaded RGB grid generation paradigm, with the intuition that the combination of these two modalities can help the reconstruction stage in inferring materials (ablated in Table 1, Fig. 3 and Video).\n\n(ii) A novel deferred shading loss that regularizes the predicted materials in an efficient manner (ablated in Table 1, A.6.3, Fig. 12 and Video).\n\n(iii) A Volumetric SDF renderer that extends Lightplane with Triton kernels for supporting large batch sizes, image resolution, and denser point sampling on rays. Additionally, we enable a direct loss on the SDF values (A.6.4), also implemented as Triton kernels for scalability (ablated in Table 3, Fig. 4 Row 4 vs 5 and Video).\n\n(iv) A novel texture refinement module that uses cross-view attention layers, which greatly improves the fidelity of reconstructed textures by operating in UV space (ablated in A.7 and Fig. 14, Fig. 4 Row 5 vs Row 6, Table 1 and Table 3, and Video).\n\n\n2. **Facilitating Comparisons**:\n\nThis is a great point and we are in discussion to open the data as much as possible. We also plan to release the PBR assets generated on the dreamfusion prompts, which are a typical benchmark for comparing text to 3D models.\nNote, however, that we do use publicly available GSO dataset for sparse-view image to 3D reconstruction. The reconstructed assets using our method on GSO will also be released.\n\n\n3. **PBR resolution**:\n\nThe resolution of our 2x2 grid is *not* 512x512, it is 512x512 per image in the grid, making the grid resolution to be 1024x1024 (albedo and shaded, each 1024 x 1024 x 3). These images are input to EmuLRM, which produces UV textures with resolution 512x512 each for albedo, metalness and roughness. These are further upscaled to 1024x1024 resolution textures by the texture refiner.\n\n\n4. **Comparison to Unique3D**:\n\nUnique3D upscales the 4 view grid to a resolution of 2048x2048, compared to our 512x512 per image resolution, which can potentially impart higher detailed textures. However, Unique3D only predicts RGB colors, not texture maps, and lacks material outputs as well. This prohibits asset use in novel environments due to baked in lighting in the albedo. \n\nFurther, we show a qualitative comparison to more recent methods like Unique3D as well as Stable 3D which came out last week in the common response PDF. The quality of assets produced by our method is visually more appealing than both.  \n\nWe will add this discussion and the comparisons to the final draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HBoEtT3aI0", "reviewer": "Reviewer_Xevg", "summary": "Emu3D is a novel approach in text-conditioned 3D generation, producing high-quality 3D meshes and materials using Physically-Based Rendering (PBR). It uses a two-stage process: first generating images from standard viewpoints, then reconstructing the 3D shape and appearance. This approach is faster and more reliable than earlier methods and creates realistic 3D graphics by modeling light interaction with surfaces. Emu3D also improves mesh generation by predicting a signed-distance field (SDF) for better quality and refining textures for greater detail. It surpasses existing methods in both image-to-3D and text-to-3D tasks, offering superior visual quality and text alignment.", "questions": "- What’s the performance of the proposed method over anisotropy materials?\n- What's the performance of the extracted mesh over transparent object like a class bottle?", "limitations": "- The generated material has significant representational limitations, as it primarily models surface reflection using only albedo, metalness, and roughness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The quality of the generated geometry, texture as well as the PBR materials offers an efficient and high quality generation workflow. The paper is well-presented, with sufficient experiments to validate the quality of the results.", "weaknesses": "- The generated PBR material includes only limited components: albedo, metallicity, and roughness. PBR materials typically also consist of other components, such as index of reflection, scattering, coat, sheen etc. Therefore, the generated material can only represent surface reflectance from diffuse to specular.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review and the feeback!\n\n1. **Lack of support for material components such as index of reflection, scattering, coat, sheen etc.**:\n\nOur model indeed only tackles essential PBR parameters (albedo, roughness and metallness) as these are the most important ones, and also the focus of 3D artists when they work on real-time applications like video games. These real-time applications are also the most likely to benefit from AI-generated content as they require a large quantity of 3D assets and, depending on the game, the quality of these assets need not to be as high as, say, a CG for a Hollywood production. Additionally, simple PBR textured assets can serve as a starting point for artists to extend the material map, making it a good place to start from.\n\nNote that in the field of generative 3D asset creation, to our knowledge, there are no methods that go beyond the simple PBR model (albedo, metallness, roughness). Our approach can in principle be extended to predict additional parameters (e.g., say the full Disney BRDF), given sufficiently representative training data.\n\nWe’ll add a note in the final version to clarify these limitations. \n\n\n2. **Performance on anisotropic and glass like materials.**\n\nWe also do not tackle transparent materials yet. Our approach could also be extended to those in principle, again given sufficient training data, but less trivially as it would require modifying the deferred shading part to support transparency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their constructive and valuable feedback. We are pleased that all four reviewers found our presentation to be excellent, the soundness of our proposed approach to be good or excellent, and our contributions to be good. 

We are glad the reviewers found our method to be “effective” (**cPQZ**), producing “efficient, high-quality results” (**Xevg**), and “favorable” (**Zgd2**), with “good performance” (**z8qF**, **cPQZ**). All reviewers appreciate the experiments, noting that the evaluation is “comprehensive“ (**cPQZ**) and “sufficient” (**Xevg**) with “many qualitative and quantitative studies” (**z8qF**, **Zgd2**), and “thorough ablations” (**Zgd2**) that “validate each component's effectiveness” (**z8qF**). The reviewers also found our design choices to be “well-motivated” (**z8qF**), “intuitive, and well-justified” (**Zgd2**). They further appreciated the writing quality as clear and easy to follow (**z8qF**, **cPQZ**) and well presented (**Xevg**).

Attached is a PDF with requested extra comparisons and visualizations, which will be included in the final draft. Due to limited space, we encourage the reviewers to zoom into the figures if necessary.

Meta-review: This paper proposes a 2-stage feed-forward framework, named EmuLRM, for text-to-3D that predicts PBR materials to support realistic relighting. In the 1st stage, 4 views with both shaded and albedo channels are generated by a text-to-image diffusion model to asist the prediction of PBR materials in the 2nd stage. In the 2nd stage, the authors extend LightplaneLRM to predict SDF together with PBR materials. The authors also introduce deferred shading, a new scalable SDF-based render, and a direct SDF loss to improve training efficiency and scalability. Finally, they propose a texture refinement step operating in the UV space to improve the sharpness and details of the textures by leveraging the reference views through cross-attention. 

This paper outperforms other feed-forward methods and produces sota results. Its results are also comparable to sota optimization based methods. Sufficient quantitative and qualitative results are presented to support and validate various design choices.

Overall, all reviewers are positive about this paper (accept, weak accept, weak accept, weak accept). The authors have addressed all the concerns rasied by the reviewers in their rebuttal. It is recommended to accept this paper.
Decision: Accept (poster)