3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion

## Abstract

Abstract The increasing demand for high-quality 3D assets across various industries necessitates efficient and automated 3D content creation.
Despite recent advancements in 3D generative models, existing methods still face challenges with optimization speed, geometric fidelity, and the lack of assets for physically based rendering (PBR).
In this paper, we introduce 3DTopia-XL, a scalable native 3D generative model designed to overcome these limitations.
3DTopia-XL leverages a novel primitive-based 3D representation, PrimX, which encodes detailed shape, albedo, and material field into a compact tensorial format, facilitating the modeling of high-resolution geometry with PBR assets.
On top of the novel representation, we propose a generative framework based on Diffusion Transformer (DiT), which comprises 1) Primitive Patch Compression, 2) and Latent Primitive Diffusion.
3DTopia-XL learns to generate high-quality 3D assets from textual or visual inputs.
We conduct extensive qualitative and quantitative experiments to demonstrate that 3DTopia-XL significantly outperforms existing methods in generating high-quality 3D assets with fine-grained textures and materials, efficiently bridging the quality gap between generative models and real-world applications.

## 1 Introduction

High-quality 3D assets are essential for various real-world applications like films, gaming, and virtual reality.
However, creating high-quality 3D assets involves extensive manual labor and expertise.
Therefore, it further fuels the demand for automatic 3D content creation techniques, which automatically generate 3D assets from visual or textual inputs by using 3D generative models.

Fortunately, rapid progress has been witnessed in the field of 3D generative models recently.
Existing state-of-the-art techniques can be sorted into three categories.
1) Methods based on Score Distillation Sampling (SDS)  (Poole et al., ; Tang et al., ) lift 2D diffusion priors into 3D representation by per-scene optimization.
However, these methods suffer from time-consuming optimization, poor geometry, and multifaceted inconsistency.
2) Methods based on sparse-view reconstruction (Hong et al., ; Xu et al., ) that leverage large models to regress 3D assets from single- or multi-view images.
Most of these methods are built upon triplane-NeRF (Chan et al., ) representation.
However, due to the triplane’s parameter inefficiency, the valid parameter space is limited to low resolutions in those models, leading to relatively low-quality 3D assets.
Plus, reconstruction-based models also suffer from a low-diversity problem as deterministic methods.
3) Methods as native 3D generative models (Yariv et al., ; Li et al., ) aim to model the probabilistic distribution of 3D assets, generating 3D objects given input conditions.
Yet, few of them are capable of generating high-quality 3D objects with Physically Based Rendering (PBR) assets, which are geometry, texture, and material packed into a GLB file.

To address the limitations above, we propose 3DTopia-XL, a high-quality native 3D generative model for 3D assets at scale.
Our key idea is scaling the powerful diffusion transformer (Peebles & Xie, ) on top of a novel primitive-based 3D representation.
At the core of 3DTopia-XL is an efficient 3D representation, PrimX, which encodes the shape, albedo, and material of a textured mesh in a compact $N\times D$ tensor, enabling the modeling of high-resolution geometry with PBR assets.
In specific, we anchor $N$ primitives to the positions sampled on the mesh surface. Each primitive is a tiny voxel, parameterized by its 3D position, a global scale factor, and corresponding spatially varied payload for SDF, RGB, and material.
Note that the proposed representation differentiates itself from the shape-only representation M-SDF (Yariv et al., ) that PrimX encodes shape, color, and material in a unified way.
It also supports efficient differentiable rendering, leading to the great potential to learn from not only 3D data but also image collections.
Moreover, we carefully design initialization and fine-tuning strategy that enables PrimX to be rapidly tensorized from a textured mesh (GLB file) which is ten times faster than the triplane under the same setting.

Thanks to the tensorial and compact PrimX, we scale the 3D generative modeling using latent primitive diffusion with Transformers, where we treat each 3D object as a set of primitives.
In specific, the proposed 3D generation framework consists of two modules.
1) Primitive Patch Compression uses a 3D VAE for spatial compression of each individual primitive to get latent primitive tokens;
and 2) Latent Primitive Diffusion leverages the Diffusion Transformers (DiT) (Peebles & Xie, ) to model global correlation of latent primitive tokens for generative modeling.
Notably, the permutation equivariance of PrimX naturally supports training Transformers without positional encoding.
The significant efficiency of the proposed representation allows us to achieve high-resolution generative training using a clean and unified framework without super-resolution or post-hoc optimization-based mesh refinement.

In addition, we also carefully design algorithms for high-quality 3D PBR asset extraction from PrimX, to ensure reversible transformations between PrimX and textured mesh.
An issue for most 3D generation models (Wang et al., ; Xu et al., ) is that they use vertex coloring to represent the object’s texture, leading to a significant quality drop when exporting their generation results into mesh format.
Thanks to the high-quality surface modeled by Signed Distance Field (SDF) in PrimX, we propose to extract the 3D shape with zero-level contouring and sample texture and material values in a high-resolution UV space.
This leads to high-quality asset extraction with considerably fewer vertices, which is also ready to be packed into GLB format for downstream tasks.

Extensive experiments are conducted both qualitatively and quantitatively to evaluate the effectiveness of our method in text-to-3D and image-to-3D tasks.
Moreover, we do extensive ablation studies to motivate our design choices for a better efficiency-quality tradeoff in the context of generative modeling with PrimX.
In conclusion, we summarize our contributions as follows:
1) We propose a novel 3D representation, PrimX, for high-quality 3D content creation, which is efficient, tensorial, and renderable.
2) We introduce a scalable generative framework, 3DTopia-XL, tailored for generating high-quality 3D assets with high-resolution geometry, texture, and materials.
3) Practical techniques for assets extraction from 3D representation to avoid quality gap.
4) We demonstrate the superior quality and impressive applications of 3DTopia-XL for image-to-3D and text-to-3D tasks.

## 2 Related Work

##### Feed-Forward 3D Generative Models.

Early works on feed-forward 3D generation involves training a GAN (Goodfellow et al., ) from 2D image datasets (Gao et al., ; Chan et al., ; Hong et al., ).
However, such methods fail to scale up to large-scale datasets with general 3D objects (Deitke et al., ; ).
Recent advancements have been focusing on feed-forward reconstruction methods that regress 3D assets from single- or multi-view images.
Large Reconstruction Model (LRM) (Hong et al., ; He & Wang, ) has shown that end-to-end training of a triplane-NeRF regression model scales well to large datasets and can be highly generalizable.
Although it can significantly accelerate generation speed, the generated 3D assets still exhibit relatively lower quality due to representation inefficiency and suffer from a low-diversity problem as a deterministic method.
Subsequent works have extended this method to improve generation quality.
For example, using multi-view images (Xu et al., ; Li et al., ; Wang et al., ; Siddiqui et al., ; Xie et al., ; Wang et al., ; Jiang et al., ; Boss et al., ) generated by 2D diffusion models as the input can effectively enhance the visual quality.
However, the generative capability is actually enabled by the frontend multi-view diffusion models (Shi et al., ; Li et al., ; Long et al., ) which cannot produce multi-view images with accurate 3D consistency.
Another direction is to use more efficient 3D representations such as Gaussian Splatting (Kerbl et al., ; Tang et al., ; Xu et al., ; Zhang et al., ; Yi et al., ; Chen et al., ) and triangular mesh (Zhang et al., ; Li et al., ; Wei et al., ; Zou et al., ).
However, few of them can generate high-quality PBR assets with sampling diversity.

##### 3D Diffusion Generative Models.

Similar to 2D diffusion models for image generation, efforts have been made to train a 3D native diffusion model on conditional 3D generation.
However, unlike the universal image representation in 2D, there are many different choices for 3D representations.
Voxel-based methods (Müller et al., ) can be directly extended from 2D methods, but they are constrained by the demanding memory usage, and suffer from scaling up to high-resolution data.
Point cloud based methods (Nichol et al., ; Nash & Williams, ) are memory-efficient and can adapt to large-scale datasets, but they hardly represent the watertight and solid surface of the 3D assets.
Implicit representations such as triplane-NeRF offer a better balance between memory and quality (Jun & Nichol, ; Gupta et al., ; Cheng et al., ; Ntavelis et al., ; Cao et al., ; Chen et al., ; Wang et al., ; Liu et al., )
There are also methods based on other representations such as meshes and primitives (Liu et al., ; Yariv et al., ; Chen et al., ; Xu et al., ; Yan et al., ).
However, these methods still struggle with generalization or producing high-quality assets.
Recent methods attempt to adapt latent diffusion models to 3D (Zhang et al., ; Zhao et al., ; Zhang et al., ; Wu et al., ; Li et al., ; Lan et al., ; Hong et al., ; Tang et al., ).
These methods first train a 3D compression model such as a VAE to encode 3D assets into a more compact form, which allows the diffusion model to train more effectively and show strong generalization.
However, they either suffer from low-resolution results or are incapable of modeling PBR materials.
In this paper, we propose a new 3D latent diffusion model based on a novel representation, PrimX, which can be efficiently computed from a textured mesh and unpacked into high-resolution geometry with PBR materials.

## 3 Methodology

### 3.1 PrimX: An Efficient Representation for Shape, Texture, and Material

Before diving into details, we outline the following design principles for 3D representation in the context of high-quality large-scale 3D generative models:
1) Parameter-efficient: provides a good trade-off between approximation error and parameter count;
2) Rapidly tensorizable: can be efficiently transformed into a tensorial structure, which facilitates generative modeling with modern neural architectures;
3) Differentiably renderable: compatible with differentiable renderer, enabling learning from both 3D and 2D data.

Given the aforementioned principles, we propose a novel primitive-based 3D representation, namely PrimX, which represents the 3D shape, texture, and material of a textured mesh as a compact $N\times D$ tensor.
It can be efficiently computed from a textured mesh (typically a GLB file) and directly rendered into 2D images via a differentiable rasterizer.

Figure: Figure 2: Illustration of PrimX. We propose to represent the 3D shape, texture, and material of a textured mesh as a compact $N\times D$ tensor (Sec. 3.1.1). We anchor $N$ primitives to the positions sampled on the mesh surface. Each primitive ${\mathcal{V}}_{k}$ is a tiny voxel with a resolution of $a^{3}$, parameterized by its 3D position ${\mathbf{t}}_{k}\in{\mathbb{R}}^{3}$, a global scale factor $s_{k}\in{\mathbb{R}}^{+}$, and corresponding spatially varied payload ${\bm{X}}_{k}\in{\mathbb{R}}^{a\times a\times a\times 6}$ for SDF, RGB, and material. This tensorial representation can be rapidly computed from a textured mesh within 1.5 minutes (Sec. 3.1.2).
Refer to caption: x1.png

#### 3.1.1 Definition

##### Preliminaries.

Given a textured 3D mesh, we denote its 3D shape as ${\mathcal{S}}\in{\mathbb{R}}^{3}$, where ${\mathbf{x}}\in{\mathcal{S}}$ are spatial points inside the occupancy of the shape, and ${\mathbf{x}}\in\partial{\mathcal{S}}$ are the points on the shape’s boundary, i.e., the shape’s surface.
We model the 3D shape as SDF as follows:

$$ $F_{{\mathcal{S}}}^{\mathrm{SDF}}({\mathbf{x}})=\begin{cases}-d({\mathbf{x}}, \partial{\mathcal{S}}),\;\;{\mathbf{x}}\in{\mathcal{S}}\\ d({\mathbf{x}},\partial{\mathcal{S}}),\;\;\mathrm{elsewise}\end{cases}\;\; \mathrm{s.t.}\;\;d({\mathbf{x}},\partial{\mathcal{S}})=\min_{{\mathbf{y}}\in{ \mathcal{S}}}||{\mathbf{x}}-{\mathbf{y}}||_{2}.$ (1) $$

Moreover, given the neighborhood of shape surface, ${\mathcal{U}}(\partial{\mathcal{S}},\delta)=\{d({\mathbf{x}},\partial{\mathcal
{S}})<\delta\}$, the space-varied color function and material function of the target mesh are defined by:

$$ $F_{{\mathcal{S}}}^{\mathrm{RGB}}({\mathbf{x}})=\begin{cases}C({\mathbf{x}}),\; \;{\mathbf{x}}\in{\mathcal{U}}\\ 0,\;\;\mathrm{elsewise}\end{cases}F_{{\mathcal{S}}}^{\mathrm{Mat}}({\mathbf{x} })=\begin{cases}\rho({\mathbf{x}}),\;\;{\mathbf{x}}\in{\mathcal{U}}\\ 0,\;\;\mathrm{elsewise}\end{cases},$ (2) $$

where $C({\mathbf{x}}):{\mathbb{R}}^{3}\xrightarrow{}{\mathbb{R}}^{3}$ and $\rho({\mathbf{x}}):{\mathbb{R}}^{3}\xrightarrow{}{\mathbb{R}}^{2}$ are corresponding texture sampling functions to get albedo and material (metallic and roughness) from UV-aligned texture maps given the 3D point ${\mathbf{x}}$.
Eventually, all shape, texture, and material information of a 3D mesh can be parameterized by the volumetric function $F_{{\mathcal{S}}}=(F_{{\mathcal{S}}}^{\mathrm{SDF}}\oplus F_{{\mathcal{S}}}^{
\mathrm{RGB}}\oplus F_{{\mathcal{S}}}^{\mathrm{Mat}}):{\mathbb{R}}^{3}
\xrightarrow{}{\mathbb{R}}^{6}$, where $\oplus$ denotes concatenation.

##### PrimX Representation.

We aim to approximate $F_{{\mathcal{S}}}$ with a neural volumetric function $F_{{\mathcal{V}}}:{\mathbb{R}}^{3}\xrightarrow{}{\mathbb{R}}^{6}$ parameterized by a $N\times D$ tensor ${\mathcal{V}}$. For efficiency, our key insight is to define $F_{{\mathcal{V}}}$ as a set of $N$ volumetric primitives distributed on the surface of the mesh:

$$ ${\mathcal{V}}=\{{\mathcal{V}}_{k}\}_{k=1}^{N},\mathrm{where}\;\;{\mathcal{V}}_ {k}=\{{\mathbf{t}}_{k},s_{k},{\bm{X}}_{k}\}.$ (3) $$

Each primitive ${\mathcal{V}}_{k}$ is a tiny voxel with a resolution of $a^{3}$, parameterized by its 3D position ${\mathbf{t}}_{k}\in{\mathbb{R}}^{3}$, a global scale factor $s_{k}\in{\mathbb{R}}^{+}$, and corresponding spatially varied feature payload ${\bm{X}}_{k}\in{\mathbb{R}}^{a\times a\times a\times 6}$ within the voxel.
Note that, the payload in PrimX could be spatially varied features with any dimensions.
Our instantiation here is to use a six-channel local grid ${\bm{X}}_{k}=\{{\bm{X}}_{k}^{\mathrm{SDF}},{\bm{X}}_{k}^{\mathrm{RGB}},{\bm{X}
}_{k}^{\mathrm{Mat}}\}$ to parameterize SDF, RGB color, and material respectively.

Inspired by Yariv et al. () where mosaic voxels are globally weighted to get a smooth surface, the approximation of a textured mesh is then defined as a weighted combination of primitives:

$$ $F_{{\mathcal{V}}}({\mathbf{x}})=\sum_{k=1}^{N}w_{k}({\mathbf{x}}){\mathcal{I}} ({\mathcal{V}}_{k},\frac{{\mathbf{x}}-{\mathbf{t}}_{k}}{s_{k}}),$ (4) $$

where ${\mathcal{I}}({\mathcal{V}}_{k},{\mathbf{x}})$ denotes the trilinear interpolant over the voxel grid ${\bm{X}}_{k}$ at position ${\mathbf{x}}$.
The weighting function $w_{k}({\mathbf{x}})$ of each primitive is defined as:

$$ $w_{k}({\mathbf{x}})=\frac{\hat{w}_{k}({\mathbf{x}})}{\sum_{j=1}^{N}\hat{w}_{j} ({\mathbf{x}})},\;\;\mathrm{s.t.}\;\;\hat{w}_{k}({\mathbf{x}})=\max(0,1-|| \frac{{\mathbf{x}}-{\mathbf{t}}_{k}}{s_{k}}||_{\infty}).$ (5) $$

Once the payload of primitives is determined, we can leverage a highly efficient differentiable renderer to turn PrimX into 2D images.
In specific, given a camera ray ${\mathbf{r}}(t)={\mathbf{o}}+t{\mathbf{d}}$ with camera origin ${\mathbf{o}}$ and ray direction ${\mathbf{d}}$, the corresponding pixel value $I$ is solved by the following integral:

$$ $I=\int_{t_{\mathrm{min}}}^{t_{\mathrm{max}}}F_{{\mathcal{V}}}^{\mathrm{RGB}}({ \mathbf{r}}(t))\frac{dT(t)}{dt}dt,\;\;\mathrm{s.t.}\;\;T(t)=\int_{t_{\mathrm{ min}}}^{t}\exp{[-(\frac{F_{{\mathcal{V}}}^{\mathrm{RGB}}({\mathbf{r}}(t))}{ \alpha})^{2}]}dt,$ (6) $$

where we use an exponent of the SDF field to represent the opacity field.
And $\alpha$ is the hyperparameter that controls the variance of the opacity field during this conversion. where we set $\alpha=0.005$.

To wrap up, the learnable parameters of a textured 3D mesh modeled by PrimX are primitive position ${\mathbf{t}}\in{\mathbb{R}}^{N\times 3}$, primitive scale $s\in{\mathbb{R}}^{N\times 1}$, and voxel payload ${\bm{X}}\in{\mathbb{R}}^{N\times a^{3}\times 6}$ for SDF, color, and material.
Therefore, each textured mesh can be represented as a compact $N\times D$ tensor, where $D=3+1+a^{3}\times 6$ by concatenation.

##### PBR Asset Extraction.

Once PrimX is constructed, it encodes all geometry and appearance information of the target mesh within the $N\times D$.
Now, we introduce our efficient algorithm to convert PrimX back into a textured mesh in GLB file format.
For geometry, we can easily extract the corresponding 3D shape with Marching Cubes algorithm on zero level set of $F_{{\mathcal{V}}}^{\mathrm{SDF}}$.
For PBR texture maps, we first perform UV unwrapping using xatlas in a 1k resolution UV space.
Then, we get sampling points in 3D using nvdiffrast and query $F_{{\mathcal{V}}}^{\mathrm{RGB}},F_{{\mathcal{V}}}^{\mathrm{Mat}}$ to get corresponding albedo and material values.
Note that, we mask the UV space to get the index of valid vertices for efficient queries.
Moreover, we dilate the UV texture maps and inpaint the dilated region with the nearest neighbors of existing textures, ensuring albedo and material maps smoothly blend outwards for anti-aliasing.
Finally, we pack geometry, UV mapping, albedo, metallic, and roughness maps into a GLB file, which is ready for the graphics engine and various downstream tasks.

#### 3.1.2 Computing PrimX from Textured Mesh

We introduce our efficient fitting algorithm in this section that computes PrimX from the input textured mesh in a short period of time so that it is scalable on large-scale datasets for generative modeling.
Given a textured 3D mesh $F_{{\mathcal{S}}}$, our goal is to compute PrimX such that $F_{{\mathcal{V}}}({\mathbf{x}})\approx F_{{\mathcal{S}}}({\mathbf{x}}),\;
\mathrm{s.t.}\;{\mathbf{x}}\in{\mathcal{U}}(\partial{\mathcal{S}},\delta)$.
Our key insight is that the fitting process can be efficiently achieved via a good initialization followed by lightweight finetuning.

##### Initialization.

We assume all textured meshes are provided in GLB format which contains triangular meshes, texture and material maps, and corresponding UV mappings.
The vertices of the target mesh are first normalized within the unit cube.
To initialize the position of primitives, we first apply uniform random sampling on the mesh surface to get $\hat{N}$ candidate initial points.
Then, we perform farthest point sampling on this candidate point set to get $N$ valid initial positions for all primitives.
This two-step initialization of position ensures good coverage of $F_{{\mathcal{V}}}$ over the boundary neighborhood ${\mathcal{U}}$ while also keeping the high-frequency shape details as much as possible.
Then, we compute the L2 distance of each primitive to its nearest neighbors, taking the corresponding value as the initial scale factor for each primitive.

To initialize the payload of primitives, we first compute candidate points in global coordinates using initialized positions ${\mathbf{t}}_{k}$ and scales $s_{k}$ as ${\mathbf{t}}_{k}+s_{k}{\bm{I}}$ for each primitive, where ${\bm{I}}$ is the unit local voxel grid with a resolution of $a^{3}$.
To initialize the SDF value, we query the SDF function converted from the 3D shape at each candidate point, i.e., ${\bm{X}}_{k}^{\mathrm{SDF}}=F_{{\mathcal{S}}}^{\mathrm{SDF}}({\mathbf{t}}_{k}+
s_{k}{\bm{I}})$.
Notably, it is non-trivial to get a robust conversion from arbitrary 3D shape to volumetric SDF function.
Our implementation is based on an efficient ray marching with bounding volume hierarchy that works well with non-watertight topology.
To initialize the color and material values, we sample the corresponding albedo colors and material values from UV space using geometric functions $F_{{\mathcal{S}}}^{\mathrm{RGB}}$ and $F_{{\mathcal{S}}}^{\mathrm{Mat}}$.
In specific, we compute the closest face and corresponding barycentric coordinates for each candidate point on 3D mesh, then interpolate the UV coordinates and sample from the texture maps to get the value.

##### Finetuning.

Even if the initialization above offers a fairly good estimate of $F_{{\mathcal{S}}}$, a rapid finetuning process can further decrease the approximation error via gradient descent.
Specifically, we optimize the well-initialized PrimX with a regression-based loss on SDF, albedo, and material values:

$$ ${\mathcal{L}}({\mathbf{x}};{\mathcal{V}})=\lambda_{\mathrm{SDF}}||F_{{\mathcal {S}}}^{\mathrm{SDF}}({\mathbf{x}})-F_{{\mathcal{V}}}^{\mathrm{SDF}}({\mathbf{x }})||_{1}+\lambda(||F_{{\mathcal{S}}}^{\mathrm{RGB}}({\mathbf{x}})-F_{{ \mathcal{V}}}^{\mathrm{RGB}}({\mathbf{x}})||_{1}+||F_{{\mathcal{S}}}^{\mathrm{ Mat}}({\mathbf{x}})-F_{{\mathcal{V}}}^{\mathrm{Mat}}({\mathbf{x}})||_{1}),$ (7) $$

where $\forall{\mathbf{x}}\in{\mathcal{U}}$, and $\lambda_{\mathrm{SDF}},\lambda$ are loss weights.
We employ a two-stage finetuning strategy where we optimize with $\lambda_{\mathrm{SDF}}=10$ and $\lambda=0$ for the first 1k iterations and $\lambda_{\mathrm{SDF}}=0$ and $\lambda=1$ for the second 1k iterations.

### 3.2 Primitive Patch Compression

Figure: Figure 3: Overview of 3DTopia-XL. As a native 3D diffusion model, 3DTopia-XL is built upon a novel 3D representation PrimX (Sec. 3.1). This compact and expressive representation encodes the shape, texture, and material of a textured mesh efficiently, which allows modeling high-resolution geometry with PBR assets. Furthermore, this tensorial representation facilitates our patch-based compression using primitive patch VAE (Sec. 3.2). We then use our novel latent primitive diffusion (Sec. 3.3) for 3D generative modeling, which operates the diffusion and denoising process on the set of latent PrimX, naturally compatible with Transformer-based neural architectures.
Refer to caption: x2.png

In this section, we introduce our patch-based compression scheme on primitives for two main purposes:
1) incorporating inter-channel correlations between geometry, color, and materials;
and 2) compressing 3D primitives to latent tokens for efficient latent generative modeling.

We opt for using a variational autoencoder (Kingma, ) (VAE) operating on local voxel patches which compresses the payload of each primitive into latent tokens, i.e., $F_{\mathrm{ae}}:{\mathbb{R}}^{D}\xrightarrow{}{\mathbb{R}}^{d}$.
Specifically, the autoencoder $F_{\mathrm{ae}}$ consists of an encoder $E$ and a decoder $D$ building with 3D convolutional layers.
The encoder $F_{\mathrm{ae}}$ has a downsampling rate of $48$ that compresses the voxel payload ${\bm{X}}_{k}\in{\mathbb{R}}^{a^{3}\times 6}$ into the voxel latent $\hat{{\bm{X}}}_{k}\in{\mathbb{R}}^{(a/2)^{3}\times 1}$.
We train this variational autoencoder $F_{\mathrm{ae}}=\{E,D\}$ with KL-regularized reconstruction loss:

$$ ${\mathcal{L}}_{\mathrm{ae}}({\bm{X}};E,D)=\frac{1}{N}\sum_{k=1}^{N}[||{\bm{X}} _{k}-D(E({\bm{X}}_{k}))||_{2}+\lambda_{\mathrm{kl}}{\mathcal{L}}_{\mathrm{kl}} ({\bm{X}}_{k},E)],$ (8) $$

where $\lambda_{\mathrm{kl}}$ is the weight for KL regularization over the latent space of the autoencoder.
Note that, unlike other works on 2D/3D latent diffusion models (Nam et al., ; Zhang et al., ; Rombach et al., ) that perform global compression over the collection of patches or global 3D latent, the proposed method only compresses each local primitive patch independently and defers the modeling of global semantics and inter-patch correlation to the powerful diffusion model.
Once the VAE is trained, we can compress the raw PrimX as ${\mathcal{V}}_{k}=\{{\mathbf{t}}_{k},s_{k},E({\bm{X}}_{k})\}$.
It leads to the low-dimensional parameter space for the diffusion model as ${\mathcal{V}}\in{\mathbb{R}}^{N\times d}$, where $d=3+1+(a/2)^{3}$.
In practice, this compact parameter space significantly allows allocating more model parameters given a fixed computational budget, which is the key to scaling up 3D generative models in high resolution.

### 3.3 Latent Primitive Diffusion

On top of PrimX (Sec. 3.1) and the corresponding VAE (Sec. 3.2), the problem of 3D object generation is then converted to learning the distribution $p({\mathcal{V}})$ over large-scale datasets.
Our goal is to train a diffusion model (Ho et al., ) that takes as input random noise ${\mathcal{V}}^{T}$ and conditions ${\mathbf{c}}$, and predicts PrimX samples.
Note that, the target space for denoising is ${\mathcal{V}}^{T}\in{\mathbb{R}}^{N\times d}$, where $d=3+1+(a/2)^{3}$.

In specific, the diffusion model learns to denoise ${\mathcal{V}}^{T}\sim{\mathcal{N}}(\mathbf{0},\mathbf{I})$ through denoising steps $\{{\mathcal{V}}^{T-1},\dots,{\mathcal{V}}^{0}\}$ given conditional signal ${\mathbf{c}}$.
As a set of primitives, PrimX is naturally compatible with Transformer-based architectures, where we treat each primitive as a token.
Moreover, the permutation equivariance of PrimX removes the need for any positional encoding in Transformers.

Our largest latent primitive diffusion model $g_{\Phi}$ is a 28-layer transformer, with cross-attention layers to incorporate conditional signals, self-attention layers for modeling inter-primitive correlations, and adaptive layer normalization to inject timestep conditions.
The model $g_{\Phi}$ learns to predict at timestep $t$ given input condition signal:

$$ $g_{\Phi}({\mathcal{V}}^{t},t,{\mathbf{c}})=\{\mathrm{AdaLN}[\mathrm{SelfAttn}( \mathrm{CrossAttn}({\mathcal{V}}^{t},{\mathbf{c}},{\mathbf{c}})),t]\}^{28},$ (9) $$

where $\mathrm{CrossAttn}({\mathbf{q}},{\mathbf{k}},{\mathbf{v}})$ denotes the cross-attention layer with query, key, and value as input.
$\mathrm{SelfAttn}(\cdot)$ denotes the self-attention layer.
$\mathrm{AdaLN}(\cdot,t)$ denotes adaptive layer normalization layers to inject timestep conditioned modulation to cross-attention, self-attention, and feed-forward layers.
Moreover, we employ the pre-normalization scheme (Xiong et al., ) for training stability.
For noise scheduling, we use discrete 1,000 noise steps with a cosine scheduler during training.
We opt for “v-prediction” (Salimans & Ho, ) with Classifier-Free Guidance (CFG) (Ho & Salimans, ) as the training objective for better conditional generation quality and faster convergence:

$$ ${\mathcal{L}}_{\mathrm{diff}}(\Phi)=\mathbb{E}_{t\sim[1,T],{\mathcal{V}}^{0},{ \mathcal{V}}^{t}}[||(\sqrt{\bar{\alpha}_{t}}{\epsilon}-\sqrt{1-\bar{\alpha}_{t }}{\mathcal{V}}^{0})-g_{\Phi}({\mathcal{V}}^{t},t,\bar{{\mathbf{c}}}(b))||_{2} ^{2}],$ (10) $$

where ${\epsilon}$ is the noise sampled from Gaussian distribution, $\bar{\alpha}_{t}=\prod_{i=0}^{t}(1-\beta_{i})$ and $\beta_{t}$ comes from our cosine beta scheduler. And $b\sim{\mathcal{B}}(p_{\mathrm{0}})$ is a random variable sampled from Bernoulli distribution taking 0, 1 with probability $p_{\mathrm{0}}$ and $1-p_{\mathrm{0}}$ respectively. And the condition signal under CFG is defined as $\bar{{\mathbf{c}}}(b)=b\cdot{\mathbf{c}}+(1-b)\cdot\varnothing$, where $\varnothing$ is the learnable embedding for unconditional generation.

## 4 Experiments

Figure: Figure 4: 3DTopia-XL can generate 3D assets directly from texts or single-view images.
Refer to caption: x3.png

### 4.1 Implementation Details

#### 4.1.1 Data Standardization

##### Datasets.

The scale and quality of 3D data determine the quality and effectiveness of 3D generative models at scales.
We filter out low-quality meshes, such as fragmented shapes and large-scale scenes, resulting in a refined collection of 256k objects from Objaverse (Deitke et al., ).
Computing PrimX on large-scale datasets involves two critical steps: 1) Instantiation of sampling functions $\{F_{{\mathcal{S}}}^{\mathrm{SDF}},F_{{\mathcal{S}}}^{\mathrm{RGB}},F_{{
\mathcal{S}}}^{\mathrm{Mat}}\}$ from a GLB file and 2) Execution of the fitting algorithm in Sec. 3.1.2.
Given the massive amount of meshes from diverse sources in Objaverse, there are challenges for properly instantiating the sampling functions in a universal way such as fragmented meshes, non-watertight shapes, and inconsistent UVs.
Our standardized procedure starts with loading the GLB file as a connected graph.
We filter out subcomponents that have less than 3 face adjacency which typically represent isolated planes or grounds.
After that, all mesh subcomponents are globally normalized to the unit cube $[-1,1]$ given one unique global bounding box.
Then, we instantiate geometric sampling functions for each mesh subcomponent for SDF, texture, and material values.

##### PrimX Hyperparameters.

To get a tradeoff between computational complexity and approximation error, we choose our PrimX to have $N=2048$ primitives where each primitive’s payload has a resolution of $a=8$.
It indicates that the sequence length of our primitive diffusion Transformer is also $2048$ where each token has a dimension of $d=3+1+(a/2)^{3}=68$.
For the rapid finetuning stage for computing PrimX, we sample 500k points from the target mesh, where 300k points are sampled on the surface and 200k points are sampled with a standard deviation of 0.01 near the surface.
The finetuning stage is run for 2k iterations with a batch size of 16k points using an Adam (Kingma & Ba, ) optimizer at a learning rate of $1\times 10^{-4}$.

#### 4.1.2 Condition Signals

##### Conditioners.

The conditional generation formulation in Sec. 3.3 is compatible with most modalities.
In this paper, we mainly explored conditional generation on two modalities, images and texts.
For image-conditioned models, we leverage pretrained DINOv2 model (Oquab et al., ), specifically “DINOv2-ViT-B/14”, to extract visual tokens from input images and take it as the input condition ${\mathbf{c}}$.
For text-conditioned models, we leverage the text encoder of the pretrained image-language model (Radford et al., ), namely “CLIP-ViT-L/14”, to extract language tokens from input texts.

##### Images.

Thanks to our high-quality representation PrimX and its capability for efficient rendering, we do not need to undergo the complex and expensive rendering process like other works (Hong et al., ), which renders all raw meshes into 2D images for training.
Instead, we opt to use the front-view image rendered by Eq. 6 which is 1) efficient enough to compute on-the-fly, and 2) consistent with the underlying representation compared with the rendering from the raw mesh.

##### Text Captions.

We sample 200,000 data points from Objaverse to generate text captions. For each object, six different views are rendered against a white background. We then use GPT-4V to generate keywords based on these images, focusing on aspects such as geometry, texture, and style. While we pre-define certain keywords for each aspect, the model is also encouraged to generate more context-specific keywords. Once the keywords are obtained, GPT-4 is employed to summarize them into a single sentence, beginning with ’A 3D model of…’. These text captions are subsequently prepared as input conditions.

#### 4.1.3 Model Details

##### Training.

We train the latent primitive diffusion model $g_{\Phi}$ using a Transformer-based architecture for scalability.
Our final model is built with 28 layers with 16-head attentions and 1152 hidden dimensions, leading to a total number of $\sim$1B parameters.
We train $g_{\Phi}$ with a batch size of 1024 using an AdamW (Loshchilov & Hutter, ) optimizer.
The learning rate is set to $1\times 10^{-4}$ with a cosine learning rate warmup for 3k iterations.
The probability for condition dropout is set to $b=0.1$.
During training, we apply EMA (Exponential Moving Average) on the model’s weight with a decay of 0.9999 for better stability.
The image-conditioned model is trained on 16 nodes of 8 A100 GPUs for 350k iterations, which takes around 14 days to converge.
The text-conditioned model is trained on 16 nodes of 8 A100 GPUs for 200k iterations, which takes around 5 days to converge.

The 3D VAE for patch-wise primitive compression is built with 3D convolutional layers.
We train the VAE on a subset of the entire dataset with 98k samples, finding it generalizes well on unseen data.
The training takes 60k iterations with a batch size of 256 using an Adam (Kingma & Ba, ) optimizer with a learning rate of $1\times 10^{-4}$.
Note that, this batch size indicates the total number of PrimX samples per iteration.
As our VAE operates on each primitive independently, the actual batch size would be $N\times 256$.
We set the weight for KL regularization to $\lambda_{\mathrm{kl}}=5\times 10^{-4}$.
The training is distributed on 8 nodes of 8 A100 GPUs, which takes about 18 hours.

##### Inference.

By default, we evaluate our model with a 25-step DDIM (Song et al., ) sampler and CFG scale at 6.
We find the optimal range of the DDIM sampling steps is 25 $\sim$ 100 while the CFG scale is $4\sim 10$.
The inference can be efficiently done on a single A100 GPU within 5 seconds.

### 4.2 Representation Evaluation

Figure: Figure 5: Evaluations of different 3D representations. We evaluate the effectiveness of different representations in fitting the ground truth’s shape, texture, and material (right). All representations are constrained to a budget of 1.05M parameters. PrimX achieves the highest fidelity in terms of geometry and appearance with significant strength in runtime efficiency (Table 1) at the same time.
Refer to caption: x4.png

**Table 1: Quantitative evaluations of different 3D representations. We evaluate the approximation error of different representations for shape, texture, and material. All representations adhere to a parameter budget of 1.05M. PrimX shows the best fitting quality, especially for the geometry (also shown in Figure 5), while having the most speedy fitting runtime. The top three techniques are highlighted in red, orange, and yellow, respectively.**
| Representation | Runtime | CD $\times 10^{-4}\downarrow$ | PSNR-$F_{{\mathcal{S}}}^{\mathrm{SDF}}$ $\uparrow$ | PSNR-$F_{{\mathcal{S}}}^{\mathrm{RGB}}$ $\uparrow$ | PSNR-$F_{{\mathcal{S}}}^{\mathrm{Mat}}$ $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| MLP | 14 min | 4.502 | 40.73 | 21.19 | 13.99 |
| MLP w/ PE | 14 min | 4.638 | 40.82 | 21.78 | 12.75 |
| Triplane | 16 min | 9.678 | 39.88 | 18.28 | 16.46 |
| Dense Voxels | 10 min | 7.012 | 41.70 | 20.01 | 15.98 |
| PrimX | 1.5 min | 1.310 | 41.74 | 21.86 | 16.50 |

##### Evaluation Protocol.

We first evaluate different designs of 3D representations in the context of 3D generative modeling.
Our evaluation principles focus on two aspects: 1) runtime from GLB mesh to the representation, and 2) approximation error for shape, texture, and material given a fixed computational budget.
Given 30 GLB meshes randomly sampled from our training dataset, we take the average fitting time till convergence as runtime, which is measured as the wall time on an A100 GPU.
For geometry quality, we evaluate the Chamfer Distance (CD) between the ground truth mesh and extracted mesh after the fitting and the Peak Signal-to-Noise Ratio (PSNR) of SDF values of 500k points sampled near the shape surface.
For appearance quality, we evaluate the PSNR of RGB (albedo) and materials values of 500k points sampled near the surface.

##### Baselines.

Given our final hyperparameters of PrimX, where $N=2048,a=8$, we fix the number of parameters of all representations to $2048\times 8^{3}\approx 1.05\mathrm{M}$ for comparisons.
We compare four alternative representations:
1) MLP: a pure Multi-Layer Perceptron with 3 layers and 1024 hidden dimensions;
2) MLP w/ PE: the MLP baseline with Positional Encoding (PE) (Mildenhall et al., ) to the input coordinates;
3) Triplane (Chan et al., ): three orthogonal 2D planes with a resolution of $128\times 128$ and 16 channels, followed by a two-layer MLP decoder with 512 hidden dimensions.
4) Dense Voxels: a dense 3D voxel with a resolution of $100\times 100\times 100$. All methods are trained with the same objectives (Eq. 7) and points sampling strategy (Sec. 4.1.1) as ours.

##### Results.

Quantitative results are presented in Table 1, which shows that PrimX achieves the least approximation error among all methods, especially for geometry (indicated by CD).
Besides the best quality, the proposed representation demonstrates significant efficiency in terms of runtime with nearly 7 times faster convergence speed compared with the second best, making it scalable on large-scale datasets.
Figure 5 shows qualitative comparisons.
MLP-based implicit methods appear to have periodic artifacts, especially for the geometry.
Triplane and dense voxels yield bumpy surfaces as well as grid artifacts around the shape surface.
Instead, PrimX produces the best quality with smooth geometry and fine-grained details like the thin and tapering beard.

Figure: Figure 6: Image-to-3D comparisons. For each method, we take the textured mesh predicted from the input image into Blender and render it with the target environment map. We compare our single-view conditioned model with sparse-view reconstruction models and image-conditioned diffusion models. 3DTopia-XL achieves the best visual and geometry quality among all methods. Thanks to our capability to generate spatially varied PBR assets shown on the rightmost, our generated mesh can also produce vivid reflectance with specular highlights and glossiness.
Refer to caption: x5.png

### 4.3 Image-to-3D Generation

In this section, we compare our single-view conditioned generative model with state-of-the-art methods suitable for image-to-3D synthesis.

##### Comparison Methods.

We run evaluations against two types of methods: 1) sparse-view reconstruction models, and 2) image-conditioned diffusion models.
The reconstruction-based models, like LGM (Tang et al., ), InstantMesh (Xu et al., ), Real3D (Jiang et al., ), CRM (Wang et al., ), are deterministic methods that learn to reconstruct 3D objects given four or six input views.
They enable single-view to 3D synthesis by leveraging pretrained diffusion models (Shi et al., ; Li et al., ) to generate multiple views from the input single image.
However, as methods for reconstruction heavily rely on the input multi-view images, those methods suffer from multi-view inconsistency caused by the frontend 2D diffusion models.
The feed-forward diffusion models, like CraftsMan (Li et al., ), Shap-E (Jun & Nichol, ), LN3Diff (Lan et al., ), are probabilistic methods that learn to generate 3D objects given input image conditions.
All methods above only model the shape and color without considering roughness and metallic while our method is suitable to produce those assets.

##### Results.

Figure 6 demonstrates qualitative results.
To fairly compare the capability of generating 3D assets ready for rendering, we take the exported textured mesh from each method into Blender (Community, ) and render it with the target environment map.
For methods that cannot produce PBR materials, we assign the default diffuse material.
Existing reconstruction-based models fail to produce good results which may suffer from multiview inconsistency and incapability to support spatially varied materials.
Moreover, these reconstruction models are built upon triplane representation which is not parameter-efficient.
This downside limits the spatial resolution of the underlying 3D representation, leading to the bumpy surface indicated by the rendered normal.
On the other hand, existing 3D diffusion models fail to generate objects that are visually aligned with the input condition.
While CraftsMan is the only method that has comparable surface quality as ours, they are only capable of generating 3D shapes without textures and materials.
In contrast, 3DTopia-XL achieves the best visual and geometry quality among all methods.
Thanks to our capability to generate spatially varied PBR assets (metallic/roughness), our generated mesh can also produce vivid reflectance with specular highlights even under harsh environmental illuminations.

##### User Study.

We conduct an extensive user study to quantitatively evaluate image-to-3D performance.
We opt for an output evaluation (Bylinskii et al., ) for user study, where each volunteer is shown with a pair of results comparing a random method against ours, and asked to choose the better one in four aspects: 1) Overall Quality, 2) Image Alignment, 3) Surface Smoothness, and 4) Physical Correctness.
A total number of 48 paired samples are provided to 27 volunteers for the flip test.
We summarize the average preference percentage across all four dimensions in Figure 7.
3DTopia-XL is the best one among all methods.
Although the image alignment of our method is only a slight improvement against reconstruction-based methods like CRM, the superior quality of geometry and the ability to model physically based materials are the keys to producing the best overall quality in the final rendering.

Figure: Figure 7: User Study. We quantitatively evaluate comparison methods by conducting preference tests against our method on four dimensions. The results show that 3DTopia-XL has the highest preference rate compared with each of other methods.
Refer to caption: x6.png

### 4.4 Text-to-3D Generation

We also demonstrate our capability for native text-to-3D generation as shown in Figure 4.
Note that, as a 3D native diffusion model, our text-driven generation is done by directly conditioning the model on textual inputs, without relying on complicated text-to-multiview followed by reconstruction models.

Moreover, we also conduct quantitative evaluations against native text-to-3D generative models.
Given a set of unseen text prompts, we take the CLIP Score as the evaluation metric which is the cosine similarity between the text embedding and image embedding in the joint text-image space of the CLIP model (Radford et al., ).
We take the front-view rendering from each method to compute the image embedding.
We mainly compare two methods with open-source implementations: Shap-E (Jun & Nichol, ) and 3DTopia (Hong et al., ).
Shap-E directly generates implicit functions of 3D objects conditioned on texts.
3DTopia adopts a hybrid 2D and 3D diffusion prior by using feedforward triplane diffusion followed by optimization-based refinement.
As shown in Table 5, our method achieves better alignment between input text and rendering of generated asset.

### 4.5 Further Analysis

#### 4.5.1 Number and Resolution of Primitives

As a structured 3D representation and a serialized representation, the number of primitives $N$ and the resolution of each primitive $a$ are two critical factors for the efficiency-quality tradeoff in PrimX.
More and larger primitives often lead to better approximation quality.
However, it results in a longer sequence length and deeper feature dimensions, causing inefficient long-context attention computation and training difficulty of the diffusion model.
Therefore, we explore the impact of the number and resolution of primitives on different parameter budgets.
We evaluate the PSNR of SDF, albedo, and material values given 500k points sampled near the surface.
As shown in Table 2, given a fixed parameter count, a longer sequence of primitives appears to have a better approximation of SDF, texture, and material.
Moreover, increasing the resolution of each primitive can reduce the approximation error.
However, its benefit is marginal as the number of primitives is enough.
The visualization in Figure 8 also confirms this observation.
The alternative with $(N=64,a=32)$ produces poor geometry even with more parameter count since larger local primitives have higher chances to waste parameters in empty space.
Furthermore, a longer sequence will increase the GFlops of DiT which also leads to better generation quality (Table 3).
Therefore, we tend to use a longer sequence of primitives with a relatively small local resolution.
Note that, the compression rate of our VAE (Sec. 3.2) also influences the hyperparameter of PrimX, which we will investigate in the next section.

Figure: Figure 8: Ablation studies of the number and resolution of primitives. Our final setting $(N=2048,a=8)$ has the optimal approximation quality of ground truth, especially for fine-grained details like thin rotor blades. We visualize the corresponding PrimX at the bottom.
Refer to caption: x7.png

**Table 2: Quantitative analysis of the number ($N$) and resolution ($a$) of primitives.**
| # Primitives | Resolution | #  Parameters | PSNR-$F_{{\mathcal{S}}}^{\mathrm{SDF}}$ $\uparrow$ | PSNR-$F_{{\mathcal{S}}}^{\mathrm{RGB}}$ $\uparrow$ | PSNR-$F_{{\mathcal{S}}}^{\mathrm{Mat}}$ $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| $N=64$ | $a^{3}=32^{3}$ | 2.10M | 61.05 | 22.18 | 18.10 |
| $N=256$ | $a^{3}=16^{3}$ | 1.05M | 59.05 | 23.50 | 18.61 |
| $N=512$ | $a^{3}=8^{3}$ | 0.26M | 59.57 | 22.58 | 18.50 |
| $N=512$ | $a^{3}=16^{3}$ | 2.10M | 62.89 | 23.92 | 18.21 |
| $N=2048$ | $a^{3}=8^{3}$ | 1.05M | 62.52 | 24.23 | 18.53 |

#### 4.5.2 Patch Compression Rate

The compression rate of our primitive patch-based VAE is also an important design choice.
Overall, as a patch-based compression, we aim to do spatial compression of each primitive to save computation instead of semantic compression (Rombach et al., ).
Empirically, a higher compression rate leads to a more efficient latent diffusion model, indicating larger batch sizes or larger models when scaling up.
On the contrary, extreme compression often accompanies loss of information.
Therefore, we analyze different compression rates given two different sequence lengths $N=256$ and $N=2048$ with the same parameter count of PrimX.
For the evaluation metric, we compute the PSNR between the VAE’s output and input on 1k random samples from the dataset to measure its reconstruction quality.
Table 5 shows the results where the final choice of $N=2048$ with compression rate $f=48$ achieves the optimal VAE reconstruction.
The setting with $N=256,f=48$ has the same compression rate but lower reconstruction quality and a latent space with higher resolution, which we find difficulty in the convergence of the latent primitive diffusion model $g_{\Phi}$.

**Figure 9: Scaling up 3DTopia-XL improves FID.**
| Setting | Rendering-FID $\downarrow$ |
| --- | --- |
| $N=256$ | 76.31 |
| $N=2048$ | 16.16 |

**Table 4: Analysis of different compression rates for VAE. $f$ stands for the compression rate between input and latent.**
| # Primitives | VAE input | Latent | $f$ | PSNR $\uparrow$ |
| --- | --- | --- | --- | --- |
| $N=256$ | $6\times 16^{3}$ | $6\times 4^{3}$ | 64 | 22.92 |
| $N=256$ | $6\times 16^{3}$ | $1\times 4^{3}$ | 384 | 19.80 |
| $N=256$ | $6\times 16^{3}$ | $1\times 8^{3}$ | 48 | 23.33 |
| $N=2048$ | $6\times 8^{3}$ | $1\times 2^{3}$ | 384 | 18.48 |
| $N=2048$ | $6\times 8^{3}$ | $1\times 4^{3}$ | 48 | 24.51 |

#### 4.5.3 Scaling

We further investigate the scaling law of 3DTopia-XL against model sizes and iterations.
For metrics, we use Fréchet Inception Distance (FID) computed over 5k random samples without CFG guidance.
Specifically, we consider Latent-FID which is computed in the latent space of our VAE and Rendering-FID which is computed on the DINO (Oquab et al., ) embeddings extracted from images rendered with Eq. 6.
Figure 9 shows how Latent-FID and Rendering-FID change as the model size increases.
We observe consistent improvements as the model becomes deeper and wider.
Table 3 also demonstrates that longer sequence (smaller patches) leads to better performance, which may come from the findings in the vanilla DiT that increasing GFlops leads to better performance.

#### 4.5.4 Sampling Diversity

At last, we demonstrate the impressive sampling diversity of 3DTopia-XL as a generative model, as shown in Figure 10.
Given the same input image and varying random seeds, our model can generate diverse high-quality 3D assets with different geometry and spatially varied PBR materials.

Figure: Figure 10: Sampling diversity. Given the same input image, 3DTopia-XL can generate diverse 3D assets by varying random seeds only. Zoom in for diverse spatially varied PBR materials and shapes.
Refer to caption: x9.png

## 5 Discussion

We present 3DTopia-XL, a native 3D diffusion model for PBR asset generation given textual or visual inputs.
Central to our approach is PrimX, an innovative primitive-based 3D representation that is parameter-efficient, tensorial and renderable.
It encodes shape, albedo, and material into a compact $N\times D$ tensor, enabling the modeling of high-resolution geometry with PBR assets.
Tailored for PrimX, several training and inference techniques are introduced to ensure the generated results can be packed into the GLB file in high quality, ready for downstream applications in the graphics engine.
Extensive evaluations demonstrate the superiority of 3DTopia-XL in text-to-3D and image-to-3D tasks, showing its great potential for 3D generative foundation models.

##### Limitations and Future Works.

It is important to note that 3DTopia-XL has been trained on a considerably large-scale dataset.
However, there is still room for improvement in terms of quality.
Different from existing high-quality 3D diffusion models Yariv et al. (); Zhang et al. () which operate on 3D representations that are not differentiably renderable, 3DTopia-XL maintains the ability to directly learn from 2D image collections thanks to PrimX’s capability of differentiable rendering (Eq. 6).
This opens up new opportunities to learn 3D generative models from a mixture of 3D and 2D data, which can be a solution to the lack of high-quality 3D data.
Moreover, as an explicit representation, PrimX is interpretable and easy to drive.
By manipulating primitives or groups of primitives, it is also fruitful to explore dynamic object generation and generative editing.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Questions：
See weakness 1, 2.
soundness: 2
strengths: Strengths:
1）It extends the 3D representation from M-SDF to support color and PBR materials and integrates it into the generative framework, enabling the generation of 3D assets containing PBR materials from text and image conditions. This introduces a new approach to 3D representation for native 3D generative tasks. 
2）Ablation studies on the number and resolution of primitives validated the selection of resolution. 
3）The presentation of the paper is good.
summary: This paper introduces 3DTopia-XL, a 3D generative model designed to create 3D assets from both images and text. It extends 3D representation from M-SDF to support color and PBR materials, encoding shape, albedo, and material fields into a compact tensor format. The model framework, built on the Diffusion Transformer (DiT), incorporates Primitive Patch Compression and Latent Primitive Diffusion. Extensive experiments show that 3DTopia-XL effectively generates convincing 3D assets from both textual and visual inputs.
weaknesses: Weakness：
1）My concern is regarding the accuracy of the PBR materials. The fourth row in Figure 7 shows noticeably incorrect PBR materials. The paper claims to generate 3D assets with PBR textures, so it should include experiments on a 3D synthetic dataset where the ground truth metallic and roughness data are available.  For example, quantitative evaluations can be handily conducted on a synthetic dataset with ground truth PBR data using  PSNR or SSIM for albedo, metallic, and roughness maps.
2）As mentioned in line 292, the paper introduces patch-based compression aimed at incorporating inter-channel correlations between geometry, color, and materials. However, there is no experimental evidence to support this. As shown in Table 3, comparing line 1 and line 2, when the grid compression rate remains the same, reducing the feature dimensions representing different geometry, color, and materials from 6 to 1 actually results in a decrease in PSNR.  Authors could provide additional experiments or analyses that directly demonstrate the benefits of incorporating these correlations, such as comparing the proposed approach to a baseline that processes geometry, color, and materials independently.
3）It seems that the experimental results are missing an ablation study where patch-based compression is not applied. It would be even better if qualitative comparison results could be added.  The author could include an ablation study that compares their patch-based compression approach to a baseline without compression, showing both quantitative metrics and qualitative visual comparisons.
4）The current generated results lack diversity. It is better to include quantitative measures of diverse objects with complex topologies, such as challenging chairs, plants, and buildings, 

In summary, I have concerns about the novelty of the paper. The 3D representation primarily extends existing work on M-SDF, and the performance of the extended features, such as PBR material, is not satisfactory. Additionally, despite the use of patch-based compression, the reported computational cost is substantial, requiring 16 nodes of 8 A100 GPUs around 14 days to converge. In comparison, InstantMesh only needs 8 NVIDIA H800 GPUs. Moreover, compared with instant mesh, the generated geometric results in this paper tend to be overly smooth, losing geometric details, as shown in Figure 5.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: Please see the weaknesses.
soundness: 3
strengths: 1. The paper introduces a novel tensor-based 3D representation called PrimX, which efficiently encodes geometry, albedo, and material properties. This representation allows for high-quality, physically-based rendering (PBR) assets with smooth geometry and intricate texture details, offering a more compact and efficient solution compared to traditional methods.

2. The proposed 3DTopia-XL model leverages a Diffusion Transformer to perform 3D generation from textual or visual inputs. This framework, combined with PrimX, enables high-quality and large-scale 3D asset generation, accommodating complex tasks like image-to-3D and text-to-3D conversion with better quality and efficiency compared to existing approaches.

3. The paper provides a detailed and efficient method for converting the PrimX representation back into a textured mesh in GLB format. This process ensures high-quality asset extraction by applying techniques like UV unwrapping, dilating, and inpainting, making the output ready for various downstream applications in graphics engines, with minimal additional processing required.
summary: 3DTopia-XL is a high-quality 3D generative model designed to meet the growing demand for efficient 3D asset creation in fields such as gaming, virtual reality, and film. It introduces an innovative representation, PrimX, which encodes complex shapes, textures, and materials into a compact tensor format, supporting Physically Based Rendering (PBR) for realistic visuals. The model employs a Diffusion Transformer framework that facilitates efficient 3D asset generation from text or image inputs through its unique Primitive Patch Compression and Latent Primitive Diffusion techniques. Additionally, 3DTopia-XL includes optimized algorithms for extracting detailed PBR assets, ensuring easy integration into graphics engines. Experiments demonstrate that 3DTopia-XL significantly outperforms existing methods in producing high-resolution, finely detailed 3D assets, making it a promising foundation for advanced 3D generative modelling applications.
weaknesses: 1. In Eq. 3, v is defined as a set of N volumetric primitives; however, it is not clear what each primitive denotes.

2. It’s not immediately clear what N and D represent. Are these dimensions related to the number of primitives and data channels (e.g., geometry and texture details), or is there another interpretation? Further defining these terms would add clarity.

3. The paper mentions using initialized positions and scales, where I is a unit local voxel grid. However, it’s not entirely clear how these initialized positions and scales contribute to the overall volumetric representation in the PrimX model. More context on what this setup accomplishes geometrically would help clarify this step.

4. Unclear about the role of permutation equivariance in PrimX and its impact on the Transformer model's design. Clarifying how permutation equivariance inherently maintains structure among primitives or why it aligns well with Transformer processing would be helpful.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - After generating the latent PrimX via latent primitive diffusion (**Fig.** 3), is it necessary to convert PrimX to mesh representations for the generation of 3D PBR assets in practical applications?
- In **Sec.** A.2.4,  the runtime of the "PrimX to Mesh" process (which includes UV unwrapping, Rasterizer, and Marching Cubes) is missed.
soundness: 3
strengths: **1.** The paper is well-motivated, effectively addressing the PBR rendering challenges for 3D asset generation. 

**2.** The design of the model and its pre-processing pipeline (including "Mesh to PrimX", Prim Patch Encoder, and Primitive Diffusion) are well-conceived.

**3.**  The paper is well-written and provides extensive qualitative results to demonstrate its superiority (e.g., high geometric fidelity of PBR assets). The authors also include comprehensive ablation study and user study, which confirm the effect of each component and highlight the model's performance.
summary: - This paper introduces a 3D Physically-Based Rendering (PBR) assets  generation method utilizing an expressive and efficient 3D representation, dubbed PrimX.
- The method leverages two main techniques:  Primitive Patch Compression and Latent Primitive Diffusion, effectively balancing generation speed with quality, achieving PBR asset creation within just 5 seconds of denoising.
- The proposed method consistently outperforms baseline methods in text-to-3D and image-to-3D settings.
weaknesses: **1.** The primary concern is the $\textcolor{blue}{\text{apparent similarity}}$ between PrimX (the core design of this work) and PrimDiffusion (Chen et al., 2023b), which is scarcely mentioned, even in the "difference with related work"  ( **Sec.** A.1.1). The authors should clarify the fundamental differences between  two methods and prove (or clarify) why extending 3D **Primitives** to **PrimX** which including the PBR attribute (Material $\in \mathbb{R}^{a^{3}\times 2}$) is non-trivial.

**2.** Although the authors provide geometric and texture reconstruction results (e.g., **Fig.** 1, **Fig.** 5, and **Fig.** 7), they only show the predicted RGB images (texture) from the front view which could potentially be directly derived from the input image. The authors should showcase texture rendering from diverse viewpoints, including  side and  $\textcolor{blue}{\textbf{back views}}$, to better validate the model's capabilities.

**3.** Compared to cutting-edge 3D generation works (e.g., Real3D, LGM, CRM, InstantMesh), 3D Topia-XL requires substantial training resources and time, taking approximately **128** NVIDIA A100 GPUs and **14** days to converge.  Apart from the mentioned "v-prediction", could pre-trained 2D models or other training strategies help accelerate convergence? How can the authors reduce training costs to democratize the proposed 3D generative model?

**4.** In the Image-to-3D setting (**Sec.** 4.2), 3DTopia-XL achieves overwhelming superiority in PBR rendering due to baseline methods' incapability to support spatially varied materials. It may be beneficial for the authors to render objects without reflections in a fixed and simplified lighting environment, then quantitatively compare with other methods on a common GSO dataset and report PSNR, SSIM, LPIPS, and CD metrics.

If the authors could address my concerns by providing corresponding quantitative or qualitative results based on **the weaknesses** and **review feedback**, I will consider improving my score.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. The proposed method does not need differentiable rendering. It is also unsure how to train a latent 3D diffusion model using the so-called direct 2D supervision. That said, the claim of the possibility to learn from 2D images using PrimX is over-claimed.
2. How are the primitives ordered for a single object during the diffusion process? Will the order affect generation quality?
3. What is the advantage of using a local VAE instead of a global one, does it bring better quality? Or is it just cheaper to train? If you can train a DiT of ~1B, training a global VAE shouldn't be a big burden. The authors should provide this additional ablation on the VAE.
4. Although the authors claimed to be “rapidly tensorizable”, the speed for texture mesh to PrimX is actually very slow, as each shape takes 1.5 min for the conversion. I wonder how long it takes to process all these 250k training objects.
5. 4.1 representation evaluation: why not compare with sparse voxel and vecset? Also, what if the number of params of all methods is increased, like 5M and even 25M? Will the proposed representation still have privilege? The proposed representation also failed to reproduce texture details, e.g. the eyes in Fig 4. Can this problem be alleviated with more parameters? Besides, I think a more valuable comparison would be fixing the number of parameters in the encoded/latent space, as this is where the diffusion model runs.

Given the limited improvement over existing methods, the untenable motivation, and the missing evaluations, I lean to reject this paper. However, I'm willing to raise my score if the authors provide justifications for my above questions, and add missing results as listed in the weaknesses.
soundness: 2
strengths: 1. The writing is clear and easy to follow.
2. Presented a novel representation for 3D object generation.
3. The proposed method can also generate PBR.
4. Thorough ablations are done regarding the hyper-parameters of the proposed method.
5. The scaling result of the method shows its potential to further increase quality by increasing data and model size.
summary: The paper presents a method for 3D shape and PBR texture generation. The key to the method is the proposed representation PrimX — each object is represented by a set of primitive cubes on the surface, with each primitive containing position, scale, and volumetric SDF and PBR values, thus forming a tensorized representation. This representation can be converted to and from textured meshes, and compressed into a more compact latent space using a primitive-level VAE. A DiT is trained on this latent space for conditioned 3D generation. The authors conducted some comparisons and ablations to demonstrate the superiority of the method.
weaknesses: 1. Discussion and comparison to sparse-voxel based generation methods are omitted, including but not limited to:
    1. Generalized Deep 3D Shape Prior via Part-Discretized Diffusion Process, CVPR 2023
    2. SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation, CVPR 2023
    3. Locally Attentional SDF Diffusion for Controllable 3D Shape Generation, SIGGRAPH 2023
    4. One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion, CVPR 2024
    5. Make-A-Shape: a Ten-Million-scale 3D Shape Model, ICML 2024
    6. XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies, CVPR 2024
    7. MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model, NeurIPS 2024

    These methods are also parameter-efficient thanks to sparsity.
2. The shape and texture generation quality is lower than sota methods like CLAY, MeshLRM, and MeshFormer. The PBR also lacks high-frequency details.
3. Factual error in A.1.1: vecset can be extended to store color or PBR properties, by adding these properties along with PE in the encoding stage and decoding them by querying with positions. They are also differentiable renderable when combined with a differentiable marching cube algorithm and a differentiable rasterizer.
4. 4.2 Fig. 5 comparison with other methods: in the qualitative result part, all the meshes are not aligned, placed at different locations, and with different scales. Also, quantitative results on the image-to-3D task besides user studies are not reported.
5. Although claimed in the abstract, no real-world test cases are shown.