Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer
Authors: Shuang Wu
Youtian Lin
Yifei Zeng
Feihu Zhang
Jingxi Xu
Philip Torr
Xun Cao
Yao Yao
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Generation
Diffsion Model
Primary area: generative_models
TL;DR: We propse a novel approach for direct 3D shape generation from a single image, bypassing the need for multi-view reconstruction.

Abstract: Generating high-quality 3D assets from text and images has long been challenging, primarily due to the absence of scalable 3D representations capable of capturing intricate geometry distributions. In this work, we introduce Direct3D, a native 3D generative model scalable to in-the-wild input images, without requiring a multi-view diffusion model or SDS optimization. Our approach comprises two primary components: a Direct 3D Variational Auto-Encoder (D3D-VAE) and a Direct 3D Diffusion Transformer (D3D-DiT). D3D-VAE efficiently encodes high-resolution 3D shapes into a compact and continuous latent triplane space. Notably, our method directly supervises the decoded geometry using a semi-continuous surface sampling strategy, diverging from previous methods relying on rendered images as supervision signals. D3D-DiT models the distribution of encoded 3D latents and is specifically designed to fuse positional information from the three feature maps of the triplane latent, enabling a native 3D generative model scalable to large-scale 3D datasets. Additionally, we introduce an innovative image-to-3D generation pipeline incorporating semantic and pixel-level image conditions, allowing the model to produce 3D shapes consistent with the provided conditional image input. Extensive experiments demonstrate the superiority of our large-scale pre-trained Direct3D over previous image-to-3D approaches, achieving significantly better generation quality and generalization ability, thus establishing a new state-of-the-art for 3D content creation. Project page: https://www.neural4d.com/research/direct3d.

Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer
Shuang Wul
Youtian Lin
Feihu Zhang'
Yifei Zeng" 1.2
Jingxi Xu'
Philip Torr
Xun Cao
Yao Yao
Tech
Nanjing University
University of Oxford
(sushuang,
anjaailnnj...... edu, on
yaoyao@nju. edu. cn
Abstract
Generating high-quality 3D assets from text and images has long been challenging, primarily due to the absence of scalable 3D representations capable of capturing intricate geometry distributions. In this work, we introduce Direct3D, a native 3D generative model scalable to in-the-wild input images, without requiring multi- view diffusion model or SDS optimization. Our approach comprises two primary components: Direct 3D Variational Auto-Encoder (D3D- VAE) and Direct 3D Diffusion Transformer (D3D-DiT). D3D-VAE efficiently encodes high-resolution 3D shapes into compact and continuous latent triplane space. Notably, method directly supervises the decoded geometry using a semi-continuous surface sampling strategy, diverging from previous methods that rely rendered images as supervision signals. D3D-DiT models the distribution of encoded 3D latents and is specifically designed to fuse positional information from the three feature maps the triplane latent, enabling a native 3D generative model scalable to large-scale 3D datasets. Additionally, we introduce an innovative image-to-3D generation pipeline incorporating semantic-level and pixel-Ievel image conditions, allowing the model to produce 3D shapes consistent with the provided conditional image input, Extensive experiments demonstrate the superiority of our large-scale pre-trate tre-trained Direct3D over previous image-to-3D approaches, achieving significantly better generation quality and generalization ability. thus establishing a new state-of- thee--f--teerrt for 3D content creation. Project page: htts://www.neuralae hww.://wwww....ooo////e/e/eeaaaa
Introduction
In recent years, substantial advancements have been made in 3D shape generation through the utilization of diffusion models [13, 51]. Inspired by the efficacy demonstrated in text-to-2D image generation, these methods seek to extend capabilities diffusion models to the realm 3D shape generation through extensive training diverse 3D datasets. Various approaches have explored diverse 3D representations, including point clouds [37, 38], voxels [45], and SDFs [35], aiming not only to faithfully capture object appearance but also preserve intricate geometric details. However, existing large-scale 3D datasets, such as ObjverseXL [6], are constrained both in the quantity and
'Equal contribution.
Chief scientific advisor of Dream'Tcch, all work was done at DreamTech.
Corresponding author. This research was supported by DreamTech, and belongs to DreamTech.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Figure 1: Direct3D is novel image-to-3D generation method that directly trains on larger-scale 3D datascts and performs state-of-the-art generation quality and generalizability. We achieve this by designing novel 3D latent diffusion model take image the prompt and generate high-quality 3D shapes that highly consistent with input images. shown ahove, our method can generate shapes from existing text-to-image diffusion models, which indicates that our method generalizes to in-the-wild images, while it only trains on 3D data.
diversity of shapes compared to their 2D counterparts like Laion5B [47], which contains 5 billion images, while ObjverseXL only comprises 10 million 3D shapes.
To address this limitation, many existing methods [5, 4, 9, 26, 27, 28, 29, 31, 34, 43, 65] employ pipeline where multi-view images an object are initially generated from a single image using multi-view diffusion model, Subsequently, techniques such sparse view reconstruction methods [23, 32, 55, 62] or score distillation sampling (SDS) optimization [42, 43, 49, 54] are applied to fuse these multi-view images into 3D shapes. While this pipeline can result in high-quality 3D shape creation, indirect generation from multi-view images raises efficiency concerns. Additionally, the quality of the resulting shape is heavily dependent on the fidelity of the multi-view images, often leading to detail loss or reconstruction failures.
In this paper, we eschew the conventional approach of indirectly generating multi-view images and instead advocate the direct generation of 3D shapes from single-view images, leveraging a native 3D diffusion model. Inspired by the success of latent diffusion models in 2D image generation, we propose the utilization of 3D variational auto-encoder (VAB) [20] to encode space, followed by diffusion transformer model (DiT) [40] to generate 3D shapes from this latent space, conditioned on an image input. However, efficiently encoding a 3D shape into latent space conducive to diffusion model training is challenging, as is decoding the latent representation back into 3D geometry. Previous approaches have employed multi-view images as indirect supervision [19, 22, 35, 61] through differentiable rendering, but still encounter accuracy and efficiency issues. To
shapes into a latent

---

address these challenges, we employ transformer model to encode high-resolution point clouds into an explicit triplane latent, which has been widely used in 3D reconstruction methods [2] for its efficiency. While the latent triplane is intentionally set with a low resolution, we introduce convolutional neural network to upsample the latent resolution and decode it into a high-resolution 3D occupancy grid. Furthermore, to ensure precise supervision of the 3D occupancy grid, we adopt semi-continuous surface sampling strategy, enabling the sampling and supervision of surface points in both continuous and discrete manners. This approach facilitates the encoding and reconstruction of 3D shapes within a compact and continuous explicit latent space.
For image-to-3] generation, we further leverage an image input as a condition to the 3D diffusion transformer. This involves arranging the 3D latent space as combination of three orthogonal views of a 3D shape and incorporating pixel-level image information into each DiT block to enhance conditional consistency. Furthermore, we introduce cross-attention layers into each DiT block to incorporate semantic-level image information, therehy facilitating the generation of high-quality 31) shapes consistent with input images.
We demonstrate the high-quality 3D generation and strong generalization abilities of the proposed Direct3D approach through extensive experiments. Figure illustrates the 3D generation results of our method on the in-the-wild images generated from text-to-image model. To summarize, the major contributions of this work include:
We introduce Direct3D, to our best knowledge, the first native 3D generative model scalable to in-the-wile wild input images (e-g.. from Flux [21], Hunyuan-Din [24] or SDXL [4]]). This enables high-fidh figh-fideitty image-to-3D generation without the need for multi-view diffusion models or SDS optimization.
We propose D3D-VALE a novel 3D variational auto-encoder effectively encoding a 3D point cloud into a triplane latent. Instead of using rendered images supervision signals, we supervise the decoded geometry directly using a semi-continuous surface sampling strategy preserve detailed 3D information in the latent triplane.
We present D3D-DiT, scalable mnndecconditioned 3D diffusion transformer capable of generating 3D asserts consistent with input images. The D3D-DiT is specially designed better fuse the positional information from the latent triplane and effectively integrates pixel-level and semantic-level information from the input image.
We demonstrate through extensive experiments that our large-scale pre-trained Direct3D model surpasses previous image-to-3D approaches in terms of generation quality and generalization ability, setting a new state-of-the-art for the task of 3D content creation.
2 Related Work
2.1 Neural 3D Representations for 3D Generation
Neural 3D representations are essential for 3D generation tasks. The introduction of Neural Radiance Fields (NeRF) [36] has significantly advanced 3D generation. Building on NeRF, DreamFusion [42] introduced Score Distillation Sampling (SDS) method generate 3D shapes using an off-the- shelf 2D diffusion model from any text prompt. Many subsequent methods have explored various representations to enhance the speed and quality of 3D generation. For instance, Magic3D [25] improves generation quality by introducing a second stage using the DMtet [48] representation, which combines Signed Distance Function (SDF) with a tetrahedral grid to represent the 3D shape.
Beyond SDS-based methods, some approaches use directly trained networks to generate different representations [16, 17, 59]. For example, LRM [16] uses triplane NeRF representations network outputs. significantly speeding up the generation process, albeit with some loss in quality. Another approach, One-2-3-45++ [26], proposes use a 3D occupancy grid the output representation to enhance geometric quality.
2.2 Multi-view Diffusion
Following the success of novel view prediction methods using diffusion models, such as 7ero123 [28]. which generates different unknown views of object from single image and text guidance.
3

---

MVDream [49] extends novel view diffusion to generate multiple views of an object at once, improving consistency across views. Imagedream [54] further enhances generation quality by introducing a novel image conditional module. Some methods adopt this approach to first generate multi-view images of an object and then reconstruct the 3D shape from these views using sparse reconstruction [23, 31, 55, 62]. Instant3D [23] proposes a reconstruction model that takes four multi- view images as input and reconstructs a NeRF representation of the 3D shape. Many subsequent methods have improved on this by enhancing multi-view or reconstruction models [53, 57, 58).
2.3 Direct 3D Diffusion
Despite the challenges of directly training a 3D diffusion model, such as the lack of a diffusible 3D representation, various strategies have been explored. One line of work fits multiple NeRFs to obtain a neural representation of 3D datasets and then applies a diffusion model to generate NeRFs from this learned representation [50]. However, separate training of NeRFs
hinder the diffusion model's ability to generalize to more diverse 3D shapes. 3euienNeural [50] proposes joint training of triplane fitting of the 3D shape with occupancy as direct supervision to train the triplane reconstruction model.
Another line of work leverages VAEs to encode 3D shapes into a latent space and trains a diffusion model on this latent space generate 3D shapes [15, 19, 22, 61J. For instance. Shap-E [19] uses pure transformer VAE to encode a point cloud and image of 3D shape into an implicit latent space, which is then recovered into NeRF and SDF field. 3DGen [10] encodes only the point cloud of a 3D shape into an explicit triplane latent space, enhancing generation efficiency. Similar to previous works that fit multiple NeR Fs, 3DTopia [15] fits multiple triplane NeRFs and encodes the triplane into latent space for which a diffusion model is trained to generate 3D shapes. 2eShappeVecSee [60] and Michelangelo [64] employ 3D occupancy as the output representation for the VAE but use multiple ID vectors as implicit latent space instead of a triplane.
However, these methods often rely rendering loss supervise the VAE reconstruction, resulting in suboptimal reconstruction and generation quality. Additionally, using implicit latent representations not designed for efficient encoding and lacking compact explicit 3D representations for diffusion further limits their performance. Our 3D VAE combines the advantages of explicit 3D latent representation and direct 3D supervision to achieve high-quality VAE reconstruction, ensuring robust 3D shape generation. Furthermore, our design for the diffusion architecture specifically addresses conditional 3D latent generation. Our 3D DIT facilitates pixel-level and semantic-level 31D-spccffic image conditioning, allowing the diffusion process to generate highly detailed 3D shapes consistent with the condition images.
3 Methods
Inspired by LDM [46], we train a latent diffusion model for 3D generation within a 3D latent space. Unlike previous methods [19, 64] that typically rely on ID implicit latent space for generative models, our approach addresses two crucial limitations: 1) the struggle of the implicit latent repr esentation to capture structured information inherent in 3D space, leading to sub-optimal quality of decoded 3D shapes; 2) the challenge of training and sampling from the latent distribution, given that the implicit latent space is unstructured and coddrrcoontr..ned.
To mitigate these issues, adopt an explicit triplane latent representation, utilizing a triplane of three feature maps to represent the 3D geometry latent. The design draws inspiration from LDM, which applies feature maps to represent the 2D image latent. Figure illustrates the overall framework of our proposed method, which comprises a two-step training process: I) the D3D-VAE is first trained to convert 3D shapes into 3D latents, which is described in Sec. 3.1; 2) the image-conditioned D3D-DiT is then trained to generate high-quality 3D assets, which is detailed in Sec. 3.2.
3.1 Direct 3D Variational Auto-Encoder
The proposed D3D-VAL consists of three components: a point-to-latent encoder, a latent-to-triplanee decoder, and geometry mapping network. Meanwhile, we design a semi-continuous surface sampling strategy that utilizes both continuous and discrete supervision to ensure the high-frequency geometric details of the decoded 3D shape.

---

Direct 3D Variational Aufo-Encoder (D3D-VAE)
Learnable Tokens
Transformer Encoder
Geometric Mapping
Network
Crocc Artentonn
Self Attenton
Solf Annomto
X
Convolutional Decoder
MLF
x.V
Point Cloud
Latent
Triplane T
Mesh
Semantic- Levell C.
Add Noise
CLIP
Pixel-Level Co
Nosiy Latent
DiT Block
x.N
DIT Block
DINO
Time
Conditional Image
(b) Connagg-ooddiiooeed.co diffusion Transformer (D3D-DiT)
The framework of our Direct3D, (a) We utilize transformer to encode point cloud sampled from model, along with a set of learnable tokens, into an explicit triplane latent space. Subsequently, a CNN-based decoder is employed to upsample these latent representations into high-resolution triplane feature maps. The occupancy values of queried points be decoded through geometric mapping network. (b) Then train the image conditional latent diffusion transformer the latent space obtained by VAE. Pixel-level information and semantic-lewe information from images are extracted using DINO-v2 and CLIP, respectively, and then injected into each DiT block.
Figure 2:
Point-to-latent encoder. In order to obtain robust representations in the latent space that effec- tively capture intricate geometry, we uniformly sample high-resolution point clouds from the surface of 3D objects, which is then encoded an explicit latent representation z ((3xxxr)xd where and dz denotes the resolution and channel dimensional of the latent representation, respectively, To be specific, given set of point clouds P E R (3+33 sampled from 3D models, where Np denotes the number of points, the channel dimension (3 3) comprises of the normalized position and normal of each point, we first use Fourier features [18] to represent the position structure of point clouds. Then we introduce a series of learnable tokens e E (B(xxxr)xxd. to query the point cloud features using a cross-attention layer, where de denotes the channel dimensional e. This enables the injection of 3D information from the point clouds into the latent tokens. Subsequently. multiple self-attention layers are employed to enhance the representation of these tokens. ultimately yielding the latent representation
xdd, where d, represents the channel dimensional of
Latent-to-triplane decoder. After obtaining the latent representation z, we reshape it to the triplane representation. Inspired by RODIN [56], we concatenate the three planes vertically along the height dimension, yielding E R (3xr)xx)xd2 to prevent incorrect blend of the planes across the channel dimension. Afterwards, the latent-triplane decoder upsamples z to high-resolution triplane feature maps with upsampling factors f. In contrast to the transformer architecture used in the encoder, our decoder model employs convolutional networks to progressively upsample the explicit latent representation and obtain the final triplane T (Txy, TYz, Txz).
Semi-continuous surface sampling. We employ a Multi-Layer Perceptron (MLP) as the geometric mapping network to predict the occupancy of queried points via features interpolated from the triplane. The MLP contains multiple linear layers with ReLU activation. Typical occupancy is represented by discrete binary value of 0 and to indicate whether point is inside an object. However, when query point is very close to the object surface, it can result in abrupt gradient changes that affect model optimization. In this work, we adopt semi-continuous occupancy, using both continuous and discrete supervision to ensure smooth gradient. Specifically, given a query point x in 3D space, when its distance to the surface is greater than small threshold value 512
the occupancy value
5

---

remains either 0 or 1. When the distance is less than s, continuous value ranging from to is assigned it. The formula for the semi-continuous occupancy o(x) follows:
1,
if sdf (x) < -8 if S sdf (x) 8
o(x)
(x 0.5
(I)
0,
if (x)
where (x) denotes the Signed Distance Function (SDF) value of x.
End-to-end optimization. During the training process, we uniformly sample points from the space and sample points proximate to the object surface to predict their semi-continuous occupancy.
We utilize Binary Cross-Entropy (BCE) loss LBCE to supervise the predictions. Additionally, we
employ KL loss LKL to prevent excessive variance in the latent space. Thus, our D3D-VAE is optimized by minimizing:
LAaDVAR LBCE + LKI
(2)
where ÀKI. denotes the weight of KL regulaizzation.
3.2 Image-conditioned Direct3D Diffusion Transformer
After training the D3D-VAE, we have access con- tinuous and compact latent space, upon which we train the latent diffusion model. Since the obtained latent embedding is an explicit triplane representa- tion, naive approach would to directly use well- designed 2D U-Net as the diffusion model. However, this would result lack communication between the three planes, thus failing to capture the structured and intrinsic properties required for 3D generation. Therefore, we build the generation model based on the architecture of the Diffusion Transformer (DiT), utilizing the transformer to better extract spatial posi- tional information among the planes, Mcanwhile, we
Conditional Image
CLIP VIT-L/14
Noisy Tokens
DINOv2
VIT-L/14
Scale, Shift 71, 81
Cp
Cls
Concal
Token
Multi-Head Self-Atication
propose to incorporate pixel-level and semantic-level information of the image in each DiT block, thereby aligning the image feature space and latent space generate 3D assets consistent with the conditional im- age content. The framework of latent diffusion model is shown in Figure 2 (b) and the architecture of each DiT block is illustrated in Figure
Remove
Scale
MLP
Multi-Head Cross-
Pixel-level alignment module. To ensure the high- frequency details of 3D assets generated by the diffu- sion model are aligned with the conditional images, design pixel-level alignment module to inject pixel-level information from the images into the la- tent space. We employ the pre-trained DINO-v2 [39] (ViT-L/14) as the pixel-level image encoder, which has been revealed in previous work [1] outperform other pre-trained vision models in extracting structural information beneficial for 3D tasks. Specifically, we first use two linear layers with GeLU [12] activation project the image tokens Cp extracted by DINO-v2 to match the channel dimension of the noisy latent to- kens Z₁- Then in each DiT block, we concatenate them with the flattened z, and feed them into a self self-attention layer to model the intrinsic relationship between cp and Subsequently, we eliminate the part of image tokens and only reserve the part of noisy tokens for input to the next module.
Time
Scale, Shift 72. /3
Pointwise Feedforward
Scale 0c2
X XN
Figure 3: The architccture our DiT. We employ the pre-trained DINO-v2 and CLIP vision model to extract tokens from conditional images respectively, then incorporate the pixel-level and semantic-level information into each DiT block.

---

Table 1: Quantitative results Google Seanned Objects dataset.
Methods
Chamfer Distance 4 Volume IoU f
F-Score t
Shap-E [19]
0.0585
0,2347
0.3474
Michelangelo [64]
0,0441
0.1260
0.4371
One-2-3-45 [27]
0.0513
0,2868
0.3149
InstantMesh [57]
0.0327
0.4105
0.5058
Ours (trained on Objaverse)
0,0296
0,4307
0.5356
Ours (trained on Objaverse internal data)
0.0271
0.4323
0.5624
Semantic-level alignment module. We devise a semantic- aligmment module to ensure semantic consistency between the generated 3D models and the conditional images. We employ the pre-trained CLIP [44] visual model (ViT-I/14) to extract semantic image tokens c, from the conditional images, and then utilize cross-attention layer within each DiT block to facilitate the interaction between C, and noisy latent token z₁, Meanwhile, unlike the original class conditional DIT, our image- conditioned diffusion model no longer utilizes class embedding. Instead, we use the classification token from the semantic image tokens after projection and add it to the time embedding to enhance semantic features. addition, to reduce the number of parameters and computational cost, we employ adaLN-single,
proposed in Aix [3], which predicts set of global shift and scale parameters P = [71, 81, 0t1, 72, $2, arg] using time embeddings, then sets a trainable embedding and adds it to P for adjustment in each block.
Training. Following LDM [46], our 3D latent diffusion transformer model predicts the noise of the noisy latent representation at time t, conditioned on image C. When training the diffusion model, we randomly zero the conditional input cp and c, with a probability of 10% to use classifier-free guidance [14] during inference, thereby improving the quality of conditional generation.
4 Experiments
4.1 Dataset
Our Direct3D is trained on a filtered subset of the Objaverse [7] dataset which consists of 160K high-quality 3D assets. To evaluate the scalability of our Direct3D, we also employ additional internal data for training. Each 3D model is normalized to unit sphere centered at the world origin. To construct conditional images for training the 3D latent diffusion transformer, we randomly render 24 views at resolution of 512 x 512 using Blender for each 3D model. Additionally, we employ depth-conditiones conditioned ControlNet [63] to generate 16 diverse images to ensure the generalization of the diffusion model. To evaluate the performance of our Direct3D, from the Google Scanned Objects (GSO) [8] dataset for Image-to-3D experiments. For the text-to-3D task, we utilize existing text-to-image models like Hunyuan-DiT [24] to generate images with several classic text prompts as conditional inputs for qualitative comparisons with other methods. The ablation studies for each component are presented in the Appendix.
randomly select 30 3D models
4.2 Implementation Details
D3D- VAE. Our D3D- VAE takes as input 81,920 point clouds with normal uniformly sampled from the 3D model, along with learnable latent token of resolution 32 and a channel dimension de = 768. The encoder network consists each attention layer comprising 12 heads of dimension 64. The channel dimension of the latent representation is dg = 16. The decoder network comprises self-attention layer and ResNet [1]] blocks to upsample the latent representation into triplane feature maps with resolution of 256 x 256 and channel dimension of 32. The geometric mapping network consists of linear layers with hidden dimension 64. During training, supervision. The KL regularization weight is set to ARI. le 6. We use the Adam [33] optimizer with learning rate le 4 and batch size of 16 per GPU.
cross-attention layer and self-attention layers, with
sample 20,480 uniform points and 20,480 near-surface points for
D3D-DIT. Our diffusion model adopts the network configuration of DiT-XL/2 [40], which consists of 28 layers of DiT blocks. Each attention layer includes 16 heads with a dimension of 72. We train the
7

---

Iaput Image
Shap-E
One-2-3-45
Michelangelo
InstantMesh
Ours
Figure 4: Qualitative comparisons with different baseline methods on GSO dataset.
Table 2: User study on the quality of meshes. The higher the score, ranging from to 5, the better.
Shap-E [19]
One-2-3-45 [27]
Michelangelo [64]
InstantMesh [57]
Ours
Quality Consistency 1.19
1.18
1.24
2.51
2.53
4.41
1.28
2.32
2.66
4.35
diffusion model with 1000 denoising steps using a linear variance scheduler ranging from 1e 4 to 2e 2. We employ the AdamW optimizer with batch size of 32 per GPU and train for 800K steps. During inference, we apply 50 steps of DDIM [52] with the guidance scale set to 7.5.
4.3 Image and Text to 3D Generation
Image-to-3D. We conduct qualitative and quantitative comparisons of our Direct3D with other baseline methods on the GSO dataset for the image-to-3D task, as illustrated in Figure and Table 1,
respectively. Shap-E [19], 3D diffusion model trained on millions of 3D assets, capable of producing plausible geometry, but it suffers from artifacts and holes in the meshes. Michelangelo [64] performs diffusion process on ID implicit latent space, and fails align the generated mesh with the semantic content of the conditional images. Multi-view based approaches such as One-2-3-45 [27] and InstantMesh [57] heavily rely on the performance of multi-view 2D diffusion model. One-2-3-45 directly employs SparseNeuS [32] for reconstruction, resulting in coarse geometry. The meshes generated by InstantMesh perform decent quality, but lack consistency with the input images in certain details like the water spout the sink and the windows of the school bus. It also produces some failure cases such as merging the hind legs of the horse together, due to the limitation of multi-view diffusion model. In contrast, our Direct3D consistently generates high-quality meshes that align with the conditional images in most cases. In Table 1, we report the Chamfer Distance, Volume IoU and F-Score to compare the quality of the generated meshes with other methods. It can be observed that our Direct3D achieves state-of-the-un performance across all metrics when trained on Objaverse dataset. Integrating our internal data for training further enhances the model's performance, validating the scalability of our approach.
Text-to-3D. Our Direct3D can produce 3D assets from text prompts by incorporating text-to-image models, such as Flux [21] and Hunyuan-DiT [24]. Figure 5 illustrates the qualitative comparisons of our Direct3D and other baseline methods on the text-to-3D task. To ensure a fair comparison, all methods utilize the same generated image as input. can be observed that these baseline methods fail in almost all cases, while our Direct3D is still able to generate high-quality meshes, demonstrating

---

Iaput
Shap-E
One-2-3-45
Michelangelo
InstantMeshi
Ours
Spider-man, photorealistic
Flat fyyng dragon
3D model of adorable cottage wich thatched roof An astronaut riding horse
A pair of sunglasses
A realistic phoenix the fairytale
police car
Iron man in ice and snow armor
Figure 5: Qualitative comparisons of the meshes generated from text. We employ the existing text-to-image
models (c.g. Hunyuan-DiT) to produce highly detailed images as the inputs of each method.
the generalizability of our approach. We also conducted a user study to quantitatively compare our D3D-DiT with other methods. We render videos of meshes generated by each method rotating 360 degrees, and ask 46 volunteers to rate each mesh based on its quality and consistency with the input images. The results Table indicate that our D3D-DiT perform superior mesh quality and consistency compared to other baseline methods.
Generation of textured mesh. Benefited from the smooth and detailed geometry produced by our Direct3D, we can easily dress up the mesh using existing texture synthesis methods. As shown in Figure 6, we utilize SyneMVD [30] obtain exquisite textured meshes.
Conclusion
In conclusion, our paper introduces novel approach for direct 3D shape generation from a single image, bypassing need for multi-view reconstruction. Leveraging a hybrid architecture, our proposed D3D- VAE the generated shapes. Our image-conditioned 3D diffusion transformer (D3D-DiT) further improves the generation quality by integrating image information both pixel and semantic levels, ensuring
encode 3D shapes into compact latent space, enhancing the fidelity of

---

An elegant chess queen with floral engraving
mystic torem with intricate animal carvings
Futuristic mech suit advanced weaponry armor
Miniature teapot shaped like an elephant
Figure 6: Visualizations of the textured meshes. We employ SyncMVD [30] to generate texture for the meshes produced by our Direct3D.
high consistency between generated 3D shapes and conditional images. Extensive experiments on the image-to-3D and text-to-3D tasks demonstrate the superior performance of Direct3D in 3D generation, surpassing existing methods in quality and generalizability.
Limitations. Despite the capability of our Direct3D produce high-fidelitg 3D assets, it is currently limited to the generation individual or multiple objects and cannot generate large-scale scenes. We will focus it in future research.
Acknowledgment. This work was mainly supported by DreamTech, and in part by the National Natural Science Foundation of China (62441204, 62472213).
References
Mohamed Banani, Amit Raj, Kokis-Ko Maninis, Abhishck Kar, Yuanzhen L.,, Michael Rubinstein, Deqing Sun, Lconidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3d awareness visual foundation models. arXiv preprint arXiv:2404.08633 2024.
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay. Sameh Khamis, et Efficient geometry-awan 3d generative adversarial networks. In CVPR, 2022.
[3] Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR, 2024.
[4] Luxi Chen, Zhengyi Wang, Chongxuan Li, Tingting Gao, Hang Su, and Jun Zhu. Microdreamer: Zero-shot 3d generation in 20 seconds by score-based iterative reconstruction. arXiv preprint arXiv:2404.1 19525.
2024.
[5] Yabo Chen, Jiemin Fang, Yuyang Huang, Taoran Yi, Xiaopeng Zhang, Lingxi Xie, Xinggang Wang. Wenrui Dai, Hongkai Xiong, and Qi Tian. Cascade-zero 123; One image to highly consistent 3d with self-prompted nearby views. arXiv preprint arXiv:2312.044: 04.4444, 2023.
[6] Mau Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, al. Objaverse-xl: A universe of 10m+ 3d objects.
In NewrIPS, 2024.
[7] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Bli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, 2023.
[8] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In ICRA, 2022.
Ruiqi Gao, Aleksander Holynski, Philipp Henzler. Arthur Brussee, Ricardo Martin Brualla, Pratul Srini- vasan, Jonathan Barron, Ben Paole. Cat3d: Create anything 3d with multi-view diffusion models. arXiv preprint 4405v:2405. 10314, 14, 2024.
[10] Anchit Gupta, Wenhan Xiong. Yixin Nie, Ian Jones, and Barlas Ouz. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint arXiv:2303.0523 05377, 2023.
[11] Kaiming He. Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
10

---

[12] Dan Hendrycks and Kevin Gimpel. Gaussian emror linear units (gelus). arXiv preprint arXiv: arXiv:1606.00000 160..0466.
2016.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[14] Jonathan Ho and Tim Salimans. Classifier- diffusion guidance. arXiv preprint arXiv:2207.12594 2022.
[15] Fangzhou Hong, Jiaxiang Tang. Ziang Cao, Min Shi, Tong Wu, Zhaoxi Chen, Tengfei Wang, Liang Pan, Dahua Lin, and Ziwei Liu. 3dtopia: Large text-to-3d generation model with hybrid diffusion priors. arXiv preprint 2r0:3 022.34, 2024.
[16] Yicong Hong, Kai Zhang. Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. In ICLR, 2024.
[17] Ka-Hei Hui, Aditya Sanghi, Arianna Rampini, Kamal Rahimi Malekshan, Zhengzhe Liu, Hooman Shayani, and Chi- Wing Fu. Make-a-shape: a ten ten-million scalle 3d shape model. arXiv preprint arXiv:2401. 2011:2401111067, 2024.
[18] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In ICML, 2021.
[19] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint STti:23055024665 2023.
[20] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
[21] Black Forest Labs. Flux, 2024, https://baccfforestabbbadd
[22] Yushi Lan, Fangzhou Hong. Shuai Yang. Shangchen Zhou, Xuyi Meng. Bo Dai, Xingang Pan, and
Chen Change Loy. Ln3diff: Scalable latent neural fields diffusion for speedy 3d generation. arXiv preprint 2024.
arXiv:2403.1200
[23] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. In ICLR, 2024.
[24] Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang. Xingchao Liu, Minbin Huang. Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine mee-grained chinese understanding. arXiv e-prints, pages arXiv- 2405, 2024.
[25] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang. Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d 30-3d content creation. In CVPR, 2023.
[26] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45 Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint urXiv:2311.0735
2023.
[27] Minghua Liu, Chao Xu, Haian Jin. Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. 0-e-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. In NeurlPS, volume 36, 2024. [28] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-33 Zero-shot one image to 3d object. In ICCV, pages
2023.
[29] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Synedreamer: Generating multiview--onsiiseett images from single-view image. In ICLR, 2023.
[30] Yuxin Liu, Minshan Xie, Hanyuan Liu, and Tien-Tsin Wong. Text-guided texturing by synchronized multi-view diffusion. arXiv preprint arXiv:2311. 1.2899,, 2023.
[31] Xiaoxiao Long. Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song- Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross doma-n diffusion. arXiv preprint arXiv:2310. 15008, 2023.
[32] Xiaoxiao Long, Cheng Lin, Peng Wang. Taku Komura, and Wenping Wang. Sparseneus: Fast generalizable neural surface reconstruction from sparse views. In ECCV, 2022.
[33] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0511 2017.
[34] Yuanxun Lu, Jingyang Zhang, Shiwei Li, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan, Xun Cao, and Yao Yao. Direct2. 5: Diverse text-to-3d generation via multi-viww 2.5 d diffussion. arXiv preprint arXiv:2311.15980 2023.
[35] Zhaoyang Lyu, Ben Fei, Jinyi Wang, Xudong Xu, Ya Zhang, Weidong Yang, and Bo Dai. Getmesh: A controllable model for high-quality mesh generation and manipulation. arXiv preprint arXf::403.111990 2024.
[36] B Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ramamoorthi, and R Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[37] Shentong Mo. Enze Xie, Ruihang Chu. Lanqing Hong. Matthias Niessner, and Zhenguo Li. Dit-3d: Exploring plain diffusion transformers for 3d shape generation. In NeurlPS, 2023.
[38] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for 2022.
generating 3d point clouds from complex prompts. arXiv preprint
11

---

[39]
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual
features without supervision. Transactions on Machine Learning Research, 2023.
[40]
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
[41]
Dustin Podell, Zion English, Kyle Laccy, Andreas Blatmmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:3377.0995 2023.
[42]
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Drcamfusion: Text-to-3d using 2d diffusion.
In ICLR, 2022.
[43] Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu. Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. arXiv preprint arXiv:2311.16918, 2023.
[44] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[45] Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams. Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies. arXiv preprint arXiv:2312.03806, 2023.
[46]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[47] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-tex text models. In NeurIPS, 2022.
[48] Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In NewrIPS, 2021.
[49] Yichun Shi, Peng Wang. Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mydream: Multi-view diffusion for generation. In ICLR, 2023.
[50] Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In CVPR, 2023.
[51] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
[52] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020.
[53] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwci Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054, 2024.
[54] Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312. 022011, 2023.
[55] Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-Irm: Pose-free large reconstruction model for joint pose and shape prediction. In ICLR, 2024.
[56] Tengfei Wang. Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. In CVPR, 2023.
[57] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient
3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.0719/ 2024.
[58] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang. Sida Peng. Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient reconstruction and generation. arXiv preprint (tti:22003.142222 2024.
[59] Lior Yariv, Omri Puny. Natalia Neverova, Oran Gafni, and Yaron Lipman. Mos for 3d generative models. arXiv preprint arXiv:2312.094 09222,, 2023.
[60] Biao Zhang. Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. ACM Transactions Graphics (TOG), 2023.
[61] Bowen Zhang, Tianyu Yang. Yu Li, Lei Zhang, and Xi Zhao. Compress3d:; a compressed latent space for
3d generation from single image. arXiv preprint arXiv:2403. /35524 2024.
[62] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. arXiv preprint arx4:22404. 19702, 2024.
[63] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
[64] Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng. Rui Wang, Pei Cheng, Bin Fu, Tao Chen, Gang Yu, and Shenghua Gao. Michelangelo: Conditional 3d shape generation based on shape-image image-text aligned latent representation. In NeurIPS, 2023.
12

---

[65] Xin-Yang Zheng, Hao Pan. Yu-Xiao Guo, Xin Tong, and Yang Liu. Mvd: Efficient multiview 3d reconstruction for multiview diffusion, arXiv preprint arXiv:2402. 14253, 2024.
13

---

A Appendix
A.1 Ablation Studies
Explicit triplane latent. Unlike typical approaches like Michelangelo [64] which employs VAE to encode inputs into ID implicit latent space, our D3D-VAE compresses high-resolution point clouds into an explicit triplane latent representation. We conduct comparative experiments on the Objaverse [7] dataset, and the 3D models for validation are not included in the training set. Figure and Table illustrate the comparison of the reconstruction results of VAE between these two approaches, demonstrating that our adopted explicit triplane representation more capable of recovering high-frequency geometric details.
Eaplic Tiipiaas lopplal
Figure 7: Qualitative comparisons of reconstruction with different latent representation.
Table 3: Quantitative comparisons of reconstruction with different latent representations Objaverse evaluation
Methods
Chamfer Distance 4 Volume IoU f F-Score f
Implicit ID Space (Shape2VecSet)
0.0057
0.8794
0.9416
Explicit Triplane (Ours)
0.0042
0.9409
0.9835
Semi-continuous surface sampling strategy. We adopt a semi-continuous surface sampling strategy during the training of D3D- VAE to alleviate the optimization difficulty caused by abrupt changes in occupancy near the object surface. To evaluate the effectiveness this strategy, we train D3D- -VE with and without this sampling strategy separately and compare the reconstructed results. As shown in Figure and Table 4, can be observed that the reconstruction performance is unsatisfactory when directly training with the original occupancy in some thin structures, but is improved when the semi-continuous sampling strategy is utilized.
Table Quantitative results of semi-continuous surface sampling strategy.
Methods
Chamfer Distance
Volume IoU t
F-Score t
w/o semi-continuous sampling
0.0060
0.8723
0.9192
w/ semi-continuous sampling
0.0057
0.8794
0.9416
2D U-Net vs D3D-DIT. To demonstrate the superiority of our D3D-DiT network architecture, we
conduct experiments to compare with 2D U-Net. We train diffusion models on the roll-out triplane latent representation using network architectures of SD 1.5 [46] and SD 2.1, respectively. Figure illustrates the qualitative comparisons using the conditional images generated by Hunyuan-DiT [24]. It can be observed that neither SD 1.5 or SD 2.1 is able to produce satisfactory meshes, while our D3D-DiT, due its powerful scalability and generalization, capable of generating high-quality shapes that align with the content of the conditional images.
14

---

w/o semb-con oce
semi-con
GT
Figure 8: Ablation study for the semi-continuone surface sampling strategy.
D3D-DiT w/o pixel alignment
Input Image
D3D-SD
D3D-SD
D3D-DIT
Figure 9: Qualitative comparisons of diffusion models with different network architectures.
Effectiveness of the pixel-level alignment module. We perform ablation experiments to validate the effectiveness of the pixel-level alignment module used in our D3D-DIT. As illustrated in Figure 9, D3D-DiT can still generate meshes of relatively high quality without this module. However, it does not align well with the conditional images, such as the external structure of the coffee machine and the wings the statue, By injecting the pixel-level information into each DiT block through the pixel-level alignment module, the produced meshes can also maintain consistency with the conditional images terms of details.
A.2 More Visualizations
We present more visualizations in Figure 10.
A.3 Broader Impacts
Like other creative tools, our project is susceptible misuse, such as improper or harmful use of the generated characters. To address these risks, it is crucial to establish and implement ethical guidelines and content moderation policies.
15

---

Figure 10: More Visualizations.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We explicitly delineate the contributions of our approach in both the abstract and the introduction in Section. of paper. The experimental results, which substantiate our claims, are comprehensively detailed and demonstrate the efficacy of our methods. For in de-tepth analysis of the experimental outcomes, refer to Section.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We mention in Section 5 that the paper discusses the limitations of the work performed by the authors.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
17

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not contain the theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides detailed descriptions of the experimental setup, including the algorithms used, parameter settings, datasets, and evaluation metrics. Additionally, it outlines the procedures followed to obtain the results, ensuring that other researchers can replicate the experiments and verify the findings. We mention this in Section. 4
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
18

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: While portion the data used in this research comprises private assets with significant commercial value, releasing this information would also violate the research contract signed by the authors.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ public/guides/ httpee///ooeSSumiiiiiooooooo...
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /CodeSubmieeeeeeeece / /nips. wttpcccpppcccccccccccccccccccctt http:////nuddees/deeuuitiiiii ismiiissoonn onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We specify all the training and testing details in Section. 4.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: resources.
Repeated training runs of large transformer models strain available GPU
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have carefully reviewed the details provided in the paper, and it includes comprehensive information on the type of computing workers, memory, and time of execu- tion required for each experiment. We mention this in Section. 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt////eerippp.eee
ines?
Answer: [Yes]
Justification: We have carefully reviewed the NeurIPS Code of Ethics, and we confirm that research adheres all outlined ethical guidelines, including considerations for faimess, transparency, and respect for all participants involved.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
20

---

Answer: [Yes]
Justification: The paper discusses both potential positive societal impacts, such as ad- vancements in technology and potential benefits various industries, and negative societal impacts, including ethical considerations and the environmental impact increased compu- tational resource usage. We mention this in Section A.3.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This question is not applicable because the paper does not involve the release data or models that pose high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators original owners of all assets used in the paper are properly credited. The licenses and terms use for these assets are explicitly mentioned and have been respected in accordance with their respective requirements.
21

---

Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license ge.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi theode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The paper includes comprehensive documentation for new assets introduced. This documentation is provided alongside the assets, ensuring that users have access to detailed descriptions, usage guidelines, and any necessary supporting information.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: The paper does not include the full text of instructions given to participants, screenshots, or details about compensation.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
22

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: We request several volunteers to participate in simple anonymous question- naire for user study, with no potential risks involved for the volunteers.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "TOuZobk1Fl", "reviewer": "Reviewer_3ERi", "summary": "Generating high-quality 3D assets from text and images has been difficult due to the lack of scalable 3D representations. This paper, Direct3D, addresses this by introducing a native 3D generative model that scales to real-world input images without needing multiview diffusion models or SDS optimization. The method includes two main components:\n\nDirect 3D Variational Auto-Encoder (D3D-VAE): Encodes high-resolution 3D shapes into a compact latent triplane space, supervising the decoded geometry using a semi-continuous surface sampling strategy instead of relying on rendered images.\n\nDirect 3D Diffusion Transformer (D3D-DiT): Models the distribution of encoded 3D latents and fuses positional information from the triplane latent's three feature maps, creating a scalable native 3D generative model for large datasets.\n\nThe approach also introduces an innovative image-to-3D generation pipeline that incorporates semantic and pixel-level conditions from images, enabling the generation of 3D shapes consistent with the provided images. Experiments show that Direct3D outperforms previous methods in generation quality and generalization, setting a new benchmark for 3D content creation.", "questions": "1. What data is Direct3D being trained on? I do wonder whether the comparison is fair, since the paper did not elaborate on the details of data filtering pipeline / dataset size. If Direct3D wants to show its advantages against existing methods, it should be trained on the same dataset (e.g., Objaverse) and do thorough comparison.\n\n2. Whether you can include comparison with other 3D origin generative-based methods, e.g., LN3Diff and Crafsman? Since this line of work is very sparse, I think a more thorough comparison is well required.\n\n2. Whether the trained models will be released?", "limitations": "Overall, though the paper shows great scalability and astounding performance over 3D AIGC using a DiT pipeline, I still find many concerns regarding this paper. I hope the reviewer can address my concerns during the rebuttal stage, and I am very happy to increase my rating.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The origin, generative-based 3D AIGC pipeline is absolutely missing. Though LRM-based methods have achieved great success in scaling up 3D models, it is still reconstruction-based and heavily relying on the first-stage MV-Diffusion for high-quality 3D objects synthesis. I really like the pipeline proposed here, and believe it is a firm step towards a converged 3D AIGC framework.\n\n2. The performance is good, the generated mesh is clean, and the user study shows the proposed method is better against Shape-E, InstantMesh etc.\n\n3. This pipeline shows that scaling up 3D DiT is also a feasible way towards high-quality AIGC when data is enough.", "weaknesses": "Though performance is good, I still have some concerns regarding the proposed method:\n\n1. This idea is not new. Shape-E /Meta 3DGen first verified the VAE + LDM pipeline, and recently, LN3Diff (ECCV 24') also verifies this pipeline on Objaverse with tri-plane latent space. More difference / discussions with the previous methods shall be discussed, e.g., compared with LN3Diff, any more differences beyond \"combines the advantages of explicit 3D latent representation and direct 3D supervision to achieve high-quality VAE reconstruction\". Ln3diff also has an explicit latent tri-plane latent space and direct depth supervision.\n2. The proposed VAE requires high-quality 3D point clouds and mesh as the ground truth, and can only model geometry information. Compared against Ln3Diff and CLAY which can also model texture, this is definitely a drawback. Another problem of the proposed design is that Direct3D cannot benchmark its performance on the GSO dataset with PSNR/LPIPS, which is a canonical evaluation pipeline for existing 3D AIGC method.\n3. No thorough quantitative comparison against the existing methods, just user study. MMD/COV are also reasonable metrics if no texture is available here.\n4. The proposed DiT pipeline cannot be seen as a contribution, since it shares very similar design with DiT-3D / PixArt.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback and thorough review of the our paper. We carefully respond to each of the concerns and questions below.\n\n**[Q1]:** The distinctions between our Direct3D and LN3Diff.\n\n**[A1]:** We thank reviewer 3ERi for bringing up the concurrent work. While LN3Diff was accepted to ECCV after the NeurIPS submission deadline, we are happy to discuss the work and will add this part to our final version.\n\nAlthough both LN3Diff and Direct3D utilize the triplane latent, the VAE of these two methods are fundamentally different. As LN3Diff uses rendered RGB images as input, it is challenging for its VAE to encode high-fidelity geometric information compared to Direct3D, which directly applies dense point clouds with normal information as input. Furthermore, while our D3D-VAE employs direct supervision on the ground truth mesh, LN3Diff relies on supervision from rendered depth maps, which strictly speaking cannot be considered as direct 3D supervision. While this approach may lead to smoother generated mesh surfaces, it still struggles to capture the intrinsic structural information of the underlying 3D data.\n\n\n**[Q2]:** Training on Objaverse dataset and quantitative evaluation.\n\n**[A2]:** We re-train our model on the Objaverse dataset and conduct quantitative comparison with other methods, please refer to the **global rebuttal or Table 1 of the uploaded PDF**.\n\n**[Q3]:** The requirement of high-quality 3D point clouds, and can only model geometry information.\n\n**[A3]:** Direct3D is specifically focused on geometry generation, and we deliberately decouple geometry from texture to streamline the learning process for better geometry generation. In fact, we regard the exclusion of texture generation not as a drawback but as an advantage of the proposed pipeline: the separation introduces more flexibilities to the texture generation part as we can immediately apply advanced texture generation methods such as SyncMVD to our pipeline. It's note worthy that CLAY also employs a similar strategy by separating geometry and texture, by using a post-processing technique of normal-conditioned MVDream [1] for texture generation. Similar texture generation methods can be integrated into our approach easily if necessary.\n\n**[Q4]:** The proposed DiT shares very similar design with DiT-3D / PixArt.\n\n**[A4]:** We apologize for any confusion raised in the manuscript and we will provide more details in the final version. The proposed DiT is fundamentally different from DiT-3D and PixArt. DiT-3D is specifically designed for 3D voxel data and utilizes 3D attention to capture spatial relationships. In contrast, our DiT is designed for triplane latent space, which we treat as 2D images, as illustrated in Section 3.2 of the main manuscript, and therefore does not require 3D attention. Regarding PixArt, it is designed for 2D images and combines cross-attention with DiT to handle conditional information. However, we observe that the mesh generated solely based on these semantic-level conditions lacks detail alignment with the input image. To address this, we introduced the additional pixel-level conditions using DINO, which has been revealed in previous work [2] that outperform other pre-trained vision models in extracting structural information beneficial for 3D tasks. Specifically, we concatenate the image tokens extracted by DINO with the noisy latent tokens, and feed them into the self-attention layer. This approach significantly enhances generation quality, allowing for a more detailed recovery of the condition input. We believe this finding is crucial for advancing 3D generation research, as it explores the integration of image conditions with the current state-of-the-art DiT architecture.\n\n\n**[Q5]:** Whether you can include comparison with other 3D origin generative-based methods, e.g., LN3Diff and CraftsMan?\n\n**[A5]:** As LN3Diff only released the text-to-3D model, it is challenging for us to train and re-implement it from scratch within such a short timeframe. Therefore, we apologize for not being able to present a comparative analysis with LN3Diff and we will make every effort to train LN3Diff and include a comparison in our final version. Additionally, **CraftsMan was publicly released after the NeurIPS submission deadline (on 23 May, 2024)**, yet we still conducted a comparison, with the results presented in **Figure 1 of the uploaded PDF**. It can be observed that our Direct3D demonstrates superior generalization compared to CraftsMan, and the generated meshes exhibit greater detail. On one hand, the VAE employed by CraftsMan still utilizes the implicit 1D latent space proposed by 3DShape2VecSet [3], resulting in lower reconstruction quality compared to the explicit latent representation that we have adopted. We conduct a quantitative comparison of the VAE with different latent representations in the Objaverse evaluation set, and the results are presented in the table below. On the other hand, CraftsMan relies on the multi-view diffusion model for conditional generation, leading to significant instability in their results.\n\n| Methods  |Chamfer Distance ↓|Volume IoU ↑| F-Score ↑ |\n------------ | :----: | :----: | :----:\n| Implicit 1D Space (3Dshape2VecSet)      | 0.0057 | 0.8794 | 0.9416 |\n| Explicit Triplane (ours) | **0.0042** | **0.9409** | **0.9835** |\n\n\n**[Q6]:** Whether the trained models will be released?\n\n**[A6]:** Yes, we will release the models trained on the Objaverse dataset upon paper acceptance to promote openness and reproducibility in our research.\n\n\n**Reference:**  \n\n[1]. Shi et al., MVDream: Multi-view Diffusion for 3D Generation.\n\n[2]. Banani et al., Probing the 3d awareness of visual foundation models.\n\n[3]. Zhang et al., 3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gP9xOxqoZ8", "reviewer": "Reviewer_HuD3", "summary": "This work presents Direct3D, a new approach for scalable image-to-3D generation using a 3D Latent Diffusion Transformer. This method enables the generation of high-quality 3D assets from text and images without the need for complex optimization techniques. Direct3D introduces a native 3D generative model that efficiently produces 3D shapes consistent with conditional image input. By incorporating semantic-level and pixel-level image conditions, Direct3D demonstrates the efficacy of its approach through detailed experimental results and discussions on limitations and theoretical considerations.", "questions": "What is the reconstruction error of the proposed 3D-VAE?", "limitations": "My biggest concern comes from the usage of private data. While the algorithm looks novel and interesting (and I personally like it very much), I can not recommend acceptance to a paper without a clear justification of where the improvement comes from. I'd love to change my mind if the author is willing to 1) train Direct3D without internal data and put it on the main manuscript while leave the internal data version to supplementary; or 2) open source the dataset (including category of dataset, its sampled visual examples, and the way of collection). I guess 1) is easier to achive.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Direct3D can handle in-the-wild input images without the need for multi-view diffusion models or complex optimization techniques.\n\nDirect3D surpasses previous image-to-3D approaches in terms of generation quality and generalization ability. \n\n- Direct3D comprises two primary components - a Direct 3D Variational Auto-Encoder (D3D-VAE) and a Direct 3D Diffusion Transformer (D3D-DiT). These components work together to encode high-resolution 3D shapes into a compact latent space and model the distribution of encoded 3D latents, respectively.", "weaknesses": "- The model is trained on 500k internal data, which is not fair to compare with other methods. It's good to see quality improvement and very interesting pipeline. But as an academic publication, it's not acceptable to me, as a reviewer, that we dont know it enough whether the improvement comes from algorithm or newly added dataset. It's just not scientific.\n\n- There is very limited qualitative evaluation conducted to demonstrate the effectiveness, E.g., CLIP-score, LPIPS, FID.\n\n- Eq (2) lacks input of $L_{BCE} $  and $L_{KL}$, which cause trouble in understanding the pipeline.\n\n- typo in figure 2, \"Nosiy Latent\"\n\n- Better to denote the shape of noisy latent and the output of DiT in the figure 2 to help reader better understand it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback and thorough review of our paper. We carefully respond to each of the concerns and questions below.\n\n**[Q1]:** Training on Objaverse dataset and quantitative evaluation.\n\n**[A1]:** Thanks for your comment. To ensure a fair comparison with other methods, we re-train our D3D-VAE and D3D-DiT on the Objaverse dataset. Following prior approaches such as InstantMesh and Wonder3D [1], we adopt the Google Scanned Objects dataset for evaluation, and report the Chamfer Distance, Volume IoU and F-Score to compare the quality of the generated meshes. The results are presented in the table below, which illustrates that generative-based methods like Shap-E and Michelangelo exhibit significantly lower accuracy compared to reconstruction-based methods such as InstantMesh. Nonetheless, our Direct3D achieves state-of-the-art performance across all metrics. Integrating our internal data for training further enhances the model’s performance, validating the scalability of our approach. Finally, we will release the model trained on the Objaverse dataset upon paper acceptance to further promote openness and reproducibility in our research.\n\n| Methods  |Chamfer Distance ↓|Volume IoU ↑| F-Score ↑ |\n------------ | :----: | :----: | :----:\n| Shap-E       | 0.0585 | 0.2347 | 0.3474 |\n| Michelangelo | 0.0441 | 0.1260 | 0.4371 |\n| One-2-3-45   | 0.0513 | 0.2868 | 0.3149 |\n| InstantMesh  | 0.0327 | 0.4105 | 0.5058 |\n| Ours (trained on Objaverse)        |0.0296|0.4307|0.5356|\n| Ours (trained on internal data)        |**0.0271**|**0.4323**|**0.5624**|\n\n**[Q2]:** Eq (2) lacks input of $L_{BCE}$ and $L_{KL}$, which cause trouble in understanding the pipeline.\n\n**[A2]:** Thanks for pointing this out. $L_{BCE}=BCE(o_{pr}(\\textbf{x}), o_{gt}(\\textbf{x}))$, where $o_{pr}(\\textbf{x})$ denotes the predicted occupancy of the given points $\\textbf{x}$, which is the output of the decoder. And $o_{gt}(\\textbf{x})$ is the ground truth semi-continuous occupancy. $L_{KL}=KL(\\textbf{z})$, where $\\textbf{z}$ denotes the latent representation obtained by the encoder.\n\n**[Q3]:** Better to denote the shape of noisy latent and the output of DiT in Figure 2 to help readers better understand it.\n\n**[A3]:** Thank you for your valuable comment. We will reorganize Figure 2 in the final version of the paper to enhance its clarity.\n\n**[Q4]:** What is the reconstruction error of the proposed 3D-VAE?\n\n**[A4]:** Here we evaluate the reconstruction loss of our D3D-VAE. For a given sampled point, we compute the Binary Cross-Entropy (BCE) between the VAE-predicted occupancy and the ground truth occupancy as the reconstruction error; Our VAE employs an explicit triplane latent representation, and we conducted a quantitative comparison in our split Objaverse evaluation set with methods such as Michelangelo [2] and 3DShape2VecSet [3], which utilize an implicit 1D latent space. As shown in the table below, our VAE demonstrates superior reconstruction quality compared to the implicit 1D latent space.\n\n| Methods  |Chamfer Distance ↓|Volume IoU ↑| F-Score ↑ |\n------------ | :----: | :----: | :----:\n| Implicit 1D Space (3DShape2VecSet)       | 0.0057 | 0.8794 | 0.9416 |\n| Explicit Triplane (ours) | **0.0042** | **0.9409** | **0.9835** |\n\n**Reference:**  \n\n[1]. Long et al., Wonder3D: Single Image to 3D using Cross-Domain Diffusion. \n\n[2]. Zhao et al., Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation. \n\n[3]. Zhang et al., 3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I86GS1H73w", "reviewer": "Reviewer_U1iF", "summary": "Summary: This manuscript focuses on the task of 3D generation, including image-to-3D and text-to-3D tasks. The framework is structured in two stages. In the first stage, a geometry VAE is trained using 3D ground-truth occupancy as supervision. Triplane representations are leveraged as the explicit 3D format, which are acquired through a Transformer processing dense point clouds. A decoder is then used to map the interpolated features into occupancy fields. In the second stage, conditioned images are encoded using CLIP and DINO to provide pixel-level and semantic-level conditions, respectively. DIT blocks are utilized to denoise the latent space of the geometry VAE. Extensive experiments validate the effectiveness of the proposed framework, demonstrating promising capabilities in conditional 3D generation.", "questions": "I recommend a detailed discussion of how pixel-level and semantic-level conditions are implemented, particularly since both conditions are introduced at the token-level. The distinction between the two is crucial for understanding the underlying mechanisms of the model. Specifically, the manuscript mentions that the CLIP token is integrated using AdaLayerNorm, while the DINO token is incorporated through self and cross-attention. It would be beneficial for the manuscript to elaborate on why these specific implementations represent pixel-level and semantic-level conditions, respectively. Such a clarification will not only enhance the clarity of the manuscript but also substantiate the design choices made in the architecture, providing readers with a deeper understanding of how each contributes to the model's performance.\n\n2.I would like to see a detailed discussion regarding the differences between the vector set representation used in 3D Shape2VecSet and the triplane representation employed as the latent representation within the framework. This discussion should include an analysis of how each representation impacts the training of the geometry VAE and the conditional generation process. Such a comparative analysis would not only clarify the choice of representation in the current framework but also provide valuable insights for future research in 3D object generation. This discussion could significantly strengthen the manuscript by providing a clearer justification for the architectural decisions and their implications on the model's performance.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The proposed Semi-continuous Surface Sampling technique is technically sound.\n2.The DIT model is innovatively applied for the first time in conditional 3D generation.\n3.The manuscript is well-organized and clearly written, facilitating easy comprehension of the complex concepts discussed.\n4.The overall framework exhibits promising capabilities in conditional 3D generation, suggesting strong potential for future applications and research.", "weaknesses": "1.The manuscript lacks a robust quantitative evaluation metric for the geometry VAE and conditional generation, particularly in comparison with state-of-the-art models and within its own ablation study. For instance, while the effectiveness of the explicit triplane latent representation and the semi-continuous surface sampling strategy is demonstrated, it is shown only through the visualization of three or four examples without any quantitative metrics. However, these results could be quantitatively evaluated using the Chamfer Distance by comparing the reconstructed mesh with the ground truth (GT) mesh, both of which are readily accessible. Additionally, as other methods also utilize 3D GT for supervision, a quantitative comparison using the Chamfer Distance should be provided to establish the proposed method's competitiveness and efficacy more definitively. \n2. The manuscript currently lacks sufficient detail in the \"Implementation Details\" section, particularly regarding the near-surface sampling strategy. For the sake of reproducibility, it is crucial to provide more information:\n\n+Sampling Strategy: What specific strategy is employed for near-surface sampling?\n+Is the loss weight of BCE loss the same across near-surface points and uniformly distributed points?\n+How many points are sampled per iteration?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback and thorough review of our paper. We carefully respond to each of the concerns and questions below.\n\n**[Q1]:** Quantitative comparison with state-of-the-art methods.\n\n**[A1]:** To ensure a fair comparison with other methods, we re-train our D3D-VAE and D3D-DiT on the Objaverse dataset. Following prior approaches such as InstantMesh and Wonder3D [1], we adopt the Google Scanned Objects dataset for evaluation, and report the Chamfer Distance, Volume IoU and F-Score to compare the quality of the generated meshes. The results are presented in the table below, which illustrates that generative-based methods like Shap-E and Michelangelo exhibit significantly lower accuracy compared to reconstruction-based methods such as InstantMesh. Nonetheless, our Direct3D achieves state-of-the-art performance across all metrics. Integrating our internal data for training further enhances the model’s performance, validating the scalability of our approach. As we replied in the global rebuttal part, **we will release the model trained on the Objaverse dataset** upon paper acceptance to further promote openness and reproducibility in our research.\n\n| Methods  |Chamfer Distance ↓|Volume IoU ↑| F-Score ↑ |\n------------ | :----: | :----: | :----:\n| Shap-E       | 0.0585 | 0.2347 | 0.3474 |\n| Michelangelo | 0.0441 | 0.1260 | 0.4371 |\n| One-2-3-45   | 0.0513 | 0.2868 | 0.3149 |\n| InstantMesh  | 0.0327 | 0.4105 | 0.5058 |\n| Ours (trained on Objaverse)        |0.0296|0.4307|0.5356|\n| Ours (trained on internal data)        |**0.0271**|**0.4323**|**0.5624**|\n\n**[Q2]:** Quantitative metrics of the explicit triplane latent representation.\n\n**[A2]:** To validate the effectiveness of the explicit latent representation we employed, we established a validation set from the Objaverse dataset that does not overlap with the training set. We compared the reconstruction metrics of the VAE using explicit latent representation against methods such as 3DShape2VecSet [2] that utilize implicit 1D latent space. As shown in the table below, the explicit latent representation outperforms the implicit 1D latent space across all metrics.\n\n| Methods  |Chamfer Distance ↓|Volume IoU ↑| F-Score ↑ |\n------------ | :----: | :----: | :----:\n| Implicit 1D Space (3Dshape2VecSet)      | 0.0057 | 0.8794 | 0.9416 |\n| Explicit Triplane (ours) | **0.0042** | **0.9409** | **0.9835** |\n\n**[Q3]:** Quantitative metrics of the semi-continuous surface sampling strategy.\n\n**[A3]:** We also conducted ablation experiments of the semi-continuous surface sampling strategy on the Objaverse evaluation set. As shown in the table below, our proposed method demonstrates a notable improvement in the reconstruction quality of VAE.\n\n| Methods  |Chamfer Distance ↓|Volume IoU ↑| F-Score ↑ |\n------------ | :----: | :----: | :----:\n| w/o semi-continuous sampling  | 0.0060 | 0.8723 | 0.9192 |\n| w/ semi-continuous sampling | **0.0057** | **0.8794** | **0.9416** |\n\n**[Q4]:** Implementation details of near-surface sampling strategy.\n\n**[A4]:** Actually in L258 of the original manuscript, we have mentioned that during the training of the VAE, we sampled 20,480 uniform points and 20,480 near-surface points for supervision. For the near-surface sampling strategy, we first randomly sample points on the surface of the mesh, then add a Gaussian perturbation with a std of 0.02 to each point to obtain the near-surface points. During training, the weights of BCE loss for both the near-surface points and the uniformly distributed points were set to 1.\n\n**[Q5]:** A detailed discussion of how pixel-level and semantic-level conditions are implemented.\n\n**[A5]:** We integrated the CLIP condition using cross-attention, a common approach in conditional diffusion models. However, we observed that meshes generated solely based on these semantic-level conditions lacked detailed alignment with the input image. To address this, we introduced additional pixel-level conditions using DINO, which previous research [2] has shown to outperform other pre-trained models in extracting structural information for 3D tasks. Specifically, we concatenated image tokens extracted by DINO with the noisy latent tokens and fed them into the self-attention layer, differing from the CLIP conditioning. This approach significantly enhances generation quality, allowing for a more detailed recovery of the conditioned input. We have conducted a qualitative comparison in Section A.3 and Figure 9 of the appendix.\n\n**[Q6]:** A detailed discussion regarding the differences between the vector set representation used in 3DShape2VecSet and the triplane representation employed as the latent representation.\n\n**[A6]:** Thanks for your valuable comment. In **[A2]**, we conducted a quantitative comparison of the VAE reconstruction accuracy between the explicit latent representation and the vector set representation employed by 3DShape2VecSet. The results indicate that our explicit latent representation outperforms the vector set representation across various metrics, including Chamfer Distance, Volume IoU, and F-Score. In addition to geometric quality, another advantage of the explicit latent representation is its significant improvement in training efficiency for diffusion models. A recent work CLAY [3] also utilizes the vector set representation, and the training time of their diffusion model is approximately **ten times longer than ours**. Given the extensive computational requirements, with training necessitating thousands of GPU hours, it is challenging for us to complete the experiments within such a short timeframe. We will include a detailed discussion and comparative results in the final version of the paper.\n\n**Reference:**  \n\n[1]. Long et al., Wonder3D: Single Image to 3D using Cross-Domain Diffusion. \n\n[2]. Banani et al., Probing the 3d awareness of visual foundation models.\n\n[3]. Zhang et al., CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4D8yxOcLEn", "reviewer": "Reviewer_yUww", "summary": "This paper proposes an image-to-3D generation model without using SDS optimization. It includes a 3D-VAE that encodes 3D shapes into latent space and a 3D DiT that models the latent distributions.", "questions": "Please see the questions in the weaknesses section. I will be willing to raise my rating if these questions are resolved.", "limitations": "Yes, it is discussed.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Single-step 3D generation is a relatively unexplored area.\n- Exploring the DiT model in 3D tasks is interesting.", "weaknesses": "- Lack of novelty. Specifically, 1) this paper mainly uses a 3D-VAE and a 3D-DiT to encode a 3D point cloud into latent space and model the distribution, which has been explored in previous methods like Point-E/Shap-E. This pipeline is widely used in image diffusions, and this paper extends it to 3D, which does not seem novel to me. 2) One novelty the authors mention is the semi-continuous occupancy used for supervision, which, compared with standard 3D supervision, only adds a continuous value between [0,1] when it is close to the boundary. 3) The semantic-level 3D-specific image conditioning is provided by DINO.\n\n- What is the dataset used for training? It seems that a private dataset is used for training, so I am not sure if the examples shown in the paper were seen during training. In addition, since the paper emphasizes that they can use in-the-wild input images for 3D generation, results on in-the-wild input images would be good to show.\n\n- There is a paper in CVPR 2024 named \"DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data\" [1]. They are also doing single-step 3D generation without SDS. This submission uses the same model name as the CVPR 2024 paper. This is very confusing. I think the authors should consider changing the method name. For me, Direct3D or D3D are both not good names. In addition, considering this submission and the CVPR paper are doing the same thing, the authors may also consider discussing the differences.\n\n[1] Liu, Qihao, et al. \"DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback and thorough review of our paper. We carefully respond to each of the concerns and questions below.\n\n**[Q1]:** Similar architectures to image generation and video generation.\n\n**[A1]:** We appreciate the insightful question raised by Reviewer yUww. Our primary contribution lies in the innovative unification between the latent DiT framework and 3D content creation task, where previous methods, such as Shape-E and Michelangelo, have struggled to scale effectively for generalized 3D generation. As noted by Reviewer 3ERi, the 'origin, generative-based 3D AIGC pipeline is absolutely missing', and we address this gap by introducing the native 3D generation pipeline leveraging the specially designed 3D VAE (D3D-VAE) and the image-conditioned 3D latent DiT (D3D-DiT). This pipeline not only outperforms existing multi-view diffusion-based methods but also shows promise for scalability with increased 3D data availability. \n\nThe innovative nature of our work has been recognized by Reviewer 3ERi, who described it as 'a firm step towards a converged 3D AIGC framework,' and by Reviewer U1iF, who highlighted 'The DiT model is innovatively applied for the first time in conditional 3D generation'. We believe these points underscore the novelty and significance of our contribution to the field of 3D content creation.\n\n\n**[Q2]:** It seems that a private dataset is used for training.\n\n**[A2]:** To ensure a fair comparison with other methods, we re-train our D3D-VAE and D3D-DiT on the Objaverse dataset. Following prior approaches such as InstantMesh and Wonder3D [1], we adopt the Google Scanned Objects dataset for evaluation, and report the Chamfer Distance, Volume IoU and F-Score to compare the quality of the generated meshes. The results are presented in the table below, which illustrates that generative-based methods like Shap-E and Michelangelo exhibit significantly lower accuracy compared to reconstruction-based methods such as InstantMesh. Nonetheless, our Direct3D achieves state-of-the-art performance across all metrics. Integrating our internal data for training further enhances the model’s performance, validating the scalability of our approach. As replied above, **we will release the model trained on the Objaverse dataset** to further promote openness and reproducibility in our research.\n\n| Methods  |Chamfer Distance ↓|Volume IoU ↑| F-Score ↑ |\n------------ | :----: | :----: | :----:\n| Shap-E       | 0.0585 | 0.2347 | 0.3474 |\n| Michelangelo | 0.0441 | 0.1260 | 0.4371 |\n| One-2-3-45   | 0.0513 | 0.2868 | 0.3149 |\n| InstantMesh  | 0.0327 | 0.4105 | 0.5058 |\n| Ours (trained on Objaverse)        |0.0296|0.4307|0.5356|\n| Ours (trained on internal data)        |**0.0271**|**0.4323**|**0.5624**|\n\n\n**[Q3]:** Results on in-the-wild input images.\n\n**[A3]:** We present the results using in-the-wild images as conditional inputs in **Figure 2 of the uploaded PDF**, demonstrating that our method is capable of generating high-quality meshes. We will include more visualization results in the final version of the paper.\n\n\n**[Q4]:** Another paper named ’DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data’.\n\n**[A4]:** Thanks for the reminder. We also noted the naming problem. Actually, the paper mentioned was made public on 6 June, 2024, after the NeurIPS submission deadline, and we had conducted a thorough search to ensure that the name ’Direct3D’ was not already in use before submitting our paper to NeurIPS. We will cite and discuss the paper to help readers to distinguish the two works.\n\n**Reference:**  \n\n[1]. Long et al., Wonder3D: Single Image to 3D using Cross-Domain Diffusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r2VChPIBSj", "reviewer": "Reviewer_FNua", "summary": "An image-conditioned 3D generation method with two parts: 1. 3D triplane VAE taking point cloud as input and generate the semi-continuous occulancy, which includes a point2latent encoder and a latent2triplane decoder 2. A  direct 3D transformer diffusion model conditioned on single image and generate tokens of the latent, with two input heads: DINO and CLIP. A wide range of evaluation with input from GSO object rendering to 2D diffusion generated images.", "questions": "See the weakness.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Well-written maniscript explains motivations and methods clearly.\n2. A novel 3D diffusion transformer to generate triplane token, then decode the semi-continuous occupancy.\n3. Good qualitative results given image rendered from GSO dataset or text conditioned image diffusion model.", "weaknesses": "1. Lack of analysis of your data's contribution on method performance. I'm curious whether your good quality is because of 500K internal data since most baselines are only trained on objaverse. It would be great if I can see the result of baselines trained on your data (If this is too expensive, maybe training your method only trained on objaverse small subset is easier).\n\n2. Lack of more quantitative results: The only quantitative result is the user study, which may be not enough. It's natural to comare with other baselines on chamfer distance, PSNR or SSIM of depth and normal maps from some views, and also compare on a larger set (not as few what you mentioned as 30 GSO objects)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback and thorough review of the our paper. We carefully respond to each of the concerns and questions below.\n\n**[Q1]** Training on Objaverse dataset and quantitative comparison with state-of-the-art methods.\n\n**[A1]** Thank you for your valuable comment. To ensure a fair comparison with other methods, we re-train our D3D-VAE and D3D-DiT on the Objaverse dataset. Following prior approaches such as InstantMesh and Wonder3D [1], we adopt the Google Scanned Objects dataset for evaluation, and report the Chamfer Distance, Volume IoU and F-Score to compare the quality of the generated meshes. The results are presented in the table below, which illustrates that generative-based methods like Shap-E and Michelangelo exhibit significantly lower accuracy compared to reconstruction-based methods such as InstantMesh. Nonetheless, our Direct3D achieves state-of-the-art performance across all metrics. Integrating our internal data for training further enhances the model’s performance, validating the scalability of our approach.\n\n| Methods  |Chamfer Distance ↓|Volume IoU ↑|F-Score ↑|\n------------ | :----: | :----: | :----:\n| Shap-E       | 0.0585 | 0.2347 | 0.3474 |\n| Michelangelo | 0.0441 | 0.1260 | 0.4371 |\n| One-2-3-45   | 0.0513 | 0.2868 | 0.3149 |\n| InstantMesh  | 0.0327 | 0.4105 | 0.5058 |\n| Ours (trained on Objaverse)        |0.0296|0.4307|0.5356|\n| Ours (trained on internal data)        |**0.0271**|**0.4323**|**0.5624**|\n\n**[Q2]** Lack of more quantitative results.\n\n**[A2]** We add more quantitative results in Table 2-3 of the uploaded PDF.\n\n**Reference:**  \n\n[1]. Long et al., Wonder3D: Single Image to 3D using Cross-Domain Diffusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their constructive and insightful suggestions. We are encouraged that all reviewers consider the proposed method novel and interesting: 

- Reviewer 3ERi: "It is a firm step towards a converged 3D AIGC framework"
    
- Reviewer HuD3: "The algorithm looks novel and interesting" 
    
- Reviewer U1iF: "The DiT model is innovatively applied for the first time in conditional 3D generation" 
    
- Reviewer yUww: "Exploring the DiT model in 3D tasks is interesting"

- Reviewer FNua: "A novel 3D diffusion transformer to generate triplane token"

Meanwhile, we acknowledge the reviewers' primary concern regarding the model training on our private dataset. To address this, **we have retrained our model on the Objaverse dataset**, which is publicly available and widely adopted in recent generative 3D research. This action ensures our methodology is transparent and reproducible. Following prior approaches such as InstantMesh and Wonder3D [1], we adopt the Google Scanned Objects dataset for evaluation, and report the Chamfer Distance, Volume IoU and F-Score to compare the quality of the generated meshes. The results are presented in the table below, which illustrates that generative-based methods like Shap-E and Michelangelo exhibit significantly lower accuracy compared to reconstruction-based methods such as InstantMesh. Nonetheless, our Direct3D achieves state-of-the-art performance across all metrics. Despite a performance decrease, the **Objaverse-trained Direct3D model still significantly surpasses previous SOTA methods**, confirming its superior performance and scalability. Additionally, **we will release the model trained on the Objaverse dataset** upon paper acceptance to further promote openness and reproducibility in our research. 

| Methods  |Chamfer Distance ↓|Volume IoU ↑| F-Score ↑ |
------------ | :----: | :----: | :----:
| Shap-E       | 0.0585 | 0.2347 | 0.3474 |
| Michelangelo | 0.0441 | 0.1260 | 0.4371 |
| One-2-3-45   | 0.0513 | 0.2868 | 0.3149 |
| InstantMesh  | 0.0327 | 0.4105 | 0.5058 |
| Ours (trained on Objaverse)        |0.0296|0.4307|0.5356|
| Ours (trained on internal data)        |**0.0271**|**0.4323**|**0.5624**|


We have addressed the other concerns raised by each reviewer in the corresponding official comments. Finally, we would like to express our sincere gratitude to the reviewers and remain open to further improvements in all aspects of our work.

**Reference:**  

[1]. Long et al., Wonder3D: Single Image to 3D using Cross-Domain Diffusion.

Meta-review: This paper receives two borderline rejects, two weak accepts and one borderline accept. AC have carefully read the paper, as well as reviews, rebuttal and discussion between reviewers and authors. 

The biggest concern among reviewers is the reliance on internal data, which prevents others to faithfully justify the effectiveness and contribution of this paper. This concern has been addressed by providing experiment results on Objaverse. 

Another concern is the novelty of proposed architecture. While reviewer yUww still holds concerns after discussion, reviewer 3ERi (addressed after discussion), HuD3, U1iF, and FNua agree it's innovative, being an early DiT-like structural exploration in direct 3D generation. AC agrees with reviewer 3ERi, HuD3, U1iF and FNua on this point. 

There are other concerns regarding insufficient comparison, evaluation metrics and ablation studies. Authors have provided corresponding responses, and AC believes strengthes of this paper outweigh its weaknesses.

Therefore, AC believes this paper is insightful to relevant community and the decision is accept. In the meantime, authors are required to revise the paper according to the discussion between authors and reviewers, especially the inclusion of results on Objaverse.
Decision: Accept (poster)